I'm searching for a Python 2 library for serializing values as JSON that does not require the entire value to be calculated & in memory before serialization. I'd prefer something streaming/event-based (like ijson but in reverse), but I'd also settle for something that can intelligently serialize generators as lists.

Simple example use:

with open('outfile.json', 'w') as outfile:
    with magic_json_library.begin_object(outfile) as obj:
        obj.set_key("foo", 3.14)
        for item in retrieve_values_from_large_database():
        for field, value in retrieve_mapping_from_large_website():
            obj.add_key(field, value)

JSON's encoding rules are really, really simple. The majority of the effort involved in writing a JSON codec is to deal with the arbitrarily-recursive nature of objects and with organizing the representation of the output (e.g. sorting the keys, etc.).

But in your case, with a streaming output engine, much (if not all) of that complexity goes away, and all that's left is basic formatting and correctness.

I imagine you could write such a tool in under 100 lines. Functions like begin_list are more-or-less as simple as print "[", plus some state maintenance for accuracy guarantees. The data types are simple to represent, the escaping rules straight-forward. There's not really anything to it.

Have a look at encoder.py from simplejson to get an idea of all that goes in to writing a full-featured JSON encoder. You'll be surprised at how little there is to do.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.