Specifically, is there a Python 2.7-compatible serialization library that meets the following criteria (most important criteria listed first):

  1. Has an API that allows me to customize serialization of certain types by providing my own serialization function:

    # Override serialization of Foo types
    def serialize_foo(pickler, obj):
        pickler.say_hello()     # prints "Hello from pickler at 0x465DED00"
        return pickler.dumps({k: v for k, v in foo.__dict__ if k != 'my_unpicklable_attr'})

    This API should must also expose the serializer itself (i.e. for the pickler argument above)

    • cPickle fails this requirement
  2. The above requirement but for the deserializer

    • cPickle fails this as its Unpickler cannot be subclassed
    • Can be done with pickle but messy -- must subclass Unpickler and re-implement its load_reduce() method
  3. Must also handle circular references: e.g. must be able to pickle the following x:

    x = {}
    x['ref'] = x
    • JSON serialization libraries (e.g. json, ujson) fail this
  4. Consistent: two separate serialization calls on the same input object must deserialize into the same output object (see here for an example of why this is important). I.e. the following must be True

    unpickler.loads(pickler.dumps(x)) is unpickler.loads(pickler.dumps(x))
    • pickle/cPickle achieve this via an internal memo dictionary (although this is actually a memory leak because the memo never releases objects and only ever grows).
    • AFAIK JSON libraries also fail at this
  5. Fast: there are plenty of python serialization performance benchmarks on google e.g. here. The library needn't be as fast as msgpack but should still keep good pace

    • pickle lags behind here

Nice to have:

  1. Batteries included: can serialize arbitrary Python objects out-of-the-box
    • e.g. dill (but dill fails in all the other places pickle does because it subclasses pickle.Pickler/Unpickler)

Not necessary for now:

  1. Compatible: needn't be cross platform compatible. I will only be using this serialization in Python

  2. Secure: I will be deserializing from trusted sources only

  3. Human-readable: Twisted Jelly emphasises this but I just don't care

My usecase:

I'm writing a lightweight twisted protocol to stream data between machines in my local network. As such I need to serialize/deserialize objects that are sent on the wire. The twisted API (e.g. AMP/PB Jelly) emphasises security (which I don't care about) and trades that off against flexibility and performance (which I do care about). I also find the Deferred API very ugly for my purposes.

migrated from stackoverflow.com Feb 26 '16 at 10:38

This question came from our site for professional and enthusiast programmers.

  • As a side note, if you want circular references, you want YAML. – Nic Hartley Feb 26 '16 at 12:04
  • @QPaysTaxes thanks but I think YAML fails #4: file = StringIO(); x = [1, 2, 3]; d = Dumper(file); d.open(); d.represent(x); d.represent(x); file.getvalue() gives me '[1, 2, 3]\n--- [1, 2, 3]\n' which will clearly deserialize to two separate objects as the id of the source object is not serialized. – mchen Feb 26 '16 at 15:03
  • I don't speak Python, and I meant the format, not the package. – Nic Hartley Feb 26 '16 at 16:12
  • Also, I'm fairly sure the reason that happens is because you serialize two objects into it – x, twice. If you serialized [x, x] it would be a single object. – Nic Hartley Feb 26 '16 at 16:14
  • @QPaysTaxes: what you're describing is correct handling of circular references. However, I'm talking about consistency of output between calls. This is important because I might choose to serialize x = [1, 2, 3] and then later serialize y = [x, 4, 5, 6]. When i deserialize, I want to make sure that y_out[0] is x_out -- this is what i mean by consistency – mchen Feb 26 '16 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.