Object persistence via traits-cereal

[johntyree]

Now's as good a time as any to start kicking the tires.

[jwiggins]

Can you separate this from the apptools imports and make it a relative import?

[jwiggins]

Oops

[johntyree]

Not sure where this even came from. Never seen it before in my life!

[johntyree]

Docstrings and cleanup are coming!

[jwiggins]

Are you planning to pass something other than set() in the future?

[johntyree]

I'm not sure, honestly. That's part of why it's tucked away. It assumes, for example that all keyed objects are hashable. Maybe that's not safe?

Does this irk you?

[rkern]

I'm really not a fan of the flat blob structure. The result is opaque and unreadable by anything but traits_cereal even if the container format is ostensibly neutral. I'd rather just use moderately customized versions of jsonpickle or hdf5pickle for any of these use cases.

[johntyree]

Is the argument that hdf5pickle and jsonpickle are more readable or that neither is readable?

hdf5pickle's output doesn't seem particularly easy to deal with to me, although their decision to use a node for each class attribute rather than attributes on a group may have been a better choice for container types.

jsonpickle apparently works around the problem of redundant data by completely ignoring it. decode(encode([x, x])) doesn't give you back a list with two references to the same instance (hdf5pickle does). If we use jsonpickle it seems like we'll have to pass in our own encoder and decoder hooks as well and enforce that all dictionaries only use strings as keys.

I'm also not clear on how we would handle versioning using either of these libraries without creating an intermediate representation... which smells like Blob again.

Perhaps I just don't know enough about how to squeeze the best mileage out of it, but I'd at least suggest yaml over json. I'm happy to ditch this if it's not going to work, but it seems like these two give us more readability on disk (kind of...) in exchange for not doing the things we care about. Do you see how they could satisfy our use cases (or corran's... or Ioannis/CanopyGeo's) without rewriting them to work more like traits_cereal does? Granted that might be a better idea than traits_cereal itself anyway.

[jwiggins]

Could you move the demos out into the examples directory?

[jwiggins]

Now that I've read the code a bit more closely, I think I agree with @rkern about the flatness of blobs. It would be nice if blobs could maintain the DAG structure of the original object. A shared object could be serialized in place where it is first encountered and stored as a reference after that. Does that seem reasonable?

Also, as a general code organization comment, it would be nice if all of the public methods of classes had docstrings. I'm thinking specifically of StorageManager and IObjectStore (and possibly the implementors of IObjectStore)

[johntyree]

Yes we could leave the first reference to an object in-place and reference it from elsewhere, although as I mentioned elsewhere, we're moving closer and closer to what hdf5pickle does on its own so it might be less work to fork and extend that than this (huge error bars around that statement).

The other option is to leave it flat at the top level and use h5 links to make it feel like a tree when you're browsing around. That is probably signficantly less work than special casing the first appearance of each serialized object and was on the list of features-to-attempt anyway.

If that turns out to be a failure or doesn't work the way we'd like, I'm not super certain traits-cereal is worthy of existence at all in its current form.

[johntyree]

Closing this PR for now, as there's no demand for this at this time.