Improve serialization logic
feanil
released this
26 Sep 20:30
Improve Serialization Logic (#82)
* Drop items that can't be serialized.
As a part of the python3 conversion, we added logic to try to convert
bytestrings to unicode strings. This had a problem because not all
bytestrings that are produced as a part of the context passed into and
out of the jailed code execution is actually unicode strings. In some
cases this was resulting in a unicode decode error.
Before this code existed we used to just drop any nonstandard types and
also only keep complex types(list,dict,tuples) that were easily
jsonable. In python2 this was fine because byte strings would
automatically get coerced to unicode strings. However in python3 this
will throw errors. So if we don't explicitly try to convert all the
byte strings to unicode, we could stop passing back data that we
previously used to pass in.
The first iteration of this code assumed all byte arrays were unicode
strings. From evidence in production this is not always the case. So
we fall back to the behaviour we had before this change where if an item
can't successfully be converted to something we can serialize to json,
we just drop that item from the list of content being passed back
through the serialization straw.
* Bump the version.