Skip to content

Commit

Permalink
Add order kwarg to encoders
Browse files Browse the repository at this point in the history
This adds an `order` kwarg to all encoders for configuring how unordered
collections/objects are encoded. Options are:

- `None`: the default. All objects are encoded in the most efficient
  manner corresponding to their in-memory representation.
- `'deterministic'`: Unordered collections (sets, dicts) are sorted
  before encoding. This ensures a consistent output between runs, which
  may be useful when comparing/hashing the encoded binary
  representation.
- `'sorted'`: same as `'deterministic'`, but *all* objet-like objects
  will have their fields encoded in alphabetical order by name. This is
  more expensive than `'deterministic'`, but may be useful for making
  the output more human readable.

The `'deterministic'` output has been heavily optimized - given the work
required to accomplish this feature, I wouldn't expect we can speed up
this operation much more. The `'sorted'` option has not been fully
optimized (the assumption being a human-readable output is rarely perf
sensitive). If needed, there are some rather simple optimizations we can
add here to speed this up further.

In general, `msgspec.json.encode(obj, order="deterministic")` should be
as fast or faster than `orjson.dumps(obj, option=orjson.OPT_SORT_KEYS)`.
For common small object sizes we average a ~25% speedup over `orjson`
for key sorting.
  • Loading branch information
jcrist committed Jan 15, 2024
1 parent a5674c1 commit c83ec7e
Show file tree
Hide file tree
Showing 13 changed files with 1,294 additions and 251 deletions.
1 change: 1 addition & 0 deletions msgspec/__init__.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@ def to_builtins(
str_keys: bool = False,
builtin_types: Union[Iterable[type], None] = None,
enc_hook: Optional[Callable[[Any], Any]] = None,
order: Literal[None, "deterministic", "sorted"] = None,
) -> Any: ...
@overload
def convert(
Expand Down
Loading

0 comments on commit c83ec7e

Please sign in to comment.