Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BSON support #1320

Merged
merged 38 commits into from
Oct 27, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
f06c8fd
BSON: serialization of non-objects is not supported
julian-becker Sep 14, 2018
5f5836c
BSON: Support empty objects
julian-becker Sep 14, 2018
9a0dddc
BSON: Object with single boolean
julian-becker Sep 15, 2018
0c0f2e4
BSON: support doubles
julian-becker Sep 15, 2018
6c447de
BSON: Support objects with string members
julian-becker Sep 15, 2018
c5ef023
BSON: support objects with null members
julian-becker Sep 15, 2018
7ee361f
BSON: support objects with int32 members
julian-becker Sep 15, 2018
c0d8921
BSON: support objects with int64 members
julian-becker Sep 15, 2018
83b427a
BSON: unsigned integers
julian-becker Sep 15, 2018
5ce7d6b
BSON: support objects with objects as members
julian-becker Sep 15, 2018
120d1d7
BSON: test case for a more complex document
julian-becker Sep 15, 2018
cf485c2
BSON: Support for arrays
julian-becker Sep 15, 2018
df33a90
BSON: Bugfix for non-empty arrays
julian-becker Sep 15, 2018
763705c
Fix: Add missing `begin()` and `end()` member functions to `alt_string`
julian-becker Sep 24, 2018
bce4816
BSON: Added test case for the different input/output_adapters
julian-becker Sep 24, 2018
ef358ae
BSON: Fixed hangup in case of incomplete bson input and improved test…
julian-becker Sep 25, 2018
0a09db9
BSON: Extend `binary_reader::get_number` to be able to hanlde little …
julian-becker Sep 29, 2018
e8730e5
BSON: Reworked `binary_reader::get_bson_cstr()`
julian-becker Sep 29, 2018
81f4b34
BSON: Improved documentation and error handling/reporting
julian-becker Oct 7, 2018
062aeaf
BSON: Reworked the `binary_writer` such that it precomputes the size …
julian-becker Oct 7, 2018
df0f612
BSON: allow and discard values and object entries of type `value_t::d…
julian-becker Oct 7, 2018
5bccacd
BSON: throw json.exception.out_of_range.407 in case a value of type `…
julian-becker Oct 16, 2018
daa3ca8
BSON: Adjusted documentation of `binary_writer::to_bson()`
julian-becker Oct 16, 2018
978c3c4
BSON: throw `json.exception.out_of_range.409` in case a key to be ser…
julian-becker Oct 16, 2018
2a63869
Merge branch 'develop' of https://github.com/nlohmann/json into featu…
julian-becker Oct 17, 2018
8de10c5
BSON: Hopefully fixing ambiguity (on some compilers) to call to strin…
julian-becker Oct 17, 2018
5ba812d
BSON: fixed incorrect casting in unit-bson.cpp
julian-becker Oct 18, 2018
ad11b6c
BSON: Improved exception-related tests and report location of U+0000 …
julian-becker Oct 18, 2018
bba1591
Merge branch 'feature/bson' of https://github.com/julian-becker/json …
nlohmann Oct 24, 2018
e2c5913
:construction: some changes to the BSON code
nlohmann Oct 24, 2018
4d1eaac
:hammer: fixed fuzz code to avoid false positives in case of discarde…
nlohmann Oct 24, 2018
1968e5c
:art: clean up binary formats
nlohmann Oct 24, 2018
6212627
:hammer: added fix for arrays
nlohmann Oct 25, 2018
19647e0
:rotating_light: fixed compiler warnings
nlohmann Oct 25, 2018
7ce720b
:rotating_light: fixed coverage
nlohmann Oct 25, 2018
d97fa30
:ok_hand: fixed comment #1320
nlohmann Oct 25, 2018
544150d
:rotating_light: fixed another linter warning
nlohmann Oct 26, 2018
6384fe2
:rotating_light: fixed another linter warning
nlohmann Oct 26, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ all:
@echo "cppcheck - analyze code with cppcheck"
@echo "doctest - compile example files and check their output"
@echo "fuzz_testing - prepare fuzz testing of the JSON parser"
@echo "fuzz_testing_bson - prepare fuzz testing of the BSON parser"
@echo "fuzz_testing_cbor - prepare fuzz testing of the CBOR parser"
@echo "fuzz_testing_msgpack - prepare fuzz testing of the MessagePack parser"
@echo "fuzz_testing_ubjson - prepare fuzz testing of the UBJSON parser"
Expand Down Expand Up @@ -220,6 +221,14 @@ fuzz_testing:
find test/data/json_tests -size -5k -name *json | xargs -I{} cp "{}" fuzz-testing/testcases
@echo "Execute: afl-fuzz -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer"

fuzz_testing_bson:
rm -fr fuzz-testing
mkdir -p fuzz-testing fuzz-testing/testcases fuzz-testing/out
$(MAKE) parse_bson_fuzzer -C test CXX=afl-clang++
mv test/parse_bson_fuzzer fuzz-testing/fuzzer
find test/data -size -5k -name *.bson | xargs -I{} cp "{}" fuzz-testing/testcases
@echo "Execute: afl-fuzz -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer"

fuzz_testing_cbor:
rm -fr fuzz-testing
mkdir -p fuzz-testing fuzz-testing/testcases fuzz-testing/out
Expand Down
16 changes: 13 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
- [JSON Merge Patch](#json-merge-patch)
- [Implicit conversions](#implicit-conversions)
- [Conversions to/from arbitrary types](#arbitrary-types-conversions)
- [Binary formats (CBOR, MessagePack, and UBJSON)](#binary-formats-cbor-messagepack-and-ubjson)
- [Binary formats (CBOR, BSON, MessagePack, and UBJSON)](#binary-formats-bson-cbor-messagepack-and-ubjson)
- [Supported compilers](#supported-compilers)
- [License](#license)
- [Contact](#contact)
Expand Down Expand Up @@ -874,14 +874,22 @@ struct bad_serializer
};
```

### Binary formats (CBOR, MessagePack, and UBJSON)
### Binary formats (CBOR, BSON, MessagePack, and UBJSON

Though JSON is a ubiquitous data format, it is not a very compact format suitable for data exchange, for instance over a network. Hence, the library supports [CBOR](http://cbor.io) (Concise Binary Object Representation), [MessagePack](http://msgpack.org), and [UBJSON](http://ubjson.org) (Universal Binary JSON Specification) to efficiently encode JSON values to byte vectors and to decode such vectors.
Though JSON is a ubiquitous data format, it is not a very compact format suitable for data exchange, for instance over a network. Hence, the library supports [BSON](http://bsonspec.org) (Binary JSON), [CBOR](http://cbor.io) (Concise Binary Object Representation), [MessagePack](http://msgpack.org), and [UBJSON](http://ubjson.org) (Universal Binary JSON Specification) to efficiently encode JSON values to byte vectors and to decode such vectors.

```cpp
// create a JSON value
json j = R"({"compact": true, "schema": 0})"_json;

// serialize to BSON
std::vector<std::uint8_t> v_bson = json::to_bson(j);

// 0x1B, 0x00, 0x00, 0x00, 0x08, 0x63, 0x6F, 0x6D, 0x70, 0x61, 0x63, 0x74, 0x00, 0x01, 0x10, 0x73, 0x63, 0x68, 0x65, 0x6D, 0x61, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00

// roundtrip
json j_from_bson = json::from_bson(v_bson);

// serialize to CBOR
std::vector<std::uint8_t> v_cbor = json::to_cbor(j);

Expand Down Expand Up @@ -1138,6 +1146,8 @@ I deeply appreciate the help of the following people.
- [Henry Schreiner](https://github.com/henryiii) added support for GCC 4.8.
- [knilch](https://github.com/knilch0r) made sure the test suite does not stall when run in the wrong directory.
- [Antonio Borondo](https://github.com/antonioborondo) fixed an MSVC 2017 warning.
- [efp](https://github.com/efp) added line and column information to parse errors.
- [julian-becker](https://github.com/julian-becker) added BSON support.

Thanks a lot for helping out! Please [let me know](mailto:mail@nlohmann.me) if I forgot someone.

Expand Down
5 changes: 4 additions & 1 deletion include/nlohmann/detail/exceptions.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ json.exception.parse_error.109 | parse error: array index 'one' is not a number
json.exception.parse_error.110 | parse error at 1: cannot read 2 bytes from vector | When parsing CBOR or MessagePack, the byte vector ends before the complete value has been read.
json.exception.parse_error.112 | parse error at 1: error reading CBOR; last byte: 0xF8 | Not all types of CBOR or MessagePack are supported. This exception occurs if an unsupported byte was read.
json.exception.parse_error.113 | parse error at 2: expected a CBOR string; last byte: 0x98 | While parsing a map key, a value that is not a string has been read.
json.exception.parse_error.114 | parse error: Unsupported BSON record type 0x0F | The parsing of the corresponding BSON record type is not implemented (yet).

@note For an input with n bytes, 1 is the index of the first character and n+1
is the index of the terminating null byte or the end of file. This also
Expand Down Expand Up @@ -236,6 +237,7 @@ json.exception.type_error.313 | invalid value to unflatten | The @ref unflatten
json.exception.type_error.314 | only objects can be unflattened | The @ref unflatten function only works for an object whose keys are JSON Pointers.
json.exception.type_error.315 | values in object must be primitive | The @ref unflatten function only works for an object whose keys are JSON Pointers and whose values are primitive.
json.exception.type_error.316 | invalid UTF-8 byte at index 10: 0x7E | The @ref dump function only works with UTF-8 encoded strings; that is, if you assign a `std::string` to a JSON value, make sure it is UTF-8 encoded. |
json.exception.type_error.317 | JSON value cannot be serialized to requested format | The dynamic type of the object cannot be represented in the requested serialization format (e.g. a raw `true` or `null` JSON object cannot be serialized to BSON) |

@liveexample{The following code shows how a `type_error` exception can be
caught.,type_error}
Expand Down Expand Up @@ -278,8 +280,9 @@ json.exception.out_of_range.403 | key 'foo' not found | The provided key was not
json.exception.out_of_range.404 | unresolved reference token 'foo' | A reference token in a JSON Pointer could not be resolved.
json.exception.out_of_range.405 | JSON pointer has no parent | The JSON Patch operations 'remove' and 'add' can not be applied to the root element of the JSON value.
json.exception.out_of_range.406 | number overflow parsing '10E1000' | A parsed number could not be stored as without changing it to NaN or INF.
json.exception.out_of_range.407 | number overflow serializing '9223372036854775808' | UBJSON only supports integers numbers up to 9223372036854775807. |
json.exception.out_of_range.407 | number overflow serializing '9223372036854775808' | UBJSON and BSON only support integer numbers up to 9223372036854775807. |
json.exception.out_of_range.408 | excessive array size: 8658170730974374167 | The size (following `#`) of an UBJSON array or object exceeds the maximal capacity. |
json.exception.out_of_range.409 | BSON key cannot contain code point U+0000 (at byte 2) | Key identifiers to be serialized to BSON cannot contain code point U+0000, since the key is stored as zero-terminated c-string |

@liveexample{The following code shows how an `out_of_range` exception can be
caught.,out_of_range}
Expand Down
Loading