Speed up BSON unmarshalling #410
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
If you close and reopen the PR, travis will rerun this build and thanks to #462 , your PR should pass all tests on all mongodb versions |
|
oh, probably need to rebase... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently when unmarshalling into a struct which is missing a field or into bson.Raw, the unmarshaller reads the parts that should be skipped into blackHole. This allows for the entire document to be checked for corruption at the expense of speed. In particular for bson.Raw, it also means that every document is unmarshalled twice, once at the initial time of reading, and later when the bson.Raw is unmarshalled into its final form. Effectively, this PR stops doing that. It verifies the element that is getting skipped but doesn't descend into them. This is particularly relevent for containers like arrays and documents.
The effect is a massive speedup (I've measured up to 6x) depending on the complexity of documents when using commands that returns cursors as arrays. This would be the new find command and the aggregate command. The downside is that the corruption message appears later in the program than it used to and sometimes a corruption message may not occur if a field is ignored or a bson.Raw is never ultimately unmarshalled. I feel these are acceptable trade-offs.
As part of verifying, I've implemented the entire bson_corpus as generated code which is checked in (so as long as the corpus doesn't change, no need to regenerate).