Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate simdjson with VAST #1230

Merged
merged 48 commits into from
Jan 13, 2021
Merged

Integrate simdjson with VAST #1230

merged 48 commits into from
Jan 13, 2021

Conversation

dominiklohmann
Copy link
Member

@dominiklohmann dominiklohmann commented Dec 14, 2020 •
edited
Loading

📔 Description

This PR serves as a merge for base @ngrodzitski's work on integrating simdjson with VAST.

All CI changes have to take place in this PR, because GitHub does not use Workflow changes from forks.

📝 Checklist

  • All user-facing changes have changelog entries.
  • The changes are reflected on docs.tenzir.com/vast, if necessary.
  • The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

n/t

@dominiklohmann dominiklohmann added the feature New functionality label Dec 14, 2020
@dominiklohmann dominiklohmann self-assigned this Dec 16, 2020
ngrodzitski and others added 23 commits December 18, 2020 14:53
@ngrodzitski
Add simdjson dependency to libvast
56776ab
@ngrodzitski
Introduce simdjson to nix builds.
49ac953
@ngrodzitski @tobim
Update nix/overlay.nix
1326483 
Co-authored-by: tobim <tobim+github@fastmail.fm>
@ngrodzitski
Fix typo
876783e
@ngrodzitski
Add a copy of json reader routine for future transformation to simdjs…
f390ff6 
…on reader
@ngrodzitski
First compilable version of simdjson (most of conversions to be imple…
7673863 
…mented)
@ngrodzitski
Add simdjson test (a copy of original json test)
853e8a5
@ngrodzitski
First version with working conversion.
f1fca14
@ngrodzitski
Add --simdjson flag to import json/import suricata commands
2db927e
@ngrodzitski
Use internal parsing for numbers coming as strings
5a96a19
@ngrodzitski
Merge branch 'dev-add-libsimdjson-phase1' of github.com:ngrodzitski/v…
ec940e2 
…ast into dev-add-libsimdjson-phase1
@ngrodzitski @dominiklohmann
Improve simdjson option
3510d14 
Co-authored-by: Dominik Lohmann <mail@dominiklohmann.de>
@ngrodzitski
Fix warnings on clang
3880158
@ngrodzitski
Remove braces from integration test names
30cd126
@ngrodzitski
Merge branch 'dev-add-libsimdjson-phase2' into dev-add-libsimdjson-ph…
933754b 
…ase1
@ngrodzitski
Fix spelling
a762152
@ngrodzitski
Fix passing params
368de17
@ngrodzitski
Remove the writer from json
81a7c4e
@ngrodzitski
Save temp changes
ed3ed89
@ngrodzitski
Refactoring convertion routines
b091cb5 
Make convertion function to be associated with types rather then indexes.
@ngrodzitski
Add integration files for simdjson
bfd1c20
@ngrodzitski
Add \r\n json-string test
46076ae
@ngrodzitski
Remove dubug code
f86a9d2
ngrodzitski and others added 17 commits January 9, 2021 21:13
@ngrodzitski
Add hex number to test
acf17a2
@ngrodzitski
Fix \r\n => \x0D\x0A missmatch
7e1b078
@ngrodzitski
Fix pivot simdjson test reference data
b5af424
@ngrodzitski
Add simdjson reference files for explore-suricata-with-overlap
126ddb6
@ngrodzitski
Fix \r\n
1458aae
@ngrodzitski
Add non ascii symbol to test
9dbbd13
@ngrodzitski
Fix '\' escaping handling
068a828
@ngrodzitski
Merge remote-tracking branch 'upstream/master' into dev-add-libsimdjs…
c943a57 
…on-phase1
@ngrodzitski
Polishing the stuff
c6e04d3
@ngrodzitski
Fix reference files to account \r\n as \x0D\x0A for simdjson
2c65736
@dominiklohmann
Merge pull request #1246
9629bd8 
Add simdjson dependency to libvast
@dominiklohmann
Satisfy clang-format checks
5bcb82e
@dominiklohmann
Merge branch 'master' into epic/simdjson
ad4cf20
@dominiklohmann
Adapt code to follow style guide
5171ac1
@dominiklohmann
Fix installation of simdjson in Dockerfile_prebuilt
16472c6
@dominiklohmann
Add changelog entry
874b202
@dominiklohmann
Merge remote-tracking branch 'origin/master' into epic/simdjson
7977197
@dominiklohmann dominiklohmann marked this pull request as ready for review January 13, 2021 13:06
dominiklohmann
dominiklohmann commented Jan 13, 2021
View reviewed changes
.github/workflows/vast.yaml Outdated Show resolved Hide resolved
libvast/vast/format/json/default_selector.hpp Outdated Show resolved Hide resolved
libvast/vast/format/json/default_selector.hpp Outdated Show resolved Hide resolved
libvast/vast/format/json/field_selector.hpp Outdated Show resolved Hide resolved
libvast/vast/format/simdjson.hpp Show resolved Hide resolved
libvast/src/format/simdjson.cpp Outdated Show resolved Hide resolved
mavam
mavam approved these changes Jan 13, 2021
View reviewed changes
Copy link
Member

@mavam mavam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super excited to see this landing in master!

CHANGELOG.md Outdated Show resolved Hide resolved
@mavam mavam added the performance Improvements or regressions of performance label Jan 13, 2021
@dominiklohmann dominiklohmann merged commit d9a3b99 into master Jan 13, 2021
@dominiklohmann dominiklohmann deleted the epic/simdjson branch January 13, 2021 16:27
@dominiklohmann dominiklohmann mentioned this pull request Jan 13, 2021
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mavam mavam mavam approved these changes

Assignees

@dominiklohmann dominiklohmann

Labels
feature New functionality performance Improvements or regressions of performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dominiklohmann @mavam @ngrodzitski