Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PoC to better streamline the import of NSRL data. #4

Merged
merged 3 commits into from
Jul 29, 2021
Merged

Conversation

cudeso
Copy link
Contributor

@cudeso cudeso commented Jul 27, 2021

Still requires some work but basic concept works.
Currently only tested with Android

cudeso added 2 commits July 27, 2021 18:37
PoC to better streamline the import of NSRL data.
Still requires some work but basic concept works.
Currently only tested with Android
@adulau adulau merged commit c54375c into adulau:main Jul 29, 2021
@adulau
Copy link
Owner

adulau commented Jul 29, 2021

Thanks a lot for the contribution. I'll do some tests.

@cudeso
Copy link
Contributor Author

cudeso commented Jul 29, 2021

Thanks. There are also three additional elements added

drecords['source'] = "NSRL"
drecords['db'] = rdbkey
drecords['insert-timestamp'] = time.time()

rdbkey comes from the dataset identifier <- defined in config.json ; provided as argument to __process_nsrl_base

@adulau
Copy link
Owner

adulau commented Jul 29, 2021

Thanks, it makes sense and I think this can be easily compressed on the RocksDB size (as it's the often the same data). I'll add a doc with the database structure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants