Improve hash calculation to ensure a list of 128.660 are all uniquely hasheable #5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Owner
onno-vos-dev
commented
Feb 4, 2024
•
onno-vos-dev
commented
- Grab more bytes if possible when calculating the hash. This helps prevent collisions as well as improves speed
- Move find_cities/1 to after spawning workers. It can be done in the background as it only takes a very short amount of time
- Read less bytes in find_cities/1 (2MB to be precise)
- In-line do_process_line/2 and add_to_state/3
- Do not generate home-grown half-baked City names in PropEr test but use an input file taken from: https://public.opendatasoft.com/explore/dataset/geonames-all-cities-with-a-population-1000
- Save test files generated by PropEr so they can be repeated from a shell for debugging rather than losing them upon shrinking
- Do not shrink at all as it never really leads to anything anyway
- Add eunit test which tries to hash all 128.660 cities and ensures uniqueness
- Add runtime results on Mac M1 Pro to README
- Add build badge to README
onno-vos-dev
force-pushed
the
minor-improvements
branch
from
7efcb7f to
d25f91e
Compare
… hasheable - Grab more bytes if possible when calculating the hash. This helps prevent collisions as well as improves speed - Move find_cities/1 to after spawning workers. It can be done in the background as it only takes a very short amount of time - Read less bytes in find_cities/1 (2MB to be precise) - In-line do_process_line/2 and add_to_state/3 - Do not generate home-grown half-baked City names in PropEr test but use an input file taken from: https://public.opendatasoft.com/explore/dataset/geonames-all-cities-with-a-population-1000 - Save test files generated by PropEr so they can be repeated from a shell for debugging rather than losing them upon shrinking - Do not shrink at all as it never really leads to anything anyway - Add eunit test which tries to hash all 128.660 cities and ensures uniqueness - Add runtime results on Mac M1 Pro to README - Add build badge to README
onno-vos-dev
force-pushed
the
minor-improvements
branch
from
d25f91e to
de831b2
Compare
onno-vos-dev
added a commit
to onno-vos-dev/1brc_erl_ex_test
that referenced
this pull request
Feb 4, 2024
IceDragon200
added a commit
to IceDragon200/1brc_erl_ex_test
that referenced
this pull request
Feb 4, 2024
…to-latest-main Bump onno-vos-dev submodule to latest merge of onno-vos-dev/1brc#5
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.