Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added the Compact data modules #2520

Merged
merged 19 commits into from
Oct 26, 2021
Merged

Added the Compact data modules #2520

merged 19 commits into from
Oct 26, 2021

Conversation

TimSheard
Copy link
Contributor

Some new data structures to be used instead of Data.Map
Take advantage of some properties

  1. Keys are Cryptographic Hashes, so they are evenly distributed, and never have collisions
  2. Aggressively using {-# UNPACK #-} to reduce space
  3. Use Trie based structures based on SmallArray

TimSheard and others added 18 commits October 26, 2021 19:11
@TimSheard @lehins
Added the Compact data modules
4f0106c
@TimSheard @lehins
Added delete and foldWithKey operations
532e4b2
@TimSheard @lehins
added DomainRestrict, starting splitHashMap.
6f2f372
@TimSheard @lehins
added splitHashMap and intersect.
864087d
@TimSheard @lehins
cleaned up, got rid of nextBits.
841ebfd
@TimSheard @lehins
Cleaned up added the HashMap module, same names ad Data.Map.Strict.
a757041
@TimSheard @lehins
changes from fixing bugs on Alexi's branch.
db587e0
@TimSheard @lehins
Added UnionWith, needs work on keys.
0e894da
@TimSheard @lehins
Fixed unionWith to track depth, to safey handle (Leaf k v) values.
b59dd47
@TimSheard @lehins
Added a TODO note
c8bdefe
@TimSheard @lehins
Fixed bug in fullNodeMask, consoidated 'bits' and 'bitsPerSubmask' in…
1a61e97 
…to 'bitsPerSegment'.
@TimSheard @lehins
Making union tests.
f4090cf
@lehins
Move new KeyMap/HashMap into its own package compact-map
a351b87
@lehins
Add first property test
96f049e
@TimSheard @lehins
Debugged UnionWith.
728d543
@TimSheard @lehins
Redid insertWithKey and lookupHM to no longer use BitState, addes str…
7623d8b 
…ictness ! to Two and One, wrote a bulk insertion.
@TimSheard @lehins
Cleaned up, debugged mylub, removed BtState.
1eb88a6
@lehins
Ormolize
cebce4c
@lehins lehins force-pushed the ts-compact branch 2 times, most recently from 9b283cc to 3ed31b6 Compare October 26, 2021 16:15
@lehins lehins self-requested a review October 26, 2021 17:49
lehins
lehins approved these changes Oct 26, 2021
View reviewed changes
Copy link
Collaborator

@lehins lehins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still work in progress, but is already functional and shows really good results with respect to reducing memory overhead when comparing to Data.Map.Strict from containers. We still need to add tests and reduce clutter by removing unused code. We will continue improving this compact-map package. However, considering that it is not being used by any other packages it is already safe to merge it in, despite its WIP state.

@lehins lehins merged commit cdbf64f into master Oct 26, 2021
@iohk-bors iohk-bors bot deleted the ts-compact branch October 26, 2021 17:54
@lehins lehins restored the ts-compact branch October 26, 2021 17:55
@lehins lehins deleted the ts-compact branch October 26, 2021 17:56
@lehins lehins mentioned this pull request Oct 26, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lehins lehins lehins approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@TimSheard @lehins