Use LMDB and sparse vectors in nn_ensemble backend #381
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ating large vectors in RAM (#363)
Codecov Report
@@ Coverage Diff @@
## master #381 +/- ##
==========================================
+ Coverage 99.38% 99.39% +<.01%
==========================================
Files 59 59
Lines 3738 3794 +56
==========================================
+ Hits 3715 3771 +56
Misses 23 23
Continue to review full report at Codecov.
|
|
This pull request introduces 1 alert when merging 41d2953 into 21d92bf - view on LGTM.com new alerts:
|
|
Still need to make sure the LMDB dependencies are included in Docker images. |
|
Kudos, SonarCloud Quality Gate passed!
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this PR, the nn_ensemble backend collected score and label (target) vectors in memory, which took a lot of RAM when there are thousands of documents, multiple source projects and a large vocabulary such as YSO. This PR makes the nn_ensemble backend instead spool those vectors into an on-disk, memory-mapped LMDB database instead (implemented as a keras.utils.Sequential subclass). The vectors are also compressed by representing them as SciPy sparse vectors.
The LMDB training data can also be reused in later training rounds by using the recently introduced
--cachedoption, although this functionality is not yet implemented in the initial commit.Fixes #363