Fix lru leak #2
Merged
Fix lru leak #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Rather than supplying a magic constant of 2, we can pull the contant value off of the record.
The Bucket Index in the riak_kv_lru cahce is of the ETS type
bag. Hence if we store {BKey, Key} into the table, its key will be
BKey only. Trying to call ets:delete(BucketIdx, {BKey, Key}) will then
fail to match since we are searching for {Bkey, Key} == BKey. Changing
the deletion routine to be ets:delete_object/2, fixes this problem.
The consequence is that riaks LRU does not leak memory anymore when
doing extensive Map/Reduce queries.
While here, provide a test case for the problem: Introduce a
consistency variant -- all tables must be of the same size when
full. Fill the table and test that the invariant is true. This fails
before this patch, but succeeds after.
|
Thanks for catching the leak. I've merged your fix into master. |
jtuple
added a commit
that referenced
this pull request
Nov 2, 2012
Active anti-entropy is a process through which multiple replicas periodically sync with each other and automatically repair any keys that are missing or have divergent data. For example, a user could delete all data from a node and Riak would automatically regenerate the data from the other replicas. This implementation uses hash trees / Merkle trees to perform lightweight key exchange, with work proportional to the number of divergent keys rather than the size of the overall keyspace. This implementation meets several design goals: 1. The underlying hash trees are on-disk structures, and can scale to billions+ keys. This is in contrast to in-memory trees that require significant RAM to support massive keyspaces. 2. The underlying hash trees are persistent. Riak nodes can be restarted without fear of hash tree data being lost and needing to be rebuilt. 3. As new data is written to Riak, the hash trees associated with the various partitions are kept up-to-date. Each write in Riak triggers an asynchronous write to one or more hash trees. Combined with #2, this enables trees to be built once through a scan over existing data, and then maintained in real-time. In reality, trees are expired over time and rebuilt to ensure the hash tree and backend data stay in sync, and also to identify bit rot / disk failure. 4. The entire implementation is designed to be non-blocking. For example, a snapshot of a hash tree is generated before performing an exchange with other replicas, therefore allowing concurrent inserts of new key/hash pairs as new writes occur. The current implementation triggers read repair for each key difference identified through the hash exchange. This is a reasonable approach as read repair is stable, production-tested mechanism in Riak. However, the read repair approach leads to slower replica repair in cases where there are a large number of key differences. This is an area for future improvement.
Closed
hmmr
pushed a commit
that referenced
this pull request
Apr 4, 2016
…develop amend create_table code path to take into account WITH-props
This pull request was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi,
This patch set fixes a (memory) leak in the LRU cache which I saw when running some Erlang Map/Reduce queries through the riakc_pb_socket interface. The commit message describes the problem and the one-line fix that plugs the leak in question.
Test case also provided.