This repository has been archived by the owner on Aug 23, 2023. It is now read-only.
add metricdefs to cassandra if memory idx DID succeed #560
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
we have 3 cases where we update cassandra idx after attempting to update memory idx. A) in memory && same partition, but old lastUpdate B) in memory but different partition C) not in memory yet B and C were correct, but in A, via 40c2f15 the following bug was introduced: 1) we saved defs to cassandra when the update in memory did *not* succeed. With the current code MemoryIdx.AddOrUpdateDef always returns error=nil, so this case could not manifest itself, although until recently (008737c) we returned idx.BranchUnderLeaf and idx.BothBranchAndLeaf errors. So for a while we saved defs with those errors to the cassandra index 2) we did not update cassandra when the memory AddOrUpdate did succeed. The consequence here is that the lastUpdate field in cassandra was out of date, so that when new instances start from scratch, loading the index, they may have excluded metrics from render requests when they were supposed to be included, but only for inactive series for which it did not receive any new points (since when it receives points, it'll update the LastUpdate in memory)
|
Nice find |
woodsaj
approved these changes
Mar 15, 2017
|
LGTM |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
we have 3 cases where we update cassandra idx after attempting to
update memory idx.
A) in memory && same partition, but old lastUpdate
B) in memory but different partition
C) not in memory yet
B and C were correct, but in A, via
40c2f15 in #504
the following bug was introduced:
we saved defs to cassandra when the update in memory did not succeed.
With the current code MemoryIdx.AddOrUpdateDef always returns error=nil,
so this case could not manifest itself, although until recently
(008737c) we returned
idx.BranchUnderLeaf and idx.BothBranchAndLeaf errors.
So for a while we saved defs with those errors to the cassandra index
we did not update cassandra when the memory AddOrUpdate did succeed.
The consequence here is that the lastUpdate field in cassandra was out
of date, so that when new instances start from scratch, loading the
index, they may have excluded metrics from render requests when they
were supposed to be included, but only for inactive series for which it
did not receive any new points (since when it receives points, it'll
update the LastUpdate in memory)