Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ring buffer gap cleanup code #578

Conversation

matthias-wende-frequenz
Copy link
Contributor

@matthias-wende-frequenz matthias-wende-frequenz commented Aug 10, 2023

When cleaning up gaps the case that a gaps start date can become older then the oldest stored value in the buffer wasn't covered. Furthermore the cases for cleaning up a single gap didn't run for the last gaps, since the loop was stopping to early.

@matthias-wende-frequenz matthias-wende-frequenz requested a review from a team as a code owner August 10, 2023 09:25
@github-actions github-actions bot added part:tests Affects the unit, integration and performance (benchmarks) tests part:data-pipeline Affects the data pipeline part:docs Affects the documentation labels Aug 10, 2023
@matthias-wende-frequenz matthias-wende-frequenz added this to the v0.25.0 milestone Aug 10, 2023
@@ -328,17 +328,24 @@ def _cleanup_gaps(self) -> None:
self._gaps = sorted(self._gaps, key=lambda x: x.start.timestamp())

i = 0
while i < len(self._gaps) - 1:
while i < len(self._gaps):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Def should have a test for this and the other new case

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a case for the new case at least

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added another test case for the "old" bug.

@Marenz
Copy link
Contributor

Marenz commented Aug 10, 2023

Your commit message claims

When cleaning up gaps the case that a gaps end date can become older
then the oldest stored value in the buffer wasn't covered.

But I don't think it's true. It's covered right here:

            # Delete out-of-date gaps
            if w_1.end <= self._datetime_oldest:
                del self._gaps[i]

@Marenz
Copy link
Contributor

Marenz commented Aug 10, 2023

About having gaps whos start time is older than the oldest.. I am not sure that's really important, but I guess it doesn't hurt to clamp it

@Marenz
Copy link
Contributor

Marenz commented Aug 10, 2023

Also, in your commit message you're wrongly using "then" when it should be "than" :)

@matthias-wende-frequenz
Copy link
Contributor Author

Your commit message claims

When cleaning up gaps the case that a gaps end date can become older
then the oldest stored value in the buffer wasn't covered.

But I don't think it's true. It's covered right here:

            # Delete out-of-date gaps
            if w_1.end <= self._datetime_oldest:
                del self._gaps[i]

No, that case deletes the whole gap once it's fully outside of the buffer, i.e. it looks only at the end of the gap. The new case looks at and updates the beginning of the gap.

@matthias-wende-frequenz
Copy link
Contributor Author

matthias-wende-frequenz commented Aug 10, 2023

About having gaps whos start time is older than the oldest.. I am not sure that's really important, but I guess it doesn't hurt to clamp it

We need that in the moving window. But maybe it's not a bug, but more like we covering an unforeseen use case. We want to know the number of missing items in the window and now we can simply sum up the length of all gaps.

@Marenz
Copy link
Contributor

Marenz commented Aug 10, 2023

I am sorry, but I am confused. The commit message talks literally about the end date being older than oldest timestamp

When cleaning up gaps the case that a gaps end date can become older
then the oldest stored value in the buffer wasn't covered.

And this matches also what you say my case is..

No, that case deletes the whole gap once it's fully outside of the buffer, i.e. it looks only at the end of the gap.

This though

The new case looks at and updates the beginning of the gap.

is true, however the message should then say "Clean up gap starts if they are before the oldest timestamp" or so

@Marenz
Copy link
Contributor

Marenz commented Aug 10, 2023

PR LGTM, just the first commit message greatly confuses

@matthias-wende-frequenz
Copy link
Contributor Author

matthias-wende-frequenz commented Aug 10, 2023

I am sorry, but I am confused. The commit message talks literally about the end date being older than oldest timestamp

When cleaning up gaps the case that a gaps end date can become older
then the oldest stored value in the buffer wasn't covered.

And this matches also what you say my case is..

No, that case deletes the whole gap once it's fully outside of the buffer, i.e. it looks only at the end of the gap.

This though

The new case looks at and updates the beginning of the gap.

is true, however the message should then say "Clean up gap starts if they are before the oldest timestamp" or so

Did I ever mention that I'm having troubles with signs :D. Indeed the the start time of the gap is meant. I'll update the message accordingly.
To be fair. The code tells the truth :).

When cleaning up gaps the case that a gaps start date can become older
than the oldest stored value in the buffer wasn't covered. Furthermore
the cases for cleaning up a single gap didn't run for the last gaps,
since the loop was stopping too early.

Signed-off-by: Matthias Wende <matthias.wende@frequenz.com>
Adds a test case for updating the start time of gaps when it's older than
the oldest stored element in the buffer.

Signed-off-by: Matthias Wende <matthias.wende@frequenz.com>
Signed-off-by: Matthias Wende <matthias.wende@frequenz.com>
Signed-off-by: Matthias Wende <matthias.wende@frequenz.com>
@matthias-wende-frequenz matthias-wende-frequenz added this pull request to the merge queue Aug 10, 2023
Merged via the queue into frequenz-floss:v0.x.x with commit 7bbfd10 Aug 10, 2023
@matthias-wende-frequenz matthias-wende-frequenz deleted the fix_ringbuffer branch August 10, 2023 12:45
Comment on lines +19 to +22

## Bug Fixes

- Fixes a bug in the ring buffer updating the end timestamp of gaps when they are outdated.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are a bad bad boy. You should have first approved and merged #574.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wops

@llucax llucax mentioned this pull request Aug 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
part:data-pipeline Affects the data pipeline part:docs Affects the documentation part:tests Affects the unit, integration and performance (benchmarks) tests
Projects
Development

Successfully merging this pull request may close these issues.

3 participants