Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: errors reported by the race detector #13174

Merged
merged 9 commits into from
Jun 10, 2024
Merged

Conversation

bboreham
Copy link
Contributor

@bboreham bboreham commented Jun 7, 2024

What this PR does / why we need it:

Race conditions lead to hard-to-understand bugs in production.
While nearly all the code touched by this PR is for tests or diagnostics, it is a great help to run the race detector regularly, to avoid new races being added in production.

All tests now pass on my local machine with -race, apart from:

  • TestOwnsRetention/multiple_compactors, which fails with "Condition never satisfied"

TestGenerateDataSize is extremely slow under the race detector, taking ~10 minutes on my machine. Perhaps it should use smaller payloads?

Which issue(s) this PR fixes:
Fixes #8586

Special notes for your reviewer:
This PR is broken down into individual commits; it will be easier to review one at a time.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • NA Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • NA Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • NA For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • NA If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

cpuUsage remains as a global variable, but it is implemented using atomic values.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Use two different variables to silence the report.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
With a Mutex.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
@pull-request-size pull-request-size bot added size/L and removed size/M labels Jun 7, 2024
@bboreham bboreham changed the title Fix races fix: errors reported by the race detector Jun 7, 2024
With an atomic variable.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Avoid changing a global variable.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
We cannot share querier or block objects across goroutines, because
they have state that is updated. Instead, copy the blocks and create a
new querier each time the mock `FetchBlocks` is called.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Give everyone their own variable.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Using an atomic variable.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Using atomic variables.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
@bboreham bboreham marked this pull request as ready for review June 7, 2024 19:20
@bboreham bboreham requested a review from a team as a code owner June 7, 2024 19:20
Copy link
Collaborator

@paul1r paul1r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice. lgtm!

@chaudum chaudum merged commit 2b19dac into grafana:main Jun 10, 2024
59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

go test -race detects significant amount of data races
3 participants