Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

osquery-perf fails with fatal error: concurrent map read and map write #24381

Open
getvictor opened this issue Dec 4, 2024 · 0 comments
Open
Assignees
Labels
~backend Backend-related issue. bug Something isn't working as documented ~dogfood Issue resulted from Fleet's product dogfooding. #g-mdm MDM product group :incoming New issue in triage process. :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. ~released bug This bug was found in a stable release.

Comments

@getvictor
Copy link
Member

Fleet version:
4.60.0
Web browser and operating system:


💥  Actual behavior

In loadtest, after enrolling 60k MDM hosts, I'm occasionally seeing osquery-perf containers die and restart.
From the logs for a stopped container, I see:
image.png

The full loadtest doc is at https://docs.google.com/document/d/1HafECokrZ3jnzRskxMtJwp4k1E2uBTbO9vfKEUtyykI

🧑‍💻  Steps to reproduce

  1. See loadtest doc.

🕯️ More info (optional)

The message points at this line in agent.go:

			if existingEntry, ok := a.scheduledQueryData[scheduledQueryName]; ok {

I recommend using sync.Map for scheduledQueryData struct member.

@getvictor getvictor added #g-mdm MDM product group :incoming New issue in triage process. :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. bug Something isn't working as documented ~backend Backend-related issue. ~dogfood Issue resulted from Fleet's product dogfooding. ~released bug This bug was found in a stable release. labels Dec 4, 2024
@dantecatalfamo dantecatalfamo self-assigned this Dec 5, 2024
@georgekarrv georgekarrv added this to the 4.62.0-tentative milestone Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
~backend Backend-related issue. bug Something isn't working as documented ~dogfood Issue resulted from Fleet's product dogfooding. #g-mdm MDM product group :incoming New issue in triage process. :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. ~released bug This bug was found in a stable release.
Development

No branches or pull requests

3 participants