-
Notifications
You must be signed in to change notification settings - Fork 429
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Scenario file scanner scans inside Kubernetes configmap mount dirs #3134
Comments
@W1zzardTPU: Thanks for opening an issue, it is currently awaiting triage. In the meantime, you can:
DetailsI am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository. |
Hi, thanks for reporting the issue. Actually the failure comes from the files being in a subdirectory (..data), which is not part of the expected hierarchy. This can be replicated by creating scenarios/somedir/somefile.yaml We can nerf the errors because these are valid configurations, and I'll leave them as warnings, even with commands like "cscli hub list". Ignoring altogether the directories starting with a dot can be a future improvement. The issue should be fixed in #3150 but since we just released 1.6.3-rc2 today, you can test it by deploying the crowdsecurity/crowdsec:dev image which follows the master branch. Let me know if it works and we can ship it in the final release. Thanks again! |
Indeed, you are right. The underlying problem is that when mounting a configmap with multiple entries I can't "merge" directories, so I mounted into a subdir. Is there any way to change the directory parser to understand nested directories? Maybe just one additional level? Maybe mark as "author" like for hub? If I remember correctly, the documentation doesn't mention that subdirectories are not supported for these files. hmm .. then why even scan into subdirs? |
There are collections/somefile.yaml, scenarios/somefile.yaml, parsers/stage/somefile.yaml... no subdirectories are recognized except for parsers/postoverflows, but linking to them is fine, so your configuration should work. The file is read twice and the symlink is the one that should be used, you'll get a benign warning for the other one. I'll remove the warning later by ignoring directories starting with '..' |
These symlinks are generated automatically by Kubernetes, I have no control over them. When mounting a configmap with multiple entries, an empty directory gets created, the actual files are mounted into these "..*" directories and then symlinked to the target directory. I can't mount multiple files into just /scenarios/ (unless I mount them one-by-one, which is impractical once you have a significant number of them |
Hi @W1zzardTPU we have a work in progress to allow subdirectories and link nesting but we are too close to 1.6.3. Could you mount your files one-by-one (hence no hidden directories or links) until 1.6.4 is out? Thanks |
That's what I'm doing while growing my set of rules, not running in prod yet anyway. Thanks! |
@W1zzardTPU this should do it. You mount a single configmap per type, in a directory (one for scenarios, one per parsers..) /etc/crowdsec/scenarios/myconfig/foo.yaml and the name of the scenario will be "myconfig/foo.yaml" instead of "foo.yaml". It will work with files or links to ..data directories, which are ignored if nothing links to them. |
What happened?
When Kubernetes mounts a configmap as folder inside a pod, it uses these directories prefixed with "..".
Crowdsec will try to scan into this directory and fail with "failed to sync items: failed to scan /etc/crowdsec: unknown configuration type for file '/etc/crowdsec/scenarios/foo/..2024_07_16_10_46_34.998354025/http-401.yaml'
In itemVisit(), can we ignore files and directories with leading '.' ?
crowdsec/pkg/cwhub/sync.go
Line 177 in c4bfdf1
What did you expect to happen?
Crowdsec should start without error and load the scenarios
How can we reproduce it (as minimally and precisely as possible)?
Not sure how to repro without a working Kubernetes setup
Anything else we need to know?
No response
Crowdsec version
cscli version
version: v1.6.2-16bfab86
Codename: alphaga
BuildDate: 2024-06-05_14:25:55
GoVersion: 1.22.3
Platform: docker
libre2: C++
User-Agent: crowdsec/v1.6.2-16bfab86-docker
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
OS version
Enabled collections and parsers
Acquisition config
On Windows:
C:> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml
paste output here
Config show
Prometheus metrics
Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.
The text was updated successfully, but these errors were encountered: