No space left on device

[aukfood]

What happened?

When I start to run crowdsec I have these error

time="2024-02-02T16:22:43+01:00" level=error msg="error in stop : no space left on device" type=file

What did you expect to happen?

A normal start of crowdsec :)

How can we reproduce it (as minimally and precisely as possible)?

We add acquisition for all apache log of all web site

---
filenames:
  - /var/www/*/var/log/apache2/access*
  - /var/www/*/var/log/apache2/error*
labels:
  type: apache2
---

Anything else we need to know?

No response

Crowdsec version

$ cscli version
2024/02/12 15:03:07 version: v1.6.0-debian-pragmatic-amd64-4b8e6cd7
2024/02/12 15:03:07 Codename: alphaga
2024/02/12 15:03:07 BuildDate: 2024-01-24_11:01:12
2024/02/12 15:03:07 GoVersion: 1.21.3
2024/02/12 15:03:07 Platform: linux
2024/02/12 15:03:07 libre2: C++
2024/02/12 15:03:07 Constraint_parser: >= 1.0, <= 3.0
2024/02/12 15:03:07 Constraint_scenario: >= 1.0, <= 3.0
2024/02/12 15:03:07 Constraint_api: v1
2024/02/12 15:03:07 Constraint_acquis: >= 1.0, < 2.0

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux ceciaa2 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64 GNU/Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Enabled collections and parsers

$ cscli hub list -o raw
name,status,version,description,type
crowdsecurity/apache2-logs,enabled,1.4,Parse Apache2 access and error logs,parsers
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers
crowdsecurity/http-logs,enabled,1.2,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers
crowdsecurity/iptables-logs,enabled,0.5,Parse iptables drop logs,parsers
crowdsecurity/mysql-logs,enabled,0.4,Parse MySQL logs,parsers
crowdsecurity/pkexec-logs,enabled,0.1,Parse pkexec logs specifically for CVE-2021-4034,parsers
crowdsecurity/segfault-logs,enabled,0.4,Parses segfault kernel side,parsers
crowdsecurity/sshd-logs,enabled,2.2,Parse openSSH logs,parsers
crowdsecurity/syslog-logs,enabled,0.8,,parsers
crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers
mywhitelists,"enabled,local",,,parsers
crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.5,Detect cve-2021-44228 exploitation attemps,scenarios
crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios
crowdsecurity/CVE-2021-4034,enabled,0.2,Detect CVE-2021-4034 exploits,scenarios
crowdsecurity/CVE-2022-26134,enabled,0.2,Detect CVE-2022-26134 exploits,scenarios
crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios
crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios
crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios
crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios
crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios
crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios
crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios
crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios
crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios
crowdsecurity/CVE-2023-22518,enabled,0.2,Detect CVE-2023-22518 exploits,scenarios
crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios
crowdsecurity/CVE-2023-4911,enabled,0.5,exploitation of CVE-2023-4911: segfaulting in dynamic loader,scenarios
crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.2,Detect cve-2020-5902 exploitation attemps,scenarios
crowdsecurity/fortinet-cve-2018-13379,enabled,0.3,Detect cve-2018-13379 exploitation attemps,scenarios
crowdsecurity/grafana-cve-2021-43798,enabled,0.2,Detect cve-2021-43798 exploitation attemps,scenarios
crowdsecurity/http-admin-interface-probing,enabled,0.3,Detect generic HTTP admin interface probing,scenarios
crowdsecurity/http-backdoors-attempts,enabled,0.5,Detect attempt to common backdoors,scenarios
crowdsecurity/http-bad-user-agent,enabled,1.1,Detect usage of bad User Agent,scenarios
crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios
crowdsecurity/http-cve-2021-41773,enabled,0.2,cve-2021-41773,scenarios
crowdsecurity/http-cve-2021-42013,enabled,0.2,cve-2021-42013,scenarios
crowdsecurity/http-generic-bf,enabled,0.6,Detect generic http brute force,scenarios
crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios
crowdsecurity/http-path-traversal-probing,enabled,0.3,Detect path traversal attempt,scenarios
crowdsecurity/http-probing,enabled,0.3,Detect site scanning/probing from a single ip,scenarios
crowdsecurity/http-sensitive-files,enabled,0.3,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios
crowdsecurity/http-sqli-probing,enabled,0.3,A scenario that detects SQL injection probing with minimal false positives,scenarios
crowdsecurity/http-xss-probing,enabled,0.3,A scenario that detects XSS probing with minimal false positives,scenarios
crowdsecurity/iptables-scan-multi_ports,enabled,0.2,ban IPs that are scanning us,scenarios
crowdsecurity/jira_cve-2021-26086,enabled,0.2,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios
crowdsecurity/mysql-bf,enabled,0.2,Detect mysql bruteforce,scenarios
crowdsecurity/netgear_rce,enabled,0.3,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios
crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.3,Detect cve-2019-11510 exploitation attemps,scenarios
crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios
crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios
crowdsecurity/thinkphp-cve-2018-20062,enabled,0.4,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios
crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios
crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.2,Detect VMSA-2021-0027 exploitation attemps,scenarios
ltsich/http-w00tw00t,enabled,0.2,detect w00tw00t,scenarios
crowdsecurity/firewall_base,enabled,0.2,,contexts
crowdsecurity/http_base,enabled,0.2,,contexts
crowdsecurity/apache2,enabled,0.1,apache2 support : parser and generic http scenarios ,collections
crowdsecurity/base-http-scenarios,enabled,0.8,http common : scanners detection,collections
crowdsecurity/http-cve,enabled,2.5,Detect CVE exploitation in http logs,collections
crowdsecurity/iptables,enabled,0.2,iptables support : logs and port-scans detection scenarios,collections
crowdsecurity/linux,"enabled,tainted",0.2,core linux support : syslog+geoip+ssh,collections
crowdsecurity/linux-lpe,"enabled,tainted",0.2,Linux Local Privilege Escalation collection : detect trivial LPEs,collections
crowdsecurity/mysql,enabled,0.1,mysql support : logs and brute-force scenarios,collections
crowdsecurity/sshd,"enabled,tainted",0.3,sshd support : parser and brute-force detection,collections

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* #Generated acquisition file - wizard.sh (service: apache2) / files : /var/log/apache2/error.log /var/log/apache2/other_vhosts_access.log /var/log/apache2/access.log filenames: - /var/log/apache2/error.log - /var/log/apache2/other_vhosts_access.log - /var/log/apache2/access.log - /var/www/*/var/log/apache2/access.* - /var/www/*/var/log/apache2/error.* labels: type: apache2 --- #Generated acquisition file - wizard.sh (service: sshd) / files : /var/log/auth.log filenames: - /var/log/auth.log labels: type: syslog --- #Generated acquisition file - wizard.sh (service: mysql) / files : /var/log/mysql/error.log filenames: - /var/log/mysql/error.log labels: type: mysql --- #Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog /var/log/kern.log /var/log/messages filenames: - /var/log/syslog - /var/log/kern.log - /var/log/messages labels: type: syslog --- # BEGIN ANSIBLE MANAGED BLOCK filenames: - /var/www/*/var/log/apache2/access* - /var/www/*/var/log/apache2/error* labels: type: apache2 --- # END ANSIBLE MANAGED BLOCK # Mise en place manuelle --- filenames: - /var/www/*/var/log/apache2/access* - /var/www/*/var/log/apache2/error* labels: type: apache2 ---

On Windows:

C:> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml

paste output here

Config show

$ cscli config show
# paste output here

Prometheus metrics

$ cscli metrics
FATA[2024-02-12T15:05:08+01:00] failed to fetch prometheus metrics: executing GET request for URL "http://127.0.0.1:6060/metrics" failed: Get "http://127.0.0.1:6060/metrics": dial tcp 127.0.0.1:6060: connect: connection refused 

Because crowdsec doesn't start

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

Best regards

[github-actions]

@aukfood: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[LaurenceJJones]

Did you try the steps of expanding the inotify limit?

How many files do you have within folders? because what is the value of monitoring really old files that wont be written too again?

[aukfood]

@LaurenceJJones yes I try this : https://discourse.crowdsec.net/t/problem-config-for-acquisition/1257/4

With 8192 and

fs.inotify.max_user_instances=16384

And same error.

I have more than 13594 files :

# ls -la /var/www/*/var/log/apache2/ |wc -l
13594

[aukfood]

@LaurenceJJones there is another solution ?

[LaurenceJJones]

@LaurenceJJones there is another solution ?

Not really, reduce the amount of logs that match the globing pattern as CrowdSec is monitoring old files that will never be used.

[LaurenceJJones]

Unless @blotus has anymore ideas?

[aukfood]

yes I think I have to modify my acquisition to match the daily file and not old files

[aukfood]

@LaurenceJJones it's possible in acquisition to have this format ?

---
filenames:
  - /var/www/*/var/log/apache2/access.%Y.%m.%d
  - /var/www/*/var/log/apache2/error.%Y.%m.%d
labels:
  type: apache2

[blotus]

Hello @aukfood,

Can you try to increase more (at least double) the max amount of user watches ?

You have around 13k files, but crowdsec will also add a watch on each file by default to get notified where there's a new line (you can disable this behavior by setting poll_without_inotify: true, but crowdsec will revert to calling stat() very frequently on each file, which will make your CPU usage explode.

[aukfood]

@blotus I try 32000 but no results.

Where to add poll_without_inotify: true ??? I try in section common in config.yaml

[LaurenceJJones]

@blotus I try 32000 but no results.

Where to add poll_without_inotify: true ??? I try in section common in config.yaml

You add it like this

---
filenames:
  - /var/www/*/var/log/apache2/access*
  - /var/www/*/var/log/apache2/error*
poll_without_inotify: true
labels:
  type: apache2

[aukfood]

@LaurenceJJones @blotus no change with 32000 files and poll_without_inotify: true

poll_without_inotify: true

[LaurenceJJones]

@LaurenceJJones @blotus no change with 32000 files and poll_without_inotify: true

poll_without_inotify: true

And you dont have any duplicate entries?

cat /etc/crowdsec/acquis.yaml
cat /etc/crowdsec/acquis.d/*.yaml

[aukfood]

@LaurenceJJones no i have no configuration in acquis.d directory

cat /etc/crowdsec/acquis.d/*.yaml
cat: '/etc/crowdsec/acquis.d/*.yaml': Aucun fichier ou dossier de ce type

[LaurenceJJones]

This issue been open for sometime, I will class it as not planned but if you still have an issue then please reopen with additional details.