getDecisionStream does not return custom scope decisions by default #2617

laurentgoudet · 2023-11-27T13:01:11Z

What happened?

Without the scopes parameter set, getDecisions returns Ip, Range, and custom scopes (UserAgent here) decisions:

curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions?origins=cscli%2Ccrowdsec' | jq .
[
  {
    "duration": "3h50m28.543341271s",
    "id": 6177362,
    "origin": "cscli",
    "scenario": "manual 'ban' from ''",
    "scope": "UserAgent",
    "type": "ban",
    "value": "crowdsec"
  },
  {
    "duration": "3h50m42.543340911s",
    "id": 6177363,
    "origin": "cscli",
    "scenario": "manual 'ban' from ''",
    "scope": "Ip",
    "type": "ban",
    "value": "1.2.3.4"
  },
  {
    "duration": "3h59m54.543340425s",
    "id": 6177364,
    "origin": "cscli",
    "scenario": "manual 'ban' from ''",
    "scope": "Range",
    "type": "ban",
    "value": "1.2.3.0/24"
  }
]

However, by default (i.e. without scopes being set) getDecisionsStream does not return the custom scopes decisions, acting as in an scopes=Ip,Range filter was set:

curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions/stream?startup=true&origins=cscli%2Ccrowdsec' | jq .
{
  "deleted": null,
  "new": [
    {
      "duration": "3h49m55.067403477s",
      "id": 6177363,
      "origin": "cscli",
      "scenario": "manual 'ban' from ''",
      "scope": "Ip",
      "type": "ban",
      "uuid": "dff4470c-6e10-4b75-acf1-3a5d7dd4084c",
      "value": "1.2.3.4"
    },
    {
      "duration": "3h59m7.067401742s",
      "id": 6177364,
      "origin": "cscli",
      "scenario": "manual 'ban' from ''",
      "scope": "Range",
      "type": "ban",
      "uuid": "798e70b5-2d84-43aa-8cfd-8abe42b2ab71",
      "value": "1.2.3.0/24"
    }
  ]
}

This prevents me from using https://github.com/crowdsecurity/cs-custom-bouncer for acting on custom scope decisions, since those are not returned by default by the stream API that the bouncer is using.

Note that manually setting scopes to Ip,Range,UserAgent makes getDecisionsStream return those decisions:

curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions/stream?startup=true&scopes=Ip,Range,UserAgent&origins=cscli%2Ccrowdsec' | jq .
{
  "deleted": null,
  "new": [
    {
      "duration": "3h44m57.2303507s",
      "id": 6177362,
      "origin": "cscli",
      "scenario": "manual 'ban' from ''",
      "scope": "UserAgent",
      "type": "ban",
      "uuid": "ae12d4ba-fedf-42d2-8e5e-20aaafeab32e",
      "value": "crowdsec"
    },
    {
      "duration": "3h45m11.230350125s",
      "id": 6177363,
      "origin": "cscli",
      "scenario": "manual 'ban' from ''",
      "scope": "Ip",
      "type": "ban",
      "uuid": "dff4470c-6e10-4b75-acf1-3a5d7dd4084c",
      "value": "1.2.3.4"
    },
    {
      "duration": "3h54m23.230349652s",
      "id": 6177364,
      "origin": "cscli",
      "scenario": "manual 'ban' from ''",
      "scope": "Range",
      "type": "ban",
      "uuid": "798e70b5-2d84-43aa-8cfd-8abe42b2ab71",
      "value": "1.2.3.0/24"
    }
  ]
}

What did you expect to happen?

I would expect all scopes to be returned by default if scopes is not set, matching how the non-stream endpoint behaves.

How can we reproduce it (as minimally and precisely as possible)?

Add decisions with

cscli decision add -i 1.2.3.4
cscli decision add -r 1.2.3.0/24
cscli decision add --scope UserAgent --value crowdsec

Register an API key

cscli bouncer add MyTestBouncer

Use that API key to cURL the getDecisionsStream endpoint:

curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions?origins=cscli%2Ccrowdsec' | jq .

Notice that the custom scope (UserAgent) decision is missing from the list returned by getDecisionsStream

Anything else we need to know?

No response

Crowdsec version

$ cscli version
2023/11/27 13:00:52 version: v1.5.5-d2d788c5dc0a9e387635276623c6781774a9dfd4
2023/11/27 13:00:52 Codename: alphaga
2023/11/27 13:00:52 BuildDate: 2023-10-24_08:13:35
2023/11/27 13:00:52 GoVersion: 1.21.3
2023/11/27 13:00:52 Platform: docker
2023/11/27 13:00:52 libre2: C++
2023/11/27 13:00:52 Constraint_parser: >= 1.0, <= 2.0
2023/11/27 13:00:52 Constraint_scenario: >= 1.0, < 3.0
2023/11/27 13:00:52 Constraint_api: v1
2023/11/27 13:00:52 Constraint_acquis: >= 1.0, < 2.0

OS version

# On Linux:
$ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.18.4
PRETTY_NAME="Alpine Linux v3.18"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
$ uname -a
Linux ip-172-18-18-8 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 Linux

Enabled collections and parsers

$ cscli hub list -o raw
# paste output here

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* filenames: - /var/log/nginx/*.log - ./tests/nginx/nginx.log #this is not a syslog log, indicate which kind of logs it is labels: type: nginx --- filenames: - /var/log/auth.log - /var/log/syslog labels: type: syslog --- filename: /var/log/apache2/*.log labels: type: apache2 cat: can't open '/etc/crowdsec/acquis.d/*': No such file or directory

Config show

$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log/
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              :
  - Hub Folder              : /etc/crowdsec/hub
API Client:
  - URL                     : https://127.0.0.1:8080/
  - Login                   :
  - Credentials File        : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : 0.0.0.0:8080
  - Profile File            : /etc/crowdsec/profiles.yaml
  - Cert File : /tmp/server.pem
  - Key File  : /tmp/server-key.pem
  - CA Cert   : /tmp/ca_bundle.pem
  - Allowed Agents OU       : Freelancer Ltd
  - Allowed Bouncers OU       : Freelancer Ltd

  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : mysql
      - Host                : db-crowdsec.syd1.fln-dev.flnltd.com
      - Port                : 3306
      - User                : crowdsec
      - DB Name             : crowdsec
      - Flush age           : 7d
      - Flush size          : 5000

Prometheus metrics

$ cscli metrics
Local API Metrics:
╭──────────────────────┬────────┬──────╮
│        Route         │ Method │ Hits │
├──────────────────────┼────────┼──────┤
│ /v1/alerts           │ GET    │ 4    │
│ /v1/alerts           │ POST   │ 5    │
│ /v1/decisions        │ GET    │ 10   │
│ /v1/decisions/stream │ GET    │ 244  │
│ /v1/heartbeat        │ GET    │ 78   │
│ /v1/watchers/login   │ POST   │ 12   │
╰──────────────────────┴────────┴──────╯

Local API Machines Metrics:
╭──────────────────────────────┬───────────────┬────────┬──────╮
│           Machine            │     Route     │ Method │ Hits │
├──────────────────────────────┼───────────────┼────────┼──────┤
│ crowdsec_agent@127.0.0.1     │ /v1/alerts    │ GET    │ 2    │
│ crowdsec_agent@127.0.0.1     │ /v1/alerts    │ POST   │ 4    │
│ crowdsec_agent@172.18.14.178 │ /v1/heartbeat │ GET    │ 37   │
│ crowdsec_agent@172.18.17.100 │ /v1/alerts    │ GET    │ 2    │
│ crowdsec_agent@172.18.17.100 │ /v1/alerts    │ POST   │ 1    │
│ crowdsec_agent@172.18.17.100 │ /v1/heartbeat │ GET    │ 37   │
╰──────────────────────────────┴───────────────┴────────┴──────╯

Local API Bouncers Metrics:
╭────────────────────────────┬──────────────────────┬────────┬──────╮
│          Bouncer           │        Route         │ Method │ Hits │
├────────────────────────────┼──────────────────────┼────────┼──────┤
│ MyTestClient               │ /v1/decisions/stream │ GET    │ 12   │
│ MyTestClient               │ /v1/decisions        │ GET    │ 10   │
│ crowdsec_bouncer@127.0.0.1 │ /v1/decisions/stream │ GET    │ 232  │
╰────────────────────────────┴──────────────────────┴────────┴──────╯

Local API Bouncers Decisions:
╭──────────────┬───────────────┬───────────────────╮
│   Bouncer    │ Empty answers │ Non-empty answers │
├──────────────┼───────────────┼───────────────────┤
│ MyTestClient │ 0             │ 10                │
╰──────────────┴───────────────┴───────────────────╯

Local API Decisions:
╭────────────────────────────────────────────┬──────────┬─────────┬───────╮
│                   Reason                   │  Origin  │ Action  │ Count │
├────────────────────────────────────────────┼──────────┼─────────┼───────┤
│ crowdsecurity/f5-big-ip-cve-2020-5902      │ CAPI     │ ban     │ 22    │
│ crowdsecurity/ssh-slow-bf                  │ CAPI     │ ban     │ 12    │
│ crowdsecurity/CVE-2022-41082               │ CAPI     │ ban     │ 367   │
│ crowdsecurity/CVE-2022-42889               │ CAPI     │ ban     │ 4     │
│ crowdsecurity/http-cve-2021-41773          │ CAPI     │ ban     │ 20    │
│ crowdsecurity/http-path-traversal-probing  │ CAPI     │ ban     │ 24    │
│ crowdsecurity/CVE-2022-37042               │ CAPI     │ ban     │ 14    │
│ crowdsecurity/CVE-2023-22518               │ CAPI     │ ban     │ 15    │
│ crowdsecurity/http-backdoors-attempts      │ CAPI     │ ban     │ 255   │
│ crowdsecurity/netgear_rce                  │ CAPI     │ ban     │ 6     │
│ flnltd/http-bruteforce-login               │ crowdsec │ captcha │ 1     │
│ crowdsecurity/apache_log4j2_cve-2021-44228 │ CAPI     │ ban     │ 306   │
│ crowdsecurity/http-probing                 │ CAPI     │ ban     │ 742   │
│ crowdsecurity/grafana-cve-2021-43798       │ CAPI     │ ban     │ 57    │
│ crowdsecurity/http-sensitive-files         │ CAPI     │ ban     │ 8     │
│ crowdsecurity/jira_cve-2021-26086          │ CAPI     │ ban     │ 19    │
│ manual 'ban' from ''                       │ cscli    │ ban     │ 4     │
│ crowdsecurity/thinkphp-cve-2018-20062      │ CAPI     │ ban     │ 14    │
│ crowdsecurity/CVE-2019-18935               │ CAPI     │ ban     │ 32    │
│ crowdsecurity/CVE-2022-35914               │ CAPI     │ ban     │ 45    │
│ crowdsecurity/fortinet-cve-2018-13379      │ CAPI     │ ban     │ 76    │
│ crowdsecurity/http-bad-user-agent          │ CAPI     │ ban     │ 2402  │
│ crowdsecurity/http-crawl-non_statics       │ CAPI     │ ban     │ 147   │
│ crowdsecurity/http-open-proxy              │ CAPI     │ ban     │ 267   │
│ crowdsecurity/nginx-req-limit-exceeded     │ CAPI     │ ban     │ 47    │
│ crowdsecurity/CVE-2022-26134               │ CAPI     │ ban     │ 165   │
│ crowdsecurity/CVE-2023-22515               │ CAPI     │ ban     │ 4     │
│ crowdsecurity/http-generic-bf              │ CAPI     │ ban     │ 8     │
│ crowdsecurity/ssh-bf                       │ CAPI     │ ban     │ 9922  │
╰────────────────────────────────────────────┴──────────┴─────────┴───────╯

Local API Alerts:
╭──────────────────────────────┬───────╮
│            Reason            │ Count │
├──────────────────────────────┼───────┤
│ flnltd/http-bruteforce-login │ 2     │
│ manual 'ban' from ''         │ 4     │
╰──────────────────────────────┴───────╯

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

The text was updated successfully, but these errors were encountered:

github-actions · 2023-11-27T13:01:26Z

@laurentgoudet: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

Check Crowdsec Documentation to see if your issue can be self resolved.
You can also join our Discord.
Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

LaurenceJJones · 2023-12-14T09:13:10Z

Hi thank you for your report,

We intentionally set those scopes if non is provided

crowdsec/pkg/apiserver/controllers/v1/decisions.go

Lines 353 to 355 in 67cdf91

    
           if _, ok := filters["scopes"]; !ok { 
        
           	filters["scopes"] = []string{"ip,range"} 
        
           }

I will ask the team if this should be the same as non stream or if we should not do this.

laurentgoudet · 2023-12-14T20:13:29Z

Thanks. Arguably the inconsistency between getDecisions & getDecisionsStream is just a minor annoyance, what I am really trying to achieve here is being able to use https://github.com/crowdsecurity/cs-custom-bouncer with custom scopes, without having to rebuild the whole thing myself just to pass a custom scopes filter.

LaurenceJJones · 2023-12-15T09:09:23Z

Thanks. Arguably the inconsistency between getDecisions & getDecisionsStream is just a minor annoyance, what I am really trying to achieve here is being able to use https://github.com/crowdsecurity/cs-custom-bouncer with custom scopes, without having to rebuild the whole thing myself just to pass a custom scopes filter.

You shouldnt have to rebuild it, our go SDK already has scopes yaml keys.

https://github.com/crowdsecurity/go-cs-bouncer/blob/ff0ac9af53a6110b88e415b0a0e876cfc82341ea/stream_bouncer.go#L39C2-L39C8

So within all bouncers you can add the following to the configuration as an example

scopes:
  - ip
  - range
  - useragent

laurentgoudet added the kind/bug Something isn't working label Nov 27, 2023

github-actions bot added version/1.5.5 needs/triage labels Nov 27, 2023

LaurenceJJones assigned blotus Dec 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

getDecisionStream does not return custom scope decisions by default #2617

getDecisionStream does not return custom scope decisions by default #2617

laurentgoudet commented Nov 27, 2023 •

edited

Loading

github-actions bot commented Nov 27, 2023

LaurenceJJones commented Dec 14, 2023

laurentgoudet commented Dec 14, 2023

LaurenceJJones commented Dec 15, 2023 •

edited

Loading

getDecisionStream does not return custom scope decisions by default #2617

getDecisionStream does not return custom scope decisions by default #2617

Comments

laurentgoudet commented Nov 27, 2023 • edited Loading

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Crowdsec version

OS version

Enabled collections and parsers

Acquisition config

Config show

Prometheus metrics

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions bot commented Nov 27, 2023

LaurenceJJones commented Dec 14, 2023

laurentgoudet commented Dec 14, 2023

LaurenceJJones commented Dec 15, 2023 • edited Loading

laurentgoudet commented Nov 27, 2023 •

edited

Loading

LaurenceJJones commented Dec 15, 2023 •

edited

Loading