[prometheus] Consider adding native support for cs_lapi_decision

[DuvelCorp]

What happened?

Just installed Crowdsec for Caddy.
Everything seems to work fine
I triggered myself from a VPN IP an alert by scanning manually several sensitive files, and I got banned as expected :

tom@cerbere:/opt/crowdsec/config$ docker exec crowdsec cscli decisions list
+-------+----------+------------------+------------------------------------+--------+---------+---------------------------------------+--------+------------+----------+
|   ID  |  Source  |    Scope:Value   |               Reason               | Action | Country |                   AS                  | Events | expiration | Alert ID |
+-------+----------+------------------+------------------------------------+--------+---------+---------------------------------------+--------+------------+----------+
| 84721 | crowdsec | Ip:45.134.79.139 | crowdsecurity/http-sensitive-files | ban    | FR      | 39486 HostRoyale Technologies Pvt Ltd | 6      | 2h49m26s   | 8        |
+-------+----------+------------------+------------------------------------+--------+---------+---------------------------------------+--------+------------+----------+

tom@cerbere:/opt/crowdsec/config$ docker exec crowdsec cscli alerts list
+----+------------------+------------------------------------+---------+---------------------------------------+-----------+-----------------------------------------+
| ID |       value      |               reason               | country |                   as                  | decisions |                created_at               |
+----+------------------+------------------------------------+---------+---------------------------------------+-----------+-----------------------------------------+
| 8  | Ip:45.134.79.139 | crowdsecurity/http-sensitive-files | FR      | 39486 HostRoyale Technologies Pvt Ltd | ban:1     | 2024-10-19 16:04:34.205253523 +0000 UTC |
+----+------------------+------------------------------------+---------+---------------------------------------+-----------+-----------------------------------------+

Prometheus seems to retrieve correctly most of crowdsec metrics :

However I dont get any metric cs_lapi_decision
According to me I should have received it as there is an active decision on-going.
This prevent me to add a list of banned IPs and their Geomap in Grafana, which probably the most important information.

Is there something I missed?

What did you expect to happen?

Get metric data cs_lapi_decision

How can we reproduce it (as minimally and precisely as possible)?

I dont know

Anything else we need to know?

No response

Crowdsec version

version: v1.6.3-4851945a
Codename: alphaga
BuildDate: 2024-09-12_09:39:08
GoVersion: 1.22.6
Platform: docker
libre2: C++
User-Agent: crowdsec/v1.6.3-4851945a-docker
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Enabled collections and parsers

$ cscli hub list -o raw
# paste output here

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* # paste output here

On Windows:

C:> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml

paste output here

Config show

$ cscli config show
# paste output here

Prometheus metrics

$ cscli metrics
# paste output here

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

[github-actions]

@DuvelCorp: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[DuvelCorp]

Hello

After a few days, my lapi and caddy/crowdsec seems to work fine. I've got real alerts :

Some of my grafana dashboards works perfectly:

But, still : other dashboard that are supposed to report the IPs banned and their geoloc are not working because I am missing some cs_lapi metrics.

This all what I have :

And thus I am missing a part of those listed in this doc : https://docs.crowdsec.net/docs/next/observability/prometheus/

Local API
cs_lapi_route_requests_total : number of calls to each route per method
cs_lapi_machine_requests_total : number of calls to each route per method grouped by machines
cs_lapi_bouncer_requests_total : number of calls to each route per method grouped by bouncers
cs_lapi_decisions_ko_total : number of unsuccessfully responses when bouncers ask for an IP.
cs_lapi_decisions_ok_total : number of successfully responses when bouncers ask for an

Any hint please

[LaurenceJJones]

Hey 👋🏻

We dont provide the actual decisions (ip address and geo location data) via the Prometheus endpoint we only provide metrics about counts (EG: scenarios has trigger X times).

Thing to note is Prometheus will only expose metrics it has counts for so the list you provided has some:

cs_lapi_route_requests_total : number of calls to each route per method
cs_lapi_machine_requests_total : number of calls to each route per method grouped by machines

However I dont see the ones below:

cs_lapi_bouncer_requests_total : number of calls to each route per method grouped by bouncers
cs_lapi_decisions_ko_total : number of unsuccessfully responses when bouncers ask for an IP.
cs_lapi_decisions_ok_total : number of successfully responses when bouncers ask for an IP.

So the question is do you have a remediation component (bouncers) interfacing with this machine LAPI?

[DuvelCorp]

@LaurenceJJones Thank you for your answer.

If that was not clear I obviously have a bouncer embedded in Caddy using custom built image
this https://github.com/hslatman/caddy-crowdsec-bouncer

But now that you state that you dont expose this metric I have understood my mistake

I imported this dashboard in Grafana : https://grafana.com/grafana/dashboards/21689-crowdsec-cyber-threat-insights/
And I didnt read the doc correctly. They use a VictoriaMetrics component between Crowdsec and prometheus to enrich crowdsec metrics with notably Geoip, and they label it "cs_lapi_decision" : https://freefd.github.io/articles/8_cyber_threat_insights_with_crowdsec_victoriametrics_and_grafana/

 {
    "metric": {
        "__name__": "cs_lapi_decision",
        "instance": "host00.domain.tld",
        "country": "SG",
        "asname": "Tencent Building, Kejizhongyi Avenue",
        "asnumber": "132203",
        "latitude": "1.2868",
        "longitude": "103.8503",
        "iprange": "129.226.0.0/16",
        "scenario": "crowdsecurity/ssh-slow-bf",
        "type": "ban",
        "duration": "4h",
        "scope": "Ip",
        "ip": "129.226.84.230"
    },
    "values": [1],
    "timestamps": [1723013301000]
}

My mistake was that seing the name, I had expected that this was coming directly from Crowdsec metrics.
Sorry for the useless bug report :-)

On a side note, its a bit over killing to setup an additional metric component like Victoria to enrich and serve that information, considering that all data needed, including Maxmind GeoIP, is already available in Crowdsec LAPI.
So I hope you will consider to add that kind of metrics at some point, as it is extremely helpful... and you can turn my bug report into a feature request !

[FlorentLM]

Hah, I was looking for this too!

Following this issue

[joshoram80]

I am also following this. I have everything setup exactly.

cs_lapi_decision{instance=~".*"}[86400s:1m]
WARNING: No match! This query hasn't selected any time series from database. Either the requested metrics are missing in the database, or there is a typo in series selector

It seems like

type: http
name: http_default
log_level: info
format: >
  {{- range $Alert := . -}}
  {{- range .Decisions -}}
  {"metric":{"__name__":"cs_lapi_decision","instance":"HTPC","country":"{{$Alert.Source.Cn}}","asname":"{{$Alert.Source.AsName}}","asnumber":"{{$Alert.Source.AsNumber}}","latitude":"{{$Alert.Source.Latitude}}","longitude":"{{$Alert.Source.Longitude}}","iprange":"{{$Alert.Source.Range}}","scenario":"{{.Scenario}}","type":"{{.Type}}","duration":"{{.Duration}}","scope":"{{.Scope}}","ip":"{{.Value}}"},"values": [1],"timestamps":[{{now|unixEpoch}}000]}
  {{- end }}
  {{- end -}}
url: http://victoria-metrics:8428/api/v1/import
method: POST
headers:
  Content-Type: application/json

is not sending any data to the VictoriaMetrics instance

[LaurenceJJones]

I am also following this. I have everything setup exactly.

cs_lapi_decision{instance=~".*"}[86400s:1m] WARNING: No match! This query hasn't selected any time series from database. Either the requested metrics are missing in the database, or there is a typo in series selector

It seems like

type: http
name: http_default
log_level: info
format: >
  {{- range $Alert := . -}}
  {{- range .Decisions -}}
  {"metric":{"__name__":"cs_lapi_decision","instance":"HTPC","country":"{{$Alert.Source.Cn}}","asname":"{{$Alert.Source.AsName}}","asnumber":"{{$Alert.Source.AsNumber}}","latitude":"{{$Alert.Source.Latitude}}","longitude":"{{$Alert.Source.Longitude}}","iprange":"{{$Alert.Source.Range}}","scenario":"{{.Scenario}}","type":"{{.Type}}","duration":"{{.Duration}}","scope":"{{.Scope}}","ip":"{{.Value}}"},"values": [1],"timestamps":[{{now|unixEpoch}}000]}
  {{- end }}
  {{- end -}}
url: http://victoria-metrics:8428/api/v1/import
method: POST
headers:
  Content-Type: application/json

is not sending any data to the VictoriaMetrics instance

Did you add http_default under notifications to the profiles.yaml? and if so did you restart crowdsec after the update?

[joshoram80]

Yes to both

…

On Fri, Jan 24, 2025, 9:16 AM Laurence Jones ***@***.***> wrote: I am also following this. I have everything setup exactly. cs_lapi_decision{instance=~".*"}[86400s:1m] WARNING: No match! This query hasn't selected any time series from database. Either the requested metrics are missing in the database, or there is a typo in series selector It seems like type: http name: http_default log_level: info format: > {{- range $Alert := . -}} {{- range .Decisions -}} {"metric":{"__name__":"cs_lapi_decision","instance":"HTPC","country":"{{$Alert.Source.Cn}}","asname":"{{$Alert.Source.AsName}}","asnumber":"{{$Alert.Source.AsNumber}}","latitude":"{{$Alert.Source.Latitude}}","longitude":"{{$Alert.Source.Longitude}}","iprange":"{{$Alert.Source.Range}}","scenario":"{{.Scenario}}","type":"{{.Type}}","duration":"{{.Duration}}","scope":"{{.Scope}}","ip":"{{.Value}}"},"values": [1],"timestamps":[{{now|unixEpoch}}000]} {{- end }} {{- end -}} url: http://victoria-metrics:8428/api/v1/import method: POST headers: Content-Type: application/json is not sending any data to the VictoriaMetrics instance Did you add http_default under notifications to the profiles.yaml? and if so did you restart crowdsec after the update? — Reply to this email directly, view it on GitHub <#3290 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAP7FG4MPMLBNEIMO5LWCFD2MFS3FAVCNFSM6AAAAABQHVD3SCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJRGEZDOOJYGM> . You are receiving this because you commented.Message ID: ***@***.***>