tuxvador@blog:~/blog/crowdsec-agent-detecting-nothing$

~/blog/crowdsec-agent-detecting-nothing

My CrowdSec Agent Detected Nothing for Four Months

4 min readLire en français →

#security#crowdsec#linux#self-hosting

My public server is the only machine in the lab that faces the internet directly. It runs a CrowdSec agent that reads its SSH, mail and web logs, and a firewall bouncer that drops banned IPs. For four months the bouncer worked perfectly. The agent did nothing at all, and nothing said so.

Why it looked healthy

My CrowdSec decisions are centralised: every agent sends its alerts to one Local API in the homelab, and every bouncer pulls the resulting bans from it. So the public server’s bouncer was enforcing bans that other agents had created, plus the community blocklist. Its nftables set held about 17,800 addresses. systemctl status crowdsec was green. The dashboards showed bans landing on that machine every day.

None of that required the local agent to detect anything. I found out while auditing the SOC for gaps, with one command:

cscli scenarios list

It printed an empty table. No scenarios, no parsers. The agent was tailing the logs and matching them against nothing.

CrowdSec’s hub items (parsers, scenarios, collections) are installed as symlinks: /etc/crowdsec/scenarios/ssh-bf.yaml points into a hub directory that holds the actual file. On this server every link pointed into /usr/share/crowdsec/hub/, which no longer existed.

find /etc/crowdsec /var/lib/crowdsec/hub -xtype l | wc -l    # 52 broken links

The package history explained it:

Date What happened
June 3 CrowdSec installed from Debian’s repository. Debian’s package ships a hub snapshot in /usr/share/crowdsec/hub/, and the item links point into it.
June 29 Switched to CrowdSec’s upstream repository for a newer version. Upstream’s package has no hub snapshot, so the directory went away with the old package, and every link broke.
September A routine apt upgrade took a later upstream version. Still nothing loaded.

CrowdSec handles a missing hub item by not loading it. With every item missing, it starts cleanly with zero parsers and zero scenarios, and never logs an error worth noticing.

Both repositories were also still configured at the same apt priority, so apt picked whichever offered the higher version. Upstream won every time, which is what I wanted, but by accident rather than by pin. Debian’s old bouncer package was still installed alongside upstream’s, unused, which made dpkg -l | grep crowdsec confusing to read.

The fix

Back up first. /etc/crowdsec and the hub directory both go into a tarball, so the whole change can be undone by extracting it.

Then remove the broken links and install from the live hub:

find /etc/crowdsec /var/lib/crowdsec/hub -xtype l -delete
cscli hub update
cscli collections install crowdsecurity/linux crowdsecurity/sshd \
    crowdsecurity/nginx crowdsecurity/postfix crowdsecurity/dovecot
crowdsec -t && systemctl restart crowdsec

Five collections pulled in their dependencies: 59 scenarios and 13 parsers. Upstream’s package also ships a daily crowdsec-hubupdate.timer, so they stay current.

Read each event once

While I was there, the acquisition file had grown into a list of everything that looked like a log. That matters more than it seems: rsyslog writes mail lines to both /var/log/mail.log and /var/log/syslog, and sshd writes to both auth.log and the journal. An agent that reads both copies sees every event twice, so a scenario that should fire after ten failed logins fires after five.

The new acquis.yaml reads one copy of each:

filenames:
  - /var/log/auth.log
  - /var/log/mail.log
labels:
  type: syslog
---
filenames:
  - /var/log/nginx/access.log
  - /var/log/nginx/error.log
labels:
  type: nginx

The whitelist that shadowed a hub item

The server had a local parser called whitelists.yaml, my own addresses from the lockout post-mortem. The hub has an item, crowdsecurity/whitelists, that installs a file with the same name in the same directory. Mine was hiding it. I renamed mine to home-whitelist.yaml (name: local/home-whitelist), kept only my egress IP and the tailnet ranges in it, and let the hub item cover the private ranges.

Proving it detects

cscli metrics would normally show lines parsed per source. On this server it fails (“prometheus is not enabled”), because the agent’s metrics endpoint is off. cscli explain works without it and shows a sample line’s path through the parsers and scenarios:

cscli explain --type syslog \
  --log 'Oct 19 01:20:00 mail sshd[1234]: Invalid user admin from 203.0.113.50 port 50000'

That line now reaches ssh-bf, ssh-slow-bf and their user-enumeration variants. A Postfix SASL failure reaches postfix-spam, and an nginx 404 probe reaches http-probing.

The real test came four and a half minutes after the restart, when a scanner probed the web server for sensitive files and path traversal. The agent raised http-sensitive-files, http-path-traversal-probing and http-probing. The central API turned that into a seven-day ban, and the address appeared in the server’s own nftables set. Those were the first alerts that machine had sent to the central API.

What I check now

The lesson is not about symlinks. Enforcement and detection are separate, and a working bouncer hides a dead agent. A green service and a full ban list say nothing about whether the local agent is contributing.

The checks I now run on every agent:

cscli scenarios list | wc -l              # not zero
find /etc/crowdsec -xtype l               # empty
cscli alerts list --since 168h -l 0 -o json | jq '[.[] | .machine_id] | group_by(.) | map({(.[0]): length}) | add'

The last one runs on the central API and counts recent alerts per agent. An internet-facing agent with zero alerts in a week isn’t quiet. It’s broken.

Pitfalls

  • Debian’s CrowdSec package and upstream’s lay out the hub differently. Switching between them breaks every installed item. Reinstall the collections after any repository change.
  • A missing hub item is skipped, not an error. An agent with nothing loaded still starts and looks healthy.
  • Two repositories at the same priority means the higher version wins. Pin the one you mean.
  • Reading the same event from two files doubles the counts, which halves every scenario’s threshold.
  • A local file named like a hub item shadows it. Give local items their own names.
  • cscli metrics needs the Prometheus endpoint. Without it, test with cscli explain.

cd ~/blog