If you run multiple containers, you need observability. When something breaks — or when someone’s probing your services — you want to know about it before it becomes a problem.

My monitoring stack combines three tools:

• Loki for log aggregation
• Grafana for dashboards and alerting
• CrowdSec for intrusion detection and crowd-sourced threat intelligence

Architecture

Container A ──→ Promtail ──→ Loki ──→ Grafana
Container B ──→ Promtail ──→   ↑        ↑
Container C ──→ rsyslog  ──→   └── CrowdSec (LAPI)
                                       │
                                  bouncer (nginx)
                                       ↓
                                  Block malicious IPs

Every container ships its logs via Promtail to Loki. CrowdSec parses logs for attack patterns, shares anonymised data with the community, and blocks repeat offenders via a bouncer on the reverse proxy.

Loki + Promtail

Loki is a horizontally-scalable log aggregation system from Grafana Labs. Unlike Elasticsearch, it doesn’t index the log content — just the metadata (labels), which makes it much cheaper to run.

Installing Loki

On a dedicated container:

wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64.deb
dpkg -i loki-linux-amd64.deb

Basic config (/etc/loki/config.yml):

auth_enabled: false
server:
  http_listen_port: 3100
ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
  chunk_idle_period: 5m
schema_config:
  configs:
    - from: 2024-01-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

Promtail on Each Container

Promtail scrapes log files and pushes them to Loki. Config (/etc/promtail/config.yml):

clients:
  - url: http://mon-srv:3100/loki/api/v1/push
scrape_configs:
  - job_name: syslog
    static_configs:
      - targets: [localhost]
        labels:
          job: syslog
          __path__: /var/log/syslog

Grafana

Grafana connects to Loki as a data source and lets you build dashboards. I have three:

1. System Overview — CPU, memory, disk across all containers
2. Security Events — CrowdSec alerts, failed SSH logins, HTTP 4xx/5xx rates
3. Service Health — Uptime, response times, certificate expiry

Alert Rules

Grafana can alert via email, Slack, or webhook. I have rules for:

• SSH brute force attempts exceed threshold
• Certificate renewing within 7 days
• Container down for more than 5 minutes
• Disk usage above 85%

CrowdSec

CrowdSec is a modern, community-driven IPS. It detects attacks by parsing logs, then blocks the offending IP via a bouncer.

The setup consists of:

• LAPI (Local API): The central service that processes alerts and serves decisions
• Scenarios: Attack detection rules (SSH brute force, HTTP scanning, port scans)
• Bouncers: Components that actually block IPs (iptables, NGINX, nftables)

After installation, CrowdSec starts learning your traffic patterns immediately:

cscli alerts list
cscli decisions list
cscli metrics

The community blocklist is what makes CrowdSec powerful — when someone attacks any CrowdSec user, every other user benefits from that intelligence.

Putting It Together

My monitoring container (mon-srv) runs all three services using about 256 MB RAM total. Promtail instances on each container add negligible overhead — about 10 MB each.

For a homelab with 10-15 containers, this stack is more than sufficient and entirely free. In production, you’d add load balancers, replication, and retention policies, but for a lab it works beautifully out of the box.