Monitoring — Loki, Alloy, Prometheus, Grafana, Blackbox Exporter

Status: deployed and verified. Confirmed by the user on 2026-09-10, replacing the Uptime-Kuma-only setup this file previously described and superseding the Netdata experiment (see ../archivo/netdata_2026-09-09.md). Uptime Kuma is substituted, not yet decommissioned — see the closing section of this file and open-items.md.

Why this exists

Nextcloud AIO’s app containers were restarting intermittently with no way to inspect why — docker logs is lost on restart, and nothing collected logs or metrics centrally. Uptime Kuma alone answers “is it up”; it cannot answer “why did it go down”. Netdata was tried as the metrics half of the answer and dropped (unused, hard to operate day to day) — see the archived file above for that reasoning.

Stack

docker-host (VM 100), /opt/stacks/monitoring/, Docker network `proxy`

  Alloy   ──(docker.sock, ro)──►  discovers every container
            ships logs, tagged by container name  ──►  Loki  (filesystem, 14d retention)

  Prometheus  ──scrapes──►  node-exporter        (host metrics)
              ──scrapes──►  cAdvisor              (per-container metrics)
              ──scrapes──►  Blackbox Exporter      (external HTTP/TLS probes)

  Grafana  ──queries──►  Loki, Prometheus
           ──alerts──►   Discord, contact point "Chivaton"

  Caddy  ──reverse_proxy──►  grafana:3000   (grafana.balserosworld.net)

Only Grafana is reachable outside the proxy network. Why: Loki, Prometheus, cAdvisor and Blackbox Exporter have no authentication of their own — publishing any of them directly would be unauthenticated access to metrics/query APIs. Grafana (which has an admin login) already proxies access to all of them, so nothing is lost by keeping the rest internal.

Pinned images (bump deliberately, never via :latest):

Service	Image
Loki	grafana/loki:3.7.0
Alloy	grafana/alloy:v1.19.2
Prometheus	prom/prometheus:v3.14.0
node-exporter	prom/node-exporter:v1.12.1
cAdvisor	ghcr.io/google/cadvisor:v0.60.5
Grafana	grafana/grafana:13.2.1
Blackbox Exporter	prom/blackbox-exporter:v0.26.0

Why cAdvisor’s registry matters: its image moved from gcr.io/cadvisor/cadvisor to ghcr.io/google/cadvisor as of v0.53.0 — the old path with a newer tag fails outright with “not found”. None of these have a LinuxServer.io image, so the official upstream image is used throughout (the repo’s LSIO-first convention does not apply when LSIO publishes nothing).

Retention: Loki capped at 14 days (retention_period: 336h, compactor retention_enabled: true); Prometheus TSDB capped at 14 days via --storage.tsdb.retention.time. Why capped: the same disk also holds the Borg repo (see ../backup/architecture.md for the 3-2-1 limitation) — logs and metrics must not grow unbounded on it.

Deploy gotchas that will bite again if not remembered

1. Compose key is cgroup: host, not cgroupns: host. This host runs Docker Compose v5.5.0 — the newer Compose Spec renamed the service-level key for cgroup namespace mode. The old key throws “additional properties not allowed”.
2. cAdvisor’s registry moved — see the pinned-images table above.
3. Never append a new service to docker-compose.yaml with cat >>. It appends to the literal end of the file, which is not necessarily inside services: — it can land under a later top-level key (networks:, volumes:) instead, silently producing an invalid Compose file. Insert with sed -i '/^networks:$/i\...' (anchored with ^...$ so it matches only the top-level key) and always re-check the result with cat -n file | sed -n 'X,Yp' around the insertion point, not tail.
4. Blackbox Exporter defaults to preferred_ip_protocol: ip6, and docker-host has no IPv6 route out. A target with only an A record (direct DNS, grey-cloud) resolves fine — Blackbox tries IPv6, finds no AAAA record, and falls back to IPv4 automatically. A target proxied through Cloudflare (orange-cloud — Tunnel or Worker) gets a real AAAA record, so Blackbox actually attempts IPv6 and fails every time with dial tcp [...]: connect: network is unreachable. Blackbox only falls back to the other IP family when resolution finds nothing for the preferred family, never when the connection to a resolved address fails. This produced real, repeated [FIRING] Discord alerts for two orange-cloud targets while the sites were reachable in a browser — diagnosed via docker logs blackbox-exporter. Fix, already applied: in blackbox.yml, inside the module’s http: block, set preferred_ip_protocol: "ip4" and ip_protocol_fallback: false, then restart the container (config is not hot-reloaded). Any future orange-cloud target added to the probe list will hit this exact failure mode if this setting is ever reverted.

Grafana access

Published via Caddy, same DNS-01 pattern as every other service:

grafana.balserosworld.net {
  tls {
    dns cloudflare {env.CF_API_TOKEN}
  }

  reverse_proxy grafana:3000
}

Grafana has no host port published; a ssh -L 3000:localhost:3000 tunnel does not work for this reason — the SSH server forwards to the host’s own loopback, and nothing listens there since the port was never published. Caddy is the only path in.

Dashboards

Two community dashboards imported rather than built from scratch:

Dashboard	grafana.com ID	Covers
Node Exporter Full	1860 (rfmoz)	Host CPU/RAM/disk/network
cAdvisor Docker Insights	19908	Per-container CPU/memory, all containers

Why: default Prometheus/cAdvisor panels are raw; these are curated and actively maintained. Select job node-exporter in Node Exporter Full’s job dropdown — this stack’s prometheus.yml names the job node-exporter, not the dashboard’s default node.

Alerting

Grafana’s built-in alerting (no separate Alertmanager container — not justified at this scale). Contact point Chivaton → Discord, using a custom notification template (clean-container-alert template group, function clean_container_alert) that strips cAdvisor’s auto-attached Docker Compose/Docker Desktop Extension labels down to container/rule name, firing/recovered status, and the rule’s own Summary/Description. The template branches on {{ .Labels.alertname }} so one template serves every rule, current and future, without per-rule edits.

Notification policy timing: group_wait: 30s, group_interval: 1m (Grafana’s 5-minute default made resolved-notifications feel very delayed).

All rules live in folder homelab-alerts, all route to Chivaton:

#	Name	Query	Condition	For	Why
1	Container down/recovered	time() - container_last_seen{name=~".+"}	> 60	0s	Generic — any container stops reporting, any service
2	Host RAM low	(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100	< 15	5m	Sustained memory pressure
3	Host swap usage high	(node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes) / 1024 / 1024 (MB)	> 500	10m	Sustained swap pressure
4	Host disk usage high	100 - ((node_filesystem_avail_bytes{mountpoint=~"/|/mnt/storage", fstype!="tmpfs"} / node_filesystem_size_bytes{...}) * 100)	> 85	10m	Root (/) and the backup disk (/mnt/storage) — the two real mountpoints on this host
5	Monitoring target down	up{job=~"node-exporter|cadvisor|prometheus"}	== 0	2m	Detects the monitoring stack itself going blind
6	Site probe down	probe_success{job="blackbox-http"}	== 0	2m	External reachability — DNS → Cloudflare → TLS → Caddy → service, end to end. This is the rule that now does Kuma’s job
7	TLS cert expiring soon	(probe_ssl_earliest_cert_expiry{job="blackbox-http"} - time()) / 86400 (days)	< 14	0s	Cert renewal failure warning — also formerly Kuma’s job
8	Container restart loop	changes(container_start_time_seconds{name=~".+"}[15m])	> 2	0s, keep firing 5m	Catches a crash-loop that recovers in under 60s, which rule 1 would miss entirely. Most directly addresses the original Nextcloud-restart mystery
9	Container OOM killed	increase(container_oom_events_total{name=~".+"}[5m])	> 0	0s, keep firing 5m	Instant signal when the kernel OOM-kills a process inside a container

A host-CPU rule was considered and deliberately deprioritized — not justified by this workload yet.

Blackbox Exporter — the six probed targets

static_configs:
  - targets:
      - https://grafana.balserosworld.net
      - https://vault.balserosworld.net
      - https://jellyfin.balserosworld.net
      - https://portainer.balserosworld.net
      - https://cloud.balserosworld.net      # Cloudflare Tunnel
      - https://docs.balserosworld.net       # Cloudflare Worker — this site

prometheus.yml’s blackbox-http job relabels __param_target from __address__ and instance from __param_target, so each probed URL shows up under its own clean instance label rather than the exporter’s own address.

Deliberately not probed:

Target	Why not
backup.balserosworld.net	Stale DNS record from the removed Duplicati stack — needs deleting in Cloudflare, not a live service
test.balserosworld.net	Used ad hoc to test backup restores, not continuously running — probing it would alert every time it is idle
Caddy itself	No dedicated target — proven working transitively: if the six URLs above respond with the right cert and content, Caddy is doing its job for each of them

Uptime Kuma: substituted, not yet decommissioned

Kuma’s job — HTTP reachability and TLS cert-expiry checks for the public services, alerting to Discord — is now fully covered by alert rules 6 and 7 above, through the same Discord channel instead of a second one. This was validated the same day it was decided: a real Blackbox/Prometheus incident fired correctly for two services (the IPv6 bug above), proving the full pipeline — probe → Prometheus → Grafana alert → Discord, both firing and resolved — end to end.

Decision: Blackbox Exporter substitutes Uptime Kuma. Kuma’s container, kuma.balserosworld.net DNS record, and any Caddy block for it have not been removed yet — see open-items.md.

Backup

If the stack is in /opt/stacks/monitoring, its configuration already falls under Layer 2 (Borg) without extra work.

⚠️ TO CONFIRM: whether Loki’s and Prometheus’s data directories are excluded from borg-backup.sh. They should be — this is observational data, bulky and replaceable; restoring old metrics/logs adds nothing to recovery, the same reasoning already applied to Netdata’s database.

To close out this file

cd /opt/stacks/monitoring
docker compose ps
docker logs blackbox-exporter --tail 30 | grep -i cloud
grep -v -iE 'token|password|key|secret' docker-compose.yaml