Self-Monitoring and Silent Drops

• Healthy is not delivering
• The metrics that matter
• Counting across layers
• Finding the processor that ate your data
• Five classes of silent loss
• Canaries and absence
• Our alert rules, reviewed
• Related lessons

Every component can be green while nothing reaches the backend.

Collector self-monitoring has to answer one question: did what came in go out? Health endpoints do not answer it.

Healthy is not delivering

The metrics that matter

Alloy serves everything on :12345/metrics; the OTel Collector on :8888/metrics. Names below are as exposed by Alloy v1.16.1.

• Converters emit no exporter_sent. otelcol.exporter.prometheus and otelcol.exporter.loki hand data to another pipeline; follow it into prometheus_remote_storage_* or loki_write_*.
• *_failed and *_refused series appear only after the first event. Panels show “No data” rather than 0.
• Controller metrics are process-wide. alloy_component_controller_running_components tells you how many components are unhealthy, not which. For the name, use the UI or GET /api/v0/web/components.

Counting across layers

Every Alloy and every Collector exposes the same metric names. Our cluster has two layers exporting spans:

sum by (job) (rate(otelcol_exporter_sent_spans_total[10m]))
# job="alloy-collector"  0.41   (Beyla spans → gateway)
# job="alloy"            234    (gateway → Tempo)

• An unselected sum() double-counts every record that passes through two layers. Always select by job.
• Measure throughput at the last hop — the one that writes to the backend.
• Measure loss per hop as a difference, sender versus receiver:

# What the node agents sent to the gateway, minus what they failed to send
sum(rate(otelcol_exporter_sent_spans_total{job="alloy-collector"}[10m]))

compared with what the gateway accepted from them. Our gateway also receives OTLP directly from OTel Demo, so here the difference is only meaningful with a canary (below) or per-sender labels.

• The _total suffix differs by version. Alloy and current Collectors expose otelcol_exporter_send_failed_spans_total; older Collectors exposed otelcol_exporter_send_failed_spans. A query with the wrong form returns an empty vector — which looks exactly like “zero errors”.

Finding the processor that ate your data

otelcol_processor_dropped_* is emitted only by memory_limiter. A filter or transform that throws data away increments nothing called “dropped”. Compare what went in and out of each processor instead:

sum by (job, processor) (
  rate(otelcol_processor_incoming_items_total[5m])
  - rate(otelcol_processor_outgoing_items_total[5m])
) > 0

Measured on Alloy v1.16.1: four log records sent, a filter rule matching INFO|DEBUG, one record left — incoming 4, outgoing 1, receiver response {"partialSuccess":{}}, HTTP 200, every component Healthy.

💪Exercise💪 (self-guided): Silent drop hunt — a filter eats half your logs while everything is green; find it from metrics alone.

Five classes of silent loss

Classes 3–5 need end-to-end checks: the collector reports success because, from its side, the request succeeded.

Canaries and absence

A canary proves delivery end to end. It needs four things:

1. A unique marker — a service name or label nothing else uses.
2. A counter — a monotonically increasing value, so gaps are countable, not guessed.
3. The same path as real data — same receiver, same processors, same exporter. A canary that skips the gateway proves nothing about the gateway.
4. A checker outside the pipeline — an alert rule evaluated in the backend, not in the collector.

# Canary did not arrive for 10 minutes
absent_over_time(canary_heartbeat_total{job="collector-canary"}[10m])

For “a known producer went silent”, join against an inventory:

# Services that sent spans last week but not in the last 30 minutes
group by (service) (rate(traces_spanmetrics_calls_total[1h] offset 7d))
unless
group by (service) (rate(traces_spanmetrics_calls_total[30m]))

Our alert rules, reviewed

helm_values/alerts/otel-collector-alerts.yaml started as a copy of the community opentelemetry-collector-monitoring rules. Against Alloy v1.16, most of them could never fire:

The file now uses the names the metrics actually have:

One failure mode has no metric at all: a reload rejected at graph level (bad reference) leaves every component healthy and the old config running. The only trace is the log line failed to reload config. It needs a log-based alert (Loki ruler), not a PrometheusRule.

The rules are applied with kubectl apply -k helm_values/alerts/ -n monitoring. No deploy script runs that step, so check kubectl -n monitoring get prometheusrule otel-collector-alerts after a fresh cluster.

The Grafana Alloy mixin (9 dashboards, ~14 alerts) is a better starting point than the community file, with gaps of its own: no resource or config-load alerts, and ClusterConfigurationDrift joins on a cluster_name label that cluster_node_info does not carry — on our gateway it is cluster_node_info{state="participant"}.

Related lessons

• Reliability
• Config lifecycle
• Cheat sheet
• Cost Optimization — measuring costs