Cheat Sheet and "No Data" Playbook
One page to keep open during an incident. Metric names as exposed by Alloy v1.16; select by
job— every layer exposes the same names.
Ports
| Port | Alloy | OTel Collector |
|---|---|---|
| 4317 / 4318 | OTLP gRPC / HTTP (when configured) | OTLP gRPC / HTTP |
| 12345 | UI, /metrics, /-/ready, /-/healthy, /-/reload, /debug/pprof, cluster gossip |
— |
| 8888 | — | Internal metrics |
| 13133 | — | health_check extension |
| 1777 | — | pprof extension |
| 55679 | — | zPages |
Commands
# Config gates (use production flags)
alloy fmt config.alloy
alloy validate --stability.level=experimental config.alloy
timeout 15 alloy run --stability.level=experimental --storage.path=/tmp/a config.alloy; test $? -eq 124
# What did Helm actually render?
helm template alloy grafana/alloy --version 1.8.1 -f values.yaml | less
kubectl -n monitoring get pod -l app.kubernetes.io/instance=alloy -o yaml | grep -A8 livenessProbe
# Did the last reload work? (the gauge will not tell you)
kubectl -n monitoring logs deploy/alloy -c alloy --since=10m | grep "failed to reload config"
kubectl -n monitoring logs deploy/alloy -c config-reloader --tail=5
# Talk to one replica (the image has no curl)
kubectl -n monitoring port-forward pod/<pod> 12345:12345
curl -s localhost:12345/metrics | grep -E '^otelcol_exporter_(queue_size|send_failed)'
curl -s localhost:12345/api/v0/web/components | python -c "import json,sys; [print(c['localID'], c['health']['state']) for c in json.load(sys.stdin)]"
PromQL
| Question | Query |
|---|---|
| Throughput at the last hop | sum by (job) (rate(otelcol_exporter_sent_spans_total{job="alloy"}[5m])) |
| Accepted per replica (gRPC skew) | sum by (pod) (rate(otelcol_receiver_accepted_spans_total{job="alloy"}[10m])) |
Refused at the door (memory_limiter) |
sum by (job, receiver) (rate(otelcol_receiver_refused_log_records_total[5m])) |
| Which processor drops | sum by (job, processor) (rate(otelcol_processor_incoming_items_total[5m]) - rate(otelcol_processor_outgoing_items_total[5m])) |
| Permanent export failures | sum by (job, exporter) (rate(otelcol_exporter_send_failed_spans_total[5m])) |
| Queue fill | max by (job, pod, exporter) (otelcol_exporter_queue_size / otelcol_exporter_queue_capacity) |
| Remote write backlog | max by (job, pod) (prometheus_remote_storage_samples_pending) |
| Remote write retries | sum by (job) (rate(prometheus_remote_storage_samples_retried_total[5m])) |
| Loki write drops | sum by (job) (rate(loki_write_dropped_entries_total[5m])) |
| Unhealthy components | sum by (job, pod) (alloy_component_controller_running_components{health_type!="healthy"}) |
| Memory vs self-set limit | max by (job, pod) (process_resident_memory_bytes / go_gc_gomemlimit_bytes) |
| Scrape targets per replica (clustering) | sum by (pod) (prometheus_scrape_targets_gauge{job="alloy"}) |
| Cluster peers | max by (job) (cluster_node_peers{state="participant"}) |
No data — decision tree
Work from the backend towards the source.
- Is the query right? Remove every filter and label matcher. Still nothing → the data really is missing. Something now → your selector is wrong (label renamed,
_totalsuffix,le="1"vsle="1.0"). - Did the backend reject it? Check the backend’s own discard metrics (Mimir
cortex_discarded_samples_total, Lokiloki_discarded_samples_total) and its distributor logs. Rejections for out-of-order, too-old or over-limit data do not show up as collector errors. - Did the last hop send it?
otelcol_exporter_sent_*/prometheus_remote_storage_samples_total/loki_write_sent_entries_totalon the writer.- Sent is rising → the problem is at the backend or routing (tenant, index, labels). Go back to 2.
- Is the last hop failing?
- Queue growing,
send_failedflat → backend unreachable or slow; data is buffered. Check the backend. send_failedrising, queue flat → permanent error: auth, payload size, schema. Read the exporter log.
- Queue growing,
- Did a processor drop it?
processor incoming − outgoingper processor. Non-zero → read that processor’s rule. - Did it arrive at the collector?
otelcol_receiver_accepted_*and_refused_*.- Refused →
memory_limiter; check memory. - Nothing accepted → the sender is not reaching this collector: endpoint, port, protocol (gRPC vs HTTP), TLS, NetworkPolicy, gRPC connection pinned to another replica.
- Refused →
- Is the running config the one you think?
failed to reload configin the log; the config-reloader sidecar log;helm templateversus the mounted ConfigMap. - Did a restart lose it? Pod restarts, OOM kills (
kubectl describe pod→OOMKilled), rollouts in the window. In-memory queues and batches are gone.