💪Exercise💪 — silent drop hunt: half the logs are gone and everything is green
Goal
Find a data-loss bug using collector metrics alone: every component is healthy, the receiver answers 200, the backend has fewer records than you sent.
Prerequisites
MEandNSfrom First pipeline.- Grafana on the workshop cluster, Loki and Prometheus data sources.
- Your release is scraped by the shared gateway:
serviceMonitor.enabledbelow creates a ServiceMonitor, and the gateway’sprometheus.operator.servicemonitorspicks up ServiceMonitors from every namespace. The first samples arrive within about a minute.
💪Exercise💪 — steps
1. Deploy the pipeline
A colleague added “a rule to drop DEBUG noise”. Deploy their config as-is:
cat > ex-drop.values.yaml <<'EOF'
alloy:
extraPorts:
- { name: otlp-http, port: 4318, targetPort: 4318, protocol: TCP }
configMap:
content: |
otelcol.receiver.otlp "default" {
http { endpoint = "0.0.0.0:4318" }
output { logs = [otelcol.processor.filter.noise.input] }
}
// drop DEBUG noise
otelcol.processor.filter "noise" {
error_mode = "ignore"
logs {
log_record = ["severity_number < SEVERITY_NUMBER_WARN"]
}
output { logs = [otelcol.processor.attributes.loki_hints.input] }
}
otelcol.processor.attributes "loki_hints" {
action {
key = "loki.resource.labels"
action = "insert"
value = "service.name"
}
output { logs = [otelcol.exporter.loki.default.input] }
}
otelcol.exporter.loki "default" {
forward_to = [loki.write.shared.receiver]
}
loki.write "shared" {
endpoint {
url = "http://loki-gateway.monitoring.svc.cluster.local:80/loki/api/v1/push"
}
}
controller:
type: deployment
replicas: 1
serviceMonitor:
enabled: true
EOF
helm upgrade --install ex-alloy-drop-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-drop.values.yaml
kubectl -n $NS rollout status deploy/ex-alloy-drop-$ME
2. Send 20 records: 5 each of DEBUG, INFO, WARN, ERROR
kubectl -n $NS port-forward svc/ex-alloy-drop-$ME 4319:4318 12346:12345 &
sleep 3
for sev in "5 DEBUG" "9 INFO" "13 WARN" "17 ERROR"; do
set -- $sev
for i in 1 2 3 4 5; do
curl -s -o /dev/null -w '%{http_code} ' -X POST localhost:4319/v1/logs -H 'Content-Type: application/json' -d '{
"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"drop-'$ME'"}}]},
"scopeLogs":[{"logRecords":[{"timeUnixNano":"'$(date +%s)000000000'","severityNumber":'$1',"severityText":"'$2'","body":{"stringValue":"'$2' record '$i'"}}]}]}]}'
done
done
echo
Twenty 200s. The intent: drop the 5 DEBUG records, keep 15.
3. Count what arrived
Grafana → Explore → Loki:
sum(count_over_time({service_name="drop-ab"}[15m]))
(use your own suffix). Write the number down. Then check the health of the pipeline:
curl -s -o /dev/null -w 'healthy=%{http_code}\n' localhost:12346/-/healthy
and open http://localhost:12346 — every component green.
4. Localize the loss from metrics
Grafana → Explore → Prometheus, for your namespace (collectors-ab). The release is fresh, so read the raw counter values — they are exact:
sum by (receiver) (otelcol_receiver_accepted_log_records_total{namespace="collectors-ab"})
sum by (processor) (otelcol_processor_incoming_items_total{namespace="collectors-ab"})
-
sum by (processor) (otelcol_processor_outgoing_items_total{namespace="collectors-ab"})
sum(loki_write_sent_entries_total{namespace="collectors-ab"})
Build the chain: accepted → per-processor loss → sent.
Do not use increase() here. These counters are created by the first record, so the first scrape already sees 20. increase() does not count the jump from “no series” to 20, and every difference reads about 0. In production, where counters have been running for days, rate() and increase() are the right tools.
Also try the “obvious” metric:
otelcol_processor_dropped_log_records_total{namespace="collectors-ab"}
Explain why it returns nothing.
5. Read the rule and fix it
Look at the filter statement against the OTel severity numbers: DEBUG 5–8, INFO 9–12, WARN 13–16, ERROR 17–20. Fix the statement so it matches the comment, upgrade, wait for the reload, and send the 20 records again. The reload rebuilds the filter component, so its counters restart from zero: after the new batch, incoming is 20 and outgoing is 15.
Success criteria
- You found the loss without reading the config first: accepted 20, filter out 10, sent 10.
- You explained why
otelcol_processor_dropped_*is empty (onlymemory_limiteremits it). - After the fix, 15 of 20 records reach Loki.
⭐Stretch: turn it into an alert
Write an alert expression that would have caught this in production without firing for every legitimate filter. Hint: compare the drop ratio per processor with its usual value, for example (incoming − outgoing) / incoming against the same ratio offset 1d.
Clean up when done: helm -n $NS uninstall ex-alloy-drop-$ME.