Reliability: Memory, Queues, Retries
- Where data waits
- Buffering per tier
- Retryable or permanent
- Reading the queue
- Memory: memory_limiter and GOMEMLIMIT
- Persistent queue
- What is lost on restart
- Failure modes
- Related lessons
A collector loses data in three ways: it runs out of memory, the backend rejects data for good, or the process restarts with data in RAM.
Each has its own setting and its own metric. None of them turns a health check red.
Where data waits
| Buffer | Survives a pod restart | Size |
|---|---|---|
| SDK export queue in the application | No | Small (SDK default 2048 spans) |
batch processor |
No | One batch |
Exporter sending_queue, in memory |
No | queue_size requests (default 1000) |
Exporter sending_queue with storage (PVC) |
Yes | Disk |
Prometheus WAL (--storage.path) |
Only on a PVC | Until sent, at most max_keepalive_time (default 8 h) |
Log files on the node (/var/log/pods) |
Yes, until kubelet rotates them | Kubelet rotation settings |
Only the last three are real buffers. Everything else is gone the moment the pod stops.
Buffering per tier
Each tier of the reference layout scales on a different axis and buffers in a different place. Our gateway has no PVC, so today it behaves like the in-memory tiers.
flowchart TB
subgraph app ["App pod — scales with the app"]
sdk["SDK batch + export queue<br/>memory · 2048 items"]
end
subgraph node ["DaemonSet / VM — one per node"]
files[("log files on the node<br/>disk · kubelet rotation")]
agent["exporter queue<br/>memory · 1000 requests"]
end
subgraph gwbox ["Gateway — StatefulSet + HPA"]
pvc[("persistent queue<br/>PVC per replica · sized by outage")]
end
be[(Backend)]
drop(["dropped"])
files -->|"tail, re-read after restart"| agent
sdk --> pvc
agent --> pvc
pvc --> be
be -.->|"retryable error:<br/>backoff, data stays on disk"| pvc
pvc -->|"PVC full · permanent error"| drop
sdk -->|"pod restart"| drop
agent -->|"pod restart · gateway down<br/>longer than the 5 min retry"| drop
subgraph legend ["Legend"]
direction LR
lgMem["memory — lost on restart"] ~~~ lgDisk[("disk — survives restart")] ~~~ lgDrop(["data loss"])
end
be ~~~ legend
classDef mem fill:#fff,stroke:#64748b,stroke-dasharray:4 3,color:#1e293b
classDef disk fill:#334155,stroke:#0f172a,color:#fff
classDef loss fill:#dc2626,stroke:#991b1b,color:#fff
class sdk,agent,lgMem mem
class files,pvc,lgDisk,be disk
class drop,lgDrop loss
%% edge order: 0 files→agent, 1 sdk→gw, 2 agent→gw, 3 gw→backend,
%% 4 backend retry, 5-7 →dropped, 8-10 legend spacers
linkStyle 4 stroke:#e8871e,stroke-width:2px
linkStyle 5,6,7 stroke:#dc2626,stroke-width:2px
The split is deliberate. Tiers in front of the Gateway only survive a short Gateway hiccup: short retries, in-memory queues. The Gateway survives a backend outage: its queue is on a PVC and drains when the backend returns. Data in a dashed box when the pod stops is lost; data in the Gateway’s queue waits until the PVC fills.
| Collector | Scales with | Buffers in | Survives a restart | What really protects the data |
|---|---|---|---|---|
| OTel SDK in the mature team’s services | The app’s own replicas | • SDK batch processor • Export queue (default 2048 items) |
No | • Nothing — full queue → SDK drops • Never make the gateway a synchronous dependency of the app |
| VM collector (Alloy, systemd) | Nothing; one per machine | • In-memory sending_queue per exporter• Log positions file on local disk |
• Logs: yes, re-read from files • OBI traces, profiles, metrics: no |
• Logs: the VM’s log files • The rest: add otelcol.storage.file on local disk; a VM has no PVC to lose |
| Node agent (DaemonSet) | Node count; no HPA | • In-memory sending_queue• Positions under storagePath |
Only with storagePath on a hostPath volume; otherwise positions die with the pod |
• Logs: the files the kubelet keeps on the node • eBPF traces and profiles: no second copy |
| Anti-corruption layer | • One team’s volume • Small Deployment, HPA optional |
In-memory sending_queue |
No | • The legacy app’s own retry • Keep the layer stateless — it is meant to be deleted |
| Gateway | • StatefulSet + HPA on PVC usage, not only CPU/memory • Clustering shards scrape targets |
• sending_queue with storage on the replica’s PVC (fully on disk, no in-memory tier)• Remote-write WAL on the same PVC |
Yes. The replica gets its PVC back and drains it | • The PVC, sized for the longest backend outage to survive |
| Optional outputs | The vendor’s side | One persistent exporter queue each, on the Gateway’s PVC | Yes, like other Gateway queues | • The vendor’s ingest • A slow vendor fills its own queue, then pushes back on everyone (see How data gets lost) |
Retryable or permanent
The exporter decides from the response whether to retry.
| Protocol | Retried | Not retried (dropped at once) |
|---|---|---|
| OTLP gRPC | UNAVAILABLE, RESOURCE_EXHAUSTED (with retry info), DEADLINE_EXCEEDED, ABORTED, CANCELLED, OUT_OF_RANGE, DATA_LOSS |
UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT, NOT_FOUND, UNIMPLEMENTED, … |
| OTLP HTTP | 429, 502, 503, 504 | Every other 4xx/5xx |
| Prometheus remote write | 5xx, 429 (retry_on_http_429) |
Other 4xx |
Consequences:
- A wrong token loses data instantly. No queue saves you; the queue never fills.
- An unreachable backend is retried with exponential backoff until
max_elapsed_time(default 5 min). Only then is the data counted as failed. - Rotating a backend token is a data-loss event unless both old and new tokens are accepted during the switch (lesson 11).
Measured on Alloy v1.16.1 — one exporter to a closed port, one to a server answering 401, five log records sent to each:
| Exporter | otelcol_exporter_queue_size |
otelcol_exporter_send_failed_log_records_total |
/-/healthy |
|---|---|---|---|
otlp → closed port |
5 and holding | absent | 200 |
otlphttp → 401 |
0 | 5 immediately | 200 |
The retry messages are logged at level info: Exporting failed. Will retry the request after interval.
Reading the queue
| Queue | send_failed |
Meaning | Action |
|---|---|---|---|
| Growing | Flat | Retryable errors; data is buffered | Backend is slow or down. You have until the queue fills or max_elapsed_time expires |
| Full | Rising | Buffer exhausted; data is being dropped | Outage longer than the buffer |
| Flat / zero | Rising | Permanent error; data is dropped right now | Auth, schema, payload size. Read the exporter log |
| Flat / zero | Flat | Healthy — or nothing is arriving | Check receiver_accepted upstream |
queue_sizecounts requests (batches), not records. A queue of 1000 can hold 1000 spans or 8 million.send_failedcounters do not exist until the first failure. An alert onrate(...) > 0works; a dashboard panel shows “No data” instead of 0.
Memory: memory_limiter and GOMEMLIMIT
memory_limiter refuses new data when heap use crosses a limit.
otelcol.processor.memory_limiter "default" {
check_interval = "1s" // required; "0s" fails at startup
limit_percentage = 80 // hard limit: 80 % of the container limit
spike_limit_percentage = 20 // soft limit = 80 - 20 = 60 %
output { /* ... */ }
}
- Both percentages are of the total limit.
80/20means soft limit at 60 %, hard limit at 80 % — not “80 % plus a 20 % spike”. - Above soft: refuses data. Above hard: also forces GC.
- It must be the first processor. Placed later, it throws away data that was already decoded and processed.
- Refusing is only safe if the sender retries. OTLP clients get a retryable error and back off. A pull receiver (scrape) cannot push back — refused samples are gone.
check_interval = "0s"passesalloy validatebut fails atalloy runwithcheck_interval must be greater than zero. See Config lifecycle for why validate misses it.
GOMEMLIMIT makes the Go runtime collect garbage harder before hitting the container limit.
| Runtime | Who sets it |
|---|---|
| Alloy | Alloy itself, 90 % of the cgroup limit. Our gateway: limit 4 GiB → go_gc_gomemlimit_bytes = 3.87 GB |
| OTel Collector chart | useGOMEMLIMIT: true (80 % of the limit) |
| Bare Collector binary | Nobody — set GOMEMLIMIT yourself, ~80 % of the limit |
Do not add GOMEMLIMIT to extraEnv on top of what the chart does — you get a duplicate env var and one silently wins.
Our stack: neither Alloy release runs a memory_limiter. The gateway relies only on GOMEMLIMIT and the 4 GiB limit. Under a burst, it runs until the kernel kills it, and every in-memory queue on that replica goes with it.
For gateways, memory — not CPU — is the resource that fails first. Size limits and HPA on memory.
Persistent queue
# OTel Collector
extensions:
file_storage:
directory: /var/lib/otelcol/queue
compaction:
on_rebound: true # without it, the file never shrinks
exporters:
otlp:
sending_queue:
storage: file_storage
queue_size: 5000 # requests, not records
service:
extensions: [file_storage]
In Alloy: otelcol.storage.file (public preview in v1.16, so it needs --stability.level=public-preview).
- With
storageset, the whole queue is on disk. There is no in-memory tier that spills over. - Delivery becomes at-least-once. After a crash, the last requests may be sent twice.
- Size the PVC for the outage you want to survive:
PVC ≈ ingest rate (records/s) × bytes per record × outage (s) × 1.4. - Per-replica PVCs do not add up. With RWO volumes, each replica buffers only what it received. Time to first loss is set by the busiest replica, not by total capacity.
- Alert on PVC fill (
kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes), not on the queue ratio. - CPU/memory HPA misses a filling PVC. Data waits on disk, not in RAM, so nothing scales while the disk fills. Scale on PVC usage (Topologies).
- Scale-down strands data. A removed replica’s queue stays on its PVC with no consumer.
whenScaled: Deletedeletes the PVC and the unsent data;Retainkeeps it, but nothing drains it. Drain before scaling down; hold scale-down during a backend outage.
What is lost on restart
Every rollout, OOM kill and node drain drops:
- The content of every
batchprocessor. - Every in-memory
sending_queue. Shutdown gives the exporter a short chance to flush, but a queue is only full when the backend is failing. Those are exactly the requests that cannot be flushed. - The WAL on
emptyDir— our gateway is a Deployment, so its remote-write WAL is gone on every reschedule.
Also lost without a restart:
- Changing an
otelcol.*component’s arguments rebuilds it and drops its in-memory queue — a config reload can lose data. - Renaming a component’s label gives it a new storage directory. The WAL or queue under the old label is orphaned and never sent.
Our stack keeps no persistent queue anywhere. A gateway restart during a Tempo outage loses whatever was queued on that replica.
Failure modes
When one component goes down
| Down | Effect | Blast radius |
|---|---|---|
| One app pod (SDK) | Its unsent batch is lost | That pod |
| VM collector | • That machine’s telemetry stops • On restart logs are re-read from files, the rest is gone |
One machine |
| One DaemonSet pod | Logs and eBPF data from that node stop | • One node • Logs recoverable while the kubelet keeps the rotated files and the positions file survives |
| Anti-corruption layer | • The legacy team’s logs and metrics stop • The app retries, then drops |
One team |
| One gateway replica | • Senders reconnect to other replicas • Its queue waits on its PVC and drains when the pod returns |
Delayed, not lost — unless the PVC is deleted |
| Gateway (all replicas) | • Queued data is safe on the PVCs • New data: senders retry up to 5 min, SDKs buffer a little, then everyone drops |
Every producer, for new data |
| One backend (Tempo, Loki, …) | • That signal queues on the Gateway’s PVC and drains on return • Loss starts when the PVC fills |
One signal |
| Optional output | Its exporter queue fills; see one slow output below | Can spread to all outputs |
A failure in a collecting layer leaves a local gap; in the layer every path crosses, a global stop. So PDB, priorityClass, anti-affinity and a durable queue belong on the global-blast-radius tier, not everywhere.
How data gets lost
First question: is the error retried? Check the queue against send_failed (Reading the queue).
| Failure | What happens | Signal |
|---|---|---|
| Backend slow or down | • Retryable; the Gateway’s queue grows on disk • With a persistent queue, PVC size, not max_elapsed_time, sets the survivable outage |
• PVC usage (kubelet_volume_stats_used_bytes) climbing• otelcol_exporter_queue_size rising• send_failed flat |
| Wrong token, TLS mismatch | • Permanent; dropped instantly, persistent queue or not • A token rotation loses data unless old and new are both accepted |
• send_failed rising at once, queue flat• unauthenticated / x509 in the sender’s log |
| PVC or queue full | • New data rejected at the door • PVC size and queue_size are separate limits; whichever hits first stops the queue• A bigger queue only moves the problem — fix the backend |
• otelcol_exporter_enqueue_failed_* rising• PVC near capacity |
| Rollout, OOM kill, node drain | • The Gateway’s queue survives • Every batch processor and in-memory queue in front of it is lost, uncounted |
• Gaps aligned with pod restarts • memory_limiter missing or not first in the pipeline |
| gRPC imbalance | • One gateway replica does all the work • HPA adds replicas that get nothing (Topologies) |
otelcol_receiver_accepted_* very uneven per pod |
| Oversized payload | ResourceExhausted … larger than max (4 MiB default) is permanent; the batch is dropped |
The message itself in the sender’s log |
| One slow output | The gateway fans out to every exporter. One full exporter queue → error to the sender → sender retries the whole batch → healthy backends get duplicates | • enqueue_failed on one exporter• Duplicates in the others |
| Silent drop | All green, data still disappears: • New config not loaded • Drop rule too broad • Record routed to the wrong index • Backend rejects the document (Elasticsearch 400 is not in send_failed)• Bad timestamp files it under another day |
• None from the collector • Only a canary querying the backend catches it (Self-monitoring) |
Diagnose backwards, from the backend to the sender: going forwards, every component looks healthy.