Reliability: Memory, Queues, Retries

A collector loses data in three ways: it runs out of memory, the backend rejects data for good, or the process restarts with data in RAM.

Each has its own setting and its own metric. None of them turns a health check red.

Where data waits

Buffer Survives a pod restart Size
SDK export queue in the application No Small (SDK default 2048 spans)
batch processor No One batch
Exporter sending_queue, in memory No queue_size requests (default 1000)
Exporter sending_queue with storage (PVC) Yes Disk
Prometheus WAL (--storage.path) Only on a PVC Until sent, at most max_keepalive_time (default 8 h)
Log files on the node (/var/log/pods) Yes, until kubelet rotates them Kubelet rotation settings

Only the last three are real buffers. Everything else is gone the moment the pod stops.

Buffering per tier

Each tier of the reference layout scales on a different axis and buffers in a different place. Our gateway has no PVC, so today it behaves like the in-memory tiers.

flowchart TB
    subgraph app ["App pod — scales with the app"]
        sdk["SDK batch + export queue<br/>memory · 2048 items"]
    end
    subgraph node ["DaemonSet / VM — one per node"]
        files[("log files on the node<br/>disk · kubelet rotation")]
        agent["exporter queue<br/>memory · 1000 requests"]
    end
    subgraph gwbox ["Gateway — StatefulSet + HPA"]
        pvc[("persistent queue<br/>PVC per replica · sized by outage")]
    end
    be[(Backend)]
    drop(["dropped"])

    files -->|"tail, re-read after restart"| agent
    sdk --> pvc
    agent --> pvc
    pvc --> be
    be -.->|"retryable error:<br/>backoff, data stays on disk"| pvc
    pvc -->|"PVC full · permanent error"| drop
    sdk -->|"pod restart"| drop
    agent -->|"pod restart · gateway down<br/>longer than the 5 min retry"| drop

    subgraph legend ["Legend"]
        direction LR
        lgMem["memory — lost on restart"] ~~~ lgDisk[("disk — survives restart")] ~~~ lgDrop(["data loss"])
    end
    be ~~~ legend

    classDef mem fill:#fff,stroke:#64748b,stroke-dasharray:4 3,color:#1e293b
    classDef disk fill:#334155,stroke:#0f172a,color:#fff
    classDef loss fill:#dc2626,stroke:#991b1b,color:#fff
    class sdk,agent,lgMem mem
    class files,pvc,lgDisk,be disk
    class drop,lgDrop loss

    %% edge order: 0 files→agent, 1 sdk→gw, 2 agent→gw, 3 gw→backend,
    %% 4 backend retry, 5-7 →dropped, 8-10 legend spacers
    linkStyle 4 stroke:#e8871e,stroke-width:2px
    linkStyle 5,6,7 stroke:#dc2626,stroke-width:2px

The split is deliberate. Tiers in front of the Gateway only survive a short Gateway hiccup: short retries, in-memory queues. The Gateway survives a backend outage: its queue is on a PVC and drains when the backend returns. Data in a dashed box when the pod stops is lost; data in the Gateway’s queue waits until the PVC fills.

Collector Scales with Buffers in Survives a restart What really protects the data
OTel SDK in the mature team’s services The app’s own replicas • SDK batch processor
• Export queue (default 2048 items)
No • Nothing — full queue → SDK drops
• Never make the gateway a synchronous dependency of the app
VM collector (Alloy, systemd) Nothing; one per machine • In-memory sending_queue per exporter
• Log positions file on local disk
• Logs: yes, re-read from files
• OBI traces, profiles, metrics: no
• Logs: the VM’s log files
• The rest: add otelcol.storage.file on local disk; a VM has no PVC to lose
Node agent (DaemonSet) Node count; no HPA • In-memory sending_queue
• Positions under storagePath
Only with storagePath on a hostPath volume; otherwise positions die with the pod • Logs: the files the kubelet keeps on the node
• eBPF traces and profiles: no second copy
Anti-corruption layer • One team’s volume
• Small Deployment, HPA optional
In-memory sending_queue No • The legacy app’s own retry
• Keep the layer stateless — it is meant to be deleted
Gateway • StatefulSet + HPA on PVC usage, not only CPU/memory
• Clustering shards scrape targets
• sending_queue with storage on the replica’s PVC (fully on disk, no in-memory tier)
• Remote-write WAL on the same PVC
Yes. The replica gets its PVC back and drains it • The PVC, sized for the longest backend outage to survive
Optional outputs The vendor’s side One persistent exporter queue each, on the Gateway’s PVC Yes, like other Gateway queues • The vendor’s ingest
• A slow vendor fills its own queue, then pushes back on everyone (see How data gets lost)

Retryable or permanent

The exporter decides from the response whether to retry.

Protocol Retried Not retried (dropped at once)
OTLP gRPC UNAVAILABLE, RESOURCE_EXHAUSTED (with retry info), DEADLINE_EXCEEDED, ABORTED, CANCELLED, OUT_OF_RANGE, DATA_LOSS UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT, NOT_FOUND, UNIMPLEMENTED, …
OTLP HTTP 429, 502, 503, 504 Every other 4xx/5xx
Prometheus remote write 5xx, 429 (retry_on_http_429) Other 4xx

Consequences:

  • A wrong token loses data instantly. No queue saves you; the queue never fills.
  • An unreachable backend is retried with exponential backoff until max_elapsed_time (default 5 min). Only then is the data counted as failed.
  • Rotating a backend token is a data-loss event unless both old and new tokens are accepted during the switch (lesson 11).

Measured on Alloy v1.16.1 — one exporter to a closed port, one to a server answering 401, five log records sent to each:

Exporter otelcol_exporter_queue_size otelcol_exporter_send_failed_log_records_total /-/healthy
otlp → closed port 5 and holding absent 200
otlphttp → 401 0 5 immediately 200

The retry messages are logged at level info: Exporting failed. Will retry the request after interval.

Reading the queue

Queue send_failed Meaning Action
Growing Flat Retryable errors; data is buffered Backend is slow or down. You have until the queue fills or max_elapsed_time expires
Full Rising Buffer exhausted; data is being dropped Outage longer than the buffer
Flat / zero Rising Permanent error; data is dropped right now Auth, schema, payload size. Read the exporter log
Flat / zero Flat Healthy — or nothing is arriving Check receiver_accepted upstream
  • queue_size counts requests (batches), not records. A queue of 1000 can hold 1000 spans or 8 million.
  • send_failed counters do not exist until the first failure. An alert on rate(...) > 0 works; a dashboard panel shows “No data” instead of 0.

Memory: memory_limiter and GOMEMLIMIT

memory_limiter refuses new data when heap use crosses a limit.

otelcol.processor.memory_limiter "default" {
  check_interval         = "1s"   // required; "0s" fails at startup
  limit_percentage       = 80     // hard limit: 80 % of the container limit
  spike_limit_percentage = 20     // soft limit = 80 - 20 = 60 %
  output { /* ... */ }
}
  • Both percentages are of the total limit. 80 / 20 means soft limit at 60 %, hard limit at 80 % — not “80 % plus a 20 % spike”.
  • Above soft: refuses data. Above hard: also forces GC.
  • It must be the first processor. Placed later, it throws away data that was already decoded and processed.
  • Refusing is only safe if the sender retries. OTLP clients get a retryable error and back off. A pull receiver (scrape) cannot push back — refused samples are gone.
  • check_interval = "0s" passes alloy validate but fails at alloy run with check_interval must be greater than zero. See Config lifecycle for why validate misses it.

GOMEMLIMIT makes the Go runtime collect garbage harder before hitting the container limit.

Runtime Who sets it
Alloy Alloy itself, 90 % of the cgroup limit. Our gateway: limit 4 GiB → go_gc_gomemlimit_bytes = 3.87 GB
OTel Collector chart useGOMEMLIMIT: true (80 % of the limit)
Bare Collector binary Nobody — set GOMEMLIMIT yourself, ~80 % of the limit

Do not add GOMEMLIMIT to extraEnv on top of what the chart does — you get a duplicate env var and one silently wins.

Our stack: neither Alloy release runs a memory_limiter. The gateway relies only on GOMEMLIMIT and the 4 GiB limit. Under a burst, it runs until the kernel kills it, and every in-memory queue on that replica goes with it.

For gateways, memory — not CPU — is the resource that fails first. Size limits and HPA on memory.

Persistent queue

# OTel Collector
extensions:
  file_storage:
    directory: /var/lib/otelcol/queue
    compaction:
      on_rebound: true          # without it, the file never shrinks
exporters:
  otlp:
    sending_queue:
      storage: file_storage
      queue_size: 5000          # requests, not records
service:
  extensions: [file_storage]

In Alloy: otelcol.storage.file (public preview in v1.16, so it needs --stability.level=public-preview).

  • With storage set, the whole queue is on disk. There is no in-memory tier that spills over.
  • Delivery becomes at-least-once. After a crash, the last requests may be sent twice.
  • Size the PVC for the outage you want to survive: PVC ≈ ingest rate (records/s) × bytes per record × outage (s) × 1.4.
  • Per-replica PVCs do not add up. With RWO volumes, each replica buffers only what it received. Time to first loss is set by the busiest replica, not by total capacity.
  • Alert on PVC fill (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes), not on the queue ratio.
  • CPU/memory HPA misses a filling PVC. Data waits on disk, not in RAM, so nothing scales while the disk fills. Scale on PVC usage (Topologies).
  • Scale-down strands data. A removed replica’s queue stays on its PVC with no consumer. whenScaled: Delete deletes the PVC and the unsent data; Retain keeps it, but nothing drains it. Drain before scaling down; hold scale-down during a backend outage.

What is lost on restart

Every rollout, OOM kill and node drain drops:

  • The content of every batch processor.
  • Every in-memory sending_queue. Shutdown gives the exporter a short chance to flush, but a queue is only full when the backend is failing. Those are exactly the requests that cannot be flushed.
  • The WAL on emptyDir — our gateway is a Deployment, so its remote-write WAL is gone on every reschedule.

Also lost without a restart:

  • Changing an otelcol.* component’s arguments rebuilds it and drops its in-memory queue — a config reload can lose data.
  • Renaming a component’s label gives it a new storage directory. The WAL or queue under the old label is orphaned and never sent.

Our stack keeps no persistent queue anywhere. A gateway restart during a Tempo outage loses whatever was queued on that replica.

Failure modes

When one component goes down

Down Effect Blast radius
One app pod (SDK) Its unsent batch is lost That pod
VM collector • That machine’s telemetry stops
• On restart logs are re-read from files, the rest is gone
One machine
One DaemonSet pod Logs and eBPF data from that node stop • One node
• Logs recoverable while the kubelet keeps the rotated files and the positions file survives
Anti-corruption layer • The legacy team’s logs and metrics stop
• The app retries, then drops
One team
One gateway replica • Senders reconnect to other replicas
• Its queue waits on its PVC and drains when the pod returns
Delayed, not lost — unless the PVC is deleted
Gateway (all replicas) • Queued data is safe on the PVCs
• New data: senders retry up to 5 min, SDKs buffer a little, then everyone drops
Every producer, for new data
One backend (Tempo, Loki, …) • That signal queues on the Gateway’s PVC and drains on return
• Loss starts when the PVC fills
One signal
Optional output Its exporter queue fills; see one slow output below Can spread to all outputs

A failure in a collecting layer leaves a local gap; in the layer every path crosses, a global stop. So PDB, priorityClass, anti-affinity and a durable queue belong on the global-blast-radius tier, not everywhere.

How data gets lost

First question: is the error retried? Check the queue against send_failed (Reading the queue).

Failure What happens Signal
Backend slow or down • Retryable; the Gateway’s queue grows on disk
• With a persistent queue, PVC size, not max_elapsed_time, sets the survivable outage
• PVC usage (kubelet_volume_stats_used_bytes) climbing
• otelcol_exporter_queue_size rising
• send_failed flat
Wrong token, TLS mismatch • Permanent; dropped instantly, persistent queue or not
• A token rotation loses data unless old and new are both accepted
• send_failed rising at once, queue flat
• unauthenticated / x509 in the sender’s log
PVC or queue full • New data rejected at the door
• PVC size and queue_size are separate limits; whichever hits first stops the queue
• A bigger queue only moves the problem — fix the backend
• otelcol_exporter_enqueue_failed_* rising
• PVC near capacity
Rollout, OOM kill, node drain • The Gateway’s queue survives
• Every batch processor and in-memory queue in front of it is lost, uncounted
• Gaps aligned with pod restarts
• memory_limiter missing or not first in the pipeline
gRPC imbalance • One gateway replica does all the work
• HPA adds replicas that get nothing (Topologies)
otelcol_receiver_accepted_* very uneven per pod
Oversized payload ResourceExhausted … larger than max (4 MiB default) is permanent; the batch is dropped The message itself in the sender’s log
One slow output The gateway fans out to every exporter. One full exporter queue → error to the sender → sender retries the whole batch → healthy backends get duplicates • enqueue_failed on one exporter
• Duplicates in the others
Silent drop All green, data still disappears:
• New config not loaded
• Drop rule too broad
• Record routed to the wrong index
• Backend rejects the document (Elasticsearch 400 is not in send_failed)
• Bad timestamp files it under another day
• None from the collector
• Only a canary querying the backend catches it (Self-monitoring)

Diagnose backwards, from the backend to the sender: going forwards, every component looks healthy.

results matching ""

    No results matching ""