Reliability: Memory, Queues, Retries

• Where data waits
• Retryable or permanent
• Reading the queue
• Memory: memory_limiter and GOMEMLIMIT
• Persistent queue
• What is lost on restart
• Related lessons

A collector loses data in three ways: it runs out of memory, the backend rejects data for good, or the process restarts with data in RAM.

Each has its own setting and its own metric. None of them turns a health check red.

Where data waits

Only the last three are real buffers. Everything else is gone the moment the pod stops.

Retryable or permanent

The exporter decides from the response whether to retry.

Consequences:

• A wrong token loses data instantly. No queue saves you; the queue never fills.
• An unreachable backend is retried with exponential backoff until max_elapsed_time (default 5 min). Only then is the data counted as failed.
• Rotating a backend token is a data-loss event unless both old and new tokens are accepted during the switch (lesson 11).

Measured on Alloy v1.16.1 — one exporter to a closed port, one to a server answering 401, five log records sent to each:

The retry messages are logged at level info: Exporting failed. Will retry the request after interval.

Reading the queue

• queue_size counts requests (batches), not records. A queue of 1000 can hold 1000 spans or 8 million.
• send_failed counters do not exist until the first failure. An alert on rate(...) > 0 works; a dashboard panel shows “No data” instead of 0.

Memory: memory_limiter and GOMEMLIMIT

memory_limiter refuses new data when heap use crosses a limit.

otelcol.processor.memory_limiter "default" {
  check_interval         = "1s"   // required; "0s" fails at startup
  limit_percentage       = 80     // hard limit: 80 % of the container limit
  spike_limit_percentage = 20     // soft limit = 80 - 20 = 60 %
  output { /* ... */ }
}

• Both percentages are of the total limit. 80 / 20 means soft limit at 60 %, hard limit at 80 % — not “80 % plus a 20 % spike”.
• Above soft: refuses data. Above hard: also forces GC.
• It must be the first processor. Placed later, it throws away data that was already decoded and processed.
• Refusing is only safe if the sender retries. OTLP clients get a retryable error and back off. A pull receiver (scrape) cannot push back — refused samples are gone.
• check_interval = "0s" passes alloy validate but fails at alloy run with check_interval must be greater than zero. See Config lifecycle for why validate misses it.

GOMEMLIMIT makes the Go runtime collect garbage harder before hitting the container limit.

Do not add GOMEMLIMIT to extraEnv on top of what the chart does — you get a duplicate env var and one silently wins.

Our stack: neither Alloy release runs a memory_limiter. The gateway relies only on GOMEMLIMIT and the 4 GiB limit. Under a burst, it runs until the kernel kills it, and every in-memory queue on that replica goes with it.

For gateways, memory — not CPU — is the resource that fails first. Size limits and HPA on memory.

Persistent queue

# OTel Collector
extensions:
  file_storage:
    directory: /var/lib/otelcol/queue
    compaction:
      on_rebound: true          # without it, the file never shrinks
exporters:
  otlp:
    sending_queue:
      storage: file_storage
      queue_size: 5000          # requests, not records
service:
  extensions: [file_storage]

In Alloy: otelcol.storage.file (public preview in v1.16, so it needs --stability.level=public-preview).

• With storage set, the whole queue is on disk. There is no in-memory tier that spills over.
• Delivery becomes at-least-once. After a crash, the last requests may be sent twice.
• Size the PVC for the outage you want to survive: PVC ≈ ingest rate (records/s) × bytes per record × outage (s) × 1.4.
• Per-replica PVCs do not add up. With RWO volumes, each replica buffers only what it received. Time to first loss is set by the busiest replica, not by total capacity.
• Alert on PVC fill (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes), not on the queue ratio.
• whenScaled: Delete on a StatefulSet deletes the PVC — and its unsent data — on scale-down.

What is lost on restart

Every rollout, OOM kill and node drain drops:

• The content of every batch processor.
• Every in-memory sending_queue. Shutdown gives the exporter a short chance to flush, but a queue is only full when the backend is failing. Those are exactly the requests that cannot be flushed.
• The WAL on emptyDir — our gateway is a Deployment, so its remote-write WAL is gone on every reschedule.

Also lost without a restart:

• Changing an otelcol.* component’s arguments rebuilds it and drops its in-memory queue — a config reload can lose data.
• Renaming a component’s label gives it a new storage directory. The WAL or queue under the old label is orphaned and never sent.

Our stack keeps no persistent queue anywhere. A gateway restart during a Tempo outage loses whatever was queued on that replica.

Related lessons

• Topologies on Kubernetes
• Config lifecycle
• Self-monitoring and silent drops
• Mimir