Collectors — Roles and Topology

A collector is the only component that touches every signal twice: once on the way in, once on the way out.

This module covers running collectors (Grafana Alloy, OpenTelemetry Collector) as production infrastructure: where they sit, how they lose data, how you find out.

One binary, many roles

There is no “agent mode” or “gateway mode” switch. A collector’s role comes from two choices:

Choice Decides
Which components you wire What it does: tail files, scrape, receive OTLP, enrich, sample, export
How Kubernetes runs it (controller.type) Where it runs and what it sees: one per node, N stateless replicas, N replicas with stable identity and disk

One grafana/alloy (or otelcol-contrib) image runs as node agent, scraper or gateway.

Role Typical controller Why that controller
Node agent — pod logs, eBPF, kubelet/cAdvisor, node metrics DaemonSet Needs node-local access (/var/log/pods, host PID, BPF maps)
Scraper — Prometheus targets, ServiceMonitors StatefulSet + clustering WAL on a PVC, targets sharded between replicas
Gateway — OTLP from SDKs, enrichment, fan-out Deployment + HPA Stateless, scales on load
Export tier — the only writer to a backend, durable queue StatefulSet One PVC-backed queue per replica

The layers

flowchart LR
    subgraph demobox ["team namespace — mature, compliant team"]
        demo["OTel Demo services<br/>(OTel SDK)"]
    end
    demo --> gw
    demo --> gw
    demo --> gw
    subgraph vmbox ["VM outside the cluster"]
        vm["Alloy — systemd service<br/>host logs, node_exporter<br/>OBI, pyroscope.ebpf"]
    end
    vm --> gw
    vm --> gw
    vm --> gw
    vm --> gw
    subgraph node ["Node agent — DaemonSet"]
        logs["pod logs<br/>loki.source.kubernetes"]
        ebpf["beyla.ebpf / pyroscope.ebpf"]
    end
    ebpf --> gw
    subgraph aclbox ["team namespace — optional, temporary"]
        legacy["Legacy app<br/>own log and metric format"] --> acl["Alloy — anti-corruption layer<br/>maps logs and metrics to the common format"]
        legacy --> acl
    end
    acl -.-> gw
    acl -.-> gw
    subgraph gwbox ["Gateway — StatefulSet, clustering, HPA, persistent queue on PVC"]
        gw["OTLP receiver<br/>+ ServiceMonitor scrape"]
    end
    logs --> gw
    ebpf --> gw
    gw --> tempo[(Tempo)]
    gw --> prom[(Prometheus / Mimir)]
    gw --> loki[(Loki)]
    gw --> pyro[(Pyroscope)]
    subgraph extbox ["Optional outputs — any signal"]
        es[(Elasticsearch)]
        dt[(Dynatrace)]
        other[("Datadog, Splunk, …")]
    end
    gw -.-> es
    gw -.-> dt
    gw -.-> other

    subgraph legend ["Legend"]
        direction LR
        lgLogs["logs"] ~~~ lgMetrics["metrics"] ~~~ lgTraces["traces"] ~~~ lgProfiles["profiles"]
    end
    demobox ~~~ legend

    classDef logsNode fill:#2e9e5b,stroke:#1f6e3f,color:#fff
    classDef metricsNode fill:#e8871e,stroke:#a85f12,color:#fff
    classDef tracesNode fill:#3b82f6,stroke:#1d4ed8,color:#fff
    classDef profilesNode fill:#a855f7,stroke:#7e22ce,color:#fff
    class loki,logs,lgLogs logsNode
    class prom,lgMetrics metricsNode
    class tempo,lgTraces tracesNode
    class pyro,lgProfiles profilesNode

    %% edge order: 0-2 demo→gw (logs, metrics, traces),
    %% 3-6 vm→gw (logs, metrics, traces, profiles), 7 ebpf→gw (traces),
    %% 8-9 legacy→acl (logs, metrics), 10-11 acl→gw (logs, metrics),
    %% 12 logs→gw, 13 ebpf→gw (profiles), 14 gw→tempo, 15 gw→prom,
    %% 16 gw→loki, 17 gw→pyro, 18-20 gw→optional outputs, 21-24 legend spacers
    linkStyle 0,3,8,10,12,16 stroke:#2e9e5b,stroke-width:2px
    linkStyle 1,4,9,11,15 stroke:#e8871e,stroke-width:2px
    linkStyle 2,5,7,14 stroke:#3b82f6,stroke-width:2px
    linkStyle 6,13,17 stroke:#a855f7,stroke-width:2px
    linkStyle 18,19,20 stroke:#888,stroke-width:2px
Collector Runs as Does Lifetime
Node agent DaemonSet, privileged, hostPID • Pod logs from its node
• eBPF traces and profiles from its node
• Sends everything to the Gateway		 Permanent
Gateway StatefulSet with clustering and HPA; one PVC per replica • OTLP receiver for SDKs and external collectors
• Sharded ServiceMonitor scraping
• Fan-out to every backend via a persistent queue
• Only tier that survives a backend outage without loss		 Permanent
VM collector Alloy as a systemd service on a VM outside the cluster • Host logs and metrics
• OBI traces and eBPF profiles from machines Kubernetes cannot see
• OTLP to the Gateway via the ingress		 Permanent
Anti-corruption layer Alloy Deployment in the owning team’s namespace • Rewrites the team’s logs and metrics to the common format (field names, severity, resource attributes) before the Gateway
• Keeps the mapping in the team’s namespace, so the Gateway holds no per-team rules		 Temporary. Delete once the team emits the common format

A mature team’s services need no collector: the OTel SDK sends OTLP straight to the Gateway.

Star, not mesh

  • Collectors never talk sideways. Each pushes toward the backends; none forwards to a peer.
  • One layer writes to each backend. Three layers writing to Loki = three buffers, three retry configs, three places to look during an outage.
  • Diagnose backwards from the backend. Compare what each hop received with what it sent (lesson 09).

When a hop earns its place

Add a layer only if it does what the previous layer cannot:

Valid reason Example
Node-local access Tailing /var/log/pods, eBPF — must run on the node
Durable buffer PVC-backed queue that survives a backend outage longer than memory can
Affinity-dependent processing Tail sampling needs every span of a trace on one instance
Credential boundary Only the export tier holds backend tokens
Egress control One place leaves the network, one place to firewall
Format translation, time-boxed An anti-corruption layer keeping one team’s formats out of the Gateway, removed once the team migrates

“Architectural tidiness” is not a reason. Every hop adds a queue that loses data on restart, a version to upgrade and metrics to watch.

Scaling and buffering

Each collector scales on a different axis and buffers in a different place. The exporter queue lets a collector ride out a slow next hop.

flowchart TB
    subgraph app ["App pod — scales with the app"]
        sdk["SDK batch + export queue<br/>memory · 2048 items"]
    end
    subgraph node ["DaemonSet / VM — one per node"]
        files[("log files on the node<br/>disk · kubelet rotation")]
        agent["exporter queue<br/>memory · 1000 requests"]
    end
    subgraph gwbox ["Gateway — StatefulSet + HPA"]
        pvc[("persistent queue<br/>PVC per replica · sized by outage")]
    end
    be[(Backend)]
    drop(["dropped"])

    files -->|"tail, re-read after restart"| agent
    sdk --> pvc
    agent --> pvc
    pvc --> be
    be -.->|"retryable error:<br/>backoff, data stays on disk"| pvc
    pvc -->|"PVC full · permanent error"| drop
    sdk -->|"pod restart"| drop
    agent -->|"pod restart · gateway down<br/>longer than the 5 min retry"| drop

    subgraph legend ["Legend"]
        direction LR
        lgMem["memory — lost on restart"] ~~~ lgDisk[("disk — survives restart")] ~~~ lgDrop(["data loss"])
    end
    be ~~~ legend

    classDef mem fill:#fff,stroke:#64748b,stroke-dasharray:4 3,color:#1e293b
    classDef disk fill:#334155,stroke:#0f172a,color:#fff
    classDef loss fill:#dc2626,stroke:#991b1b,color:#fff
    class sdk,agent,lgMem mem
    class files,pvc,lgDisk,be disk
    class drop,lgDrop loss

    %% edge order: 0 files→agent, 1 sdk→gw, 2 agent→gw, 3 gw→backend,
    %% 4 backend retry, 5-7 →dropped, 8-10 legend spacers
    linkStyle 4 stroke:#e8871e,stroke-width:2px
    linkStyle 5,6,7 stroke:#dc2626,stroke-width:2px

The split is deliberate. Tiers in front of the Gateway only survive a short Gateway hiccup: short retries, in-memory queues. The Gateway survives a backend outage: its queue is on a PVC and drains when the backend returns. Data in a dashed box when the pod stops is lost; data in the Gateway’s queue waits until the PVC fills.

Collector Scales with Buffers in Survives a restart What really protects the data
OTel SDK in the mature team’s services The app’s own replicas • SDK batch processor
• Export queue (default 2048 items)		 No • Nothing — full queue → SDK drops
• Never make the gateway a synchronous dependency of the app
VM collector (Alloy, systemd) Nothing; one per machine • In-memory sending_queue per exporter
• Log positions file on local disk		 • Logs: yes, re-read from files
• OBI traces, profiles, metrics: no		 • Logs: the VM’s log files
• The rest: add otelcol.storage.file on local disk; a VM has no PVC to lose
Node agent (DaemonSet) Node count; no HPA • In-memory sending_queue
• Positions under storagePath		 Only with storagePath on a hostPath volume; otherwise positions die with the pod • Logs: the files the kubelet keeps on the node
• eBPF traces and profiles: no second copy
Anti-corruption layer • One team’s volume
• Small Deployment, HPA optional		 In-memory sending_queue No • The legacy app’s own retry
• Keep the layer stateless — it is meant to be deleted
Gateway • StatefulSet + HPA on PVC usage, not only CPU/memory
• Clustering shards scrape targets		 • sending_queue with storage on the replica’s PVC (fully on disk, no in-memory tier)
• Remote-write WAL on the same PVC		 Yes. The replica gets its PVC back and drains it • The PVC, sized for the longest backend outage to survive
• Delivery is at-least-once: expect duplicates after a crash
Optional outputs The vendor’s side One persistent exporter queue each, on the Gateway’s PVC Yes, like other Gateway queues • The vendor’s ingest
• A slow vendor fills its own queue, then pushes back on everyone (see failure modes)

Scaling rules (details in Topologies):

  • CPU/memory HPA misses a slow backend. Data waits on the PVC, not in RAM, so nothing scales while the disk fills. Add kubelet_volume_stats_used_bytes as an HPA metric via prometheus-adapter, threshold below the disk-full alert.
  • Cap maxReplicas. Slow next hop → retries raise CPU → HPA adds replicas → more retries hit the same backend: a retry storm.
  • gRPC does not rebalance on scale-up. Existing HTTP/2 connections stay on old replicas; new ones idle until max_connection_age forces a reconnect.
  • Memory HPA scales down slowly. The Go runtime does not always return freed memory: replicas come up fast, leave late.
  • Scale-down strands data. A removed replica’s queue stays on its PVC with no consumer. whenScaled: Delete deletes the PVC and the unsent data; Retain keeps it, but nothing drains it. Drain before scaling down; hold scale-down during a backend outage.
  • Per-replica PVCs do not add up. Each replica buffers only what it received, so the busiest one sets the time to first loss. Even load balancing matters most here.
  • queue_size counts requests (batches), not records, and ignores disk. Size the Gateway’s PVC by outage length: PVC ≈ records/s × bytes/record × outage seconds × 1.4 (Reliability).

Failure modes

When one component goes down

Down Effect Blast radius
One app pod (SDK) Its unsent batch is lost That pod
VM collector • That machine’s telemetry stops
• On restart logs are re-read from files, the rest is gone		 One machine
One DaemonSet pod Logs and eBPF data from that node stop • One node
• Logs recoverable while the kubelet keeps the rotated files and the positions file survives
Anti-corruption layer • The legacy team’s logs and metrics stop
• The app retries, then drops		 One team
One gateway replica • Senders reconnect to other replicas
• Its queue waits on its PVC and drains when the pod returns		 Delayed, not lost — unless the PVC is deleted
Gateway (all replicas) • Queued data is safe on the PVCs
• New data: senders retry up to 5 min, SDKs buffer a little, then everyone drops		 Every producer, for new data
One backend (Tempo, Loki, …) • That signal queues on the Gateway’s PVC and drains on return
• Loss starts when the PVC fills		 One signal
Optional output Its exporter queue fills; see one slow output below Can spread to all outputs

A failure in a collecting layer leaves a local gap; in the layer every path crosses, a global stop. So PDB, priorityClass, anti-affinity and a durable queue belong on the global-blast-radius tier, not everywhere.

How data gets lost

First question: is the error retried? Queue growing, send_failed at zero = retryable, data is waiting. Queue flat, send_failed rising at once = permanent, data is gone (Reliability).

Failure What happens Signal
Backend slow or down • Retryable; the Gateway’s queue grows on disk
• PVC size, not max_elapsed_time, sets the survivable outage		 • PVC usage (kubelet_volume_stats_used_bytes) climbing
• otelcol_exporter_queue_size rising
• send_failed flat
Wrong token, TLS mismatch • Permanent; dropped instantly, persistent queue or not
• A token rotation loses data unless old and new are both accepted		 • send_failed rising at once, queue flat
• unauthenticated / x509 in the sender’s log
PVC or queue full • New data rejected at the door
• PVC size and queue_size are separate limits; whichever hits first stops the queue
• A bigger queue only moves the problem — fix the backend		 • otelcol_exporter_enqueue_failed_* rising
• PVC near capacity
Rollout, OOM kill, node drain • The Gateway’s queue survives
• Every batch processor and in-memory queue in front of it is lost, uncounted		 • Gaps aligned with pod restarts
• memory_limiter missing or not first in the pipeline
gRPC imbalance • One gateway replica does all the work
• HPA adds replicas that get nothing		 otelcol_receiver_accepted_* very uneven per pod
Oversized payload ResourceExhausted … larger than max (4 MiB default) is permanent; the batch is dropped The message itself in the sender’s log
One slow output The gateway fans out to every exporter. One full exporter queue → error to the sender → sender retries the whole batch → healthy backends get duplicates • enqueue_failed on one exporter
• Duplicates in the others
Silent drop All green, data still disappears:
• New config not loaded
• Drop rule too broad
• Record routed to the wrong index
• Backend rejects the document (Elasticsearch 400 is not in send_failed)
• Bad timestamp files it under another day		 • None from the collector
• Only a canary querying the backend catches it (Self-monitoring)

Diagnose backwards, from the backend to the sender: going forwards, every component looks healthy.

What a collector does not do

  • It does not store data. WAL and sending queue are transient buffers sized in minutes, not days.
  • It does not guarantee delivery. At-least-once at best; in-flight data is lost on restart without a persistent queue (lesson 06).
  • It does not report drops unless you read the right metric (lesson 09). A green health check means the process runs, not that data arrives.

results matching ""

    No results matching ""