Collectors — Roles and Topology
- One binary, many roles
- The layers
- Star, not mesh
- When a hop earns its place
- Where reliability lives
- What a collector does not do
- Related lessons
A collector is the only component that touches every signal twice: once on the way in, once on the way out.
This module covers running collectors (Grafana Alloy, OpenTelemetry Collector) as production infrastructure: where they sit, how they lose data, how you find out.
This page shows the reference design. Our workshop stack runs a subset of it: a node-agent DaemonSet (alloy-collector) and a gateway Deployment (alloy) with no persistent queue. The gaps are listed in Topologies and Reliability.
One binary, many roles
There is no “agent mode” or “gateway mode” switch. A collector’s role comes from two choices:
| Choice | Decides |
|---|---|
| Which components you wire | What it does: tail files, scrape, receive OTLP, enrich, sample, export |
How Kubernetes runs it (controller.type) |
Where it runs and what it sees: one per node, N stateless replicas, N replicas with stable identity and disk |
One grafana/alloy (or otelcol-contrib) image runs as node agent, scraper or gateway.
| Role | Typical controller | Why that controller |
|---|---|---|
| Node agent — pod logs, eBPF, kubelet/cAdvisor, node metrics | DaemonSet |
Needs node-local access (/var/log/pods, host PID, BPF maps) |
| Scraper — Prometheus targets, ServiceMonitors | StatefulSet + clustering |
WAL on a PVC, targets sharded between replicas |
| Gateway — OTLP from SDKs, enrichment, fan-out | Deployment + HPA |
Stateless, scales on load |
| Export tier — the only writer to a backend, durable queue | StatefulSet |
One PVC-backed queue per replica |
The reference layout below folds scraper, gateway and export tier into one Gateway StatefulSet: one layer writes to every backend and holds the only durable queue. Split them into separate tiers when they must scale independently (Topologies).
The layers
flowchart LR
subgraph demobox ["team namespace — mature, compliant team"]
demo["OTel Demo services<br/>(OTel SDK)"]
end
demo --> gw
demo --> gw
demo --> gw
subgraph vmbox ["VM outside the cluster"]
vm["Alloy — systemd service<br/>host logs, node_exporter<br/>OBI, pyroscope.ebpf"]
end
vm --> gw
vm --> gw
vm --> gw
vm --> gw
subgraph node ["Node agent — DaemonSet"]
logs["pod logs<br/>loki.source.kubernetes"]
ebpf["beyla.ebpf / pyroscope.ebpf"]
end
ebpf --> gw
subgraph aclbox ["team namespace — optional, temporary"]
legacy["Legacy app<br/>own log and metric format"] --> acl["Alloy — anti-corruption layer<br/>maps logs and metrics to the common format"]
legacy --> acl
end
acl -.-> gw
acl -.-> gw
subgraph gwbox ["Gateway — StatefulSet, clustering, HPA, persistent queue on PVC"]
gw["OTLP receiver<br/>+ ServiceMonitor scrape"]
end
logs --> gw
ebpf --> gw
gw --> tempo[(Tempo)]
gw --> prom[(Prometheus / Mimir)]
gw --> loki[(Loki)]
gw --> pyro[(Pyroscope)]
subgraph extbox ["Optional outputs — any signal"]
es[(Elasticsearch)]
dt[(Dynatrace)]
other[("Datadog, Splunk, …")]
end
gw -.-> es
gw -.-> dt
gw -.-> other
subgraph legend ["Legend"]
direction LR
lgLogs["logs"] ~~~ lgMetrics["metrics"] ~~~ lgTraces["traces"] ~~~ lgProfiles["profiles"]
end
demobox ~~~ legend
classDef logsNode fill:#2e9e5b,stroke:#1f6e3f,color:#fff
classDef metricsNode fill:#e8871e,stroke:#a85f12,color:#fff
classDef tracesNode fill:#3b82f6,stroke:#1d4ed8,color:#fff
classDef profilesNode fill:#a855f7,stroke:#7e22ce,color:#fff
class loki,logs,lgLogs logsNode
class prom,lgMetrics metricsNode
class tempo,lgTraces tracesNode
class pyro,lgProfiles profilesNode
%% edge order: 0-2 demo→gw (logs, metrics, traces),
%% 3-6 vm→gw (logs, metrics, traces, profiles), 7 ebpf→gw (traces),
%% 8-9 legacy→acl (logs, metrics), 10-11 acl→gw (logs, metrics),
%% 12 logs→gw, 13 ebpf→gw (profiles), 14 gw→tempo, 15 gw→prom,
%% 16 gw→loki, 17 gw→pyro, 18-20 gw→optional outputs, 21-24 legend spacers
linkStyle 0,3,8,10,12,16 stroke:#2e9e5b,stroke-width:2px
linkStyle 1,4,9,11,15 stroke:#e8871e,stroke-width:2px
linkStyle 2,5,7,14 stroke:#3b82f6,stroke-width:2px
linkStyle 6,13,17 stroke:#a855f7,stroke-width:2px
linkStyle 18,19,20 stroke:#888,stroke-width:2px
| Collector | Runs as | Does | Lifetime |
|---|---|---|---|
| Node agent | DaemonSet, privileged, hostPID |
• Pod logs from its node • eBPF traces and profiles from its node • Sends everything to the Gateway |
Permanent |
| Gateway | StatefulSet with clustering and HPA; one PVC per replica | • OTLP receiver for SDKs and external collectors • Sharded ServiceMonitor scraping • Fan-out to every backend via a persistent queue • Only tier that survives a backend outage without loss |
Permanent |
| VM collector | Alloy as a systemd service on a VM outside the cluster | • Host logs and metrics • OBI traces and eBPF profiles from machines Kubernetes cannot see • OTLP to the Gateway via the ingress |
Permanent |
| Anti-corruption layer | Alloy Deployment in the owning team’s namespace | • Rewrites the team’s logs and metrics to the common format (field names, severity, resource attributes) before the Gateway • Keeps the mapping in the team’s namespace, so the Gateway holds no per-team rules |
Temporary. Delete once the team emits the common format |
A mature team’s services need no collector: the OTel SDK sends OTLP straight to the Gateway.
Star, not mesh
- Collectors never talk sideways. Each pushes toward the backends; none forwards to a peer.
- One layer writes to each backend. Three layers writing to Loki = three buffers, three retry configs, three places to look during an outage.
- Diagnose backwards from the backend. Compare what each hop received with what it sent (Self-monitoring).
When a hop earns its place
Add a layer only if it does what the previous layer cannot:
| Valid reason | Example |
|---|---|
| Node-local access | Tailing /var/log/pods, eBPF — must run on the node |
| Durable buffer | PVC-backed queue that survives a backend outage longer than memory can |
| Affinity-dependent processing | Tail sampling needs every span of a trace on one instance |
| Credential boundary | Only the export tier holds backend tokens |
| Egress control | One place leaves the network, one place to firewall |
| Format translation, time-boxed | An anti-corruption layer keeping one team’s formats out of the Gateway, removed once the team migrates |
“Architectural tidiness” is not a reason. Every hop adds a queue that loses data on restart, a version to upgrade and metrics to watch.
Where reliability lives
- Only the Gateway survives a backend outage. Its queue is on a PVC. Every tier in front of it buffers in memory and rides out only a short Gateway hiccup.
- A failure in a collecting layer leaves a local gap; in the Gateway, a global stop. PDB, priorityClass, anti-affinity and the durable queue belong on the Gateway, not everywhere.
Buffers per tier, blast radius per component and every way data gets lost: Reliability.
What a collector does not do
- It does not store data. WAL and sending queue are buffers sized for an outage, not for retention.
- It does not guarantee delivery. At-least-once at best; in-flight data is lost on restart without a persistent queue (Reliability).
- It does not report drops unless you read the right metric (Self-monitoring). A green health check means the process runs, not that data arrives.
Related lessons
- OTel Architecture
- Cost Optimization — the collector as a cost gateway (sampling, filtering, routing)
- Grafana Alloy in our stack
- Pipeline model
- Reliability