Collectors — Roles and Topology

A collector is the only component that touches every signal twice: once on the way in, once on the way out.

This module covers running collectors (Grafana Alloy, OpenTelemetry Collector) as production infrastructure: where they sit, how they lose data, how you find out.

This page shows the reference design. Our workshop stack runs a subset of it: a node-agent DaemonSet (alloy-collector) and a gateway Deployment (alloy) with no persistent queue. The gaps are listed in Topologies and Reliability.

One binary, many roles

There is no “agent mode” or “gateway mode” switch. A collector’s role comes from two choices:

Choice Decides
Which components you wire What it does: tail files, scrape, receive OTLP, enrich, sample, export
How Kubernetes runs it (controller.type) Where it runs and what it sees: one per node, N stateless replicas, N replicas with stable identity and disk

One grafana/alloy (or otelcol-contrib) image runs as node agent, scraper or gateway.

Role Typical controller Why that controller
Node agent — pod logs, eBPF, kubelet/cAdvisor, node metrics DaemonSet Needs node-local access (/var/log/pods, host PID, BPF maps)
Scraper — Prometheus targets, ServiceMonitors StatefulSet + clustering WAL on a PVC, targets sharded between replicas
Gateway — OTLP from SDKs, enrichment, fan-out Deployment + HPA Stateless, scales on load
Export tier — the only writer to a backend, durable queue StatefulSet One PVC-backed queue per replica

The reference layout below folds scraper, gateway and export tier into one Gateway StatefulSet: one layer writes to every backend and holds the only durable queue. Split them into separate tiers when they must scale independently (Topologies).

The layers

flowchart LR
    subgraph demobox ["team namespace — mature, compliant team"]
        demo["OTel Demo services<br/>(OTel SDK)"]
    end
    demo --> gw
    demo --> gw
    demo --> gw
    subgraph vmbox ["VM outside the cluster"]
        vm["Alloy — systemd service<br/>host logs, node_exporter<br/>OBI, pyroscope.ebpf"]
    end
    vm --> gw
    vm --> gw
    vm --> gw
    vm --> gw
    subgraph node ["Node agent — DaemonSet"]
        logs["pod logs<br/>loki.source.kubernetes"]
        ebpf["beyla.ebpf / pyroscope.ebpf"]
    end
    ebpf --> gw
    subgraph aclbox ["team namespace — optional, temporary"]
        legacy["Legacy app<br/>own log and metric format"] --> acl["Alloy — anti-corruption layer<br/>maps logs and metrics to the common format"]
        legacy --> acl
    end
    acl -.-> gw
    acl -.-> gw
    subgraph gwbox ["Gateway — StatefulSet, clustering, HPA, persistent queue on PVC"]
        gw["OTLP receiver<br/>+ ServiceMonitor scrape"]
    end
    logs --> gw
    ebpf --> gw
    gw --> tempo[(Tempo)]
    gw --> prom[(Prometheus / Mimir)]
    gw --> loki[(Loki)]
    gw --> pyro[(Pyroscope)]
    subgraph extbox ["Optional outputs — any signal"]
        es[(Elasticsearch)]
        dt[(Dynatrace)]
        other[("Datadog, Splunk, …")]
    end
    gw -.-> es
    gw -.-> dt
    gw -.-> other

    subgraph legend ["Legend"]
        direction LR
        lgLogs["logs"] ~~~ lgMetrics["metrics"] ~~~ lgTraces["traces"] ~~~ lgProfiles["profiles"]
    end
    demobox ~~~ legend

    classDef logsNode fill:#2e9e5b,stroke:#1f6e3f,color:#fff
    classDef metricsNode fill:#e8871e,stroke:#a85f12,color:#fff
    classDef tracesNode fill:#3b82f6,stroke:#1d4ed8,color:#fff
    classDef profilesNode fill:#a855f7,stroke:#7e22ce,color:#fff
    class loki,logs,lgLogs logsNode
    class prom,lgMetrics metricsNode
    class tempo,lgTraces tracesNode
    class pyro,lgProfiles profilesNode

    %% edge order: 0-2 demo→gw (logs, metrics, traces),
    %% 3-6 vm→gw (logs, metrics, traces, profiles), 7 ebpf→gw (traces),
    %% 8-9 legacy→acl (logs, metrics), 10-11 acl→gw (logs, metrics),
    %% 12 logs→gw, 13 ebpf→gw (profiles), 14 gw→tempo, 15 gw→prom,
    %% 16 gw→loki, 17 gw→pyro, 18-20 gw→optional outputs, 21-24 legend spacers
    linkStyle 0,3,8,10,12,16 stroke:#2e9e5b,stroke-width:2px
    linkStyle 1,4,9,11,15 stroke:#e8871e,stroke-width:2px
    linkStyle 2,5,7,14 stroke:#3b82f6,stroke-width:2px
    linkStyle 6,13,17 stroke:#a855f7,stroke-width:2px
    linkStyle 18,19,20 stroke:#888,stroke-width:2px
Collector Runs as Does Lifetime
Node agent DaemonSet, privileged, hostPID • Pod logs from its node
• eBPF traces and profiles from its node
• Sends everything to the Gateway
Permanent
Gateway StatefulSet with clustering and HPA; one PVC per replica • OTLP receiver for SDKs and external collectors
• Sharded ServiceMonitor scraping
• Fan-out to every backend via a persistent queue
• Only tier that survives a backend outage without loss
Permanent
VM collector Alloy as a systemd service on a VM outside the cluster • Host logs and metrics
• OBI traces and eBPF profiles from machines Kubernetes cannot see
• OTLP to the Gateway via the ingress
Permanent
Anti-corruption layer Alloy Deployment in the owning team’s namespace • Rewrites the team’s logs and metrics to the common format (field names, severity, resource attributes) before the Gateway
• Keeps the mapping in the team’s namespace, so the Gateway holds no per-team rules
Temporary. Delete once the team emits the common format

A mature team’s services need no collector: the OTel SDK sends OTLP straight to the Gateway.

Star, not mesh

  • Collectors never talk sideways. Each pushes toward the backends; none forwards to a peer.
  • One layer writes to each backend. Three layers writing to Loki = three buffers, three retry configs, three places to look during an outage.
  • Diagnose backwards from the backend. Compare what each hop received with what it sent (Self-monitoring).

When a hop earns its place

Add a layer only if it does what the previous layer cannot:

Valid reason Example
Node-local access Tailing /var/log/pods, eBPF — must run on the node
Durable buffer PVC-backed queue that survives a backend outage longer than memory can
Affinity-dependent processing Tail sampling needs every span of a trace on one instance
Credential boundary Only the export tier holds backend tokens
Egress control One place leaves the network, one place to firewall
Format translation, time-boxed An anti-corruption layer keeping one team’s formats out of the Gateway, removed once the team migrates

“Architectural tidiness” is not a reason. Every hop adds a queue that loses data on restart, a version to upgrade and metrics to watch.

Where reliability lives

  • Only the Gateway survives a backend outage. Its queue is on a PVC. Every tier in front of it buffers in memory and rides out only a short Gateway hiccup.
  • A failure in a collecting layer leaves a local gap; in the Gateway, a global stop. PDB, priorityClass, anti-affinity and the durable queue belong on the Gateway, not everywhere.

Buffers per tier, blast radius per component and every way data gets lost: Reliability.

What a collector does not do

  • It does not store data. WAL and sending queue are buffers sized for an outage, not for retention.
  • It does not guarantee delivery. At-least-once at best; in-flight data is lost on restart without a persistent queue (Reliability).
  • It does not report drops unless you read the right metric (Self-monitoring). A green health check means the process runs, not that data arrives.

results matching ""

    No results matching ""