Topologies on Kubernetes

• Choosing the controller
• Clustering — sharding scrape targets
• gRPC load balancing — the silent skew
• Traces that need affinity
• Where durability and hardening go
• Autoscaling
• Sizing rules of thumb
• Related lessons

Topology is two decisions per layer: which Kubernetes controller runs it, and how traffic is spread across its replicas.

The second one fails silently: adding replicas can add zero throughput.

Choosing the controller

In the grafana/alloy chart, controller.type picks one of three:

• The chart’s default config is empty. A fresh install collects nothing.
• controller.type cannot change in place. helm upgrade from DaemonSet to Deployment fails; run_deploy_scripts/deploy-alloy-module.sh uninstalls the old release first for exactly this reason.
• prometheus.exporter.unix in a DaemonSet without hostPath mounts reports the container, not the node.
• The WAL has no directory argument. It lives under --storage.path (chart: alloy.storagePath), one directory per remote_write. For a StatefulSet, alloy.storagePath must match the volumeClaimTemplates mount path.

A common production layout is two releases: a DaemonSet for logs and node data, and a clustered StatefulSet for metrics. Ours is a DaemonSet (alloy-collector) plus a clustered Deployment (alloy).

Clustering — sharding scrape targets

Clustering makes N replicas split one target list so each target is scraped once.

It needs both switches:

How it works:

• Peers discover each other through the headless Service <release>-cluster (ours: alloy-cluster) and gossip over the HTTP port (12345).
• Consistent hashing: each node owns 512 tokens on a ring; a target belongs to the node owning its hash.
• Membership change moves ~1/N of targets. During a rollout, expect short gaps or overlaps on the moved targets.
• Clustering does not balance incoming OTLP. Pushed data goes wherever the Service sends the connection.
• --cluster.wait-for-size holds clustered components until N peers join. The default --cluster.wait-timeout of 0 means wait forever — set both or neither.
• prometheus.exporter.self under clustering is itself one target: without a fixed instance label it looks like a different series on whichever node scrapes it.

Inspect: the UI’s Clustering page lists peers; cluster_node_info and cluster_node_peers are the metrics.

💪Exercise💪 (self-guided): Clustering split — run two replicas with and without the component block and count who scrapes what.

gRPC load balancing — the silent skew

gRPC keeps one long-lived HTTP/2 connection per client. A ClusterIP Service balances connections, not requests. Once connected, a client stays on one replica until the connection breaks.

Our gateway, 10-minute rate of accepted spans per replica, measured on the workshop cluster:

Two replicas, 74/26 split. With many senders (a DaemonSet of 50 pods), connections average out. With few senders, one replica does most of the work and a new replica from the HPA gets no traffic at all until clients reconnect.

Client-side fix (when the sender is a collector) — all three together:

otelcol.exporter.otlp "gateway" {
  client {
    endpoint      = "dns:///alloy-cluster.monitoring.svc.cluster.local:4317"  // headless Service + dns:///
    balancer_name = "round_robin"                                             // balance per request
    tls { insecure = true }
  }
}

• Headless Service — DNS returns every pod IP, not one virtual IP. Ours already exists: alloy-cluster exposes 4317.
• dns:/// scheme — makes the gRPC client resolve and track all addresses.
• round_robin — the default pick_first would still pin to one address.

Miss any one and nothing changes, with no error.

Server-side fix (when senders are SDKs you do not control) — force periodic reconnects:

otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
    keepalive {
      server_parameters {
        max_connection_age       = "5m"   // default: infinite
        max_connection_age_grace = "1m"   // let in-flight exports finish
      }
    }
  }
  // output { ... }
}

Traces that need affinity

Tail sampling decides on a whole trace, so every span of a trace must reach the same instance. Plain round-robin splits traces across replicas and each replica decides on a fragment.

Standard two-tier shape:

// Tier 1: stateless, routes by trace ID
otelcol.exporter.loadbalancing "to_sampling" {
  routing_key = "traceID"
  resolver {
    kubernetes {
      service = "sampler.monitoring"   // headless Service of tier 2
      ports   = [4317]
    }
  }
  protocol {
    otlp {
      client { tls { insecure = true } }
    }
  }
}
// Tier 2: otelcol.processor.tail_sampling on every replica

• Scaling tier 2 remaps trace IDs. Traces in flight during a scale event are split once.
• The sampling policies themselves are in Cost Optimization.

Where durability and hardening go

Three different threats, three settings:

Our gateway has none of the three. kubectl -n monitoring get pods -l app.kubernetes.io/instance=alloy -o wide currently shows both replicas on the same node.

Autoscaling

Our gateway: HPA 2–6 replicas on CPU 70 % / memory 80 %.

• CPU and memory HPA is blind to a slow backend. When Tempo slows down, data waits in the queue (or on a PVC); the collector’s CPU does not rise. Queue depth is a good alert, a poor scaling signal — more replicas send more retries to the same slow backend.
• Cap maxReplicas. Unbounded scale-out during a backend outage becomes a retry storm.
• gRPC does not rebalance on scale-up (see above). A new replica may sit idle.
• Scale on memory before CPU for gateways — memory is what kills them (lesson 06).
• For a PVC-backed export tier, scale on kubelet_volume_stats_used_bytes through prometheus-adapter, with the HPA threshold below the alert threshold.

Sizing rules of thumb

Starting points from Grafana’s documentation; measure your own.

Our gateway’s load test results (log ingestion, before and after the DaemonSet/gateway split) are in Grafana Alloy.

Related lessons

• Collectors — roles and topology
• Reliability
• Grafana Alloy in our stack — clustering internals and load tests
• Prometheus federation