💪Exercise💪 — clustering split: who scrapes what
Goal
See target sharding work, then see the two ways it silently does not: the component block missing, and the process flag missing.
Prerequisites
- Your namespace and variables from First pipeline (
ME,NS, Grafana Helm repo). - About 60 pods in
monitoringexpose a port namedhttp-metrics(Loki, Mimir, Tempo components). They are the targets. Samples are discarded (forward_to = []), so nothing is written to the shared Prometheus.
💪Exercise💪 — steps
1. Two replicas, clustering on the process, not on the component
cat > ex-cluster.values.yaml <<'EOF'
alloy:
clustering:
enabled: true # process level: --cluster.enabled
configMap:
content: |
discovery.kubernetes "pods" {
role = "pod"
namespaces { names = ["monitoring"] }
}
discovery.relabel "metrics_ports" {
targets = discovery.kubernetes.pods.targets
rule {
source_labels = ["__meta_kubernetes_pod_container_port_name"]
regex = "http-metrics"
action = "keep"
}
}
prometheus.scrape "monitoring" {
targets = discovery.relabel.metrics_ports.output
forward_to = []
scrape_interval = "60s"
// clustering { enabled = true }
}
controller:
type: deployment
replicas: 2
EOF
helm upgrade --install ex-alloy-cluster-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-cluster.values.yaml
kubectl -n $NS rollout status deploy/ex-alloy-cluster-$ME
The release name contains alloy on purpose: the chart then uses it as-is for resource names. With a name like ex-cluster, everything is called ex-cluster-alloy.
2. Count targets per replica
The Alloy image has no curl, so read each pod’s /metrics through a port-forward:
cat > count-targets.sh <<'EOF'
for pod in $(kubectl -n $NS get pods -l app.kubernetes.io/instance=ex-alloy-cluster-$ME -o name); do
kubectl -n $NS port-forward $pod 22400:12345 >/dev/null 2>&1 & PF=$!
sleep 3
echo "$pod $(curl -s localhost:22400/metrics | grep '^prometheus_scrape_targets_gauge' | awk '{print $2}')"
kill $PF
done
EOF
bash count-targets.sh
Write down both numbers. Expected: the same number on both pods (for example 59 and 59). Every target is scraped twice.
3. Add the component block
sed -i 's#// clustering { enabled = true }#clustering { enabled = true }#' ex-cluster.values.yaml
helm upgrade ex-alloy-cluster-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-cluster.values.yaml
The config-reloader applies it in place, so wait about a minute, then run bash count-targets.sh again. Expected: two numbers that add up to the earlier total, not necessarily equal (for example 21 + 38 = 59). With a few dozen targets, hashing is lumpy.
Open the UI of one pod (kubectl -n $NS port-forward <pod> 12345:12345, then http://localhost:12345/clustering) and find both peers.
4. Scale out and watch targets move
kubectl -n $NS scale deploy/ex-alloy-cluster-$ME --replicas=3
kubectl -n $NS rollout status deploy/ex-alloy-cluster-$ME
sleep 30; bash count-targets.sh
Expected: three numbers, same total (for example 16 + 28 + 15). Compare with step 3: the new pod took targets from both old ones, and most targets stayed where they were.
Run helm upgrade once more with the same values file, then count pods. The replicas: 2 in the values file overrides your manual scale — kubectl scale is drift that the next deploy removes.
Success criteria
- You showed that without the component block, every replica scrapes everything.
- You showed that with it, the per-replica counts add up to the total.
- You can say roughly what share of targets moved when the third replica joined, and why it was not all of them.
⭐Stretch: the block without the process flag
Keep clustering { enabled = true } in the component but set alloy.clustering.enabled: false in the values, upgrade, and count again.
Expected: every replica reports the full count again. There is no error and no warning. Without --cluster.enabled, each process is a cluster of one and owns every target.
Clean up when done: helm -n $NS uninstall ex-alloy-cluster-$ME.