💪Exercise💪 — reload trap: two broken configs, two opposite outcomes
AI-agent variant: Reload trap with an AI agent. Do one of the two.
Goal
Push two broken configs through helm upgrade. One is silently ignored while the old config keeps running. The other crash-loops the pod. In both cases Helm reports deployed and the config metrics report success.
Prerequisites
- The
ex-alloy-$MErelease andex-alloy.values.yamlfrom First pipeline, working (fixed after its step 4). MEandNSexported as there.
💪Exercise💪 — steps
1. Add a liveness probe on /-/healthy
This is the probe our gateway values file used to ask for, under a top-level key that the chart ignores. Here you add it where it takes effect, under alloy:.
grep -q livenessProbe ex-alloy.values.yaml || sed -i 's#^alloy:#alloy:\n livenessProbe:\n httpGet: { path: /-/healthy, port: 12345 }\n periodSeconds: 10#' ex-alloy.values.yaml
head -5 ex-alloy.values.yaml
helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-alloy.values.yaml
kubectl -n $NS rollout status deploy/ex-alloy-$ME
kubectl -n $NS get pod -l app.kubernetes.io/instance=ex-alloy-$ME -o jsonpath='{.items[0].spec.containers[0].livenessProbe.httpGet.path}{"\n"}'
The last command must print /-/healthy. The probe changed the pod template, so this upgrade did roll the pod.
Make one copy of the values file per failure:
sed 's/otelcol.exporter.debug.default.input, /otelcol.exporter.debug.defualt.input, /' ex-alloy.values.yaml > bad-reference.values.yaml
sed 's/verbosity = "detailed"/verbosity = "loud"/' ex-alloy.values.yaml > bad-argument.values.yaml
2. Broken reference: nothing happens
helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f bad-reference.values.yaml
sleep 90
kubectl -n $NS get pods -l app.kubernetes.io/instance=ex-alloy-$ME
kubectl -n $NS logs deploy/ex-alloy-$ME -c alloy --since=2m | grep "failed to reload config" | tail -1
Then check what the metrics say:
Stop port-forwards left over from earlier exercises first (kill %1 %2 in the same shell, or close that terminal) — the local port 12345 is probably still taken.
kubectl -n $NS port-forward deploy/ex-alloy-$ME 12345:12345 &
sleep 3
curl -s localhost:12345/metrics | grep -E '^alloy_config_(last_load_successful|load_failures_total)'
curl -s -o /dev/null -w 'healthy=%{http_code}\n' localhost:12345/-/healthy
Expected: pod Running, zero restarts; alloy_config_last_load_successful 1; alloy_config_load_failures_total 0; healthy=200. Only the log says the reload failed. The old config is still running.
3. Broken argument: the pod crash-loops
helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f bad-argument.values.yaml
for i in $(seq 1 8); do sleep 20; kubectl -n $NS get pods -l app.kubernetes.io/instance=ex-alloy-$ME --no-headers; done
kubectl -n $NS get events --sort-by=.lastTimestamp | grep -E "Liveness|liveness" | tail -2
Expected sequence, about 1–2 minutes after the upgrade:
- The reload is applied partially;
otelcol.exporter.debug.defaultturnsUnhealthy. /-/healthyreturns500:Liveness probe failed: HTTP probe failed with statuscode: 500.Container alloy failed liveness probe, will be restarted.- The restart is a fresh startup with the broken config → exit 1 →
CrashLoopBackOff.
Picture this in the shared gateway: every replica reloads at the same moment, so every replica crash-loops together.
4. Recover
helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-alloy.values.yaml
sleep 90; kubectl -n $NS get pods -l app.kubernetes.io/instance=ex-alloy-$ME
The pod recovers on its next restart attempt once the good ConfigMap has synced. The restart counter stays as evidence.
Success criteria
- You can name which of the two errors keeps the old config running, and which one applies a partial new graph.
- You showed that
alloy_config_last_load_successfulstayed at1for a rejected config. - You can explain the chain from
/-/healthyreturning 500 toCrashLoopBackOff, and whyhelm upgradecould not notice it.
⭐Stretch: move the probe to /-/ready
Change the probe path to /-/ready in all three values files, apply the good one, then repeat step 3.
Expected: the pod stays Running with zero restarts. /-/healthy returns 500, /-/ready returns 200, and alloy_component_controller_running_components{health_type="unhealthy"} is 1. The problem is now visible in metrics instead of fatal — which is what you want from a probe.
Restore the good values file when done.