💪Exercise💪 — reload trap: two broken configs, two opposite outcomes

AI-agent variant: Reload trap with an AI agent. Do one of the two.

Goal

Push two broken configs through helm upgrade. One is silently ignored while the old config keeps running. The other crash-loops the pod. In both cases Helm reports deployed and the config metrics report success.

Prerequisites

  • The ex-alloy-$ME release and ex-alloy.values.yaml from First pipeline, working (fixed after its step 4).
  • ME and NS exported as there.

💪Exercise💪 — steps

1. Add a liveness probe on /-/healthy

This is the probe our gateway values file used to ask for, under a top-level key that the chart ignores. Here you add it where it takes effect, under alloy:.

grep -q livenessProbe ex-alloy.values.yaml || sed -i 's#^alloy:#alloy:\n  livenessProbe:\n    httpGet: { path: /-/healthy, port: 12345 }\n    periodSeconds: 10#' ex-alloy.values.yaml
head -5 ex-alloy.values.yaml
helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-alloy.values.yaml
kubectl -n $NS rollout status deploy/ex-alloy-$ME
kubectl -n $NS get pod -l app.kubernetes.io/instance=ex-alloy-$ME -o jsonpath='{.items[0].spec.containers[0].livenessProbe.httpGet.path}{"\n"}'

The last command must print /-/healthy. The probe changed the pod template, so this upgrade did roll the pod.

Make one copy of the values file per failure:

sed 's/otelcol.exporter.debug.default.input, /otelcol.exporter.debug.defualt.input, /' ex-alloy.values.yaml > bad-reference.values.yaml
sed 's/verbosity = "detailed"/verbosity = "loud"/' ex-alloy.values.yaml > bad-argument.values.yaml

2. Broken reference: nothing happens

helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f bad-reference.values.yaml
sleep 90
kubectl -n $NS get pods -l app.kubernetes.io/instance=ex-alloy-$ME
kubectl -n $NS logs deploy/ex-alloy-$ME -c alloy --since=2m | grep "failed to reload config" | tail -1

Then check what the metrics say:

Stop port-forwards left over from earlier exercises first (kill %1 %2 in the same shell, or close that terminal) — the local port 12345 is probably still taken.

kubectl -n $NS port-forward deploy/ex-alloy-$ME 12345:12345 &
sleep 3
curl -s localhost:12345/metrics | grep -E '^alloy_config_(last_load_successful|load_failures_total)'
curl -s -o /dev/null -w 'healthy=%{http_code}\n' localhost:12345/-/healthy

Expected: pod Running, zero restarts; alloy_config_last_load_successful 1; alloy_config_load_failures_total 0; healthy=200. Only the log says the reload failed. The old config is still running.

3. Broken argument: the pod crash-loops

helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f bad-argument.values.yaml
for i in $(seq 1 8); do sleep 20; kubectl -n $NS get pods -l app.kubernetes.io/instance=ex-alloy-$ME --no-headers; done
kubectl -n $NS get events --sort-by=.lastTimestamp | grep -E "Liveness|liveness" | tail -2

Expected sequence, about 1–2 minutes after the upgrade:

  1. The reload is applied partially; otelcol.exporter.debug.default turns Unhealthy.
  2. /-/healthy returns 500: Liveness probe failed: HTTP probe failed with statuscode: 500.
  3. Container alloy failed liveness probe, will be restarted.
  4. The restart is a fresh startup with the broken config → exit 1 → CrashLoopBackOff.

Picture this in the shared gateway: every replica reloads at the same moment, so every replica crash-loops together.

4. Recover

helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-alloy.values.yaml
sleep 90; kubectl -n $NS get pods -l app.kubernetes.io/instance=ex-alloy-$ME

The pod recovers on its next restart attempt once the good ConfigMap has synced. The restart counter stays as evidence.

Success criteria

  • You can name which of the two errors keeps the old config running, and which one applies a partial new graph.
  • You showed that alloy_config_last_load_successful stayed at 1 for a rejected config.
  • You can explain the chain from /-/healthy returning 500 to CrashLoopBackOff, and why helm upgrade could not notice it.

⭐Stretch: move the probe to /-/ready

Change the probe path to /-/ready in all three values files, apply the good one, then repeat step 3.

Expected: the pod stays Running with zero restarts. /-/healthy returns 500, /-/ready returns 200, and alloy_component_controller_running_components{health_type="unhealthy"} is 1. The problem is now visible in metrics instead of fatal — which is what you want from a probe.

Restore the good values file when done.

results matching ""

    No results matching ""