💪Exercise💪 — reload trap: two broken configs, two opposite outcomes

Goal

Push two broken configs through helm upgrade. One is silently ignored while the old config keeps running. The other crash-loops the pod. In both cases Helm reports deployed and the config metrics report success.

Prerequisites

  • The ex-alloy-$ME release and ex-alloy.values.yaml from First pipeline, working (fixed after its step 4).
  • ME and NS exported as there.

💪Exercise💪 — steps

1. Add a liveness probe on /-/healthy

This is the probe our gateway values file used to ask for, under a top-level key that the chart ignores. Here you add it where it takes effect, under alloy:.

grep -q livenessProbe ex-alloy.values.yaml || sed -i 's#^alloy:#alloy:\n  livenessProbe:\n    httpGet: { path: /-/healthy, port: 12345 }\n    periodSeconds: 10#' ex-alloy.values.yaml
head -5 ex-alloy.values.yaml
helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-alloy.values.yaml
kubectl -n $NS rollout status deploy/ex-alloy-$ME
kubectl -n $NS get pod -l app.kubernetes.io/instance=ex-alloy-$ME -o jsonpath='{.items[0].spec.containers[0].livenessProbe.httpGet.path}{"\n"}'

The last command must print /-/healthy. The probe changed the pod template, so this upgrade did roll the pod.

Make one copy of the values file per failure:

sed 's/otelcol.exporter.debug.default.input, /otelcol.exporter.debug.defualt.input, /' ex-alloy.values.yaml > bad-reference.values.yaml
sed 's/verbosity = "detailed"/verbosity = "loud"/' ex-alloy.values.yaml > bad-argument.values.yaml

2. Broken reference: nothing happens

helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f bad-reference.values.yaml
sleep 90
kubectl -n $NS get pods -l app.kubernetes.io/instance=ex-alloy-$ME
kubectl -n $NS logs deploy/ex-alloy-$ME -c alloy --since=2m | grep "failed to reload config" | tail -1

Then check what the metrics say:

Stop port-forwards left over from earlier exercises first (kill %1 %2 in the same shell, or close that terminal) — the local port 12345 is probably still taken.

kubectl -n $NS port-forward deploy/ex-alloy-$ME 12345:12345 &
sleep 3
curl -s localhost:12345/metrics | grep -E '^alloy_config_(last_load_successful|load_failures_total)'
curl -s -o /dev/null -w 'healthy=%{http_code}\n' localhost:12345/-/healthy

Expected: pod Running, zero restarts; alloy_config_last_load_successful 1; alloy_config_load_failures_total 0; healthy=200. Only the log says the reload failed. The old config is still running.

3. Broken argument: the pod crash-loops

helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f bad-argument.values.yaml
for i in $(seq 1 8); do sleep 20; kubectl -n $NS get pods -l app.kubernetes.io/instance=ex-alloy-$ME --no-headers; done
kubectl -n $NS get events --sort-by=.lastTimestamp | grep -E "Liveness|liveness" | tail -2

Expected sequence, about 1–2 minutes after the upgrade:

  1. The reload is applied partially; otelcol.exporter.debug.default turns Unhealthy.
  2. /-/healthy returns 500: Liveness probe failed: HTTP probe failed with statuscode: 500.
  3. Container alloy failed liveness probe, will be restarted.
  4. The restart is a fresh startup with the broken config → exit 1 → CrashLoopBackOff.

Picture this in the shared gateway: every replica reloads at the same moment, so every replica crash-loops together.

4. Recover

helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-alloy.values.yaml
sleep 90; kubectl -n $NS get pods -l app.kubernetes.io/instance=ex-alloy-$ME

The pod recovers on its next restart attempt once the good ConfigMap has synced. The restart counter stays as evidence.

Success criteria

  • You can name which of the two errors keeps the old config running, and which one applies a partial new graph.
  • You showed that alloy_config_last_load_successful stayed at 1 for a rejected config.
  • You can explain the chain from /-/healthy returning 500 to CrashLoopBackOff, and why helm upgrade could not notice it.

⭐Stretch: move the probe to /-/ready

Change the probe path to /-/ready in all three values files, apply the good one, then repeat step 3.

Expected: the pod stays Running with zero restarts. /-/healthy returns 500, /-/ready returns 200, and alloy_component_controller_running_components{health_type="unhealthy"} is 1. The problem is now visible in metrics instead of fatal — which is what you want from a probe.

Restore the good values file when done.

results matching ""

    No results matching ""