Config Lifecycle and Upgrades

• Three gates: fmt, validate, run
• Startup vs reload
• How the Helm chart ships config
• What a reload does not change
• Rolling out config safely
• Upgrading the binary
• Related lessons

A config change is a deployment. In the default Helm setup it is a deployment with no rollout strategy, no readiness gate, and a success message that means nothing.

All behaviour on this page was measured on Alloy v1.16.1 / chart 1.8.1, the versions the workshop cluster runs.

Three gates: fmt, validate, run

• validate must use production flags. Without --stability.level=experimental, a config using otelcol.exporter.debug fails validation; with a lower flag in CI than in production, CI rejects configs production accepts — and the reverse.
• A directory is validated as one merged config. Labels must be unique across files.
• A smoke start is cheap. A config that fails to build exits within seconds; one that builds keeps running:

# CI gate: exit 124 (killed by timeout) = config built and ran for 15 s
timeout 15 alloy run --stability.level=experimental --storage.path=/tmp/alloy config.alloy
test $? -eq 124

• Run the smoke start where the components can build. Outside a cluster, discovery.kubernetes fails with unable to load in-cluster configuration and exits 1 — a false failure. Run it as a pod in a test namespace, the way the exercises in this module use their own release.

Startup vs reload

The same broken config behaves differently depending on when it is loaded.

After both failed reloads, measured:

The config metrics describe parsing, not loading. They report success for a config that was rejected. Reliable signals: the HTTP code of /-/reload, the log line failed to reload config, and alloy_component_controller_running_components{health_type!="healthy"}.

The liveness trap. With a liveness probe on /-/healthy, an argument-level error on reload makes the probe fail with 500. Kubernetes restarts the container. The restart is a fresh startup with the same broken config, so it exits 1. Result: CrashLoopBackOff, while helm upgrade reported STATUS: deployed. Point liveness at /-/ready (process up), not /-/healthy (all components healthy).

How the Helm chart ships config

The grafana/alloy chart runs a config-reloader sidecar (quay.io/prometheus-operator/prometheus-config-reloader):

1. helm upgrade changes the ConfigMap. Pods are not restarted.
2. The kubelet syncs the mounted ConfigMap — up to about a minute.
3. The sidecar sees the file change and calls POST /-/reload.
4. On 400, it retries every 5 s and logs received non-200 response: 400 Bad Request; have you set --web.enable-lifecycle Prometheus flag? — a misleading hint inherited from Prometheus.

Consequences:

• helm upgrade success proves nothing about the config. Helm only checks that Kubernetes accepted the objects.
• Every pod reloads at the same time. No maxUnavailable, no readiness gate, no canary. A bad config hits the whole fleet at once.
• checksum/config is added to the pod template only when configReloader.enabled: false. With the reloader on (the default), a config change never triggers a rollout. GitOps tools see a synced ConfigMap and report healthy.
• Pods restart only when the pod template changes: image, extraArgs, extraEnv, resources, probes.
• kubectl scale is drift. The next helm upgrade resets controller.replicas.

Chart values nest under alloy:. In chart 1.8.1, livenessProbe is alloy.livenessProbe. A top-level livenessProbe: key is ignored without a warning. Both our values files had one on /-/healthy, so the running releases had no liveness probe at all. They now set alloy.livenessProbe on /-/ready. Always confirm with helm template or kubectl get pod -o yaml that a setting reached the pod.

💪Exercise💪 (self-guided): Reload trap — push two broken configs through helm upgrade and watch one do nothing and the other crash-loop.

What a reload does not change

For credentials that rotate, read them from a file or a Kubernetes Secret component, not from sys.env().

Rolling out config safely

• Gate in CI: fmt → validate with production flags → 15-second smoke run.
• Canary with a separate release. Deploy the new config to a second, small release (same chart, own ConfigMap), watch it, then promote.
• Or turn the reloader off (configReloader.enabled: false). The chart then adds checksum/config, so every config change becomes a normal rolling update with maxUnavailable and readiness checks.
• Alert on the reload outcome, not the parse gauge: log-based alert on failed to reload config, plus unhealthy components.
• Keep old component labels when refactoring. A renamed label orphans the component’s WAL or queue directory (lesson 06).

Upgrading the binary

• Cadence: Alloy ships a minor release about every three weeks. Each one bumps the embedded OpenTelemetry Collector and Prometheus.
• Read CHANGELOG.md, not only GitHub release notes. Breaking changes hide in the embedded components.
• Known traps:
  • v1.11 moved to Prometheus v3: le="1" became le="1.0" in histogram buckets, which breaks recording rules and dashboards that match on le.
  • Default batching and queue settings change between minors — compare sending_queue defaults before and after.
• Pin both chart and image. Our deploy script pins the chart (ALLOY_VERSION="${2:-1.8.1}" in deploy-alloy-module.sh); the image follows the chart’s appVersion. For a reproducible image, set image.tag to "v1.16.1@sha256:…".
• DaemonSets cannot be canaried with Argo Rollouts. Use updateStrategy.rollingUpdate.maxUnavailable and a node label to stage.
• Fleet Management, remotecfg and OpAMP manage config, not binaries. Binary upgrades still go through your deployment pipeline.

Related lessons

• First pipeline
• Reliability
• Self-monitoring
• Hardening