💪Exercise💪 — first pipeline: OTLP in, debug and Prometheus out, then break it

You run your own Alloy release in your own namespace. The shared alloy and alloy-collector releases in monitoring are read-only for this module — breaking them breaks the cluster for everyone.

Goal

Build a pipeline that receives OTLP, prints what it gets and converts metrics into Prometheus samples. Then break one reference and see how Alloy reacts at startup.

Prerequisites

• kubectl access to the workshop cluster, helm 3.x, curl, a bash shell (Git Bash or WSL on Windows).
• Grafana on the workshop cluster, Prometheus data source.
• Pick a short, unique suffix (your initials). Every command below uses it:

export ME=ab                              # your initials
export NS=collectors-$ME
kubectl create namespace $NS --dry-run=client -o yaml | kubectl apply -f -
helm repo add grafana https://grafana.github.io/helm-charts && helm repo update

💪Exercise💪 — steps

1. Write the values file

Chart 1.8.1 = Alloy v1.16.1, the version the shared releases run.

Every release name ends with -$ME. The chart creates a cluster-scoped ClusterRole and ClusterRoleBinding named after the release, so two participants installing ex-alloy in different namespaces would collide, and the second helm install would fail.

cat > ex-alloy.values.yaml <<'EOF'
alloy:
  stabilityLevel: experimental          # otelcol.exporter.debug is experimental
  extraPorts:
    - { name: otlp-http, port: 4318, targetPort: 4318, protocol: TCP }
  configMap:
    content: |
      otelcol.receiver.otlp "default" {
        http { endpoint = "0.0.0.0:4318" }
        output {
          logs    = [otelcol.processor.batch.default.input]
          metrics = [otelcol.processor.batch.default.input]
        }
      }

      otelcol.processor.batch "default" {
        timeout = "2s"
        output {
          logs    = [otelcol.exporter.debug.default.input]
          metrics = [otelcol.exporter.debug.default.input, otelcol.exporter.prometheus.default.input]
        }
      }

      otelcol.exporter.debug "default" {
        verbosity = "detailed"
      }

      otelcol.exporter.prometheus "default" {
        forward_to = [prometheus.remote_write.shared.receiver]
      }

      prometheus.remote_write "shared" {
        endpoint {
          url = "http://prometheus-and-grafana-kub-prometheus.monitoring.svc.cluster.local:9090/api/v1/write"
        }
      }
controller:
  type: deployment
  replicas: 1
EOF

helm upgrade --install ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-alloy.values.yaml
kubectl -n $NS rollout status deploy/ex-alloy-$ME

2. Send a log and a metric

kubectl -n $NS port-forward svc/ex-alloy-$ME 4318:4318 12345:12345 &

NOW=$(date +%s)000000000
curl -s -X POST localhost:4318/v1/logs -H 'Content-Type: application/json' -d '{
  "resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"ex-'$ME'"}}]},
  "scopeLogs":[{"logRecords":[{"timeUnixNano":"'$NOW'","severityText":"INFO","body":{"stringValue":"hello from '$ME'"}}]}]}]}'
echo
curl -s -X POST localhost:4318/v1/metrics -H 'Content-Type: application/json' -d '{
  "resourceMetrics":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"ex-'$ME'"}}]},
  "scopeMetrics":[{"metrics":[{"name":"collectors_exercise_ping","gauge":{"dataPoints":[{"asDouble":1,"timeUnixNano":"'$NOW'"}]}}]}]}]}'
echo

Both calls answer {"partialSuccess":{}}. An empty partialSuccess means everything was accepted. A rejection shows up as a non-empty rejectedLogRecords / rejectedDataPoints field — with HTTP 200 either way.

3. Read what arrived

kubectl -n $NS logs deploy/ex-alloy-$ME -c alloy | grep -A3 "hello from"

Then in Grafana → Explore → Prometheus:

collectors_exercise_ping{job="ex-ab"}

(service.name became the job label — use your own suffix.)

Open the Alloy UI at http://localhost:12345 and find the graph: five components, all green.

4. Break a reference

Change one reference to a label that does not exist and roll it out:

sed -i 's/otelcol.exporter.debug.default.input, /otelcol.exporter.debug.defualt.input, /' ex-alloy.values.yaml
helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-alloy.values.yaml
kubectl -n $NS get pods

Note what helm upgrade printed (STATUS: deployed) and what the pod did: nothing — same pod, zero restarts. Wait a minute for the ConfigMap to propagate, then look at the reloader sidecar:

kubectl -n $NS logs deploy/ex-alloy-$ME -c config-reloader --tail=3

It retries every 5 seconds with received non-200 response: 400 Bad Request; have you set --web.enable-lifecycle Prometheus flag? — a misleading hint, since the sidecar is borrowed from the Prometheus Operator. The running pod keeps the old config. Now force a fresh start:

kubectl -n $NS delete pod -l app.kubernetes.io/instance=ex-alloy-$ME
sleep 20; kubectl -n $NS get pods
kubectl -n $NS logs deploy/ex-alloy-$ME -c alloy | grep -i error

Expected: the pod goes into Error / CrashLoopBackOff with

Error: /etc/alloy/config.alloy:13:16: component "otelcol.exporter.debug.defualt.input" does not exist or is out of scope
Error: could not perform the initial load successfully

A bad reference at startup is fatal — no UI, no “Unhealthy” component to look at. Why the running pod ignored the same config in the previous step is the subject of Config lifecycle.

5. Fix and clean up (or keep it for later exercises)

sed -i 's/defualt/default/' ex-alloy.values.yaml
helm upgrade ex-alloy-$ME grafana/alloy --version 1.8.1 -n $NS -f ex-alloy.values.yaml
kubectl -n $NS delete pod -l app.kubernetes.io/instance=ex-alloy-$ME

Deleting the pod kills your port-forward — start it again after every pod restart. Later exercises reuse this release. When you are done with the module: kubectl delete namespace $NS.

Success criteria

• You saw your log record in the debug exporter output and collectors_exercise_ping in Prometheus.
• You can explain why {"partialSuccess":{}} is a success.
• You saw a startup with a broken reference fail, and noticed that the helm upgrade itself reported success.

⭐Stretch: healthy is not delivering

Point prometheus.remote_write "shared" at a URL that does not exist (http://nowhere.monitoring.svc.cluster.local:9090/api/v1/write), upgrade, delete the pod, and send the metric again.

• The UI shows prometheus.remote_write.shared as Healthy.
• curl -s -o /dev/null -w '%{http_code}\n' localhost:12345/-/healthy returns 200.
• The only evidence is in metrics: curl -s localhost:12345/metrics | grep -E 'prometheus_remote_storage_samples_(pending|retried_total|failed_total)' — pending stays above 0 and retried_total climbs, while failed_total stays at 0. A retryable failure is not a failure until the data is finally given up on.

Put the original URL back before the next exercise, which reuses ex-alloy.values.yaml.

Related lessons

• Pipeline model
• Config lifecycle
• Self-monitoring