Hardening

A collector reads every signal your platform produces, and it often holds the credentials for every backend.

Most of its exposure comes from chart defaults nobody reads. Render the chart with helm template and read what you actually deploy.

Chart defaults are attack surface

Chart Default Risk Fix
grafana/alloy ClusterRole grants cluster-wide get/list/watch on secrets and configmaps (for remote.kubernetes.secret) A compromised collector pod reads every Secret in the cluster Custom ClusterRole without secrets; namespaced Role where the component needs one
grafana/alloy HTTP server on 0.0.0.0:12345 UI, /metrics, /-/reload and the config reachable from any pod NetworkPolicy; enableHttpServerPort: false only removes the container port, not the listener
open-telemetry/opentelemetry-collector Merges jaeger, zipkin and prometheus receivers into the config Extra unauthenticated ingest paths. In a measured case, zipkin accepted spans (HTTP 202) while the OTLP receiver returned 401 Remove them explicitly: config.receivers.jaeger: null, zipkin: null, prometheus: null, and drop them from pipelines

Disabling a port does not remove a receiver. Our helm_values/elasticsearch/otel-es-gateway.values.yaml used to set only ports.zipkin.enabled: false and ports.jaeger-*.enabled: false. Rendered with the chart, the config still had:

    traces:
      receivers:
      - otlp
      - jaeger
      - zipkin

The receivers still listened on the pod IP; only the Service port was gone. The file now sets jaeger: null, zipkin: null, prometheus: null under config.receivers, and traces: null, metrics: null, logs: null under service.pipelines. The rendered config passes otelcol validate on contrib 0.111.0.

Check ours:

kubectl get clusterrole -l app.kubernetes.io/name=alloy -o yaml | grep -B2 -A6 secrets
helm template g open-telemetry/opentelemetry-collector -f helm_values/elasticsearch/otel-es-gateway.values.yaml | grep -A6 "traces:"

The HTTP server

Alloy’s port 12345 serves:

Path Exposes Auth
/ (UI), /api/v0/web/components Full component graph, arguments, live debugging stream None
/metrics Self-monitoring None
POST /-/reload Re-reads config from disk None
/debug/pprof Go profiles None
  • Our gateway publishes the UI through ingress (alloy.workshop2.indexoutofrange.com, path /), so everything in the table above is reachable from the internet. Our ingress-nginx has snippet annotations disabled, so a path cannot simply be denied. A second Ingress on the same host (alloy-ui-deny in extraObjects) instead claims /-/reload (Exact) and /debug (Prefix), which beat / by path specificity. It sends them to the OTLP/HTTP port, which answers 404. The UI, /metrics and /api/v0/web/* stay public and read-only.
  • Secrets are redacted in the UI and support bundles, but not in Alloy’s own logs. A debug log or a debug exporter can print tokens and payloads.
  • Live debugging streams raw telemetry — it is off by default for that reason (livedebugging { enabled = true } turns it on; our gateway has it on).
  • Auth on /metrics breaks probes unless /-/ready and /-/healthy stay open.

Ingest endpoints

Endpoint Default auth Harden with
OTLP gRPC / HTTP None otelcol.auth.bearer (Alloy), bearertokenauth (Collector); mTLS; NetworkPolicy
Ingress in front of OTLP None nginx.ingress.kubernetes.io/whitelist-source-range, auth at the ingress

Our gateway exposes OTLP publicly without authentication on purpose — participants export from their laptops. The comment in alloy-gateway.values.yaml says so and shows the whitelist annotation to add. Anyone who knows the hostname can write into Prometheus, Mimir, Loki and Tempo.

Client side in Alloy: otelcol.auth.bearer as a client requires TLS. For plaintext inside the cluster, send the header with otelcol.auth.headers instead.

Secrets

Source Rotates without restart Notes
sys.env("TOKEN") No — read once at process start Simple; every rotation is a rollout
local.file with is_secret = true Yes — file watched Mount a Kubernetes Secret as a volume; kubelet updates the file
remote.kubernetes.secret Yes Needs RBAC on secrets — scope it to one namespace
remote.vault Yes Vault agent not needed

Collector equivalents: bearertokenauth with filename: (file is watched), and from contrib v0.122 a tokens: list on the server side.

Rotating credentials without losing data

UNAUTHENTICATED and HTTP 401 are not retried (lesson 06). A sender with the old token loses data the moment the receiver stops accepting it.

Approach How
Two valid tokens during the switch Server accepts old and new (tokens: list, or two receivers on two ports); roll senders; remove the old token
File-based tokens on both sides Update the Secret; the kubelet refreshes the file; both sides pick it up
Absent rather than rejecting If the server must restart with a new token, scale it to 0 first. Senders get UNAVAILABLE, which is retried and buffered, instead of UNAUTHENTICATED, which is dropped

Checklist

Item Our gateway Our DaemonSet
Non-root user ✅ UID 473 (set explicitly in values) ❌ root, privileged — required for eBPF
Minimal capabilities ✅ drop: [ALL] ❌ SYS_ADMIN, BPF, PERFMON, … — required for eBPF
ClusterRole without secrets ❌ chart default ❌ chart default
UI / reload not public ⚠️ UI public; /-/reload and /debug blocked at the ingress ✅ ClusterIP only
OTLP authenticated ❌ intentionally open n/a
NetworkPolicy on 12345 ❌ ❌
Image pinned by digest ❌ tag from chart appVersion ❌
Liveness probe not on /-/healthy ✅ alloy.livenessProbe on /-/ready ✅ alloy.livenessProbe on /-/ready

A workshop cluster makes different trade-offs from production. The point of the checklist is that each ❌ is a decision you should be able to name, not a default you never saw.

results matching ""

    No results matching ""