CI/CD Observability

• 🎯 Why pipelines need telemetry
• 📐 CI/CD and VCS semantic conventions
• 🧰 Three ways to get pipeline telemetry
• 🔗 Propagating context into build tools
• 🚀 Deployments as signals for runtime telemetry
• 📊 DORA metrics from telemetry
• 🚨 Common failure modes
• Related lessons

A pipeline is a distributed system: a workflow schedules jobs on runners, jobs call registries, clouds and clusters, and any step can be slow or flaky. Most teams only see a green or red icon and a log.

OpenTelemetry treats pipeline runs as traces and pipeline health as metrics, with dedicated cicd.* and vcs.* conventions.

🎯 Why pipelines need telemetry

📐 CI/CD and VCS semantic conventions

Status: Development. Attribute and metric names may still change.

Trace shape: pipeline run = root span, task (job, stage, step) = child span.

deployment.environment.name, deployment.id and deployment.status connect a pipeline run to what it deployed.

🧰 Three ways to get pipeline telemetry

Collector github receiver (contrib, alpha): a scraper for VCS metrics through the GitHub API, and a webhook endpoint that turns workflow_run / workflow_job events into traces.

receivers:
  github:
    collection_interval: 300s
    scrapers:
      scraper:
        github_org: ProtopiaTech
        search_query: "org:ProtopiaTech topic:training"
        auth:
          authenticator: bearertokenauth/github
    webhook:
      endpoint: 0.0.0.0:19418
      path: /events
      health_path: /health
      secret: ${env:GITHUB_WEBHOOK_SECRET}   # HMAC check of every delivery

extensions:
  bearertokenauth/github:
    token: ${env:GITHUB_PAT}

service:
  extensions: [bearertokenauth/github]
  pipelines:
    traces:  {receivers: [github], exporters: [otlp/tempo]}
    metrics: {receivers: [github], exporters: [prometheusremotewrite]}

Check the receiver README for your Collector version — the config layout of alpha components changes. The gitlab receiver follows the same pattern for GitLab pipeline webhooks.

The webhook endpoint must be reachable from the CI platform. Expose only that path, validate the secret, and run it on a separate Collector from your internal OTLP ingest.

otel-cli inside a job:

- name: Deploy cluster
  env:
    OTEL_EXPORTER_OTLP_ENDPOINT: https://otlp.example.com
    OTEL_SERVICE_NAME: ci-deploy
  run: |
    otel-cli exec --name "helm upgrade tempo" -- \
      helm upgrade --install tempo grafana/tempo -f helm_values/tempo/tempo.values.yaml

otel-cli exec creates a span around the command and passes TRACEPARENT to the child process.

🔗 Propagating context into build tools

Processes have no HTTP headers. The OTel spec defines environment variables as context carriers:

A parent process sets them; an instrumented child reads them and continues the trace. Tools that support it (OpenTelemetry Maven extension, pytest and Gradle plugins, otel-cli) produce one trace per run instead of disconnected spans per tool:

Deploy Kubernetes Cluster                 run      14m02s   result=success
├── Deploy to AKS                         job      13m40s
│   ├── Azure Login                       step        8s
│   ├── Install CLI tools                 step       41s
│   ├── Deploy cluster                    step    10m55s
│   │   ├── az deployment group create    otel-cli  6m10s
│   │   ├── helm upgrade tempo            otel-cli    52s
│   │   └── helm upgrade otel-demo        otel-cli  2m31s
│   ├── Configure Cloudflare DNS          step        6s
│   └── Create Grafana workshop users     step       12s

Job and step names are from this repo’s .github/workflows/deploy.yml; the otel-cli spans are illustrative. With only the webhook receiver, the tree stops at the step level.

🚀 Deployments as signals for runtime telemetry

A pipeline’s most useful output for operations is the fact that a deploy happened.

service.version must be low-cardinality: a release tag or short commit SHA, not a build timestamp.

📊 DORA metrics from telemetry

# successful production deploys per day
sum(increase(cicd_pipeline_run_duration_seconds_count{
  cicd_pipeline_name="Deploy Kubernetes Cluster", cicd_pipeline_result="success"}[1d]))

🚨 Common failure modes

Related lessons

• Instrumentation Conventions
• Traces
• Entities and the Resource Model
• Config Lifecycle and Upgrades