Instrumenting Applications

Instrumentation is the code and configuration that makes an application emit telemetry.

No instrumentation, no signals — backends like Prometheus, Loki, or Tempo only store what something already produced.

🎛️ Five ways to instrument

Instrumentation is a spectrum, ordered below from most manual to most automated.

The gradient is who does the work: at 1 a developer writes every span, at 5 nobody touches the application at all. Depth of coverage runs the other way.

These stack. A realistic service runs 2 or 3 + 1, sits inside a mesh (4), and is profiled by eBPF (5) at the same time.

What each one cannot do

🧪 What each one looks like

Minimal, runnable shapes for all five, each followed by how well it is supported per language — same language list as the maturity table in What is Observability. Java and .NET agent setup is in OTel Architecture, so the examples below use other stacks.

Legend: ✅ production-ready · ⚠️ works with caveats or still moving · ❌ not available. Support moves fast — verify against the OpenTelemetry registry before committing to a stack.

1. Manual SDK

Only you can say what a span means in business terms. Everything below is invisible to approaches 2, 3, 4, and 5.

Go — a domain span with attributes and a recorded error:

var tracer = otel.Tracer("checkout")

func Checkout(ctx context.Context, order Order) error {
    ctx, span := tracer.Start(ctx, "checkout.process")
    defer span.End()

    span.SetAttributes(
        attribute.String("tenant.tier", order.Tenant.Tier),      // yours, not conventional
        attribute.String("payment.provider", order.Provider),
        attribute.Int("cart.item_count", len(order.Items)),
    )

    if err := charge(ctx, order); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "charge failed")
        return err
    }
    return nil
}

C# — ActivitySource is the OTel tracer under a .NET name:

private static readonly ActivitySource Source = new("Checkout");

using var activity = Source.StartActivity("checkout.process");
activity?.SetTag("tenant.tier", order.Tenant.Tier);
activity?.SetTag("payment.provider", order.Provider);

A business metric — a counter no agent would ever invent (Python):

meter = metrics.get_meter("checkout")
orders = meter.create_counter("orders.completed", unit="{order}")

orders.add(1, {"payment.provider": provider, "tenant.tier": tier})   # low-cardinality labels only

Structured log correlated with the trace — emit trace/span IDs so Loki and Tempo line up:

span = trace.get_current_span().get_span_context()
logger.info("order completed", extra={
    "order_id": order.id,
    "trace_id": format(span.trace_id, "032x"),
    "span_id":  format(span.span_id, "016x"),
})

Language support — manual instrumentation is limited only by the API/SDK maturity of the language:

2. Instrumentation libraries

You choose each instrumentation and register it in bootstrap code. In Go this is the usual route: Go has no runtime agent, because there is no interpreter or VM to hook at startup.

Go — wrap the handler and the HTTP client:

import (
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
)

// inbound: every request handled here produces a server span
handler := otelhttp.NewHandler(mux, "checkout-api")
http.ListenAndServe(":8080", handler)

// outbound: propagates traceparent, produces a client span
client := http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
resp, err := client.Get("http://payment:8080/charge")

Python — the same idea, explicit instead of launcher-discovered:

from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.psycopg2 import Psycopg2Instrumentor

FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()
Psycopg2Instrumentor().instrument()     # every SQL query becomes a child span

Registering an instrumentation library and running the zero-code agent (3) produces every span twice. Pick one per library.

Language support — what matters here is the breadth of the contrib ecosystem:

3. Zero-code / agent

Nothing in the repository changes. You wrap the process and set environment variables.

Python — the opentelemetry-instrument launcher discovers installed libraries at startup:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install          # pulls instrumentation for what you already import

OTEL_SERVICE_NAME=checkout \
OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy-collector:4318 \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
opentelemetry-instrument python manage.py runserver --noreload   # the autoreloader forks and double-instruments

Node.js — a preloaded module, no require in your source:

npm install @opentelemetry/auto-instrumentations-node

OTEL_SERVICE_NAME=frontend \
OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy-collector:4318 \
node --require @opentelemetry/auto-instrumentations-node/register server.js

Kubernetes — the OpenTelemetry Operator injects the agent as an init container, so even the wrapper disappears from your image:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-python: "otel/default"   # or inject-java, inject-nodejs, inject-dotnet

Language support — this is the approach where support is simply absent for some languages. A runtime agent needs an interpreter or VM to hook into at startup; a compiled binary offers none, so the equivalent has to happen at build time:

The Operator also injects instrumentation into Apache HTTPD and Nginx, which have no OTel SDK of their own.

4. Proxy / service mesh

The sidecar already sees every request; you only tell it where to send traces.

Istio — tracing is a mesh-wide setting:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: otel
      randomSamplingPercentage: 10

ingress-nginx — one ConfigMap, and every request entering the cluster gets a root span:

data:
  enable-opentelemetry: "true"
  otlp-collector-host: "alloy-collector.monitoring.svc"
  otlp-collector-port: "4317"
  otel-sampler: "TraceIdRatioBased"     # without this the sampler stays AlwaysOn
  otel-sampler-ratio: "0.1"

otel-sampler defaults to AlwaysOn, and otel-sampler-ratio is ignored unless you switch it. Setting the ratio alone traces 100% of ingress traffic.

ingress-nginx reached end of life in March 2026. The snippet still describes a great many running clusters; for new ones the equivalent is a Gateway API controller (Envoy Gateway, Traefik, Cilium), which exports OTLP through the same mechanism.

Spans stop at the pod boundary: you see frontend → checkout took 400 ms, but not which SQL query caused it.

Language support — the proxy reads bytes on the wire, so span capture is identical in every language. What stays language-dependent is whether your application forwards the incoming traceparent on its outbound calls; without that, one trace becomes one disconnected span per service:

5. eBPF / OBI

Configuration lives next to the platform, not the application. The workshop cluster instruments the llm service this way, in alloy-collector.values.yaml:

beyla.ebpf "llm" {
  discovery {
    instrument {
      name       = "llm"
      open_ports = "8000"          // match by listening port
    }
  }

  ebpf {
    context_propagation = "disabled"   // "headers" nests spans, but rewrites proxied HTTP on the node
  }

  attributes {
    kubernetes { enable = "true" }     // decorate with pod, namespace, node
  }

  metrics {
    features = ["application", "application_span", "application_service_graph"]
  }

  output {
    traces = [otelcol.processor.batch.beyla.input]
  }
}

The llm service was never modified, never rebuilt, and never restarted — yet it reports RED metrics, spans, and a service-graph edge.

Language support — eBPF attaches to the kernel and to the process’s own symbols, so what matters is the runtime, not the SDK. Coverage is broad but shallow everywhere except Go:

• “Network level” means no in-process detail. You get POST /checkout 503 420ms, never the SQL query or the method that threw.
• Java and Node.js are special cases. The JVM terminates TLS inside itself (JSSE), so the OpenSSL uprobes that work everywhere else have nothing to attach to. OBI closes the gap by injecting a small agent at runtime — javaagent.enabled and nodejs.enabled, both on by default. The Java agent needs HotSpot or OpenJ9, Java 8+, and a writable container filesystem; a read-only root filesystem silently disables it.
• OBI does not profile. It emits spans and RED metrics only. The CPU-profile column above describes eBPF profilers in general — in this cluster that is pyroscope.ebpf, a separate component in the same Alloy DaemonSet.

⚖️ Coverage vs. control

The trade-off is the same in every language:

• Automatic approaches give breadth cheaply. One env var and 40 services report HTTP latency. Nobody had to open a pull request.
• Manual instrumentation gives depth expensively. order.value, tenant.id, payment.provider exist only because somebody wrote them.

Auto-instrumentation tells you that checkout is slow. Manual instrumentation tells you it is slow for premium customers paying with provider X.

Practical rollout order:

1. Turn on zero-code instrumentation everywhere. Accept whatever it gives you.
2. Fix naming (service.name, service.namespace, deployment.environment.name) before adding anything else.
3. Add manual spans and attributes only at points where an incident review showed you were blind.

Step 3 without step 2 produces data nobody can correlate.

🧭 Choosing an approach

Concrete setup — agent flags, SDK bootstrap code, and OBI — is in OTel Architecture.

Naming, context propagation, and the mistakes that make instrumentation unusable are the next chapter: Instrumentation Conventions.

Related lessons

• What is Observability
• Instrumentation Conventions
• Traces
• OTel Architecture