Instrumentation Conventions
Instrumentation that emits data is not the same as instrumentation you can query.
Four things decide whether telemetry is usable: who produced it, what the fields are called, whether it stays connected across hops, and which mistakes quietly ruin it.
🏷️ Resource attributes — the part people skip
Every signal an application emits carries a resource: attributes describing who produced it. This is what lets Grafana jump from a metric to the logs of the same pod.
| Attribute | Why it matters |
|---|---|
service.name |
Required. Missing it means telemetry lands under unknown_service and is effectively lost |
service.namespace |
Separates two services that share a name across teams |
service.version |
Lets you attribute a latency regression to a deploy |
deployment.environment.name |
Keeps staging noise out of production dashboards |
host.name, k8s.pod.name |
Usually filled in by the Collector or a resource detector, not by you |
Set them once, via OTEL_RESOURCE_ATTRIBUTES or the SDK, and every signal inherits them.
export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES=service.namespace=shop,service.version=1.24.3,deployment.environment.name=prod
📐 Semantic conventions
OpenTelemetry defines standard names for common attributes — http.request.method, db.system.name, messaging.destination.name, error.type. Auto-instrumentation emits them by default.
Why you should follow them in manual code too:
- Dashboards and alerts written against conventional names work across every service, in every language, without per-service tweaking.
- Backend features (Tempo’s service graph, Grafana’s Explore views, RED metrics generated by
spanmetrics) key off these exact names. - Custom names like
httpMethodorstatusCodesilently exclude your service from all of the above.
Reserve custom attribute names for things that are genuinely yours: order.id, cart.item_count, tenant.tier.
Conventions evolve. Some attributes were renamed (
http.method→http.request.method); SDKs ship both for a migration window. Pin which convention version your dashboards assume.
🔗 Context propagation
A trace only holds together if every hop passes the traceparent header along. Automatic instrumentation handles this inside supported HTTP and gRPC clients.
Where it breaks — and stays your job:
| Boundary | What happens | What to do |
|---|---|---|
| Message queues | Producer and consumer become separate traces | Inject/extract context into message headers |
| Background jobs, cron | Work runs with no parent context | Start a new root span, add a span link to the trigger |
Thread pools, async handoffs |
Context is lost on the thread switch | Use the SDK’s context-aware executor/wrapper |
| Outbound calls via raw sockets | No HTTP client to hook | Propagate manually, or accept the break |
Details on context and span links are in Traces.
🚨 Common failure modes
| Symptom | Cause | Fix |
|---|---|---|
unknown_service floods the backend |
service.name never set |
Set OTEL_SERVICE_NAME in the deployment |
| Every span appears twice | Agent and SDK instrumentation both registered | Pick one; disable the overlapping instrumentation |
| Traces stop at the queue | No context in message headers | Inject/extract manually |
| Metric cardinality explodes | User ID, request ID, or URL with IDs used as an attribute | Move high-cardinality values to span/log attributes, never metric labels |
| PII in telemetry | Auto-instrumentation captured headers or query strings | Redact in the SDK or with a Collector attributes processor |
| Noticeable latency added | Everything sampled at 100%, exporter blocking | Use batch export; sample head or tail |
Cardinality and its cost are covered in Metrics and Cost Optimization.