Entities and the Resource Model

Every span, metric and log carries a resource: a bag of attributes that says who produced it. Correlation across signals works only when every producer describes the same thing the same way.

The resource has a flaw: it does not say which attributes identify the producer and which only describe it. The OpenTelemetry entities model adds that distinction.

🧩 The resource today

A resource is a flat set of attributes, attached once per batch of telemetry:

service.name            = checkout
service.namespace       = opentelemetry-demo
service.version         = 2.1.3
service.instance.id     = 6f1c9a0e-…
deployment.environment.name = workshop
k8s.namespace.name      = otel
k8s.pod.name            = checkout-7d9f8c6b5-x2x4q
k8s.pod.uid             = 3b1e…
k8s.node.name           = aks-nodepool1-12345678-vmss000002
container.id            = 9c4e…
host.arch               = amd64
process.runtime.name    = go
telemetry.sdk.language  = go

The basics are in Instrumentation Conventions. What this flat list hides:

Problem Example
Several things in one bag Service, pod, container, node and process mixed together, with no boundaries
Identity not marked Is this producer identified by service.instance.id, k8s.pod.uid or container.id? Backends guess
Descriptive attributes change A pod label edited, a node relabelled: the resource changes, and label-based backends start a new series
No lifecycle A pod deleted an hour ago still “exists” until its series go stale

🪪 Entities: identity vs description

Status: Development. The data model is specified; SDK, Collector and backend support is partial.

An entity is one thing that produces or is described by telemetry, with a type and two attribute sets:

Entity type Identifying attributes Descriptive attributes
service service.name, service.namespace service.version
service.instance service.instance.id  
k8s.pod k8s.pod.uid k8s.pod.name, k8s.pod.label.*, k8s.pod.annotation.*
k8s.node k8s.node.uid k8s.node.name, k8s.node.label.*
container container.id container.name, container.image.name
host host.id host.name, host.arch

Rules:

  • Identifying attributes never change during the entity’s life. A new value means a new entity.
  • Descriptive attributes may change without creating a new entity. Backends should not turn them into series identity.
  • A resource is a set of entities. The checkout resource above is service + service.instance + k8s.pod + container + k8s.node.
  • Entities have relationships (container runs in pod, pod scheduled on node) and lifecycle events. The Collector’s k8s_cluster receiver can emit entity state and delete events as log records, so a backend knows when a pod is gone.

What this enables: a backend can keep one series per identity while showing the latest descriptive attributes, build a topology graph from relationships, and correlate signals from different producers that report the same entity.

🧭 Who sets which attributes

Layer Sets How
Application / build service.name, service.namespace, service.version, deployment.environment.name OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, build-time injection
SDK resource detectors process.*, host.*, os.*, container.id, telemetry.sdk.* Enabled by default or with OTEL_*_RESOURCE_PROVIDERS / detector packages
Kubernetes / Operator k8s.pod.name, k8s.namespace.name, service.instance.id Downward API env vars injected by the Operator
Collector k8s.* enrichment, cloud.*, host.* k8sattributes, resourcedetection, resource processors
  • service.instance.id must be unique and stable per instance. The OTel Operator derives it from namespace, pod and container names. A random UUID per process start is valid but breaks continuity across restarts.
  • Enrich in one place. If both the SDK and the Collector set k8s.pod.name, decide which wins (k8sattributes does not overwrite existing values by default).
  • The workshop gateway enriches logs with otelcol.processor.k8sattributes and fixes k8s.namespace.name for demo traces with an otelcol.processor.transform rule in alloy-gateway.values.yaml.

📦 How backends map the resource

None of the Grafana backends store the resource as-is. Each flattens it differently:

Backend Mapping Control
Prometheus / Mimir service.namespace/service.name → job, service.instance.id → instance; all other resource attributes go into one target_info series per producer Prometheus otlp.promote_resource_attributes; Alloy otelcol.exporter.prometheus include_target_info, resource_to_telemetry_conversion
Loki (OTLP ingest) A fixed list (service.name, k8s.namespace.name, k8s.pod.name, …) → index labels; the rest → structured metadata limits_config.otlp_config.resource_attributes
Tempo Stored as-is; queried with the resource. scope { resource.k8s.pod.name = "checkout-7d9f8c6b5-x2x4q" }
Pyroscope service_name label plus selected attributes as labels Collector / Alloy relabeling

Joining target_info in PromQL — add pod name to a metric without making it a label on every series:

sum by (job, instance, k8s_pod_name) (
  rate(http_server_request_duration_seconds_count[5m])
  * on (job, instance) group_left (k8s_pod_name) target_info
)

Prometheus 3 has an experimental info() function for the same join (--enable-feature=promql-experimental-functions):

info(rate(http_server_request_duration_seconds_count[5m]), {k8s_pod_name=~".+"})

Promote only identifying or low-churn attributes to labels. k8s.pod.name changes on every rollout; k8s.pod.label.* changes whenever someone edits a label. resource_to_telemetry_conversion = true promotes all of them and multiplies series.

🔀 One entity, two identities

Correlation breaks when two producers describe the same entity differently.

Example from the workshop stack: Pyroscope showed both checkout and otel.checkout for one pod. The Pyroscope SDK in the image reports PYROSCOPE_APPLICATION_NAME=otel.checkout; the eBPF profiler in the node DaemonSet reports the Kubernetes workload name. Same process, two names, two sets of profiles. The fix was a drop rule for SDK-instrumented pods in the eBPF relabeling (discovery.relabel "ebpf_pods" in alloy-collector.values.yaml). The otel. prefix itself stays: the Tempo → Pyroscope link in Grafana builds service_name="otel.${service.name}", so renaming one side breaks the correlation.

Producer pair Typical mismatch
SDK vs eBPF (Beyla / OBI, Pyroscope eBPF) service.name from env var vs from the workload name
SDK vs Prometheus scrape job / instance from OTLP vs from service discovery (namespace/pod:port)
App logs via OTLP vs pod logs from files service.name vs app / container labels
Two Collector tiers Both enrich, one overwrites with a different value

Pick one naming source per attribute (usually the workload: Deployment name → service.name) and make every producer use it.

🚨 Common failure modes

Symptom Cause
unknown_service:java in every backend service.name not set
Out-of-order or duplicate samples in Mimir Two instances share a service.instance.id
Series count doubles after each rollout A descriptive, high-churn attribute (k8s.pod.name, pod labels) promoted to a metric label
Metric → logs link in Grafana finds nothing Metric labels and Loki labels were built from different attributes
Same service twice in Pyroscope or the service graph Two producers with two identities for one entity
Deleted pods linger in topology views No lifecycle signal; the backend waits for staleness

results matching ""

    No results matching ""