🎯Exercises

1. Basic Queries

1.1 Basic Syntax

Base syntax:

• {} - returns all traces.
• {resource.service.name = "checkout"} - filtering by resource attribute
• {duration > 10ms} - filters spans lasting longer than 10ms. Note: { trace:duration > 10ms } filters by the duration of the entire trace, not a single span. Full list is here
• { duration > 10ms && .http.method = "GET" && .http.status_code = 200 } - conditions can also be combined. Where attributes starting with . are implicit references to both span and resource.

🎯 What does this query do:

{ resource.service.name = "cart" && (duration > 10ms || .http.status_code >= 500 ) }

⚠️ Tips and tricks:

• Analyze the results and what they have in common.
• Think about how Tempo is designed and how search works.

1.2 Data Types

They are compatible with OpenTelemetry types, meaning we have:

• String: - text, case-sensitive.
• Numeric: - supports basic comparisons: =, !=, >, >=, etc.
• Duration: - supports convenient notation like: “10ms” or “15.5s,” and comparisons: =, <, <=, etc.
• Bool: - supports comparisons like true and false
• Status: - allows comparison to built-in keywords: ok, error, and unset

❗Note❗

Since Tempo has no data schema and has no type casting:

• { .userID = 1000 } - will return results only for ints
• { .userID = "1000" } - will return results only for strings

1.3 Operators

Available:

• = (equality)
• != (inequality)
• > (greater than)
• >= (greater than or equal to)
• < (less than)
• <= (less than or equal to)
• =~ (regular expression)
• !~ (negated regular expression)

1.4 Scopes

In TraceQL we have two important objects:

• resource - describes the entity producing telemetry (service, host, process, container), e.g., resource.service.name
• span - properties of the operation itself.

Searching by them can be done as follows:

• span.xxx.yyy - specific span
• resource.xxx.yyy - resource properties
• .xxx.yyy - searches everything (span and resource)

Knowing that Tempo works on columns, which operation will be more efficient:

• { resource.service.name = "cart" && span.url.path = "/oteldemo.CartService/GetCart" }
• { .service.name = "cart" && .url.path = "/oteldemo.CartService/GetCart" } ?

1.5 Multi-Span Queries

It is possible to define the following relationships:

• { resource.service.name = "cart" } && { resource.service.name="frontend" } - finds traces that contain spans from cart and frontend. Without specifying the relationship between them.

• { resource.service.name = "cart" } || { resource.service.name="frontend" } - finds traces that contain spans from cart or frontend. Without specifying the relationship between them.

Structural Operators

Structural operators allow filtering by parent-child relationships in the trace tree. They always return spans from the right-hand side of the operator (RHS).

Examples:

• { resource.service.name="frontend" } > { resource.service.name = "checkout" } - checkout is a direct child of frontend.

• { resource.service.name="frontend" } >> { resource.service.name = "checkout" } - checkout is a descendant of frontend, but not necessarily a direct child.

• { resource.service.name="checkout" } ~ { resource.service.name="payment" } - checkout and payment have the same direct parent.

• { resource.service.name="checkout" } < { resource.service.name="frontend" } - frontend is a direct parent of checkout.

Union Operators (structural with union)

Work the same as regular structural operators, but return spans from both sides of the operator (LHS and RHS). Useful when you want to see both parent and child in results.

Example:

• { span.http.url = "/api/checkout" && status = error } &>> { status = error } - finds both the span with endpoint /api/checkout and all its descendant spans with errors.

Negations (experimental)

Invert the structural condition. Marked as experimental — may return false positives.

Example:

• { } !< { resource.service.name = "foo" } - finds leaf spans (without children) in service foo.

1.6 Pipelines

🎯 What does this query do?

{ resource.service.name = "frontend-proxy" } | count() > 2 | avg(duration) > 10ms

⚠️ Tips and tricks:

• Run the query.
• Remember how Tempo works and how it analyzes search.
• Remove and/or modify the query and analyze what changed.

Tempo has various aggregate functions

2. Exercises

2.1 Find traces that contained an HTTP error

🎯 Goal: Find traces that anywhere contain a span with an HTTP request that ended with a status code in the 500 family.

2.2 Spans with database queries

🎯 Goal: Find spans that contain PostgreSQL database query content.

⚠️ Tips and tricks:

• Query content is in span attributes. Attribute db.xxxx

2.3 More than one database query

🎯 Goal: Find traces that contain more than one call to the Redis database.

2.4 Filtering by duration and status

🎯 Goal: Find traces from the payment service that lasted longer than 0.1ms and ended with an error.

⚠️ Tips and tricks:

• Use the intrinsic field status with values: ok, error, unset
• Combine the duration condition with the status condition
• The payment service has the paymentFailure feature flag enabled (~10% of transactions end with an error)

2.5 Parent-child relationship (>>)

🎯 Goal: Find traces where frontend calls (directly or indirectly) the payment service and that payment call ends with an error.

⚠️ Tips and tricks:

• The >> operator means a parent-child relationship (not necessarily direct)
• First spanset is the parent, second is the child
• Place the status condition in the child spanset

2.6 Sibling relationship (~)

🎯 Goal: Find traces where the checkout service within a single operation calls both payment and email (as siblings — both calls are children of the same PlaceOrder span).

⚠️ Tips and tricks:

• The ~ operator means both spans have the same parent
• Think about the trace topology: checkout/PlaceOrder is the parent, and its children include calls to payment and email
• Both spans in the query should be from the checkout service — because checkout creates the spans for calls to payment and email

2.7 Pipeline — average duration and grouping

🎯 Goal: Find services where the average span duration exceeds 20ms. Group results by service name.

⚠️ Tips and tricks:

• Use avg(duration) in the pipeline
• For grouping use by(resource.service.name)
• Syntax: { CONDITIONS } | AGGREGATION | by(ATTRIBUTE)
• Check the aggregator documentation

2.8 select() — displaying selected attributes

🎯 Goal: Find traces from the product-catalog service that contain an error. Display the service name, status, and span duration in the results.

⚠️ Tips and tricks:

• select() allows choosing which attributes are visible in results
• Syntax: { CONDITIONS } | select(ATTRIBUTE1, ATTRIBUTE2, ...)
• You can use both intrinsic fields (duration, status) and attributes (resource.service.name)
• The product-catalog service has the productCatalogFailure feature flag which generates errors