Cost Optimization

💰 The Problem

Observability at production scale generates massive amounts of data. Without conscious cost management, the bill for storage and compute can quickly spiral out of control.

Where do costs come from?

Note: Without sampling, traces generate more data than logs — one HTTP request is one log line, but many spans (each hop between services, database query, external API call). That’s why trace sampling is crucial for cost control.

Traces — Most Expensive Signal Without Sampling

Problem: Without sampling, traces account for the largest data volume. Each request in a microservices architecture generates 5-20+ spans, and each span is “heavier” than a typical log line (attributes, events, timestamps, links).

Reduction strategies:

• Tail-based sampling — keep 100% of errors and slow requests, 1-5% of normal traffic
• Head-based sampling — simpler, but discards errors before seeing them
• Adaptive sampling — dynamically adjust percentage based on load
• Span filtering — discard spans from health checks and readiness probes

# OTel Collector: drop health check spans
processors:
  filter/health:
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'

Logs — Second Largest by Volume

Problem: Logs are the second largest signal, especially with verbose logging (debug/trace level in production).

Reduction strategies:

• Filtering at the Collector level — drop debug/trace logs in production before they reach the backend

  // Alloy: drop debug logs before sending to Loki
  loki.process "filter" {
    forward_to = [loki.write.default.receiver]
  
    stage.drop {
      expression = ".*level=debug.*"
    }
  }
  

• Structured logs — JSON instead of plain text, better compression ratio in Loki
• Retention — short retention for debug logs (3-7 days), longer for error/audit (30-90 days)
• Parsing at ingestion — extract only needed fields, drop the rest

Metrics — Cardinality Under Control

Problem: Each unique label combination creates a new time series. A label with user_id with 1M users = 1M series.

Reduction strategies:

• Avoid high-cardinality labels — never use user_id, request_id, trace_id as metric labels
• Aggregation in the Collector — sum/average before sending to the backend
• Native histograms (Prometheus) — one series instead of many buckets
• Recording rules — pre-aggregate frequent queries, reduce query load

# Prometheus recording rule — pre-aggregation
groups:
  - name: cost-optimization
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job, status_code)

Data Retention

Architecture and Costs

• Object storage (Azure Blob / S3) instead of local disks — 10-100x cheaper per GB
• Tiered storage — hot data on SSD, cold on object storage (Mimir, Loki, Tempo support this natively)
• Compression — Loki and Tempo compress data automatically (snappy/zstd), but structured data compresses better
• Dedicated Collectors per signal — easier scaling and cost control per pipeline