Write Format

Write Format

• Why are labels important?
  • Each unique label combination = separate stream = separate set of chunks
  • Many labels (high cardinality) means:
    • large indexes with many chunks — slower queries, more I/O operations
    • many open chunks — high RAM cost on ingesters
  • Example: adding a user_id label with 100k values = 100k streams × retention = cost explosion
• Show scrape config for Loki
• Two types of labels:
  • Static
    • E.g.: host, application, and environment
    • Limited number - around 10
  • Dynamic - high cardinality - e.g., IP addresses
• Don’t add labels until they’re needed
  • Start with brute force using filters: |= "text", |~ "regex"
  • If a label has many values and is hard to retrieve -> client-side parsing
• How to decide when to use a label vs dynamic parsing?
  • If volume is enough to fill a chunk (chunk_target_size) in less time than max_chunk_age -> consider adding additional labels
    • We want to avoid splitting logs into streams (log was written due to chunk_max_age). There’s a query for this: sum by (reason) (rate(loki_ingester_chunks_flushed_total{cluster="dev"}[1m]))
• Stream -> combination of labels with values
• Q: How many streams are here?

{job="apache",env="dev",action="GET",status_code="200"} 11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
{job="apache",env="dev",action="POST",status_code="200"} 11.11.11.12 - frank [25/Jan/2000:14:00:02 -0500] "POST /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
{job="apache",env="dev",action="GET",status_code="400"} 11.11.11.13 - frank [25/Jan/2000:14:00:03 -0500] "GET /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
{job="apache",env="dev",action="POST",status_code="400"} 11.11.11.14 - frank [25/Jan/2000:14:00:04 -0500] "POST /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"