Exporters

Exporters

• expose metrics from systems that don’t support Prometheus
• Available:
  • Official: https://github.com/prometheus
  • All: https://github.com/prometheus/prometheus/wiki/Default-port-allocations

How to write exporters

Main design decisions

Trade-off between cleanliness and ease of maintenance:

• Systems with few metrics (e.g., HAProxy) - can aim for perfect metrics
• Systems with hundreds of metrics (e.g., MySQL) - not worth engaging in getting ideal metrics
• Mixed approach (e.g., Node Exporter) - different complexity depending on module

Exporter configuration

For applications:

• Aim for exporter without configuration - just provide application location
• Optionally: filtering expensive metrics for large setups
• Optionally: disabling expensive metrics by default

For monitoring systems/protocols:

• Best scenario: similar data model to Prometheus (CloudWatch, SNMP, collectd)
• Worse scenario: custom metrics requiring transformation configuration (JMX, Graphite, StatsD)
• Standard format: YAML (as in entire Prometheus ecosystem)

Metric naming

Basic rules:

• Prefix with exporter name: haproxy_up
• Use base units (seconds, bytes) - leave conversion to graphical tools
• Expose ratios, not percentages
• Snake_case instead of camelCase
• Avoid colons (:) - reserved for rules
• Allowed characters: [a-zA-Z0-9:_]

Reserved suffixes:

• _total - for counters (COUNTER)
• _sum, _count, _bucket - for histograms/summary
• process_, scrape_ - reserved prefixes

Best practice examples:

• Requests: separate metrics for sum and errors instead of one with label
• Cache: separate metrics for sum and hits

Labels

Avoid:

• type - too general
• Conflicts with target labels: region, zone, cluster, environment
• le (histograms) and quantile (summary)

Rule of thumb: one metric should make sense after summing or averaging

Minimalism: each additional label is a complication for users in PromQL

Metric types

• Match to Prometheus types (counter, gauge)
• When in doubt: UNTYPED as safe option
• Note: counters from other systems may decrease → this is not a Prometheus counter

Collector implementation

Never use direct instrumentation and updates at each scrape!

Correct approach - create new metrics each time:

• Go: MustNewConstMetric in Collect() method
• Python: custom collectors
• Java: List<MetricFamilySamples>

Reasons:

• Avoiding race conditions during concurrent scrapes
• Preventing exposure of “dead” labels

Scrape metrics

• Duration: {exporter_name}_scrape_duration_seconds (gauge)
• Avoid additional scrape metrics (counters, histograms) - duplicate Prometheus built-in metrics

Deployment

1:1 Rule: one exporter per application instance

Exceptions:

• Black box devices (SNMP, IPMI) - physically impossible to run
• Business queries - business queries to random replica

Schedule:

• Only synchronous scrape on Prometheus request
• No timestamps - Prometheus will set them
• Cache acceptable for metrics > 1 minute
• Default timeout: 10 seconds

Scrape failures:

• Option 1: return 5xx error
• Option 2: metric {exporter}_up (0/1)

Landing page: simple HTML page with link to /metrics

Ports: register at https://github.com/prometheus/prometheus/wiki/Default-port-allocations

What to Remove

Statistics to skip:

• 1m, 5m, 15m averages (Prometheus calculates better)
• Averages since application start
• Min/max without time context
• Standard deviation
• Quantile (optionally move to Summary)

Machine metrics: remove if available through Node Exporter or similar