Collecting Azure Metrics and Logs
- Architecture
- Metrics: pulled, not pushed
- Logs: through Event Hubs
- Is the hub keeping up?
- What can go wrong
- Scaling to thousands of resources
- Alloy vs alternatives
- Related lessons
Managed services (AKS control plane, PostgreSQL Flexible Server, storage) have no node for an agent and report only to Azure Monitor. The in-cluster stack sees none of it — yet it often explains an incident.
Two paths: pull metrics, push logs.
Architecture
flowchart LR
subgraph llane ["Logs — diagnostic settings push, Alloy streams"]
direction LR
subgraph lres [" "]
direction TB
aksL["AKS control plane<br/>admin audit (writes) · scheduler<br/>controller manager · autoscaler · authorization"]
pgL["PostgreSQL<br/>server logs"]
end
hub["Event Hubs<br/>main namespace"]
hub2["Event Hubs<br/>second namespace"]
cons["Alloy<br/>Event Hubs consumer"]
loki[("Loki")]
aksL --> hub
pgL --> hub
hub -- "its own logs<br/>(never into itself)" --> hub2
hub --> cons
hub2 --> cons
cons --> loki
end
subgraph mlane ["Metrics — every resource reports to Azure Monitor, Alloy pulls"]
direction LR
subgraph mres [" "]
direction TB
aksM["AKS<br/>API server · etcd · scheduler<br/>autoscaler · nodes"]
pgM["PostgreSQL<br/>CPU · memory · connections · IOPS"]
stoM["Storage account<br/>transactions · latency · availability"]
ehM["Event Hubs, both namespaces<br/>throughput · errors · throttling"]
end
monitor["Azure Monitor<br/>metrics API"]
exp["Alloy<br/>Azure exporter, every 60 s"]
prom[("Prometheus")]
aksM --> monitor
pgM --> monitor
stoM --> monitor
ehM --> monitor
monitor -- "pull" --> exp --> prom
end
classDef metricsNode fill:#e8871e,stroke:#a85f12,color:#fff
classDef logsNode fill:#2e9e5b,stroke:#1f6e3f,color:#fff
classDef resNode fill:#475569,stroke:#1e293b,color:#fff
class monitor,exp,prom metricsNode
class hub,hub2,cons,loki logsNode
class aksM,pgM,stoM,ehM,aksL,pgL resNode
style mres fill:none,stroke:none
style lres fill:none,stroke:none
%% edges 0-5 logs lane, 6-11 metrics lane (logs defined first so metrics render on top)
linkStyle 0,1,2,3,4,5 stroke:#2e9e5b,stroke-width:2px
linkStyle 6,7,8,9,10,11 stroke:#e8871e,stroke-width:2px
Every resource has metrics; only resources with a diagnostic setting send logs. Storage is metrics-only — its read/write logs would be Loki’s, Tempo’s and Mimir’s own traffic. Both Alloy boxes are one process (alloy-azure), separate from the OTLP gateway, with workload identity and read-only rights: no secrets in the cluster.
Metrics: pulled, not pushed
- Discovery is automatic within the configured resource types.
- Every poll is an API call — throttled and billed. Poll once a minute, the metrics’ resolution.
- Gaps are normal: an idle minute is often no value, not zero.
Azure can push metrics too; neither way fits here:
| Push | Why not |
|---|---|
Diagnostic setting AllMetrics → Event Hubs |
Dimensions are flattened — Microsoft: “aggregated across dimension values” — so no per-node or per-hub series; not every metric is exportable; idle resources report up to two hours late; arrives as log records Alloy cannot turn into metrics |
| Data collection rule (DCR) export | Keeps dimensions, ~3 min latency — but supports AKS and storage, not PostgreSQL or Event Hubs; dimensions need custom conversion back into metrics |
Push wins only when the API is the bottleneck and the resource types fit. It is not free: DCR export bills per sample, Event Hubs ingestion per GB.
Logs: through Event Hubs
Logs are stored nowhere until a diagnostic setting sends them out. Of its destinations, only Event Hubs is a stream a collector outside Azure can read; Log Analytics keeps them inside Azure, storage is an archive.
- Pick categories by value — full API audit and storage read/write logs are mostly cost. The activity log (who changed what, subscription-wide) is worth adding.
- Normalize at ingest (category, resource, level) and drop sensitive events — audit logs can carry request bodies.
- The hub’s own logs go to a second namespace. Into itself they would loop (Microsoft: “an infinite loop of generating and writing data”). The second namespace is watched by metrics only — the chain ends there.
Is the hub keeping up?
No tier exposes consumer lag as a metric. Instead:
| Question | Signal |
|---|---|
| Throttled? | Throttled requests > 0; traffic vs throughput units (1 MB/s in, 2 MB/s out each) — act at ~70% |
| Consumer keeping up? | Messages in − out ≈ 0 (one consumer group) |
| Consumer delivering? | Records written to Loki > 0 |
| Every source sending? | Each log category keeps arriving |
What can go wrong
| Risk | Mitigation |
|---|---|
| Restarts after a fresh deploy — permissions need minutes | Expected; stops on its own |
| Log path fails → metrics fail too (one process) | Two collector releases |
| Long Loki outage loses data — reading is confirmed before Loki writes | Retry instead of dropping; replay by resetting the consumer group’s offset within the hub’s retention |
Scaling to thousands of resources
Azure Resource Manager allows 25 reads per second per identity (burst 250; all identities up to 15×), Resource Graph about 15 queries per 5 seconds, 20 metrics per call. Beyond that: HTTP 429, and a gap nobody fills. 5,000 resources polled every minute need ~83 calls per second — over three times the limit.
| Lever, cheapest first | Effect | Catch |
|---|---|---|
Subscription scope (Alloy regions) |
One call per subscription, region and type — independent of resource count | Not every type; subscription-wide read rights; no resource-group filter |
| Interval tiers — alerts every minute, capacity every 5–15 min | Several times fewer calls | Prometheus marks series stale after 5 min — slow tiers are sparse |
| Separate identities per collector shard | Each identity has its own budget | Sharding by resource region alone does not help; never two collectors on one shard |
| Batch API — 50 resources per call, outside the ARM limit | 5,000 resources → ~100 calls per minute | Own throttling (529); needs a batch-capable collector |
| Don’t pull the long tail — push, or Grafana’s Azure Monitor data source on demand | Fewer or no API calls | Push costs export/ingestion |
The batch collector is the OTel Collector’s azuremonitorreceiver (or Promitor; Alloy has none):
| ➕ | ➖ |
|---|---|
| Batch API, 24 h resource cache, automatic dimension split, resource-group and service filters, per-pod identity | Alpha receiver, experimental batch; cache delays new resources; split multiplies series; OTel metric names break existing dashboards; a second collector type |
A Prometheus-native batch option exists: azure-monitor-exporter (jkroepke, not webdevops’ azure-metrics-exporter) — /probe per resource type and metric over metrics:getBatch, workload identity. Still v0.1.0 (July 2024, three releases) — too early for production.
More collector replicas do not help metrics. The limit is Azure’s API budget, not collector CPU. Replicas share one identity, so one budget. Without clustering each replica pulls the same resources — N× the calls and duplicate series. With clustering the exporter is a single target: one replica scrapes, the rest idle. Scale metrics by sharding the config (resource types, subscriptions) across releases with their own identities, not by autoscaling.
Pulling is static load — manage it as a budget. Calls follow from configuration × resource count × interval; nothing to autoscale. Two things still move it: new resources of configured types (discovery adds them silently) and other tools sharing the subscription’s ARM cap. Watch estimated calls vs the limit, failed scrapes, scrape duration and throttling responses — in our stack the Azure Metrics Pull Budget dashboard.
Logs do scale with replicas. Consumers in one consumer group split the hub’s partitions — more replicas, more throughput, up to the partition count. That is another reason to run metrics and logs as separate releases: logs scale horizontally, metrics by shard.
Logs scale per region: a diagnostic setting’s hub must be in the resource’s region. Each namespace scales by throughput units (up to 40 on Standard) and partitions — fixed at creation on Standard, ~1 MB/s in each, one consumer each.
Alloy vs alternatives
Alloy’s Azure exporter is azure-metrics-exporter, built in. Vector has no Azure source.
| Alloy | azure-metrics-exporter | OTel Collector | Vector | |
|---|---|---|---|---|
| Covers | Metrics + logs | Metrics | Metrics + logs | Logs (Kafka), parsing is yours |
| Auth | Per-pod identity | Per-pod identity | Per-pod identity | Shared key, service principal, node identity |
| Strength | One component, Azure log format, subscription scope | Discovery cache, more query modes | Batch API, caching | Transformations, delivery guarantees |
| Weakness | No caching or batch, one failure domain | Extra component, no batch | Alpha, different metric names | Extra tool for metrics |
| Situation | Choose |
|---|---|
| Grafana stack, moderate volume, no secrets | Alloy |
| Thousands of resources | Alloy with subscription scope and tiers; OTel Collector batch where that does not fit |
| Metrics must survive a log failure | Two collector releases |
| Heavy log reshaping, many destinations | Vector for logs + exporter for metrics |
| Dashboards only | Grafana’s Azure Monitor data source |
In our stack: infrastructure/azure-monitoring.bicep, the alloy-azure release, dashboards in the Grafana folder Azure (AKS control plane, Event Hubs, PostgreSQL, metrics pull budget).