Collecting Azure Metrics and Logs

• What Azure exposes
• The pipeline
• Authentication: workload identity
• Metrics: prometheus.exporter.azure
• Logs: loki.source.azure_event_hubs
• Monitoring the hub itself
• The hub’s own logs
• Failure modes
• Alloy vs azure-metrics-exporter vs Vector
• Decision table
• In our stack
• Related lessons

Managed services have no node to put an agent on. The AKS control plane, a PostgreSQL Flexible Server and a storage account report only to Azure Monitor.

Collecting them means two different paths: pull metrics from the Azure Monitor API, and push logs out of Azure through Event Hubs.

What Azure exposes

• Metrics are not pushed. Something has to poll the API. Every poll is an ARM API call that counts against throttling limits.
• Resource logs are not collected by default. Without a diagnostic setting they are not stored anywhere. Event Hubs is the only destination a collector can read as a stream.
• Idle often means NULL, not 0. Each resource provider decides whether an idle minute is 0 or no value, and a dimension value has no series until it had data. Error and throttling series often appear only with the first error — query with or vector(0).

The pipeline

flowchart LR
    subgraph az ["Azure"]
        res["AKS · PostgreSQL"]
        sto["Storage account<br/>(metrics only)"]
        mon["Azure Monitor metrics API"]
        diag["Diagnostic settings"]
        eh[("Event Hubs<br/>main namespace")]
        eh2[("Event Hubs<br/>-logs namespace<br/>(metrics only)")]
        res --> mon
        sto --> mon
        eh --> mon
        eh2 --> mon
        res --> diag --> eh
        eh -- "its own logs" --> eh2
    end
    subgraph k8s ["AKS — alloy-azure (1 replica, workload identity)"]
        exp["prometheus.exporter.azure"] --> scr["prometheus.scrape"]
        src["loki.source.azure_event_hubs ×2"] --> proc["loki.process"]
    end
    mon -- "pull, every 60 s" --> exp
    eh -- "consume (Kafka, OAuth)" --> src
    eh2 -- "consume (Kafka, OAuth)" --> src
    scr --> prom[(Prometheus)]
    proc --> loki[(Loki)]

Authentication: workload identity

No secret in the cluster. The pod proves its identity with a Kubernetes token; Entra ID trusts that token through a federated credential.

1. Pod label azure.workload.identity/use: "true" + ServiceAccount annotation azure.workload.identity/client-id → the AKS webhook injects AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_AUTHORITY_HOST, AZURE_FEDERATED_TOKEN_FILE and mounts a projected ServiceAccount token.
2. Both components use the Azure SDK DefaultAzureCredential, which exchanges that token for Entra ID tokens of the managed identity — one for ARM (metrics), one for https://<ns>.servicebus.windows.net/.default (Event Hubs).
3. Entra ID accepts it because of a federated credential: issuer = the cluster’s OIDC issuer URL, subject = system:serviceaccount:monitoring:alloy-azure, audience = api://AzureADTokenExchange.
4. Event Hubs receives the token over Kafka SASL_SSL + OAUTHBEARER; Azure Monitor receives it as a bearer token.

The only key involved is the namespace’s SAS rule (Manage, Send, Listen) that diagnostic settings use to write into the hub. Alloy never sees it.

Metrics: prometheus.exporter.azure

Excerpt — one of ten exporter blocks; the real config has more metrics and concatenates all exporter targets:

prometheus.exporter.azure "aks_nodes" {
  subscriptions               = [sys.env("AZURE_SUBSCRIPTION_ID")]
  resource_type               = "Microsoft.ContainerService/managedClusters"
  resource_graph_query_filter = string.format("where resourceGroup =~ '%s'", sys.env("AZURE_RESOURCE_GROUP"))
  metrics                     = ["node_cpu_usage_percentage", "node_memory_working_set_percentage"]
  included_dimensions         = ["node"]
}

prometheus.scrape "azure" {
  targets         = prometheus.exporter.azure.aks_nodes.targets   // array.concat(...) of all blocks
  job_name        = "integrations/azure"
  scrape_interval = "60s"   // matches the 1-minute grain; each scrape = API calls
  scrape_timeout  = "50s"
  forward_to      = [prometheus.relabel.azure_dimensions.receiver]
}

Names get long: azure_microsoft_containerservice_managedclusters_apiserver_cpu_usage_percentage_average_percent. That is the price of a name that says type, metric, aggregation and unit.

Logs: loki.source.azure_event_hubs

loki.source.azure_event_hubs "resource_logs" {
  fully_qualified_namespace = sys.env("EVENTHUB_KAFKA_ENDPOINT")   // <ns>.servicebus.windows.net:9093
  event_hubs                = [sys.env("EVENTHUB_NAME")]
  group_id                  = "alloy-azure"
  use_incoming_timestamp    = true
  labels                    = { job = "integrations/azure_event_hubs", source = "azure" }
  relabel_rules             = loki.relabel.azure.rules   // __azure_event_hubs_category → category
  authentication { mechanism = "oauth" }
  forward_to = [loki.process.azure.receiver]
}

All Azure lines are under {source="azure"}, with category, resource and resource_type labels.

• Event Hubs Standard tier or higher. Basic has no Kafka endpoint.
• One Event Hubs message = many records. Diagnostic settings wrap logs in {"records": [...]}; the component emits one Loki line per record and sets __azure_event_hubs_category.
• Strict schema check. A record needs time or timeStamp, category, resourceId and operationName (field names match case-insensitively). Otherwise it becomes a raw line without a category label. Some services use an older schema (Event Hubs: EventTimeString, no time, no operationName) — take category from the JSON in loki.process.
• Derive labels from resourceId (resource_type, resource) — low cardinality, and the same names work across categories.
• Set level at ingest. Azure severities (LOG, klog I/W/E, HTTP status in audit events) are not names Loki recognises, so detected_level becomes unknown. A level structured-metadata field fixes Explore, Logs Drilldown and Grafana MCP at once.
• Choose categories by volume.
  • Storage account StorageRead/StorageWrite logs for a Loki/Tempo/Mimir backend flood the hub.
  • kube-audit is the main AKS cost driver; kube-audit-admin excludes get and list events.
  • Of the categories enabled here, guard (Entra ID authorization checks) is the noisiest.
• Know what audit events contain. The AKS audit policy logs request and response bodies (RequestResponse) for writes on core API groups. Secrets, ConfigMaps, ServiceAccounts and TokenReviews are logged at Metadata level — name and user, no body. Filter in loki.process what must not reach Loki.
• Never send a namespace’s logs into itself. Microsoft: sending logs “from a resource to the same resource would generate an infinite loop of generating and writing data”. The portal blocks it; ARM, Bicep and the CLI do not. Send them to a second namespace (below) and monitor the hub with metrics (next section).

Monitoring the hub itself

The hub cannot report on itself through itself, and Standard tier has no consumer-lag metric (ConsumerLag exists only in ApplicationMetricsLogs, Premium and Dedicated). Four independent views:

Outside the cluster:

• Activity log — who changed or deleted the namespace, hub or a diagnostic setting (management-plane operations). Collected automatically; export it with a subscription diagnostic setting.
• Resource Health / Service Health — Azure-side incidents for the namespace and region.
• The hub’s own logs — through a second namespace, see below.
• Kafka consumer-group tooling — Kafka clients and exporters can read committed offsets of the consumer group through the Kafka endpoint (Event Hubs supports only part of the Kafka admin API — test before relying on it).

The hub’s own logs

The namespace that carries every other resource’s logs cannot carry its own (infinite loop). Yet its operational logs, Kafka coordinator logs (consumer group joins, rebalances) and Kafka user error logs (failed client API calls) are what explains a failure: metrics say that a client failed or a consumer group rebalanced, not which or why.

Every option sends the logs somewhere else. The chain ends at the second destination: it has no diagnostic setting of its own and is monitored by metrics only — metrics are pulled from Azure Monitor and need no hub, so nothing recurses.

Our stack: a second namespace (<ns>-logs, Standard, 1 TU, one event hub, 1 partition). The main namespace’s diagnostic setting writes into it; a second loki.source.azure_event_hubs block in the same alloy-azure release reads it with the same identity (Data Receiver on both namespaces). The logs land in Loki under resource="<main namespace>", next to every other Azure log. Log Analytics is the alternative when the logs are rarely needed and a KQL query from Grafana is enough.

Failure modes

Alloy vs azure-metrics-exporter vs Vector

prometheus.exporter.azure uses the webdevops azure-metrics-exporter as a library — same prober, but Alloy’s own handler and defaults. Switching between them is not drop-in: dashboards and alerts change. Vector has no Azure source at all.

Alloy — pros

• One binary for both paths. No extra Deployment, Service and ServiceMonitor for the exporter; same config language, UI and upgrade cadence as the rest of the collector fleet.
• Secretless Event Hubs with per-pod identity. mechanism = "oauth" + workload identity; nothing to rotate.
• Azure log format handled. records[] unpacking, timestamps and category without custom code.
• Supported, GA components. Both Azure components are generally available in Alloy.

Alloy — cons

• Behind upstream and only partly exposed. Even v1.20 pins a July 2025 commit (upstream 25.12.0). No caching, no tag-driven or per-resource probe endpoints.
• Shared crash domain. An Event Hubs failure at startup stops metrics too; a panic in the embedded library stops the whole process.
• Weaker log delivery than Vector (offset before Loki accepts, no production disk buffer for loki.write).
• Limited transforms. Mixed Azure schemas need stage.match + templates where VRL would be one function.
• No metrics:getBatch. Neither Alloy (issue #4651, open) nor the webdevops exporter uses the batch API; Promitor and the OTel Collector azuremonitorreceiver (use_batch_api) have it as an experimental option.

Decision table

In our stack

Diagnostic settings:

loki.process drops every kube-audit-admin line that mentions flagd (the incident exercise must not be solvable by reading the audit log) and every lease renewal (coordination.k8s.io, pure noise).

Related lessons

• Alloy vs OpenTelemetry Collector vs Vector
• Topologies on Kubernetes
• Reliability
• Self-monitoring
• Cheat sheet