HolmesGPT — AI SRE Agent

• Versions
• The Investigation Loop
• Toolsets in the Lab
• Reacting to Alerts
• Asking Directly
• What It Found in the Lab
• Model Choice and API Constraints
• Security and Cost
• Related lessons

HolmesGPT is an open-source AI agent that investigates incidents. You give it an alert or a question; it decides which queries to run against your metrics, logs, traces and cluster state, runs them, and answers with a root cause and the evidence behind it. It is read-only by design: it diagnoses and proposes a fix, it never applies one.

Versions

The Investigation Loop

Holmes is an agent loop, not a fixed runbook. Every step is an LLM decision about which tool to call next, based on what the previous tools returned.

flowchart LR
    Q[Alert or question] --> L[LLM picks a tool]
    L --> T[Toolset runs a query]
    T --> E[Result added to context]
    E --> L
    E -->|enough evidence| A[Root cause + evidence + fix]

• Toolsets are the agent’s hands: each one wraps an API (Kubernetes, Prometheus, Loki, Tempo, bash with an allowlist). Holmes only knows what its toolsets can reach.
• The answer is only as good as the evidence. A good investigation lists the queries it ran and what they returned, so a human can check every claim.
• Cost scales with tool calls, not with the size of the question: every call adds its output to the context the model reads again on the next step.

Toolsets in the Lab

Holmes also accepts remote MCP servers as toolsets (e.g. the lab’s Grafana MCP), which would give it Grafana’s view with a service-account’s permissions instead of direct backend access.

Reacting to Alerts

The standalone chart has no alert receiver. Its HTTP API has one investigation entry point, POST /api/chat; the Holmes Operator runs scheduled checks and checks after Deployment rollouts, not on alerts. The lab closes the gap with Alertmanager and a small relay:

flowchart LR
    G[Grafana-managed<br/>alert rules] --> AM[Alertmanager]
    P[PrometheusRule<br/>alerts] --> AM
    AM -->|AlertmanagerConfig<br/>holmes-react| R[Relay<br/>holmes-react:8080]
    R -->|POST /api/chat| H[HolmesGPT]
    H --> R
    R -->|e-mail| M[Mailpit]

Which alerts reach Holmes (AlertmanagerConfig monitoring/holmes-react):

What the relay does:

• Switch. Reacts only while ConfigMap holmes/holmes-react-switch says react=on (GitHub workflow HolmesGPT). Off is the default: alerts are acknowledged and dropped, so Alertmanager never logs a failed notification.
• Deduplication. An alert (by fingerprint) is investigated at most once per 6 hours. Alertmanager repeats every 15 minutes, so switching on picks up alerts that were already firing.
• Namespace scope. An alert group with a namespace label other than otel gets a prompt that restricts every query to that namespace and is mailed to <namespace>@workshop2.indexoutofrange.com. This is an instruction to the model, not RBAC: the ClusterRole still reads every namespace.
• Output. The analysis goes only to the e-mail, never to stdout — pod logs land in Loki, where an agent working through Grafana MCP could read another agent’s answer.

Why the AlertmanagerConfig lives in monitoring: by default the Prometheus operator adds namespace="<object's namespace>" to every route of an AlertmanagerConfig. Grafana-managed alerts carry no namespace label, so the route would never match. The lab sets alertmanagerConfigMatcherStrategy: OnNamespaceExceptForAlertmanagerNamespace, which skips that matcher for objects in Alertmanager’s own namespace.

Asking Directly

kubectl port-forward -n holmes svc/holmes-holmes 8080:80
curl -s localhost:8080/api/chat -H 'Content-Type: application/json' \
  -d '{"ask": "Why are checkouts failing in namespace otel? List the evidence for each cause."}' \
  | jq -r .analysis

The response also carries tool_calls (every query Holmes ran) — the audit trail behind the answer.

What It Found in the Lab

Measured on 2026-10-13 against the shop with the baseline faults on (12 flagd failure flags):

• It took a shortcut. It read the flagd ConfigMap directly — any agent with ConfigMap access would. With faults injected into code instead of flags, it can name the failing span and its error, but not the line of code.
• It separated cause from symptom. The cart error says “Wasn’t able to connect to redis”; Holmes checked Valkey and showed it was healthy.
• It reported what it did not check (Kafka consumer lag, image latency). Trust an investigation that lists its gaps more than one that claims to have covered everything.

Model Choice and API Constraints

• Claude Opus 5.5 rejects sampling parameters. Holmes sends temperature=0.00000001 unless the env var TEMPERATURE is empty; the lab sets it empty. No thinking, budget_tokens or reasoning_effort either — Opus 5.5 always thinks adaptively and returns 400 on a thinking budget.
• Grafana’s LLM plugin uses Claude Opus 4.6, not 5.5. Grafana’s dashboard AI buttons send temperature 1 or 0.5, which Opus 5.5 rejects; the plugin passes request parameters through unchanged. See Grafana.
• Opus 5.5 pricing: $4 per million input tokens, $20 per million output tokens. Each tool call re-sends the growing context, so a 20–35-call investigation reads far more tokens than the question and alert contain.

Security and Cost

Related lessons

• Grafana
• Grafana Tempo
• Grafana Loki
• AI Telemetry