bet · Personal project

AgentOps — A Multi-Agent Incident-Response System

A LangGraph multi-agent system (planner, log analyst, fixer) that takes an incident from alert to root cause to a draft fix — with MCP tools, an eval harness, and human approval on every change.

Role: Sole designer and builder — agent graph, MCP servers, eval harness, and guardrails
Team: Solo
Scope: Production incident investigation: from an alert, to a root-cause hypothesis backed by evidence, to a pull request with a proposed fix
Level: Personal project

Eval dimensions per run

Method: Each run over the labelled incident scenarios is scored on root-cause accuracy, tool-call success rate, latency, and cost per run.
Window: every change

Write actions without approval

possiblenone

Method: Design property: human-in-the-loop approval is required before any write action, such as opening a pull request.
Window: by design

Observability

opaque runsevery step traced

Method: OpenTelemetry tracing is emitted for every agent step and every tool call.
Window: by design

Context

When a production alert fires, the first hour is usually the same work: figure out what to look at, dig through logs and metrics, form a hypothesis about the root cause, and draft a fix. AgentOps is a personal project to see how much of that an agent system can do reliably — and how to know when it can't.

Code: repository link coming soon.

Three agents, one explicit graph

Rather than one agent holding every tool, AgentOps is a LangGraph graph of specialized agents:

a planner that decides what to investigate,
a log analyst that gathers and interprets evidence,
a fixer that drafts a change once there's a root cause.

Each has a focused prompt and a small toolset. Transitions are explicit and the shared state is inspectable, so when a run goes wrong you can see exactly which stage failed — and each stage can be evaluated on its own.

Real systems through MCP

The agents need real access: logs, metrics, and code. I wrote custom MCP servers that expose logs, metrics, and GitHub as tools. Agents query live systems through them and open pull requests with proposed fixes. MCP gives a standard, testable boundary, and each server exposes only the operations it should.

Evals before features

Agent demos are easy to make look good. The harness is what keeps this honest: a set of labelled incident scenarios, and every change scored on root-cause accuracy, tool-call success rate, latency, and cost per run. A prompt tweak that fixes one incident and breaks three others shows up immediately.

Guardrails and humans in the loop

The fixer can open pull requests, so any write action needs human approval, guardrails bound what each agent may do, and OpenTelemetry tracing covers every agent step and tool call. Autonomous investigation, human-approved change.

What broke

Two lessons came straight out of the evals. First, the planner sometimes sent incidents that looked familiar straight to the fixer, which then confidently fixed a cause nobody had established — now the fixer is gated on an explicit, evidence-backed root-cause hypothesis, and root-cause accuracy is scored separately from fix quality. Second, an early log tool returned unbounded results that flooded the context window; tools now return bounded, summarized, paginated results, and cost per run is tracked alongside accuracy.

Outcomes

A multi-agent system that takes an incident from alert to root cause to a draft fix, measured on every change, observable end to end, and never landing a change without a human saying yes.

Decision Records

DR-001 · architecture

A planner, a log analyst, and a fixer wired together as an explicit LangGraph state machine are easier to trust, debug, and evaluate than one agent with every tool

Context

Incident response has distinct phases — decide what to look at, gather and interpret evidence, propose a change. A single agent holding every tool and every instruction tends to skip steps, mix phases, and fail in ways that are hard to locate.

Options considered

✗
One general agent with all tools in a single loop
Simple to start, but the prompt and toolset grow without bound, and when it goes wrong you can't tell which 'phase' failed.
✓
Specialized agents (planner, log analyst, fixer) as nodes in a LangGraph graph with shared state
Each agent has a focused prompt and a small toolset. Transitions are explicit, state is inspectable, and each step can be evaluated on its own.

Tradeoffs

More orchestration code and more LLM calls per incident, which costs latency and money. Worth it for reliability and debuggability.

Outcome

An incident moves through explicit stages — plan, analyse, fix — with the state at every transition available for inspection and evaluation.

DR-002 · architecture

Wrapping logs, metrics, and GitHub as MCP servers gives the agents a standard, permissioned interface to real systems instead of ad-hoc function glue

Context

To investigate real incidents, agents need to query logs, read metrics, inspect code, and open pull requests. How those capabilities are exposed determines how reusable, safe, and testable they are.

Options considered

✗
Hand-written function wrappers inside the agent code
Couples tools to one agent framework, mixes integration code with agent logic, and makes permissions hard to reason about.
✓
Custom MCP servers for logs, metrics, and GitHub
A standard protocol with a clean boundary. Tools are reusable by any MCP client, testable on their own, and each server exposes only the operations it should.

Tradeoffs

Running separate servers adds moving parts, and each tool still needs careful design so results stay small and useful. The clean boundary is worth it.

Outcome

Agents query live logs and metrics and open pull requests with proposed fixes through MCP tools.

DR-003 · bet

Measuring root-cause accuracy, tool-call success rate, latency, and cost per run over labelled incident scenarios is the only honest way to know whether a change made the agent better

Context

Agent demos are easy to make look good. Whether a prompt change, model swap, or new tool actually helps can't be judged from a couple of hand-picked runs.

Options considered

✗
Judge changes by trying a few incidents by hand
Selection bias. A change that fixes the incident you're looking at can quietly break three others.
✓
An eval harness over labelled incident scenarios with tracked metrics
Every change is scored on the same scenarios across accuracy, tool reliability, latency, and cost — so trade-offs are visible, not guessed.

Tradeoffs

Labelled scenarios take time to build and need expanding as the agent grows. That investment is what makes iteration fast and safe.

Outcome

Each change to prompts, models, or tools is measured on root-cause accuracy, tool-call success rate, latency, and cost per run.

DR-004 · bet

Human-in-the-loop approval, guardrails, and OpenTelemetry tracing on every agent step and tool call are what make an autonomous investigator acceptable near production

Context

The fixer can open pull requests against real repositories. A wrong action by an agent near production systems has a large blast radius, and an agent you can't observe is one you can't debug.

Options considered

✗
Let the agent open and merge changes on its own
Maximizes autonomy and blast radius at the same time. Trust has to be earned, not assumed.
✓
Agents investigate freely with read tools; any write needs human approval; every step is traced
Keeps the speed of autonomous investigation while a human stays accountable for changes. Tracing makes every decision reconstructable.

Tradeoffs

Approval adds a human wait to the loop, and tracing adds overhead. Both are the price of running agents responsibly.

Outcome

No change lands without human approval, guardrails bound what agents can do, and every agent step and tool call is traced with OpenTelemetry.

Where I Got It Wrong

Failure 1

Early on, the planner sometimes routed incidents that resembled earlier ones straight to the fixer, which then proposed confident fixes for a cause that the log analyst hadn't actually established

Cost: Plausible-looking but wrong draft fixes — the most dangerous kind of output for an incident tool

Lesson: Gate the fixer on an explicit root-cause hypothesis with supporting evidence in the graph state, and score root-cause accuracy separately from fix quality in the evals

Failure 2

A log-query tool initially returned raw, unbounded results, so a broad query could flood the context window and drive up cost and latency per run

Cost: Slow, expensive runs and worse reasoning on noisy incidents

Lesson: Tools should return bounded, summarized, paginated results — tool design is as much a part of agent quality as the prompt, and cost per run belongs in the evals

Long-Term Impact

A working reference for how I build agents: specialized roles, standard tool interfaces, evals first, and humans in control of every change.