Internal AI Agents for Backend Operations — Gated by Evals

Context

A lot of backend on-call work is repetitive cognition: reading through logs, noticing the anomaly, checking that data looks right, writing up what happened. It's exactly the kind of toil an LLM is good at — if you can stop it from making things up, and if you can tell when a change has made it worse.

At Bidgely I built internal AI agents on top of LLM APIs to take that toil off engineers' plates: log analysis, anomaly detection, data validation, and incident summarization.

Grounded in real evidence

A model prompted with no context will confidently invent service names, error semantics, and runbook steps. In an operational setting, confidently-wrong is worse than no tool at all.

So the agents use contextual retrieval: embeddings and search pull the genuinely relevant logs and runbook context at query time, and the model reasons over that real evidence instead of its priors. Answers stay current as the systems change, and they trace back to their sources.

Tools, not stuffed prompts

Production logs are far bigger than any context window. Instead of guessing up front what to paste into a prompt, the agents get bounded, read-only tools — query these logs, fetch these metrics, look up this record — and decide step by step what to look at. Every tool call is logged, so when an agent reaches a conclusion you can see exactly what evidence it used. Designing those tools well (tight inputs, bounded outputs, useful errors) turned out to matter more than the prompts.

Assistive, not autonomous

I deliberately kept the agents on the assistive end: they summarize, flag anomalies, validate data, and draft incident write-ups, and a human makes the call. Autonomous remediation would have raised the blast radius of a wrong answer enormously, and trust has to be earned first. That scoping is why they got adopted quickly — they removed toil without asking anyone to hand a production decision to a model.

Evals before every release

The part I'd defend hardest: every release runs against eval suites first. LLM behavior shifts with each prompt edit, model upgrade, or tool change, and the shift is usually invisible on the three examples you try by hand. The eval suites measure agent accuracy on representative cases, so a regression shows up as a number going down before anyone on the team sees a worse answer — and an improvement is measured instead of assumed.

What broke

An early version over-weighted recent logs in retrieval and would miss relevant older context, so summaries came out confident but incomplete. That's the failure mode that quietly kills trust in an internal tool. Tuning the chunking and relevance ranking fixed it — and it's why I evaluate retrieval quality directly, not just the final answer.

Outcomes

Engineers get grounded, source-cited help with the repetitive parts of operations, a human still owns every decision, and no release ships without passing the evals. The same grounded-retrieval pattern carries over to customer-facing work too — LLM-generated, personalized energy insights built over our analytics data.