A self-improving eval harness for agents that ship.

The Problem

The standard failure in agentic software:

• A prompt or tool tweak ships to production
• Nobody has a repeatable harness to check it against the edge cases that mattered
• The first signal is a customer report — or a security incident
• The team says "we should have caught that" — then ships the next change the same way

The missing piece isn't another model. It's a repeatable evaluation loop with memory — one that gets smarter between runs instead of re-testing the same cases forever.

What We Built

Component	What it does
Eval suite	Repeatable test cases per agent workflow — happy path + edge cases + failure modes
Regression gate	Every change re-runs the suite; a regression blocks the merge instead of reaching production
Postmortem loop	Every failed run writes a structured lesson — stream, run id, output, suspected cause, next-run change
Memory	The harness reads past lessons before each run, so a recurring failure gets its approach reworked — never blindly re-run
Stale-run review	A stream that fails N runs in a row is flagged for rework, not silently killed or ignored
Frozen anchors	Hard constraints (no auto-send, no auto-spend, no auto-publish) that the loop can never tune or weaken

The Loop

Runs on a schedule. Each tick:

• Reads revenue + the stream registry
• Target reached? → logs the win, exits
• Runs eligible streams in parallel — independent work fans out, never queues
• Postmortem — failed or zero-progress runs append a lesson so the loop fixes itself
• Surfaces gates — spend/publish/approve decisions go to a human, never automated

The loop optimizes the whole portfolio over many ticks — not one run's win rate. Verdicts require multiple data points; nobody kills or retunes a stream on one bad tick.

The Result

• Every failed run now produces a rework plan, not just a red X in CI
• Regressions surface at the harness, before customers do
• The eval suite compounds: each incident adds a permanent edge case
• Ops discipline — anchors, kill switches, logs — that's easy to say and rare to actually ship

What a Client Gets From This

• Audit — we instrument your agent workflows and stand up a repeatable eval suite from your real failure logs
• Harness — regression gate wired into your CI; every change runs the suite
• Loop — postmortem memory + stale-run review so the harness improves itself between releases

"If your agent works but you can't prove it, it doesn't — yet."