The Agent Evaluation Harness Spec

#SITUATION
I'm building an AI agent that takes actions (calls tools, makes decisions, answers questions). I cannot ship it on vibes. I need a structured way to evaluate whether it works and catch regressions when I change the prompt or model.

#PURPOSE
Produce an evaluation harness spec — the test cases, metrics, and pass/fail criteria — that I can implement to measure this agent's quality before every release.

#EXPECTED_OUTPUT
Structure the spec as:
1. **Agent capability summary** — what the agent is supposed to do, in 2-3 sentences.
2. **Evaluation scenarios** — 6-10 test scenarios that span the agent's expected use. For each: the input, the expected behavior or output, and why this scenario matters (edge case, common case, adversarial case).
3. **Metrics** — define 3-5 metrics. For each: what it measures, how to compute it (automated check, LLM-as-judge rubric, or human review), and the target threshold.
4. **Pass/fail criteria** — what combination of metric results means 'safe to ship' vs 'blocked.'
5. **Regression strategy** — how to run this on every change without it taking a full day.

#CONTEXT
Agent description: {{what your agent does, what tools/APIs it calls, what decisions it makes}}
Target users: {{who interacts with the agent and how}}
Current failure modes: {{what goes wrong today, or 'unknown — this is the first evaluation'}}

#STYLE
Technical and concrete. Write this for an engineer who will implement it. Specify the metric computation method, not just the metric name. No generic advice about 'measuring accuracy.'

Rules:
- At least one scenario must be an adversarial or abuse case (user tries to make the agent do something it shouldn't).
- At least one metric must be automatable without an LLM-as-judge (to keep eval cost and latency down).
- Define what 'good enough to ship' actually means in numbers.