The Prompt Regression Test Spec

#SITUATION
A prompt is running in production. Every time someone 'improves' it, something else silently breaks — a format change downstream code depends on, a tone shift users notice, a refusal where there used to be an answer. Changes get shipped on vibes. I need a test bench.

The production prompt:
{{paste_current_prompt}}

What it produces (the output consumers see):
{{output_description}}

Where its output goes next (parsed by code, read by users, fed to another model):
{{downstream_consumers}}

How often it changes: {{change_frequency}}

#PURPOSE
Design a regression testing setup for this prompt: a saved set of test cases, a scoring method, and a run procedure that answers one question before any new version ships — does this change make things better, worse, or just different?

#EXPECTED OUTPUT

## 1. Test Set Design
- 15-25 test inputs covering: common cases (the bulk of real traffic), format-edge cases (very long input, unusual characters, empty fields), tone-sensitive cases, and at least 3 failure probes (inputs that previously produced bad output)
- For each test input: the input, why it is in the set, and the expected qualities of a good output (not a golden string — a checklist: must contain X, must not contain Y, must be under Z length, format spec)
- Where to source real cases from production logs

## 2. Scoring Method
Define 2-4 checks per case, split into:
- **Automated** (exact/regex match, length bounds, JSON schema validation, required-field presence) — runnable with a script, no model judgment
- **Rubric** (tone, helpfulness, accuracy) — scored 1-5 with a one-line rubric definition each, designed for either a human or an LLM judge
Specify the exact rubric wording so scores stay stable across runs.

## 3. Run Procedure
Step-by-step: how to run the old and new prompt versions over the test set, how to present diffs side by side, and how to record scores. Include a results table template.

## 4. Ship / Fix / Roll Back Criteria
Concrete thresholds: what score profile means ship, what means iterate, what single failure auto-blocks regardless of aggregate score (e.g., any broken JSON output on format-edge cases).

## 5. Regression Log
A template for recording every prompt change: version, date, what changed, why, test results, decision. So the next person knows why the prompt looks like this.

#CONTEXT
- Who runs the tests: {{who}} (solo builder, small team)
- Tooling tolerance: {{tooling}} (happy to write a script vs. spreadsheet-only)

#STYLE
Implementable this week. If a step needs a script, provide it in pseudo-Python under 30 lines. If a spreadsheet works, say so — do not engineer a test framework nobody will maintain.