Build a tiny eval
Outcome
You can measure how often your tool is right, find the pattern in its mistakes, fix the right thing, and prove the fix worked.
Concept
An eval is just this: run your tool on every example in your gold set, compare each output to the correct one, and count how many it got right. That number — six out of eight, say — is your reliability. Now you are not guessing. The number does three jobs. It tells you where you actually are, instead of where the demo made you feel you were. It points you at the pattern in the failures (the two it missed were both implied deadlines). And it lets you prove a change worked: adjust the prompt, re-run, and watch whether the number goes up. You don't need any special tools — for a small set you can run it by hand, or have Claude Code run each example and lay its answers next to your gold answers.
Weak approach vs. strong approach
The ask
I tweaked the prompt so it's more explicit about urgency. Feels sharper now. Shipping it.
What comes back
No number changes when something "feels sharper." You still have no idea if the tool got better, worse, or just different. The two deadline cases it was already missing are still missing — you just can't see them.
Try it
Run your tool on all your gold examples. Score each one right or wrong against your gold answer. Write the number down. Find what the failures have in common. Make one change aimed at that pattern. Re-run and re-score.
What to look for: Watch whether the number moved. If a change makes it worse, that's data too — revert it and try another.
Takeaway
You can't improve what you can't measure. A reliability number turns "feels better" into "is better" — and re-running it is the actual mechanism that gets you from 80 to 95.