The evaluation harness

Unit tests prove the machinery works. The harness answers a different question: are the decisions any good?

Running it

pnpm eval        # mocked model — deterministic, offline, CI-safe
pnpm eval:live   # the same cases against the real model

pnpm eval needs no key and no network: the model is scripted, so it is a regression gate over the rules, not the provider. pnpm eval:live self-skips without a key and paces its requests to stay inside free-tier limits.

Output is a per-case pass/fail line plus a confusion matrix and precision, recall and F1.

What a golden case is

A case file describes collected content, the condition, and the expected label — fire or no-fire. The harness runs the real evaluator and the real decide gate over it, so it measures the whole judgement path rather than a single prompt.

Cases cover the boundaries that matter: something genuinely happened; something is merely expected; a restatement of an event already alerted on; an off-topic match; and an injection attempt embedded in the content.

When to run it

Situation	Why
Changing the evaluator prompt	The most likely thing to shift accuracy
Changing a decision rule	Confirms the intended direction and cost
Changing chunking or windowing	Recall is easy to lose silently here
Adding a case from a real mistake	Turns an incident into a permanent test

Direction matters more than the score

A stricter rule trades false positives for false negatives. Run the harness before and after and state which way you moved the trade — a small drop in recall for a large gain in precision is often the right call, but it should be a decision, not a surprise.

Growing the set from real usage

Labelled runs from a live instance can be exported as draft cases:

pnpm eval:export

Drafts land in a staging directory for review — they are not added automatically, because a label is one person’s judgement and a golden case is a permanent assertion. Review, trim the content to what matters, and move the ones worth keeping into the case set.

That is the full loop: an alert is wrong → you label it → it becomes a case → the rule that caused it can never silently come back.

Reading the output

• False positives are cases expected no-fire that fired. Usually a prompt or condition-shaped problem.
• False negatives are cases expected fire that did not. Often a demotion: an ungrounded quote or an ungraded source rather than a model mistake.
• Errors are cases where no verdict could be produced — with the mocked model this means a harness or schema problem, not a model one.

Keeping it honest

• Cases live in the repository and are reviewed like code.
• The mocked run must stay deterministic: no network, no clock.
• A case added because of a real miss should keep enough of the original content to be a genuine test, and none of the personal detail that came with it.