The feedback loop

Watcher does not ask you to trust a benchmark. It measures itself on your watches, from your labels.

Labelling

1. Open a watch’s run history in the panel or the app.

2. On any run, press 👍 or 👎:

   • 👍 correct — the decision was right, whether it fired or not.
   • 👎 wrong — the decision was wrong, whether it fired or not.

3. Label a mix. Wrong alerts are obvious, but labelling the runs that should have fired and did not is what makes recall meaningful.

You are labelling the decision, not the news. A run that correctly stayed quiet is a 👍.

What the numbers mean

Term	Meaning
True positive	Fired, and it was right
False positive	Fired, and it was wrong — noise
False negative	Did not fire, and it should have — a miss
True negative	Did not fire, correctly
Precision	Of the alerts you got, how many were right
Recall	Of the things worth alerting, how many you got
F1	A single number balancing the two

curl -s https://<your-host>/feedback/stats -H "authorization: Bearer $TOKEN"

The response includes recent false positives and false negatives, so you can see what went wrong rather than only how often.

Reading the result

Pattern	Likely cause	Where to go
Low precision (noisy)	Condition too loose, or broad search sources	Tune precision
Low recall (missing things)	Thresholds too high, sources too narrow, or extraction failing	Tune precision, Sources return nothing
Everything arrives unconfirmed	Sources are not graded	Manage the trust registry
Both look fine but you are unhappy	The watch is answering a slightly different question	Writing a condition

Note

With a handful of labels these numbers are noisy. Twenty or thirty labels across a couple of weeks is where they start being worth acting on.

Collection health, separately

GET /collection/stats — and the panel’s collection dashboard — reports attempted, collected, extracted and an extraction rate across recent runs.

It answers a different question from accuracy: not “did it judge well?” but “did it have anything to judge?”. A low extraction rate means pages are being fetched and not read, and no amount of threshold tuning will help.

Feeding the harness

Labelled runs can be exported as draft cases for the repository’s accuracy harness:

pnpm eval:export

They land in a staging directory for review, and once accepted they become regression cases that run offline against a mocked model. That is how a real mistake becomes a permanent test. See The evaluation harness.