The feedback loop

Watcher does not ask you to trust a benchmark. It measures itself on your watches, from your labels — and it learns from them, one watch at a time.

Labelling

1. Open the run — tap the alert on your phone, use the Rate this alert link at the end of a Telegram alert (when APP_URL is set), or open a watch’s run history in the panel.

2. Press 👍 or 👎:

   • 👍 correct — the decision was right, whether it fired or not.
   • 👎 wrong — the decision was wrong, whether it fired or not.

3. Optionally, tell Watcher why: pick the reason that fits and add a sentence in your own words. For a missed event you can paste the page it was on.

You are labelling the decision, not the news. A run that correctly stayed quiet is a 👍.

Reason	When
Not what I’m watching	It alerted on something outside the watch’s scope
Rumor / not confirmed	It treated an expectation or “coming soon” as confirmed
Old news	The event was real, but it isn’t news. Watcher won’t alert on it again, from any source
Duplicate	You were already alerted about this event. Same effect as Old news
Source I don’t trust	It came from a site you don’t rely on
Link or quote doesn’t support it	The evidence doesn’t back the alert
It missed something	A quiet run where there was something. On a held-back run, it also adds a rule that content like this counts
Should have been confirmed	A correct alert that arrived as “unconfirmed”

What Watcher does with it

Each label changes only that watch, and everything it changes is visible under What Watcher learned on the watch, where you can switch it off or undo it.

• A false alert stops haunting the watch. When you mark an alert as off-topic, a rumor, from an untrusted source, or badly evidenced, Watcher removes it from the watch’s memory — otherwise the real event, when it comes, could look like a repeat of the false one and be skipped.
• A missed item gets a second look. Mark a quiet run as “it missed something” and the content that run saw is evaluated once more on the next run (for up to a week), with whatever the watch has learned since.
• Rules for the watch. Off-topic, rumor and old-news labels become a short rule for that watch (“speculation like this is not confirmation here”). When you write a note, an AI reads it and turns it into a rule in plain words, which you can read and edit. Rules refine what counts as a match; they cannot make Watcher fire without a verbatim quote, and they never relax source-trust grading.
• Suggestions, never silent changes. When the labels point at a setting — a noisy unconfirmed band, a site that deserves “trusted”, a page that should be a source — Watcher suggests the change. Trust rules apply with one tap; anything else opens the revise flow so you approve each field.

Note

Up to six rules can be active per watch. When a seventh is learned it is saved switched off, and Watcher asks you to retire one — it never silently drops what you taught it. Pause all of a watch’s rules with one switch; the server operator can turn learning off everywhere with FEEDBACK_LEARNING=off.

Old news and held-back alerts

Every would-be alert is checked against today’s date and everything the watch already sent. When one is stopped, the run shows held back with the reason, such as Not sent: Repeats the admit-card alert from 4 Oct.

The check judges the development rather than the dates in the text. A result released today for an exam held last month is news, and so are a future date that was never alerted, the next edition of something recurring, and a real change in something ongoing. A recap, an expired deadline, or a report of an earlier cycle isn’t. A page with no date at all can still alert.

If a watch is about something slow-moving, raise its news window with maxEventAgeDays. If it held something back by mistake, mark that run 👎 → It missed something.

What the numbers mean

Term	Meaning
True positive	Fired, and it was right
False positive	Fired, and it was wrong — noise
False negative	Did not fire, and it should have — a miss
True negative	Did not fire, correctly
Precision	Of the alerts you got, how many were right
Recall	Of the things worth alerting, how many you got
F1	A single number balancing the two

curl -s https://<your-host>/feedback/stats -H "authorization: Bearer $TOKEN"

The response includes recent false positives and false negatives, so you can see what went wrong rather than only how often, and splits precision between confirmed and unconfirmed alerts. Each watch also shows its own numbers on its page (GET /watches/:id/feedback/stats). Labels on runs where nothing was judged — a collection failure, nothing new — are counted separately: they say something about your sources, not about the decision.

Reading the result

Pattern	Likely cause	Where to go
Low precision (noisy)	Condition too loose, or broad search sources	Say why on the wrong alerts; Tune precision
Low recall (missing things)	Thresholds too high, sources too narrow, or extraction failing	Mark the misses; Tune precision, Sources return nothing
Everything arrives unconfirmed	Sources are not graded	Use “Should have been confirmed”; Manage the trust registry
Both look fine but you are unhappy	The watch is answering a slightly different question	Add a rule, or rewrite the condition

Note

With a handful of labels these numbers are noisy. Twenty or thirty labels across a couple of weeks is where they start being worth acting on.

Collection health, separately

GET /collection/stats — and the panel’s collection dashboard — reports attempted, collected, extracted and an extraction rate across recent runs.

It answers a different question from accuracy: not “did it judge well?” but “did it have anything to judge?”. A low extraction rate means pages are being fetched and not read, and no amount of threshold tuning will help.

Feeding the harness

Labelled runs can be exported as draft cases for the repository’s accuracy harness:

pnpm eval:export

They land in a staging directory for review. A draft uses the text the evaluator actually saw (kept for about 30 days), carries the watch’s current rules, and is flagged when the label was given on an earlier version of the watch. Once accepted, drafts become regression cases. See The evaluation harness.