Accuracy

Asking a language model “did this happen?” is easy. Trusting the answer is the hard part. Watcher’s design assumes the model will sometimes be confidently wrong, and puts checks around it that do not depend on the model’s own judgement.

The four checks

1. The quote must be real

When the model triggers, it must copy the exact sentence that proves it. Watcher then checks that sentence against the text it actually collected. A quote that is not there — a paraphrase, an invention, or nothing at all — means the verdict cannot fire a confirmed alert.

The check tolerates real-world messiness: HTML entities, curly quotes, non-breaking spaces and reflowed whitespace are normalised first, and a near-verbatim quote that drops a word is still accepted through a bounded fuzzy match. A genuine paraphrase is not.

2. The link must be one Watcher fetched

The evidence URL must be a URL that was actually collected in that run. A fabricated link is either repaired to the real page that carries the quote, or dropped — it is never delivered, and never used as an event identity.

3. A second model, from a different family, must agree

Every triggered verdict is checked by a second-opinion model from a different family. Confident negatives skip the extra call; triggers never do, because a confident-but-wrong trigger is exactly the error class worth paying to catch.

• Both agree it fired → the alert proceeds, with the two confidences averaged.
• They disagree → the alert still reaches you, but capped at the weak tier and labelled unconfirmed.

If no second-opinion model is configured, Watcher runs single-model and says so in the logs. Adding one materially improves verification.

4. The source must be credible

An item from a site Watcher cannot grade is capped at the weak tier, no matter how confident the model is. See Source trust.

Prompt injection

Collected pages are untrusted input. The model is instructed to treat all page text as data, never as instructions, and to ignore anything inside a page claiming to be “confirmed”, “official”, or telling it to trigger. The quote and trust checks are the real defence: a page can lie in its text, but it cannot make an ungraded domain look official, and it cannot fabricate a quote into a page Watcher fetched from somewhere else.

Measuring it on your own watches

Watcher ships a feedback loop instead of asking you to trust a benchmark:

1. Label alerts 👍 or 👎 in the panel or the app.
2. Read the rolling precision, recall and F1 on the accuracy dashboard, with the recent false positives and false negatives listed.
3. Adjust thresholds, conditions or trust rules and watch the numbers move.

See The feedback loop.

Where it can still be wrong

Honest limitations

• A JavaScript-rendered page can look empty. If extraction returns nothing, there is nothing to be right about — Watcher will quietly not fire.
• Polling has gaps. An event that appears and disappears between two checks is missed entirely.
• Confidence is not calibrated. A 0.8 from the model is not a probability; treat thresholds as dials to tune, not as statistics.
• The weak tier is deliberately noisy. That is its job — it exists so borderline evidence is visible rather than silently dropped.
• A single-model setup is weaker. Without a second-opinion key, check 3 above does not happen.

Going deeper

• Decision rules — the exact gate.
• Known limitations — the full list.