Skip to content
Watcher Docs

Decision rules

This is the page to read when an alert did something you did not expect.

How an alert tier is decided A triggered verdict becomes a confirmed alert only if all six checks pass: confidence at or above the threshold, enough corroborating sources, a grounded quote, a trusted or official source, agreement from the second-opinion model, and not having come from the failover model. If any check fails, the verdict falls to the weak tier when a weak threshold is configured, or is dropped. Triggered verdict confidence 0–1 ALL SIX MUST PASS 1 · confidence ≥ threshold 2 · enough sources corroborate 3 · quote found in collected text 4 · source is official or trusted 5 · second opinion agrees 6 · not a failover-only verdict Checks 3 and 4 are the ones that most often demote a confident verdict to the weak tier. strong — confirmed alert all six passed weak — "unconfirmed" a check failed, or confidence sits between weakThreshold and threshold none — nothing is sent below weakThreshold, or no weak tier
A confirmed alert requires all six checks to pass.

A confirmed (strong) alert requires all of

Section titled “A confirmed (strong) alert requires all of”
# Check Fails when Result
1 confidence >= threshold The model was less sure than your bar Weak, or nothing
2 Enough distinct sources contributed evidence You set minSources: 2 and only one source had it Weak
3 The quote is grounded in the collected text The quote is absent, empty, or a paraphrase Weak
4 The corroborating source is official or trusted The source is unknown, low, or ungraded Weak
5 The second-opinion model agreed The two models disagreed Weak
6 The verdict did not come from the failover model alone The primary model was down and the fallback answered Weak

If all six pass → strong. If any fails → weak, provided weakThreshold is set and the confidence reaches it. Otherwise → none, and the run records below-threshold.

Check 3 — grounding, in detail

Section titled “Check 3 — grounding, in detail”

The model must copy the sentence that proves the condition. Watcher then looks for that sentence in the text it collected this run.

  • Matching is done after normalising HTML entities, Unicode forms, smart punctuation, case and whitespace — real pages mix all of these freely.
  • A near-verbatim quote (the model dropped or joined a word) still counts, via a bounded word-pair overlap check that requires at least 80% of the quote’s adjacent word pairs to appear, and only applies to quotes of five words or more.
  • A short quote, a paraphrase, or no quote at all is not grounded.

Check 4 — trust, in detail

Section titled “Check 4 — trust, in detail”

The tier is resolved per collected item from your trust registry merged over the built-in one, with an explicit trust on the source winning where it applies.

  • official or trusted → may confirm.
  • unknown or low → capped at weak.
  • Nothing graded it → capped at weak by default. Set allowUnratedStrong: true on a watch to opt out.

When several sources support the same event at different tiers, the evidence link in your alert is re-pointed at the most authoritative one.

Check 5 — the second opinion

Section titled “Check 5 — the second opinion”

Every triggered verdict is verified by a model from a different family. Negative verdicts are only re-checked when they are borderline (confidence between 0.4 and 0.85); clear and confident negatives skip the call to save quota.

  • Both models trigger → confidences are averaged, alert proceeds.
  • They disagree → the firing verdict is carried forward but marked disputed, so it can only reach the weak tier.
  • Neither triggers → no alert.

If no second-opinion model is configured, this check is not enforced and Watcher runs single-model.

Check 6 — failover verdicts

Section titled “Check 6 — failover verdicts”

If the primary model is unavailable and a secondary answers instead, the verdict is flagged. A single unverified answer from a weaker model may not fire at full strength — otherwise the same watch would flip between tiers depending on which provider had quota that hour.

A watch can forbid the secondary model entirely with allowSecondaryModel: false, which disables both the second opinion and the failover for that watch. Use it for watches over private or sensitive sources.

Deduplication happens around the gate

Section titled “Deduplication happens around the gate”

Before tiering, an event whose identity matches one already alerted on exits as already-notified. After tiering, a semantically near-identical event exits as semantic-duplicate. See Deduplication.

Reading the decision after the fact

Section titled “Reading the decision after the fact”

Every run record stores the model’s reasoning, the evidence quote and URL, the models consulted, and the highest corroborating trust tier. Open a run in the panel — or GET /watches/:id/runs — and the demotion usually explains itself.