Deduplication

A single announcement can appear on an official page, in three news write-ups and in a search result — and the same page can be re-fetched every hour for a week. Watcher is built so that all of that produces one alert.

Three mechanisms do the work, at three different stages.

1. Content memory — “have I read this already?”

Before anything is sent to a model, every collected item is hashed on its URL plus its normalised text. Items whose hash the watch has already seen are dropped.

That means:

• an unchanged page costs nothing after the first run,
• a page that genuinely changed produces a new item,
• and trivial churn — a re-encoded character, reflowed whitespace, curly quotes swapped for straight ones — does not count as a change.

If nothing survives this stage the run ends with nothing-new, which is the most common outcome on a healthy system.

Seen content is never re-read

Fresh items are marked as seen whatever the verdict. If you change a condition afterwards, Watcher will not re-examine the content it already consumed — only genuinely new content is judged against the new condition. See What Watcher remembers for how to force a re-check.

2. Event identity — “have I told you about this already?”

When a verdict does trigger, Watcher computes a stable fingerprint for the event, from the evidence URL plus the normalised quote and date — not from the model’s free-text summary.

This distinction matters in both directions:

• The same event summarised differently on two runs keeps the same fingerprint, so it is not alerted twice.
• Two different announcements posted at the same URL — a notice board that gets a new row — get different fingerprints, so both fire.

A fingerprint you have already been alerted about produces the already-notified outcome.

3. Semantic similarity — “is this the same story, reworded?”

Exact fingerprints cannot catch five different sites describing one event in five different sentences. So before delivering, Watcher embeds a canonical description of the event and compares it against recent alerts for the same watch. If the similarity is at or above 0.92, the alert is suppressed as a semantic-duplicate.

Watcher keeps the last 50 notified-event vectors per watch for this comparison.

Note

Semantic deduplication is best-effort by design. If the embedding service is unavailable, Watcher skips this check for that run rather than blocking a real alert — so a duplicate may occasionally slip through when the provider is down.

What this means in practice

You observe	Why
First run alerts, later runs are quiet	Content memory: nothing is new
A notice board fires twice in a day	Two distinct events at one URL — correct behaviour
Five sites covered one story, you got one alert	Semantic deduplication
You edited the condition and nothing happened	Seen content is not re-read
An alert repeated once after an outage	A rare duplicate when embedding was unavailable

Going deeper

• Run outcomes — nothing-new, already-notified, semantic-duplicate in context.
• Duplicate or noisy alerts — what to change.