Failures and retries

Watcher assumes things break. The design question at every layer is the same: can this failure lose you an alert? Where it could, there is a retry; where it cannot, the failure is tolerated and recorded.

The summary table

What broke	What happens	Do you lose an alert?
One source	Skipped, error recorded, other sources continue	No
Every source	Run ends collection-failed, health alert sent, watch stays due	No — retried next opportunity
Page extraction	Falls through to the next extraction tier	No
The model returned unparseable output	Retried, then failed over, then treated as a non-fatal transient	No — re-evaluated next cycle
A model key hit its rate limit	The key cools down and the pool rotates to another	No
A model key is invalid	Marked dead for the process, next key is used	No
The primary model is down	The second-opinion model answers instead, capped at the weak tier	No
The embedding service is down	Semantic deduplication is skipped for that run	No — but a duplicate may slip through
One delivery channel	Recorded; the run still counts as notified if another channel worked	No
Every delivery channel	Alert is stored and replayed on the next run	No — see below
The whole run crashed	Logged, health alert sent, other watches continue	No — state advances only at persist points

Undelivered alerts are retried

This is the guarantee worth stating precisely, because a monitoring system that silently drops the one alert you cared about is worse than useless.

When an alert passes every check but every configured channel fails to deliver it:

1. The run ends with the notify-failed outcome.

2. The formatted alert — summary, evidence quote, evidence URL, date, confidence, tier, and the event’s embedding — is written to a durable pending-delivery queue on the watch’s stored state, along with an attempt count.

3. The event is not marked as notified, so it cannot be considered “already alerted” while it is still owed to you.

4. On the next run of that watch, before any collecting happens, the queue is replayed on the watch’s current channels. No source is fetched and no model call is spent — the stored verdict is sent verbatim.

5. On success, the alert is finally recorded as notified: its fingerprint, its embedding and its ledger entry all advance exactly as a first-time delivery would, so it will not be sent twice.

6. On failure, the attempt count increases and the alert stays queued for the run after that.

The limits

Limit	Value	Why
Maximum attempts per alert	5 (the original failed send counts as the first)	Bounded, so a permanently broken channel cannot retry forever
Maximum queued alerts per watch	10 (oldest dropped first)	Bounded state
Model cost of a retry	zero	The verdict is replayed, never recomputed

Two things to know

• If a watch has no notifier wired at all for its channels, nothing is attempted and the queue simply waits — the attempt count does not increase.
• When an alert is given up on after five attempts, it is dropped silently: there is no separate health alert for it today. If a channel is broken, you find out from the repeated notify-failed outcomes in run history, not from a message. Watch for that outcome.

This guarantee is enforced by tests

The retry path is covered end to end, including across a run boundary through the real database adapter — the case where the queue must survive the process exiting between runs. See Testing.

Model failures

Watcher distinguishes two kinds:

“The model could not produce a valid answer.” Free-tier models occasionally emit malformed structured output. The evaluator re-asks a couple of times, then the cascade fails over to the other family. If nothing produces a verdict, the run is reported as evaluation unavailable — logged and health-alerted, but not treated as a hard failure, because no state was persisted and the next cycle will simply try again.

“The key is exhausted or invalid.” This is a real failure worth waking up for. Rate-limited keys are put on a cooldown (using the provider’s own retry hint when it sends one, otherwise 60 seconds) and the pool rotates. Invalid keys are marked dead for the life of the process. When a pool runs out of usable keys, the run fails loudly.

Collection failures

A single failing source is not an error — it is expected, and the reason to give important watches more than one source. The engine records what went wrong per source and continues.

Only when every source fails does the run end as collection-failed. In that case the last-run timestamp is deliberately not written, so the watch stays due and retries at the very next opportunity instead of waiting for its next cron slot.

Crash and restart

Watcher keeps nothing important in memory between runs. Everything it needs is in the database, so a crashed process, a killed container or a restarted runner simply picks up at the next run.

The one state that is lost on restart is per-process key health: a key that was cooling down will be tried again after a restart. That is harmless — it either works or it cools down again.

One narrow window

State advances only at explicit persist points. If a process is killed in the instant between a successful delivery and the write that records it, the next run can re-alert that event. The window is a single database write, and no test covers it because it cannot be induced deterministically — but it is honest to say it exists.

Going deeper

• Run outcomes — the vocabulary above, defined.
• Alerts are not arriving — the diagnosis path.
• Model and quota errors — key pools in practice.