Model and quota errors

How Watcher classifies a provider error

Every call through a key pool is classified, and the classification decides what happens next:

Class	Recognised from	Reaction
Rate limited	HTTP 429, quota or “rate limit” wording, provider plan-limit statuses	Key cools down (using the provider’s retry hint, else 60 seconds) and the pool rotates to the next key
Auth	HTTP 401/403, “invalid api key” wording	Key is marked dead for the life of the process; the next key is tried
Transient	5xx, network errors, timeouts	One quick retry on the same key, then a short cooldown and rotate
Other	Anything else, e.g. a content or validation error	Surfaced immediately — it is not a key problem, so rotating would not help

When no usable key remains, the pool raises an “all keys unavailable” error and the run fails loudly. That is a real failure worth acting on.

Symptom: runs fail with “all API keys are unavailable”

Every key in that pool is either cooling down or benched.

1. Check which. GET /admin/keys (admin role) shows per-key status — active, cooldown with a deadline, or dead — and today’s metered usage against the configured limit.
2. If everything is cooldown: you are simply out of quota for now. Add a second key, spread watches across the day, or reduce how often the instance runs.
3. If keys are dead: they failed authentication. Fix the key at the provider, then use the re-validate action — a successful probe revives the key everywhere it is pooled.

Note

Key health is per process. After a restart, a cooling key is tried again. That is harmless: it either works or it cools down again.

Symptom: “evaluation unavailable” in the logs

The model returned output that could not be parsed into a verdict — free-tier models do this occasionally. Watcher already retried, then failed over to the other model family.

This is reported as non-fatal: the run does not fail the whole scheduled pass, no state was persisted, and the watch simply re-evaluates next cycle. You get a health alert so it is visible, not silent.

If it happens constantly, the configured model for that category is a poor fit for structured output — switch the category’s model to another allowed option.

Symptom: alerts suddenly all say “unconfirmed”

Check whether the primary model is failing. When the primary is unavailable and the second-opinion model answers instead, the verdict is flagged as failover and capped at the weak tier by design.

The run record lists the models consulted, so a run showing only the secondary model explains the whole pattern.

Symptom: no second opinion is happening

If no second-opinion key is configured, Watcher logs that it is running primary-only at startup and every trigger goes unverified by a second family. Adding a key from a different provider materially improves verification — it is the single highest-value key to add after the primary.

Adding and rotating keys

Keys can come from environment variables or from the encrypted store managed through the admin API and panel. Stored keys are:

• probed live before being saved — a broken key is rejected with a reason, and a valid-but-rate-limited key is saved on cooldown rather than rejected,
• encrypted at rest with a master key that lives only in the environment,
• never returned by the API — only masked previews.

See Models and keys.

Watching usage

Each key has a daily usage ledger: requests, successes, rate limits, auth failures and token totals, with a 30-day series per key. It is locally metered — an estimate from Watcher’s own calls, not a bill from the provider — but it is enough to see which pool is under pressure before it runs dry.