Skip to content
Watcher Docs

Search and extraction

Search (Tavily)

Section titled “Search (Tavily)”
.env
TAVILY_API_KEY=<your-key>

Powers two things:

  1. search sources — Watcher runs your query and turns results into candidate items. Without a key, a search source fails at collect time.
  2. The preferred page-extraction tier — a clean parse of a known URL. Without a key, page sources fall back to fetching the HTML themselves, which is usually fine and occasionally worse.

Per-source levers: officialDomains, recencyDays, searchDepth, topic. See Sources.

Extraction (the reader tier)

Section titled “Extraction (the reader tier)”
.env
JINA_API_KEY=<optional>

The reader tier renders JavaScript-heavy pages into clean text. It is:

  • opt-in per source — hints: { reader: true } — because it draws on a shared free tier,
  • last in the ladder — only used when the tiers above returned too little,
  • usable without a key. A key only raises the rate limit, and if a keyed pool is exhausted Watcher falls back to the keyless tier rather than failing.

The full ladder

Section titled “The full ladder”
How a page becomes text Page extraction is a ladder of four tiers, tried in order and only when the previous tier returned less than 200 characters: a clean extract service, article extraction from the fetched HTML, a whole-page tag strip, and finally an opt-in reader tier that renders JavaScript-heavy pages. 1 · Extract API clean parse, needs a key 2 · Article parse readability on the HTML 3 · Strip tags saves notice boards 4 · Reader tier opt-in: hints.reader under 200 chars? under 200 chars? under 200 chars? Result: one candidate item — a URL plus its readable text A tier is only adopted if it produced more text than the tier before it. Watcher does not run a browser. A page that builds its content entirely client-side may need tier 4 — or a different source.
Each tier runs only if the previous one returned under 200 characters of text.

In words: Watcher first asks the extract service for a clean parse; failing that it fetches the page and runs article extraction; failing that it strips the whole page to text (which is what makes notice boards work); and finally, if the source opted in, it asks the reader tier to render the page.

Choosing what to configure

Section titled “Choosing what to configure”
You mostly watch Configure
Feeds and simple article pages Nothing extra — plain fetching is enough
Government or institutional portals Search (for search sources and cleaner extraction)
JavaScript-heavy dashboards or SPAs The reader tier, enabled per source
“Tell me wherever this appears” Search, pinned with officialDomains

Watching extraction health

Section titled “Watching extraction health”

GET /collection/stats reports an extraction rate: the share of collected items that yielded substantive text. A persistently low rate means Watcher is fetching pages it cannot read — the moment to enable the reader tier or find a better source. See Sources return nothing.