Skip to content
Watcher Docs

Sources

A source tells Watcher where to look. Each run asks every source for candidate items, and each item is a URL plus the readable text found there.

The three types

Section titled “The three types”
- type: rss
  value: "https://nodejs.org/en/feed/blog.xml"
  trust: official

Fetches the feed and turns every entry with a link into one item. Cheapest and most reliable source type — use it whenever a feed exists.

Feeds that publish only a one-line summary can be told to fetch the linked article as well:

  hints: { followLink: true }

How a page becomes text

Section titled “How a page becomes text”

page sources (and rss sources with followLink) go through an extraction ladder. Each tier is only used if the previous one returned too little — the bar is 200 characters of text:

  1. Tavily Extract — a clean parse of a known URL. Used first when a Tavily key is configured, unless the source sets hints: { extract: false }.
  2. Article extraction — a readability-style parse of the fetched HTML.
  3. Whole-page strip — tags removed, whitespace collapsed. This is what saves listing and notice-board pages, which have no “article” to extract.
  4. Jina Reader — renders JavaScript-heavy pages into clean markdown. Opt-in per source with hints: { reader: true }, because it consumes a shared free tier.

Collection hints

Section titled “Collection hints”

Every hint is optional, and a source without hints behaves exactly as it always has.

Hint Applies to What it does
officialDomains search Restricts results to these domains, and marks them as official
recencyDays search Only consider results from the last N days
searchDepth search basic or advanced (default advanced)
topic search general or news
extract page Set false to skip the Tavily Extract tier
reader page, rss+followLink Enable the Jina Reader tier
followLink rss Fetch each entry’s linked article for fuller text
legacyTls page Allow legacy TLS renegotiation for old portals

Failure is per source, not per watch

Section titled “Failure is per source, not per watch”

If one source throws — a 404, a timeout, a TLS error — Watcher records the error, skips that source, and continues with the others. The run only fails outright if every source failed, which produces the collection-failed outcome and a health alert.

That is why pairing an official page with a search source is a good habit: when the portal is down, the search still sees the news.

Trust travels with the source

Section titled “Trust travels with the source”

Each collected item carries a trust tier, resolved from the source and from the item’s own host. It decides whether an alert may be confirmed or only “unconfirmed”. See Source trust.

Going deeper

Section titled “Going deeper”