Add a collector

A collector turns a source into candidate items. Adding one does not touch the processing loop.

The contract

interface Collector {
  collect(source: Source): Promise<Candidate[]>
}

A Candidate is a URL, the readable text found there, when it was fetched, an optional title, and a trust tier.

Your collector must:

• Return clean text, not HTML. Downstream is a language model and a substring check for grounding; markup poisons both.
• Set a real URL per item. It becomes the evidence link and part of the event’s identity, and the loop verifies that a model’s cited URL was actually collected.
• Resolve trust from the source, so items can be graded.
• Validate its own output and skip malformed items rather than throwing.
• Throw on a genuine failure — the loop records it and continues with the other sources.
• Be injectable. Take the fetcher and the clock as options so tests need no network.

Steps

1. Decide whether you need a new source type. If your source is really “a URL that returns text”, the existing page collector plus a hint may be enough. A new type is a schema change, and every existing watch keeps working only because the type is a closed set.

2. Add the type to the source-type list in the domain models, if you do need one.

3. Write the adapter in src/adapters/collectors/. Follow the existing shape: an options object with injectable dependencies, sensible defaults, and schema-validated output.

   export class MySourceCollector implements Collector {
     constructor(private readonly options: MySourceCollectorOptions = {}) {}
   
     async collect(source: Source): Promise<Candidate[]> {
       const fetchedAt = (this.options.now ?? (() => new Date()))()
       const trust = resolveSourceTrust(source)
       const raw = await (this.options.fetchText ?? defaultFetchText)(source.value)
   
       return raw.items
         .map((item) =>
           CandidateSchema.safeParse({
             url: item.url,
             fetchedAt,
             title: item.title,
             text: item.text.trim(),
             trust,
           }),
         )
         .filter((parsed) => parsed.success)
         .map((parsed) => parsed.data)
     }
   }

4. Use the shared HTTP helper rather than calling fetch directly. It gives you a bounded timeout and error messages that name the underlying cause.

5. Register it in entrypoints/wiring.ts, keyed by source type.

6. Write the tests — see below.

7. Document it: add the type to Sources and the watch schema, including any hints it reads.

Tests to write

Test	Why
Happy path against a fixture	Proves parsing and mapping
Malformed item in the middle	Must be skipped, not fatal
Transport failure	Must throw so the loop records it
Trust resolution	An explicit trust on the source is honoured
Any hint your collector reads	Proves the lever is wired

Use an injected fake fetcher and a fixture file. No test may touch the network.

Watch the text quality

The most common bug in a new collector is text that technically parses but is useless — navigation, cookie banners, or an empty shell. The loop counts an item as “extracted” only above 200 characters, and a thin item cannot ground a quote, so it will silently never fire. Print what your collector returns for a real page before you trust it.

What you do not have to do

• Deduplicate — the loop hashes content for you.
• Filter by date — that is the condition’s job, and search sources have a recency hint.
• Retry — the loop tolerates a failed source and the watch stays due when all of them fail.
• Rate-limit — if your source needs a key, put it behind a key pool and get rotation for free.