Polling thousands of feeds is not primarily a parsing challenge. It is a scheduling and fairness problem constrained by remote servers you do not control. A naive worker that fetches every source every five minutes wastes bandwidth on dormant feeds, overloads shared hosts at the top of the minute, and repeats work when documents have not changed. A scalable poller learns source behavior, uses HTTP validators, and commits results idempotently.

The goal is not the shortest theoretical delay. It is dependable discovery with respectful network behavior and predictable cost. High-value active feeds can be checked frequently, quiet feeds can back off, and every source can receive eventual attention. When push notifications are available, polling remains the reconciliation channel that repairs missed events.

Store scheduling state per source

Each source needs a next-attempt time, last-attempt time, last-success time, last-change time, consecutive failure count, recent change intervals, ETag, Last-Modified value, and host key. Keep this state in durable storage so a worker restart does not reset every source to “fetch now.” A scheduler should claim due sources with a lease, allowing another worker to recover work if the first worker fails.

Separate source state from tenant subscriptions. Several customers may follow the same public feed, but the network fetch should usually be shared. Normalize one source snapshot, then apply tenant-specific filters and delivery rules downstream. This reduces load and prevents one popular feed from being fetched once per customer. Authorization still applies to private feeds, where credentials or network boundaries make sharing inappropriate.

Use conditional requests correctly

When a successful response provides an ETag or Last-Modified value, store it and send the corresponding conditional header on the next request. The server can respond that the representation has not changed, avoiding a full body transfer. The semantics of validators and conditions are defined in the HTTP conditional requests section of RFC 9110. Preserve the exact validator value rather than trying to interpret its internal format.

A not-modified response is a successful fetch. Update last-attempt and last-success, record latency, and schedule the next check without parsing. If the server stops sending validators, continue safely with full responses and compare a body hash. Do not send both validators blindly if your client library's precedence is unclear; test the resulting headers and document your policy. Redirects may lead to a resource with different validators, so bind validators to the resolved representation URL.

Adapt intervals to observed change

Begin with a conservative interval and adjust after collecting history. If a feed changes frequently, shorten the interval within configured limits. If it remains unchanged for many checks, lengthen the interval. Use a moving window or exponential smoothing so one burst does not permanently classify a source as active. Apply a minimum interval to protect publishers and a maximum interval that still meets your product's freshness promise.

Publisher hints can inform scheduling but should not override policy automatically. Cache headers, RSS scheduling elements, or documented update windows may be useful. Compare hints with observed behavior and cap them. A malicious or mistaken source should not be able to demand one request per second or delay checks for a year. The monitoring layer can expose both the configured interval and the reason it was chosen.

Add jitter and host-level fairness

Never align every source to round clock boundaries. Add random jitter when scheduling the next attempt so work spreads across time. Jitter should be stable enough to avoid chaotic rescheduling but large enough to break synchronized waves after a restart. Rebuild the due queue gradually during recovery instead of releasing the entire backlog at once.

Apply concurrency and request-rate limits per host or registrable domain. Thousands of feed URLs may be served by one publishing platform, CDN origin, or organization. A global worker limit does not prevent one host from receiving a burst. Use host queues, token buckets, or weighted fair scheduling so active domains cannot starve smaller ones. Respect explicit retry guidance and slow down hosts that return throttling or overload responses.

Handle failures with classified backoff

Not every failure deserves the same retry. A timeout or server error is usually transient and can use exponential backoff with jitter. A not-found response may indicate a permanent move or deleted feed and should be retried less often after checking redirects and discovery. An authorization failure should stop automated retries until credentials change. A parse failure can be retried on the normal cadence because the publisher may fix the document, but repeated identical invalid bodies do not need expensive reprocessing.

Keep a ceiling on backoff so recovery is eventually detected. Reset the failure counter only after a meaningful success. A not-modified response counts as success; a network response that cannot be parsed does not. Group failures by host and signature to identify infrastructure incidents. Circuit breakers can pause aggressive attempts to a failing host while allowing occasional probes.

Deduplicate at more than one layer

Network deduplication avoids parsing unchanged documents through validators or body hashes. Document deduplication records whether the normalized feed snapshot changed. Entry deduplication decides whether an item is new, updated, or already known. Delivery deduplication prevents the same item event from reaching a destination twice. Each layer needs its own key because one key cannot answer all four questions.

For entries, prefer a stable source-provided ID. Fall back to a canonical URL, then a documented fingerprint based on fields that are likely stable. Do not include the current fetch time. Store identity aliases when a publisher changes from one identifier style to another, and surface collisions for review. Compare a content fingerprint separately so an existing entry can be updated without being recreated.

Commit before fan-out

Parsing workers should write the source observation, snapshot, entries, and change events in a durable transaction or an equivalent atomic workflow. Only then should they enqueue downstream deliveries. If the worker crashes after commit but before enqueue, an outbox process can recover the pending events. If it crashes before commit, the source can be fetched again without creating inconsistent partial state.

Every delivery event should have a stable idempotency key. A destination retry then reuses the same key, and a consumer can safely reject duplicates. Keep source polling independent from slow destinations; otherwise one failing webhook can delay discovery for every subscriber. The delivery architecture should consume committed events from a queue and maintain its own retry schedule.

Operate the scheduler with measurable goals

Track due-source lag, fetch throughput, success rate, not-modified rate, bytes saved by conditional requests, parser time, new items per fetch, host throttling, and queue depth. Due-source lag—the difference between scheduled and actual start time—is a direct measure of whether the system is keeping up. Break it down by priority and host so a healthy global average does not hide a starved queue.

Set capacity plans using observed latency and change rates, not only source count. Ten thousand fast cached feeds can be cheaper than one thousand slow feeds with large bodies. Maintain a controlled load test with synthetic hosts that produce redirects, delays, compression, invalid documents, and changing validators. Test restart behavior and backlog recovery, because those moments create the largest accidental bursts.

Scaling checklist

  • Persist per-source schedule, validator, health, and observation state.
  • Share public-source fetches across tenant subscriptions where policy allows.
  • Use ETag and Last-Modified validators and treat not-modified as success.
  • Adapt intervals from change history within strict minimum and maximum bounds.
  • Add jitter, per-host concurrency limits, and fair scheduling.
  • Classify failures and apply policy-specific backoff with eventual probes.
  • Deduplicate documents, entries, change events, and deliveries independently.
  • Commit normalized state before fan-out and monitor due-source lag.

Scale by doing less unnecessary work

A scalable poller is not the one with the largest worker fleet. It is the one that avoids downloading unchanged bytes, avoids parsing identical documents, avoids fetching a shared source repeatedly, and avoids notifying consumers twice. HTTP already provides useful mechanisms; the system's job is to apply them consistently and measure the result.

Build fairness and recovery into the scheduler from the beginning. When the queue grows or a host fails, the poller should degrade gradually, preserve important sources, and recover without a burst. That operational behavior matters more than shaving a few seconds from the happy path.