Refresh urgent policies more often than reference material

Classify the consequence of delay

An engineering handbook updated twice a year and a frequently changing service notice have different needs. Polling both every minute wastes work on one without necessarily making the other dependable. Begin with source owners and identify which changes need rapid visibility, which can wait and which require immediate withdrawal controls.

Keep this classification small enough to operate. A few clear service classes are easier to maintain than a separate schedule for every document. Each class should define its expected freshness and what the application does when that expectation is missed.

Do not confuse publication frequency with consequence. A rarely changed emergency procedure may still need urgent removal when a defect is found.

Find the expensive stage

Measure source reads, extraction, embedding generation, index writes and reconciliation separately. A metadata-only change may not require recomputing the text embedding, while a parser change may affect every passage even if the source file is unchanged.

Use content and configuration versions to decide which work can be reused. Reuse is only valid when the relevant inputs match. A cache keyed solely by file identity can miss changes to extraction rules or permissions.

Large documents may produce many passages, so source count alone is a poor capacity estimate. Track processed bytes or tokens and resulting passage counts. A batch of ten complex reports can be more expensive than thousands of short records.

Reserve room for catch-up

A pipeline running at its maximum sustainable rate cannot quickly recover from an outage. Estimate the backlog that a plausible interruption creates and the spare capacity needed to clear it while new changes continue arriving.

For an illustrative workload, if normal processing consumes most available capacity, doubling the backlog does not imply it will clear in one normal processing interval. Only capacity above the incoming rate reduces the queue. Use observed throughput and variation rather than assuming every document costs the same.

Prioritise urgent withdrawals and access changes without starving ordinary updates indefinitely. Separate queues or weighted scheduling can help, but measure oldest-item age in each class to detect neglected work.

Compare total service cost

Include source API limits, retry traffic, duplicate processing and operator time. A cheaper polling interval may create more stale-answer incidents and manual reprocessing. Conversely, expensive continuous refresh may add little value to stable reference material.

Review the classification after observing real changes. Move a source to a faster class when evidence justifies it, and simplify overly aggressive schedules that produce no useful benefit. The aim is a predictable freshness service with enough recovery capacity, rather than the shortest interval the scheduler permits.