The hidden cost of permission-specific caches

Identify what the cache is saving

A system can cache an embedding for a question, a ranked list of document identifiers, selected passages or a completed answer. These objects have different risks. A question embedding may contain no source text, while a completed answer can include details from several restricted documents.

Choose the layer deliberately. Caching a lower-cost intermediate result may deliver a smaller saving but make access checks easier to retain. Caching an entire answer saves more model work, but it also requires enough provenance to decide whether every source remains available to the next reader.

Write down the invalidation events for the chosen object. Content updates, access changes, model changes and prompt changes may each make an entry unsuitable. A cache with no explicit invalidation story is likely to become a source of old answers.

Compare three key designs

A key based only on the question is simple and produces frequent hits. It is unsafe for answers that vary by organisation or access rights. Adding the organisation identifier prevents one class of cross-customer reuse, but it does not separate departments with different permissions inside the same organisation.

A user-specific key is easier to reason about, though it reduces sharing. It still needs to account for that user's access changing. A key based on a verified permission-set version can allow some sharing between users with equivalent access, but the application must maintain that version correctly.

Here is a conceptual key for a protected answer. The values describe the inputs to the decision, rather than prescribing one cache product.

organisation + permission-set version
+ normalised question + source collection version
+ prompt version + model configuration

Do not place raw access lists or confidential question text into an openly readable cache key. Use appropriate identifiers and keep the cache's own access controls in scope.

Estimate the benefit with real access patterns

Suppose a team asks 10,000 questions a day and 2,000 repeat an earlier question. That does not imply 2,000 reusable answers. Some repeated questions come from people with different rights, and some refer to documents that changed between requests.

Replay a representative, appropriately protected sample against the proposed key. Count hits that are valid after access and freshness checks. Measure the lookup cost as well as the model work avoided. A complex cache that misses almost every time can add latency without reducing much spend.

Look at the distribution by team and document collection. A public policy collection may support broad reuse, while client-specific advice may not. Different cache policies can be simpler than forcing both workloads through the most complicated shared design.

Make misses an ordinary outcome

When a cache entry cannot be validated, recompute the answer within the current access boundary or explain that the source is unavailable. Do not fall back to the stale answer because a dependency is slow.

Set a capacity limit and an eviction policy so unique questions cannot grow the cache indefinitely. Monitor hit rate together with invalidation failures and the cost of maintaining provenance. The useful measure is the cost of a correct, authorised answer, not the highest possible cache-hit percentage.