Permission-Aware RAG: Propagating ACLs into a Vector Index

Summary#

The Work Brain post ended with an admission: the permission model was unresolved, the prototype trusted the ingestion layer's access-control lists, and that assumption would not survive a real audit. This is the follow-up I promised, later than the six weeks I estimated. It covers why the obvious approach fails on recall rather than on security, how Jira and Confluence permissions actually resolve, the three architectures that are defensible, and the staleness window that is the part auditors care about most. Part of the AI delivery series.

The problem is not what I expected it to be#

I assumed permission propagation was a plumbing problem. Read the ACLs at ingestion, attach them to the chunk, filter on them at query time. Two prototypes later, the plumbing turned out to be the easy half.

The hard half is that a permission filter is the worst possible shape of filter for a vector index, and the failure mode is silent.

Vector search over an HNSW graph is approximate by construction. You traverse a navigable small-world graph, you visit a bounded number of candidates, and you return the best you found. Apply the filter after that traversal and you are not filtering a result set, you are filtering a sample. Qdrant published the numbers: a broad-value post-filter drops recall to 90.8%, and an AND filter over two broad values drops it to 39.7%.

A permission filter is precisely that. Broad, because most users can see most of a corpus. Conjunctive, because access is usually project AND space AND security level. And different for every user, so you cannot warm a cache and move on.

At 39.7% recall the system does not error. It answers. It answers from four of the ten chunks that mattered, and the user cannot distinguish that from a corpus which genuinely lacked the answer. In a search box that is a poor experience. In a system that people use to make decisions, it is a system that quietly misleads them, and it does so more often for the users with the most restricted access, who are frequently the ones handling the most sensitive material.

What the source systems actually give you#

Before any of this can be modelled, the source permissions have to be read correctly, and the shape is more awkward than the API documentation suggests.

In Jira, issue-level security sits in a security scheme, separate from project permissions. Sub-tasks inherit the security level of their parent issue. If a security level exists on a project but is not set on a given issue, the project permission scheme applies instead. So the effective permission on one issue is a function of the issue, its parent, the project scheme, and the groups and roles the user belongs to.

In Confluence, page restrictions only behave as expected when the user already has read access to the space. A page restriction is a narrowing of space permission, not an independent grant. Restrictions also inherit down the page tree.

Two consequences follow, and both are structural rather than incidental.

The first is that permission is a graph, not a field. It resolves through hierarchy (sub-task to parent, page to ancestor to space) and through indirection (user to group to role to permission). Flattening that graph to a list of principals per document is possible, but the flattening is a computation you now own and must keep correct.

The second is that the graph changes independently of the content. A document can sit untouched for a year while the group that grants access to it is renamed, merged, or emptied. Any ingestion pipeline keyed on content changes will never notice. This is the source of most real-world leakage, and it is invisible in testing because test fixtures rarely model a permission change without a content change.

This is the same class of problem as lineage in a data platform, and it fails the same way: not at write time, but at the point where someone asks you to prove a historical state. I wrote about that pattern in the piece on data governance for engineers.

Three architectures that are defensible#

Denormalise the ACL into the vector payload#

Resolve each document's effective principals at ingestion, store them on the chunk as an indexed payload field, and filter on them during the index traversal rather than after it.

This is the only approach that keeps recall intact at interactive latency, because the filter participates in the graph walk. Qdrant builds additional edges at index time between points sharing a value in an indexed payload field, so a filtered query still has connected paths to follow. One operational detail that is easy to get wrong: those payload indexes must exist before the vectors are uploaded, otherwise the filter-aware links are never built and you are back to a slow scan.

The cost is staleness, and it is not a rounding error. A permission revoked at 09:15 remains effective in the index until the next sync completes. That window is a number you will have to say out loud in an audit, so it should be a number you chose deliberately rather than one that emerged from a cron schedule.

Denormalisation also multiplies. A document visible to four groups, chunked into thirty pieces, is one hundred and twenty principal entries. Across a corpus in the tens of thousands of documents this is manageable. It is worth calculating before committing rather than after.

Late-binding check against the source#

Retrieve without permission filtering, then verify each candidate chunk against the source system before it reaches the model.

This is always current, which is exactly what the denormalised approach is not. There is no staleness window to disclose because there is no cached copy of the decision.

It is also a network round trip per candidate, against an API with rate limits that were not designed for this access pattern. Retrieve fifty candidates and you have fifty permission checks in the critical path of a request a user is waiting on. And it reintroduces the recall problem through the back door: you filtered after retrieval, so the sample was already drawn from an unrestricted corpus.

Late binding is correct and slow. Used alone it does not work at interactive speed. Used as a second stage it is genuinely valuable, which is the point of the hybrid below.

Externalise the permission graph#

Model permissions as relationship tuples in a dedicated authorization store rather than flattening them per document.

This is the Zanzibar design, published by Google at USENIX ATC in 2019. Its relevant idea is the tuple-to-userset, which represents hierarchy as one tuple per hop rather than as a materialised list. That is a direct answer to the inheritance problem above: sub-task to parent, page to ancestor, user to group are all hops, and changing an inheritance rule does not require rewriting the tuples underneath it. The paper reports the system scaling to trillions of access control lists and millions of authorization requests per second, so the design is not the constraint.

The honest caveat is that adding an authorization service to a retrieval system is a significant increase in operational surface, and for a corpus of tens of thousands of documents it is difficult to justify on scale alone. It becomes justified when the permission graph is complex enough that flattening it is where your bugs live, which is a different threshold and one worth naming explicitly before you commit either way.

What I would build now#

Denormalised ACLs in the payload for retrieval, then a late-binding verification on the small set of chunks that actually reach the answer.

The reasoning is that the two failure modes are asymmetric. Denormalisation fails by being stale, which means occasionally retrieving something that should no longer be visible. Late binding fails by being slow. Applying late binding to fifty candidates is unaffordable; applying it to the five or six chunks that survive reranking costs one bounded round trip and closes the staleness window at the only point where it has consequences, which is the moment content is about to be shown to a person.

That leaves the staleness window open for retrieval scoring, where a stale ACL affects ranking but never exposes content. I think that trade is defensible. I would want to state it explicitly in a design document rather than let it be discovered, because the distinction between "stale permissions affected what was ranked" and "stale permissions affected what was shown" is exactly the distinction an auditor will probe.

I am not presenting this as solved. It is the architecture I would defend today, and the third prototype rather than a finished system.

The questions to design against#

Auditors do not ask which vector database you chose. In regulated environments the questions concentrate on three points, and it is worth building against them from the start because retrofitting the evidence is expensive.

Can you demonstrate that a specific user could not retrieve a specific document on a specific date? This requires retaining the effective permission state over time, not just the current one. Most systems store the current ACL and overwrite it.

What is the maximum window between a permission being revoked at source and that revocation taking effect in retrieval? This is one number. You either know it or you do not, and "it syncs nightly" is not an answer, because it omits failure and backlog.

Can you reproduce the access decision behind a generated answer, including which chunks were excluded? This is the one most implementations fail, because retrieval is logged as a similarity query rather than as an access decision. The fix is cheap if done early: log the principal set used, the filter applied, and the candidate identifiers admitted and rejected, against a request identifier such as TICKET-4417 that ties back to the answer shown.

None of these require a particular architecture. They require deciding, before the first index is built, that permission is a first-class part of the retrieval record rather than a property of the pipeline that produced it.

Where this leaves the Work Brain#

The original build trusted the ingestion layer's ACLs, which in practice meant it inherited whatever the connector happened to expose and had no answer for permission changes that were not accompanied by content changes. That is fine for a personal index over documents you can already read. It is not fine for anything shared, and I said as much at the time.

What changed in the intervening months is my view of where the difficulty sits. I expected the hard part to be reading permissions out of the source systems. The hard part is that permission filtering and approximate nearest-neighbour search are in tension by construction, and every architecture above is a different way of paying for that tension: in staleness, in latency, or in operational surface. There is no option that avoids paying.

The broader patterns behind this sit in the production RAG guide, and the retrieval architecture it builds on is described in the Work Brain case study.

If you have taken the third route and put a real authorization store behind a retrieval system in a regulated environment, I would like to hear how the operational cost landed against the alternative. That is the part I am least able to reason about from first principles, and the part where I would most rather learn from someone else's production experience than from my own.

Sources#

• Pang, Cáceres, Burrows et al., Zanzibar: Google's Consistent, Global Authorization System, USENIX ATC 2019
• Qdrant, Pre-Filtering vs Post-Filtering (and Why Qdrant Does Neither)
• Qdrant, A Complete Guide to Filtering in Vector Search
• Atlassian, Configuring issue-level security