Building the Work Brain: A Glean Alternative in a Weekend

Summary#

An open-source alternative to Glean was built over 45,349 Jira and Confluence documents in a weekend. The local version costs nothing to run. The AWS deployment runs at €60 to €100 per month when the team needs shared access. Glean's floor for the same corpus sits at roughly €60,000 per year.

Source-adaptive chunking outperforms fixed-size chunking by a wide margin. Jira tickets are embedded whole, Confluence pages are split at 512-token boundaries with their page titles prepended, and GitHub PRs are decomposed into title-plus-description, review comments with diff hunks, and the full diff as a parent document.

Hybrid search with Reciprocal Rank Fusion (k=60, Cormack et al., 2009) is scale-invariant and consistently outperforms weighted score combination. A cross-encoder reranker on top delivers a measurable precision lift on every retrieval-quality benchmark tested.

pgvector is comfortable to roughly five million vectors before the workload justifies reaching for pgvectorscale. The commonly cited "200K limit" is incorrect. The real bottlenecks are memory and filtered-HNSW recall on low-selectivity payloads.

Every LLM-generated citation is validated before the response reaches the user. Published groundedness evaluations place the unsupported-claim rate in RAG outputs at 20 to 50% of cited sentences. Post-generation verification is the cheapest step that prevents hallucinated citations reaching users.

The retrieval problem, not a knowledge problem#

A product manager pinged me at 10:47pm on a Sunday asking for the rollback procedure on a production incident. I knew the document existed. I had written half of it. Finding it took me 22 minutes, because it was buried in a Confluence page titled "Release Notes Q3" that had been renamed twice since.

That was the moment the framing shifted. The issue was not that the knowledge was missing. The issue was that the retrieval layer we had, which is to say, Confluence search and some Slack bookmarks, could not find knowledge that existed, was indexed, and was correctly titled when written.

The corpus I was working with contained 39,446 Jira tickets across 8 projects, 5,903 Confluence pages across more than fifty spaces, and a Slack workspace with fourteen months of retention including a #general channel holding 31,000 messages. The architectural decisions that mattered most were buried in PR review comments, often as a reply to a reply, on a diff that had been force-pushed over a week later.

I do not put much stock in the McKinsey claim that people spend twenty percent of their week searching for information. My suspicion is that the true cost is higher, because most of that time does not register as searching. Engineers simply give up and re-derive the answer. I have watched teams rewrite the same forty-line retry handler three times in a year because nobody could find the original implementation.

Glean's published floor is roughly €60,000 per year at the 100-seat minimum, which sits on the wrong side of most finance directors' patience for an internal search tool. I gave myself a weekend, Friday evening through Sunday night, to see how close an open-source alternative could get with an AWS bill under €100 per month.

The result was closer than I expected, though not equivalent. The remainder of this post is the honest account: what worked, what I broke twice, and the architectural decisions I would defend in a post-mortem.

What the enterprise vendors are actually doing#

A caveat before the critique: I do not have access to Glean's source code. What follows is inferred from their public engineering posts, the behaviour of the demo tenant I had for three weeks, and a two-hour call with a Dashworks solutions engineer who described their chunking pipeline in sufficient detail to reconstruct the shape. If any specific vendor differs in implementation, I am happy to be corrected. The pattern, however, recurs in every enterprise RAG demo I have reviewed.

The first failure mode is document-level embedding. A fourteen-page Confluence runbook is passed whole through an embedding model with a 512-token context window. The model truncates. The resulting vector is, in a technical sense, mush. Ask the system for the rollback command for the payments service and it surfaces a ticket about rollback SLAs because the word "rollback" weighted the vector in that direction.

The second failure mode is single-retriever search. Most enterprise tools default to semantic-only retrieval, sometimes BM25-only with a reranker bolted on. Semantic search is useless when a user types an exact ticket ID such as TICKET-27997. There is no conceptual neighbour to a ticket ID; there is only the ticket. BM25 is useless when a user types "that ticket about the upstream timeout last quarter." Both retrievers are required, and the merging step is where most implementations go wrong.

The third mistake is subtler and only becomes visible after the second re-ingestion. A Jira ticket averages 180 to 220 tokens of structured fields plus a description. A Confluence runbook is 8,000 to 12,000 tokens of prose with tables and code blocks. A PR review thread is a sparse conversation where the signal lives in the diff hunks. Applying a uniform 500-token sliding window across all three produces chunks that begin mid-sentence inside a code fence. I know this because mine did, for the first six hours of the first re-ingestion.

The pipeline#

The architecture comprises four stages. Each one carries a decision that I would revisit today with the benefit of hindsight.

Stage 1: Source-adaptive ingestion#

The extractors live in ingest/sources/, organised as one Python module per source, each exposing an iter_documents() generator.

Jira was the most time-consuming source to wire up. Atlassian removed the legacy /rest/api/3/search endpoint on 31 October 2025 (announced October 2024 and progressively enforced from May 2025) and replaced it with POST /rest/api/3/search/jql using nextPageToken cursor pagination. There is also a separate /rest/api/3/search/approximate-count endpoint, because the paged response no longer returns total. Integrations that still pass startAt in 2026 have been silently returning zero tickets since last autumn. Dashboards continue to work while search quietly degrades, because the endpoint returns a valid empty page.

Confluence was worse. The API returns Atlassian Document Format (ADF), which is a nested JSON tree rather than markdown. I spent an hour writing my own ADF walker before discovering that atlassian-python-api's confluence.get_page_by_id(expand='body.view') returns rendered HTML. I then passed that output through markdownify. The conversion is imperfect; tables render inelegantly and code blocks lose their language hint. The output is nevertheless adequate for embedding.

Chunking differs per source because the sources themselves differ in structure and length.

Jira tickets are short. At p50 they contain 184 tokens, and at p95 they contain 612. This allows them to be embedded whole, with metadata (assignee, status, fix version, labels) stored as payload fields on the vector row rather than concatenated into the text. I initially tried concatenation. Retrieval quality degraded noticeably. The word "Closed" appears on roughly 60% of tickets and began polluting the vectors toward generic "completed work" rather than specific content.

Confluence pages are the long tail. I split them recursively on markdown headings first, then on 512-token boundaries with 64-token overlap inside sections that still overflow. Each chunk carries its parent page title prepended, separated by >. A chunk from a section titled "Rollback" inside a page titled "Deployment Runbook" therefore begins with Deployment Runbook > Rollback. This single change moved nDCG@5 on the internal evaluation set from 0.61 to 0.73. I did not expect the lift to be that large.

GitHub PRs are chunked three ways and all three representations are stored: the PR title plus description as a single document, each review comment paired with its diff hunk as an individual document, and the full diff as a parent document the generator can pull in for context expansion when a review-comment chunk is retrieved. The pattern, which favours small chunks for retrieval precision with parent linkage for context expansion at generation time, is borrowed from LlamaIndex's SentenceWindowNodeParser.

The embedding model choice consumed half a Saturday, for reasons described in the section on what I got wrong. The system shipped with nomic-embed-text-v1.5, at 768 dimensions and Matryoshka-trained, which allows truncation to 256d for fast candidate retrieval while retaining the full dimensions for reranking. An earlier draft of this post recommended bge-base-en-v1.5. That model still works, but it is a 2023 answer. The 2026 answer is an MRL-trained model that enables tiered retrieval by dimension.

Stage 2: Hybrid retrieval with Reciprocal Rank Fusion#

I have reviewed the retrieval code in three "production RAG" repositories that reached the Hacker News front page during 2025. All three made the same error. I made it too, for approximately four hours on the Saturday, before a half-remembered sentence from the Cormack paper prompted me to revisit the approach.

The naive approach combines semantic and keyword scores with a weighted average:

final_score = (semantic_score * 0.7) + (bm25_score * 0.3)

This combination is incorrect. Cosine similarity ranges from 0 to 1. BM25 (Robertson and Zaragoza's classic formulation) is unbounded and typically spans 0 to 25 for short queries. The two scales are not commensurable. The weights therefore carry no meaningful interpretation, and the results are worse than either retriever would deliver alone for most query types.

Reciprocal Rank Fusion resolves the issue:

def rrf_search(query, k=60):
    semantic_results = vector_search(query, limit=20)
    keyword_results = bm25_search(query, limit=20)

    scores = defaultdict(float)
    for rank, doc in enumerate(semantic_results, 1):
        scores[doc.id] += 1.0 / (k + rank)
    for rank, doc in enumerate(keyword_results, 1):
        scores[doc.id] += 1.0 / (k + rank)

    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

RRF is rank-based rather than score-based, which makes it scale-invariant. A cosine similarity, a BM25 score, and a reranker logit can be fused in a single operation and the result remains coherent. Cormack, Clarke, and Büttcher's 2009 paper demonstrated that it outperforms CombSUM and Borda count on TREC runs. In an offline evaluation against 200 hand-labelled queries drawn from a real engineering-support channel, switching from weighted sum to RRF moved MRR@10 from 0.54 to 0.61.

I settled on k=60 because that is the value used in the original paper. I tested 30 and 100. The difference between them lived in the third decimal. Further tuning is not warranted. OpenSearch, Elasticsearch, Qdrant, MariaDB, and LlamaIndex all default to k=60 for the same reason.

Stage 3: Contextual retrieval and reranking#

Retrieval returns the top 20 candidates. These are typically in the right neighbourhood but not precisely ordered, and strong candidates occasionally lack the surrounding context required for a non-expert reader to evaluate them.

Two techniques address the gap, applied in sequence.

Contextual embeddings represent the single largest RAG improvement published in the last eighteen months, and they are the most notable omission from the first version of the system I shipped. Anthropic's September 2024 contextual retrieval technique pre-pends a chunk-specific context blurb to each chunk before embedding. A typical blurb reads "This chunk is from the Deployment Runbook, describing the rollback procedure after a failed canary deployment." The blurb is generated by a cheap model and amortised through prompt caching down to approximately $1 per million document tokens. Anthropic's published evaluation showed contextual embeddings alone reduced retrieval failure rate by 35%. Contextual embeddings combined with contextual BM25 reduced it by 49%. The full stack including reranking reduced it by 67%. For a corpus of roughly 60 million tokens, the one-time re-embedding cost was approximately €60. Retrieval quality on my evaluation set improved by a wider margin than any other single change in the pipeline.

Cross-encoder reranking is the second technique. Unlike the bi-encoder that generated the embeddings, a cross-encoder takes the query and each candidate document together and produces a true relevance score. Inference runs once per candidate and is therefore more expensive, but twenty inferences is a trivial workload. I use FlashRank's ms-marco-MiniLM-L-12-v2, which occupies roughly 134 MB, is slower than the 4 MB Nano variant, and is stronger on long queries. It completes twenty candidates in under 100 ms on a single CPU core. The precision lift on the internal evaluation set sits between 7 and 12 points of nDCG@5, depending on query class. Practitioners should cite their own benchmarks rather than quoting a single figure. The published range across BEIR is wide, spanning 8% to 48% depending on baseline strength, reranker model, and candidate pool size, and a narrow claim will invite disagreement.

Stage 4: Generation with verified citations#

The retrieved chunks feed a Claude model with a specific prompt structure:

system_prompt = """You are a knowledge base assistant. Answer questions based ONLY on the provided source documents.

Rules:
- Cite sources using [1], [2], etc.
- If the sources don't contain enough information, say so honestly.
- Be concise and direct.
- Include specific details (ticket numbers, dates, names) from the sources."""

user_message = f"""Sources:
[1] {source_project}/{file_path}: {content}
[2] {source_project}/{file_path}: {content}
...

Question: {question}"""

Citations are validated before the response is returned. This step was almost omitted from the first release.

During the first end-to-end test I asked the chatbot which ticket covered the partner webhook retry fix. It answered confidently with a cited ticket ID, TICKET-14822 [1], and a clean summary. The ticket did not exist. The correct identifier was TICKET-14228. Claude had transposed two digits and produced a citation that pointed at chunk 1, which was a genuine chunk about a different ticket entirely. This is the canonical RAG failure mode, and it destroys user trust within the first week of operation.

Published groundedness evaluations (Liu et al., 2023; Gao et al., 2023; and more recent deep-research-agent audits) place the rate of unsupported or partially-supported claims in RAG outputs at 20 to 50% of cited sentences, depending on model and chunk quality. My informal failure rate on the first fifty queries was approximately 30%.

The fix is mechanical. For every [N] in the response, the raw text of chunk N is retrieved, and a cheap model is asked whether the chunk supports the claim, returning yes or no. I use Haiku for this role, at a verification cost of roughly €0.003 per answer. If the answer is no, the citation is stripped and a note is appended that supporting evidence was not found. In the first week of operation, the validator caught unsupported citations in 18% of responses. The same principle applies elsewhere in agent architectures: the pattern of validating agent tool calls against a known schema treats hallucination as a structural problem solved by verification, not by prompt engineering.

The vector database decision#

The vector-database debate is polarising, and most teams over-engineer it. Vendor documentation is confident and short; comparison threads on the internet are rarely either. The same database selection framework that applies to OLTP versus OLAP decisions applies here: choose the specialised tool only when the workload demands it.

The decision tree I use today is straightforward.

| Corpus size | Recommendation | Reasoning | |---|---|---| | 0 to 5M vectors | pgvector (PostgreSQL) | Already installed. HNSW works. Hybrid search via tsvector. Sub-50 ms queries comfortable across this range. | | 5M to 50M | pgvector + pgvectorscale | DiskANN-backed index, streaming graph storage, iterative index scans for filtered queries. Published benchmarks show 28x lower p95 latency than Pinecone s1 on similar workloads. | | 50M+ or regulated multi-tenant | Qdrant | Filterable-HNSW inside the index scan, first-class multi-tenancy, native dense plus sparse plus late-interaction fusion in a single query. |

For the Work Brain at 59,629 embeddings, pgvector is the correct choice. I do not plan to switch to Qdrant until the workload genuinely justifies it.

One note on a figure that appears in older commentary. The "200,000 document limit" for pgvector is incorrect. The genuine bottlenecks are RAM (HNSW must fit in memory) and filtered-HNSW recall collapse on low-selectivity payload filters. Both have been addressed by pgvector 0.8+ and by pgvectorscale. Numerous teams are running five million or more vectors on plain pgvector in production.

What it costs#

Running the system locally during development requires no infrastructure expenditure. Docker Compose on a laptop covers the entire stack. The Anthropic API for the chatbot adds roughly €5 to €10 per month at current usage levels.

The planned AWS deployment is as follows:

• Aurora Serverless v2 with pgvector at the 0.5 ACU floor: roughly €33 to €44 per month. Scale-to-zero is available in newer regions and reduces idle cost close to zero for bursty workloads. Storage is billed separately at approximately €0.09 per GB-month.
• S3 for the markdown source of truth: roughly €1 to €2 per month.
• Bedrock Titan Text Embeddings v2 at current pricing ($0.02 per one million input tokens, or roughly half that in batch mode): re-embedding the full 60 million-token corpus costs approximately €1.20, and incremental ingestion sits in the single-digit euros per month.
• Lambda plus API Gateway for the query API: €0, because the free tier covers this scale.
• Lightsail Container for the Outline wiki UI: €10 per month.
• Lightsail Managed DB for Outline metadata: €15 per month.
• MWAA for ingestion orchestration (already running elsewhere): no incremental cost.

The total sits between €60 and €100 per month.

Against Glean's floor of €60K per year, the ratio approaches 800x. That figure requires an honest caveat. Glean includes connectors, an administrative UI, a support organisation, and vendor accountability that a weekend build does not replicate. When procurement cycles cost more than the SaaS bill, Glean is the right purchase. Where they do not, the self-hosted alternative is often preferable, not because it is better in absolute terms, but because it is yours to control.

What I got wrong#

Every RAG article I read while building the Work Brain was glossy. This section is the counterweight.

The first setback consumed most of a Saturday afternoon and was caused by the embedding model choice. I had decided to use BGE-M3 because it topped the MTEB leaderboard, which is the kind of decision an engineer makes when reaching for the best tool before the right tool. BGE-M3 is a 4 GB model. On my M2 Max with 64 GB of unified memory, the first batch of 32 Confluence chunks attempted to allocate a 26.3 GB attention buffer, and Metal Performance Shaders crashed with MPSNDArray allocation failure. I spent 90 minutes investigating what I assumed was a PyTorch-nightly bug before batch_size=1 succeeded and revealed that the model was simply too large for MPS's memory architecture. GPU-class models require GPU-class hardware. I migrated to nomic-embed-text-v1.5 on CPU at approximately 120 chunks per second, and on the 200-query evaluation set the difference from BGE-M3 was within noise. Leaderboards measure something; that something is rarely the workload you actually have.

The second setback was genuinely embarrassing and concerned the Outline wiki. I wanted to backfill 45,349 markdown files into Outline quickly, and the API processes one document at a time at approximately three requests per second, a four-hour job. I decided to INSERT directly into Postgres on the grounds that the schema looked simple.

It was not simple. Outline tracks a sort field (a float used for ordering siblings), a urlId (a short random identifier that cannot be null), and per-document documentStructure JSON on the parent collection that must stay consistent with the document tree. I populated none of these fields. The ingestion reported success. Every collection page then threw Cannot read properties of null (reading 'sort') in the React sidebar, and the application became unusable. I spent Sunday morning writing a reconciliation script. The correct answer was the API the entire time, at four hours of wall-clock cost. Product invariants live in application code, not in database schemas. I shipped via the API on Sunday evening.

The third setback was smaller in terms of time lost but warrants inclusion because it will happen to anyone running the same stack on a laptop. When BGE-M3 ran out of memory during Saturday's experiment, the failure did not remain contained. It exceeded Docker Desktop's VM memory ceiling, the OOM killer terminated Postgres, and Outline lost its Redis connection pool, producing a cascading failure on a single laptop. docker compose down && docker compose up -d restored the stack, but the ingestion progress was lost. Adding mem_limit: 2g and oom_score_adj: 500 to the embedding service in docker-compose.yml would have confined the blast radius to one container. That is three lines of YAML. Resource limits belong on every container from the first commit. A pre-commit hook now rejects any compose service that lacks them.

What I am less sure about#

The architecture works. What I remain uncertain about is whether the gap I am aiming at constitutes a real market or simply my own frustration.

Glean raised at $7.2 billion in 2024 and has more than 600 customers. The capital is real, and the pain they address is real. A weekend project does not replace a hundred-person company.

What I think the build demonstrates, at least to me, is that the technical moat is thinner than the pricing implies. The Work Brain draws almost entirely on public research: pgvector, nomic-embed, FlashRank, Reciprocal Rank Fusion from a 2009 paper, and contextual retrieval from a 2024 Anthropic blog post. The difficult parts sit elsewhere. The integrations are non-trivial. The user experience is work. The permission model is unresolved; my version trusts the ingestion layer's access-control lists, and that assumption will not survive a real enterprise audit. The trust that comes with being a named vendor cannot be compressed into a weekend.

For a mid-market team looking at a €60,000 quote, the calculation is straightforward. Is it more economical to pay Glean, or to have one of the team's engineers spend two weekends and €75 per month? For most of the teams I speak with, the weekend version is the right call.

For a 5,000-person organisation where procurement cycles cost more than the SaaS bill, the calculation inverts. Glean is a good product. This post is not addressed to that buyer.

What comes next#

The Work Brain is one layer of a larger argument about modern data architecture patterns, which is that platforms should be built for the workloads they actually carry rather than the workloads a vendor pitch deck promised. The broader reference is my post on production RAG patterns. This article is the worked case study for a specific mid-market scale under a specific regulated-industry constraint, and it serves as the capstone of the AI delivery series where theory meets a concrete implementation.

A follow-up on the permission model is planned for roughly six weeks from now. That is the component I am currently stuck on. If you have solved row-level ACL propagation from Jira through to a vector index in a way that survives audit, I would genuinely welcome the conversation. I am two prototypes in and satisfied with neither.

This project sits at the intersection of several areas Emanuel Mallia works in, from data platform design to AI delivery to the observability that connects them. A fuller picture of that scope is on the expertise page.

The GitLab repository is public. If you are wrestling with the same problem, particularly in a regulated industry, I would rather hear from you directly than guess at what you need. The email address is in the footer.