Enterprise RAG Knowledge Base: Architecture and Cost

Why a plain LLM is not enough

Plain ChatGPT or Claude access carries a hallucination risk in 60-70% of enterprise questions. This is because the model does not know your company's latest contract, yesterday's ticket, or a procedure that was updated three weeks ago. The model is trained on general knowledge, not on your enterprise data.

There are three solution paths: fine-tuning, pasting all documents into the context window, and RAG. Fine-tuning is expensive and updates slowly, while pasting documents is impossible across thousands of pages. RAG sits between these two extremes. It indexes enterprise documents, retrieves the most relevant pieces for a question, and passes them to the model as context. The model then produces a response grounded in enterprise data.

In this article, I explain the seven layers of an enterprise RAG system, the technical decisions for each, KVKK compliance, and the real cost. Where to begin in the first 90 days, in what order to expand, and which pitfalls to avoid.

Layer 1: Document Ingestion

Which sources will you draw data from? A typical enterprise inventory:

• Structured: SharePoint, Confluence, Notion, Jira, ticket systems
• Semi-structured: PDF contracts, Word policy documents, email archives
• Conversational: Slack/Teams channels, customer call transcripts
• Database: CRM records, ERP modules, product catalogs

Each source requires a different connector. Microsoft Graph API for SharePoint, REST for Confluence, the Notion API for Notion, pypdf2 or unstructured.io for PDF, IMAP for email. Open-source ingestion frameworks (LlamaIndex, Haystack, Unstructured) provide most of these connectors out of the box.

Typical mistake: Saying "let's index everything" and loading 500 thousand documents. The result: too much noise, low quality, high cost. The right approach is to start with 5 to 10 thousand high-value documents first. FAQs, product documentation, policy documents that do not change often. Then expand based on usage metrics.

KVKK / PII filtering: An automatic PII/PHI scanner should run at the ingestion layer. National ID numbers, phone numbers, email addresses, and card numbers can be caught with regex patterns; names and addresses require spaCy plus a Turkish NER model. Documents that continuously contain personal data (for example, personnel files, patient reports) are placed in a separate bucket, and access control is applied much more strictly.

Layer 2: Chunking Strategy

A 50-page PDF cannot be vectorized directly. It is first split into chunks. Chunk size directly affects the quality of the result.

Three main strategies:

1. Fixed-size: Each chunk is 500 tokens with 20% overlap. The simplest, fastest to index. But there is a high risk of cutting off in the middle of a sentence.

2. Sentence / paragraph based (semantic-aware): Splits at natural boundaries. Sentence integrity is preserved with sentence-transformers. Better for SQL contracts and policy documents.

3. Hierarchical (parent-child): Search is performed with small chunks (200 tokens), and the large parent paragraph of the found chunk (1500 tokens) is passed to the model as context. High precision plus high context. LlamaIndex's HierarchicalNodeParser provides this out of the box.

Practical recommendation: Hierarchical for contracts and long policy documents. Sentence-based for FAQs and product descriptions. Fixed-size for Slack messages (already short).

Typical mistake: Splitting every document with the same strategy. If a 200-page ISO 27001 document and a 3-sentence FAQ are processed with the same strategy, one becomes too general and the other too fragmented.

Layer 3: Embedding Model Selection

Chunks are converted into vectors. An embedding model is evaluated across three main categories:

Practical recommendation for Turkish content: Cohere multilingual-v3, or BGE-M3 locally. OpenAI models support Turkish but miss fine distinctions in some domain-specific terms (legal, medical, engineering).

Domain-specific fine-tuning: If your enterprise terminology is very specialized (for example, a cardiology clinic, the defense industry, law firms), contrastive fine-tuning on BGE-M3 with 5 to 10 thousand Turkish domain examples can provide a 15-25% improvement in semantic search.

Cost calculation: 50 thousand documents x an average of 800 tokens = 40 million tokens. A one-time 800 USD with OpenAI text-embedding-3-small. This is only the initial index. As documents are updated, re-embedding costs are added, but in a typical enterprise this is around 30-50 USD per month.

Layer 4: Vector Database Selection

You need an infrastructure that can search vectors and retrieve them at query time. Five main options:

Practical recommendation for small to mid-sized enterprises (50K-1M chunks): pgvector. Because:

• It uses your existing PostgreSQL, with little additional operational cost
• With ACID compliance, metadata plus vector in a single transaction
• For hybrid search, keyword search with pg_trgm or tsvector in the same DB

Large scale (10M+ chunks): Qdrant or Weaviate. More optimized HNSW index, faster queries.

KVKK / location compliance: If data residency is mandatory in Turkey (health data, defense), a self-hosted option is the only choice. US-only managed services such as Pinecone run into the cross-border transfer conditions of KVKK Article 9.

Layer 5: Hybrid Search

Pure semantic search is not enough. This is because users sometimes search for "exactly this number." Like "decision TS-2024-1547."

Hybrid approach:

1. The user query is both embedded (semantic) and parsed as keywords (BM25 / tf-idf).
2. The two searches run in parallel, each returning the top 50 results.
3. They are merged with Reciprocal Rank Fusion (RRF).

def rrf(semantic_results, keyword_results, k=60):
    scores = {}
    for rank, doc in enumerate(semantic_results):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
    for rank, doc in enumerate(keyword_results):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda x: -x[1])

Practical gain: Compared to semantic only, precision@10 improves by 15-25%. Especially for queries containing technical terms, product codes, and regulation numbers.

Layer 6: Reranking

Hybrid search returns 50 results. Passing all of them to the LLM as context is both expensive and distracting. The reranker reduces 50 to 5-10 and reorders them.

Options:

• Cohere rerank-3: Managed, very powerful, ~2 USD / 1K queries
• BGE-reranker-v2-m3: Open source, runs on local GPU, free
• Cross-encoder (sentence-transformers): Open source, runs even on CPU, slower

Gain: Without reranking versus with reranking, a 20-30% improvement in final LLM response quality metrics (groundedness, faithfulness) is typical.

Layer 7: LLM Response Generation

The final layer: the retrieved 5-10 chunks plus the system prompt plus the user question go to the model. A response is generated.

System prompt skeleton:

You are the [company name] enterprise knowledge assistant. Answer the
user's question using the documents below.

RULES:
1. Use only the information in the provided documents. Do not add general knowledge.
2. For every claim, cite the source document number in the format [#1], [#2].
3. If the answer is not in the provided documents, say "I do not have enough
   information on this topic," do not make things up.
4. Keep the response in Turkish, clear, and concise.

DOCUMENTS:
{retrieved_chunks}

QUESTION: {user_question}

Model selection:

• High speed + cheap: GPT-4o-mini, Claude Haiku, Gemini Flash. Sufficient for most queries.
• Complex reasoning: Claude Sonnet 4 or GPT-4.1. Legal interpretation, multi-step reasoning.
• Local (confidential data): Llama 3.3 70B, Qwen 2.5 72B. When sensitive data must not travel through the cloud.

Citation enforcement: The [#1], [#2] references the model produces should be turned into clickable source document links in the frontend. The user sees the source, trust increases, and model hallucination is caught early.

Total Cost of Ownership (TCO)

Mid-sized scenario (50 thousand documents, 200 daily active users, an average of 8 queries per user per day):

These figures are typical for a mid-sized enterprise in Turkey. In large companies (1000+ active users), it rises to 60 thousand - 150 thousand USD per year. Still below the salary of a full-time AI engineer, but with operational control kept in-house.

KVKK and Access Control

A three-layer approach:

1. Ingestion layer (filter): Documents containing PII are automatically detected and anonymized or routed to a different bucket.
2. Query layer (RBAC): The document pool a user can access is restricted according to their role. Metadata filter in the vector DB: WHERE org_id = ? AND user_role IN (...).
3. Response layer (audit): Every query, user, retrieved chunks, and generated response is logged. This is mandatory for the processing activity record under KVKK Article 12.

Typical mistake: Applying access control only in the prompt ("user X should not see this, do not answer if a certain document appears"). The LLM cannot enforce this reliably. The filter must be physical, at the vector DB query level.

First 90 Days Roadmap

Days 1-15: Discovery + scope

• Which 5-10 use-case scenarios produce the highest value?
• Which 3-5 document sources feed these scenarios?
• Access rights and KVKK requirements are documented.

Days 16-45: MVP

• Start with a single source (for example, SharePoint) plus a single scenario (for example, HR FAQ).
• Stack: pgvector + OpenAI embedding + Claude Sonnet + a simple web UI.
• 5 internal pilot users.

Days 46-75: Expansion

• 3-5 source connectors are added.
• Hybrid search + reranker are integrated.
• Citation UI is added.
• 30-50 active users.

Days 76-90: Production

• Monitoring (latency, faithfulness, user feedback score) is set up.
• The KVKK audit log is completed.
• It is opened to the entire target user base.
• An improvement loop (weekly prompt + chunking + retrieval tuning) is started.

Conclusion

When set up correctly, enterprise RAG provides 40-60% time savings in information access and a 20-30% reduction in load on call center/support teams. When set up incorrectly, it produces hallucination, loss of trust, and a quietly abandoned project.

The difference is hidden in the architectural decisions, the chunking and embedding strategy, the hybrid search + reranker layer, and the KVKK-compliant access control. It is a seven-layer system beyond a plain LLM call; each layer has its own optimization and cost trade-off.

Starting with a small scope in the first 90 days, getting quick feedback, and expanding is the strongest antidote to the "do everything at once" mistake.

For support in your organization's AI transformation, you can review our professional consulting services.