Building a Local RAG System with Go, PostgreSQL, and Ollama: No Cloud APIs Required

Introduction

In an era where data privacy and cost control are paramount, many organizations are moving away from cloud-dependent AI solutions. A recent article on Habr (July 2026) details how a team of developers built a fully local Retrieval-Augmented Generation (RAG) system using Go, PostgreSQL, and Ollama—without relying on any external cloud APIs. This approach offers complete control over data, lower latency, and predictable costs. In this article, we break down the architecture, implementation challenges, and practical results from this real-world case study.

Why Go Local RAG?

The core motivation behind this project was to eliminate dependence on third-party cloud providers for both embedding generation and large language model (LLM) inference. The developers wanted a system that could run entirely on-premises or on a single machine, processing sensitive documents without sending data to external servers. They chose Go for its performance and concurrency, PostgreSQL with the pgvector extension for vector storage and similarity search, and Ollama for running open-source LLMs locally.

The Architecture

The system follows a standard RAG pipeline but is implemented entirely with open-source tools:

The entire stack runs on a single Linux server with a GPU (NVIDIA RTX 4090), though CPU-only setups are also feasible for smaller datasets.

Implementation Details

The developers encountered several practical challenges during implementation:

1. Document Chunking

They experimented with different chunk sizes (256, 512, and 1024 tokens) and found that 512-token chunks with 50-token overlap provided the best balance between retrieval accuracy and context window usage. Smaller chunks improved precision but increased the number of database queries.

2. Vector Storage in PostgreSQL

Using the pgvector extension, they created a simple table:

CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding vector(768)
);

They built an index using the IVFFlat algorithm for efficient approximate nearest neighbor search, which reduced query time from 2 seconds to under 100 milliseconds for a dataset of 50,000 chunks.

3. Embedding with Ollama

Ollama's API was called via Go's HTTP client to generate embeddings. The developers noted that nomic-embed-text produced 768-dimensional vectors, which matched well with pgvector's default settings. They also implemented a caching layer to avoid re-embedding duplicate content.

4. Retrieval and Generation

For each user query, the Go application:
- Generates an embedding of the query using the same Ollama model
- Performs a cosine similarity search in PostgreSQL (LIMIT 5)
- Concatenates the retrieved chunks with the original query
- Sends the prompt to Ollama for answer generation
- Returns the answer along with source references

They used a prompt template like:

"You are a helpful assistant. Use the following context to answer the question. If the context does not contain the answer, say you don't know. Context: {context} Question: {query} Answer:"

Results and Performance

The team tested the system on a corpus of 10,000 technical documents (approximately 5 million tokens). Key metrics:

The system achieved a recall@5 of 92% on a custom test set of 500 queries, meaning the relevant chunk was in the top 5 results in 92% of cases.

Lessons Learned

The developers shared several practical insights:

• Batch embedding is critical: Processing documents one by one is slow. They implemented batch embedding (up to 32 chunks per request) which improved throughput by 4x.
• Ollama's context window: Some models have a limited context window (e.g., 4096 tokens). They had to ensure the combined context + query fit within this limit, sometimes truncating older chunks.
• Indexing strategy: The IVFFlat index requires careful tuning of the lists parameter. They found that lists = sqrt(number_of_rows) worked well in practice.
• Error handling: Ollama occasionally returns empty responses or timeouts. The Go code implements retries with exponential backoff.

Comparison with Cloud-Based RAG

While cloud services like OpenAI's GPT-4 with retrieval offer higher accuracy, the local RAG system provides:
- Data sovereignty: All data stays on-premises
- Zero API costs: After hardware investment, operational costs are only electricity and maintenance
- Customizability: Any open-source model can be swapped in
- Latency predictability: No network jitter

The trade-off is lower accuracy on complex queries (about 15% lower than GPT-4 based on their internal tests) and the need for technical expertise to maintain the infrastructure.

Use Cases

This architecture is particularly suitable for:
- Legal firms: Processing confidential contracts and case files
- Healthcare: Analyzing patient records without exposing data to external APIs
- Enterprise knowledge bases: Internal documentation search for employees
- Edge devices: Running on laptops or servers without internet connectivity

For example, a law firm could index thousands of past cases and use natural language queries to find relevant precedents—all while keeping client data completely private.

Conclusion

The local RAG system built with Go, PostgreSQL, and Ollama proves that high-quality, private AI-powered document search is achievable without cloud dependency. The developers successfully created a production-ready system with sub-2-second response times and 92% retrieval accuracy. While not a replacement for cutting-edge cloud models in every scenario, it offers a compelling alternative for organizations prioritizing data privacy and cost control. The full source code and detailed benchmarks are available in the original article on Habr.

Source