Show HN: Offline RAG System Using Docker and Llama 3 (No Cloud APIs)

6 points by PhilYeh 2 hours ago

I'm sharing a fully offline RAG (Retrieval-Augmented Generation) stack I built to solve a crucial problem in industrial environments: data privacy and recurring API costs.

We deal with sensitive proprietary datasheets and schematics daily, making cloud-based LLMs like ChatGPT non-compliant.

The Solution: A containerized architecture that ensures data never leaves the local network.

The Stack: LLM: Llama 3 (via Ollama) Vector DB: ChromaDB Deployment: Docker Compose (One-click setup) Benefit: Zero API costs, no security risks, fast local performance. The code and architecture are available here: https://github.com/PhilYeh1212/Local-AI-Knowledge-Base-Docke...

Happy to answer questions about the GPU passthrough setup or document ingestion pipeline.