Building an internal chatbot with LLMs: lessons from a real project

When our client — a manufacturing company with two thousand employees — asked us to build an internal chatbot, their initial goal was simple: reduce the number of times employees called HR to ask about leave policies, bonuses, and internal procedures.

The first challenge had nothing to do with technology. Their internal documentation was a mix of Word files, scanned PDFs, and archived emails going back ten years. Before building the chatbot, we needed to normalize and clean this knowledge base.

Prompt design was the most time-consuming part. An internal chatbot needs a clear system prompt defining its scope — only answer questions related to company policy — and must transparently decline or escalate when a question falls outside that scope.

The hallucination problem was more serious than we anticipated. In an HR context, a wrong answer about leave policy could trigger a labor dispute. Our solution: every response must include a citation to a specific source document, and a disclaimer is added when confidence is low.

After eight weeks, the chatbot was handling seventy-five percent of common HR questions without human involvement. Response time dropped from several hours (calling HR, waiting for a reply) to under thirty seconds.

The biggest lesson: an internal chatbot is not a technology project — it is a knowledge management project. Chatbot quality depends directly on the quality and organization of internal documents. Invest in the knowledge base first, technology second.

Share