AI that gives staff trusted answers
My role: Technical product manager. I managed the knowledge base's development: defining what it should answer, setting up human-in-the-loop review and testing answers for accuracy. I also build and test AI agents myself.
- Role
- Technical product manager
- Approach
- LLMs, RAG, human-in-the-loop review
- Users
- Staff across business units
- Tools
- Claude, AI agents
- Outcome
- Training cut from weeks to days
The problem
People across the Port need reliable answers from the Port's own plans and procedures, especially during incidents. New starters took two to four weeks to learn those procedures, and information spread across many documents could overlap or contradict itself.
Key decisions
RAG, grounded in a vetted handbook. We chose retrieval-augmented generation because it suits a knowledge base: the assistant answers only from a curated handbook, the single source it cites, rather than from general knowledge.
Nothing is trusted until a person approves it. New information lands in a staging area first. AI proposes how to merge, update or retire handbook entries, and nothing is written until a knowledge administrator approves it. This human-in-the-loop step keeps errors to a minimum.
Flag conflicts, don't guess. When the knowledge base detects information that contradicts itself, it flags the conflict to a moderator instead of picking an answer.
Let users tell us when answers miss. A direct feedback loop lets people mark whether they're happy with each answer, so we know where to improve.
What I did
- Managed development of a knowledge base built on large language models and RAG, now used across business units.
- Defined how knowledge is captured, curated and approved, with human-in-the-loop review, conflict detection and a user feedback loop.
- Included an emergency assistant that answers Incident Commanders' questions only from the Port's own plans.
- Built and tested AI agents, prototyped products with Claude and vibe-coded websites.
- Embedded AI into engineering to enforce the Port's coding standards.
The result
New starters used to take two to four weeks to learn the Port's procedures. Now it takes a few days. Answers are grounded in the Port's own approved documents, so people can trust them.
What I learned
We treated the first knowledge base as an experiment, and it became our blueprint: it taught us how to build the knowledge bases that followed across other business units. It also revealed uses we hadn't planned for, such as using the same approach to create business documents, not just answer questions.