Building A Knowledge Agent That Remembers

Knowledge without memory is guesswork. We take a hard look at why most workflow agents stall at triage and show how to turn them into knowledge agents that deliver trusted, context-rich answers drawn from your organisation’s best thinking. 

Starting with the real cost of lost information and context switching, we map the path from scattered wikis and chat threads to a reliable institutional memory powered by retrieval augmented generation and hybrid search.

At a Glance / TLDR:

• the memory gap between task routing and problem solving
• why hybrid retrieval outperforms pure vector in enterprise settings
• practical chunking strategies and metadata fields for authority and recency
• architecture choices across vector stores, hybrid search, and connectors
• governance, citations, accuracy monitoring, and freshness controls
• case studies: hours saved, quality gains, and revenue impact
• failure patterns: infra overruns, integration debt, and weak curation
• four principles: exec sponsorship, domain experts, user focus, workflow redesign

We break down the decisions that matter: how to chunk documents so the agent can both recall facts and reason across context, how to enrich content with metadata that signals authority and freshness, and how to fuse vector semantics with keyword precision for queries that mix intent with exact terms like product codes and financial acronyms. 

On the engineering side, we cover architecture trade‑offs between vector databases and native hybrid search, secure connectors into CRM and ERP systems, and the governance needed for citations, audits, accuracy monitoring, and content freshness. 

You’ll hear where teams slip - capacity spikes, weak document prep, brittle identity integrations - and how to design for elasticity and compliance from day one.

The proof is in production. Uber’s engineering co‑pilot reclaimed thousands of hours and raised answer quality; JP Morgan Chase scaled insights to more than two hundred thousand employees and unlocked major business value; Goldman Sachs is pushing beyond retrieval to application, where the agent drafts, analyses, and accelerates financial workflows. 

Across these stories, a shared blueprint emerges: executive sponsorship, domain expert curation, user‑centred iteration, and workflow redesign that embeds the agent into daily decisions. If you’re ready to turn proprietary knowledge into a real moat and to build a platform that compounds value across use cases this conversation offers the playbook.

Enjoyed the episode? Follow, rate, and share with a colleague who’s building AI into their workflow, and leave a review with the biggest knowledge challenge you want us to tackle next.

Like some free book chapters?  Then go here How to build an agent - Kieran Gilmurray

Want to buy the complete book? Then go to Amazon or  Audible today.

Chapters

• 0:00 The Memory Gap In Agents
• 1:14 RAG As Organisational Memory
• 2:20 Hybrid Search Beats Pure Vector
• 4:02 Preparing Documents For Retrieval
• 6:21 Metadata For Authority And Freshness
• 7:06 Architecture And Connectors
• 8:16 Governance And Accuracy Monitoring
• 10:06 Uber’s Engineering Co‑Pilot Results
• 11:24 JP Morgan’s Scale And ROI
• 12:49 Goldman’s Shift To Knowledge Application
• 14:10 Why Successful Programmes Scale
• 14:50 ROI, Productivity, And Ramp Time
• 16:19 Failure Modes And Mitigations
• 18:02 Four Principles For Enterprise Success
• 19:32 Building Institutional Intelligence

Transcript

SPEAKER_00: 0:00

Chapter 6. Giving Your Agent Memory The Knowledge Agent. As your most experienced engineers retire, they take irreplaceable knowledge and expertise with them. Your workflow agents could coordinate tasks brilliantly, but they operate in a knowledge vacuum, unable to tap into the institutional wisdom that separates efficient task processing from intelligent problem solving. The workflow agent we built in chapter 5 is an intelligent coordinator, but it operates with a limited memory. It can assess the urgency of a new request, but it cannot access the decades of accumulated expertise trapped within your organization's documents, support cases, and engineering notes. Consider a high-value customer asking a complex question that references a past conversation with a business. Our Chapter 5 agent was able to correctly identify it as a high priority ticket and route it to an expert. However, it couldn't provide the substantive guidance needed to actually solve the customer's unique problem. This gap between workflow intelligence and institutional wisdom is what distinguishes task processors from true problem solvers. This issue extends far beyond a minor concern. It represents a crisis in how knowledge is accessed. Research shows the average knowledge worker spends 2.5 hours daily simply searching for information. While some searching is unavoidable, most of those 2.5 hours represent wasted effort. For an organization with 1,000 employees, that could mean millions of dollars in lost productivity every year. Critical insights are scattered across wikis, emails, and Slack, forcing teams to waste time and money just to find what they need. RAG bridges this gap by providing agents with access to organizational memory. Like a skilled consultant with perfect recall, a RAG-enabled agent is able to consult your company's entire knowledge base and then provide informed, contextually relevant answers. This transformation converts a workflow agent into a knowledge agent, a system that provides actionable guidance based on your organization's collective intelligence. How agents remember Our goal is to give our agent an institutional memory, and, as we know from earlier chapters, retrieval augmented generation is the key. By connecting our agent to a vector database of company documents, we allow it to perform intelligent, semantic searches that understand context and keywords. While the vector databases that power RAG are excellent at understanding semantic meaning, e.g., connecting customer churn to retention strategies, a purely semantic approach can fall short in a business context. It can sometimes miss critical keywords, specific product names, or financial acronyms. Microsoft's Azure AI search combines two different methods, vector plus keyword, of retrieving information, and this hybrid approach often outperforms pure vector or keyword methods. In independent evaluations, hybrid models have achieved relative gains of about 20% over pure deep retrieval methods in cross-domain settings. This is why leading enterprise implementations of knowledge agents now use a hybrid search approach. The practical impact becomes clear in enterprise scenarios. A query for Q3 EBITDA projections requires both semantic understanding, financial forecasting concepts, and keyword precision. Hybrid search ensures that both dimensions are captured, delivering more accurate and comprehensive results. Building your knowledge agent. Transforming the workflow agent from Chapter 5 into a knowledge-based system requires careful attention to how organizational information is prepared, stored, and retrieved. Our technical implementation builds on familiar patterns while adding the memory layer that enables institutional intelligence. Document preparation. The effectiveness of our knowledge agent rises or falls based on how well information is prepared for retrieval. Two levers dominate outcomes. How we chunk content and how we enrich it with metadata. How information is broken down and labeled makes a big difference in how well an AI system retrieves answers. Break text into pieces that are too small, and the system may miss the bigger picture, make them too large, and it may overlook key details. The right balance depends on your business context. So it's essential to start with a sensible default and adjust it based on your specific needs. Multi-dataset analyses show a clear trade-off. Smaller chunks, about 64 to 128 tokens, tend to excel on concise, fact-based queries, while larger chunks, about 512 to 1024 tokens perform better when answers require broader context. Practical starting points from industry guidance begin around 250 tokens, about 1000 characters, and adjust based on evals. Financial documents Approximately 1,800 characters provide the necessary context for complex financial relationships and calculations, while avoiding the 10-20% performance degradation that occurs with chunks exceeding 14,400 characters. Domain specifics matter. In a finance slash SEC filing setting, about 1,800 characters worked well, while about 14,400 character chunks diluted relevance and drove performance drops of around 10 to 20%. Metadata enrichment transforms static documents into intelligently searchable assets. Each chunk should be enriched so the agent can filter by what's authoritative and current, not just what's semantically similar. Recommended fields include title, summary, keywords slash entities, and machine generated attributes for filtering, e.g., document type, created or updated date, owning team or department, approval state. These fields enable hybrid retrieval, combining keyword and AI-powered search for better accuracy, and policy-aware filtering, which ensures results respect business rules such as approvals and ownership. Implementation architecture. Building your knowledge agent requires integrating several components, such as vector databases, hybrid search tools, and secure connectors, into a cohesive system that can scale with your organization's needs while maintaining response quality and security standards. Vector database and hybrid search considerations. Pinecone has been benchmarked to achieve low latencies, often under 120 milliseconds at the 95th percentile, meaning almost all queries complete this fast, at tens of millions of vectors, making it a reliable choice when performance and scale matter. Weaviate supports true hybrid search, Vector plus BM25, a proven keyword search algorithm, by running both searches in parallel and fusing results via configurable strategies, e.g., ranked fusion or relative score fusion. The choice between them depends on trade-offs, pure vector for raw speed and scale, or hybrid support, when exact matching and semantic context both matter. Integration architecture. Your agent must interface securely with existing document stores, CRM slash ERP systems, and identity slash access control. You'll need API connectors, pull, push, streaming, to sync content updates, authentication and authorization layers respecting your access model, e.g. RBAC, OAuth, Skim, Audit, Logging, and Versioning for traceability and compliance. Quality control and governance. Enterprise knowledge systems demand rigorous controls to ensure data quality, reliability, and accountability. Each response should include source attribution to ensure auditability and allow user verification. Accuracy monitoring involves tracking response quality through user feedback, automated evaluation metrics, and periodic human review. Some high-performing systems aim for 85% accuracy rates for factual queries. Systems that fail to meet these thresholds require optimization before production deployment. Content freshness management ensures that a knowledge base accurately reflects the current organizational state. This includes automated updates when source documents change, depreciation handling workflows for outdated information, and conflict resolution when multiple sources provide contradictory user guidance. The monitoring framework should track key performance indicators that correlate with business value, accuracy, user satisfaction, attribution completeness, and system availability. Enterprise Success Stories. Uber's Engineering Knowledge Revolution Like many tech companies, Uber's critical engineering knowledge was trapped in thousands of Slack conversations, making it inaccessible during production incidents or key architectural discussions. To solve this, they implemented an AI-powered engineering co-pilot using an advanced RAG architecture. Uber's system now processes 45,000 questions per month, providing engineers with instant access to curated technical documentation and institutional expertise. The business impact has been profound. Uber documented 13,000 engineering hours saved annually, the equivalent of adding six major engineers to the team. Crucially, the system also improves quality, delivering a 27% increase in acceptable answers, and a 60% reduction in incorrect advice compared to standard RAG implementations. The key insight from Uber's success is its domain-specific focus. By focusing on high-quality engineering content and learning from user interactions, the agent developed a deep understanding of the technical context that a general-purpose system could never achieve. JP Morgan Chase, scaling knowledge to 200,000 employees. JP Morgan Chase's implementation of its knowledge agent demonstrates knowledge management at an unprecedented scale, serving over 200,000 employees with its LLM Suite Knowledge Platform. For expensive asset management analysts who previously spent hours on manual research, the RAG enhanced platform now delivers comprehensive insights in minutes. The documented business impact is transformative. The bank achieved$1.5 billion in annual business value, beyond savings. AI-powered advisory tools have driven a 20% increase in gross sales and are projected to enable a 50% increase in client roster capacity within five years. The bank currently operates over 450 proofs of concepts, with plans for 1,000 plus AI use cases by 2026, suggesting that the knowledge platform serves as the foundational infrastructure for a broader AI transformation. This scaling approach, which involves building robust knowledge capabilities first, and then expanding use cases, represents a repeatable pattern for achieving enterprise AI success. Goldman Sachs, projecting transformational productivity. Goldman Sachs AI assistant platform, currently serving 10,000 plus employees with company-wide expansion plan for 2025, illustrates the next evolution of enterprise knowledge systems. The investment bank expects 3 to 4x productivity gains through agentic AI integration that combines institutional knowledge with specialized financial workflows. The platform demonstrates how knowledge systems can move beyond information retrieval to actively create work products. This represents a fundamental shift from knowledge access to knowledge application. The system not only finds relevant information, but also uses it to generate initial drafts, analysis frameworks, and implementation approaches. Goldman Sachs's strategic vision envisions AI systems functioning as seasoned Goldman employees within three to five years, fundamentally transforming trading and investment banking operations through integration with proprietary data and specialized financial workflows. This ambition reflects confidence in knowledge systems as a foundation technology that can scale across diverse business functions while maintaining the precision and expertise that defines the firm's competitive position. The pattern of success. These successful implementations share a common DNA. They are driven by executive sponsorship that guarantees resources, built on the knowledge of domain experts who ensure content quality, and launched using a phased approach that starts with a focused pilot to prove value before scaling. Critically, they are treated as living systems, with continuous optimization through user feedback and performance monitoring, ensuring they evolve and maintain high accuracy over time. Business impact and ROI from knowledge systems. Implementing a knowledge agent can yield compounding value across operations, talent, and strategic layers, although exact returns depend heavily on context, baseline processes, and scale. Productivity gains manifest via reduced search time, fewer context switches, and faster decision support. In many knowledge management initiatives, organizations report that employees spend a non-trivial portion of their day, e.g., multiple hours weekly, locating information. Some estimates suggest that as much as eight hours per week is spent searching for information across systems, and that effective KI systems can reduce this burden by 30 to 35%. For talent acceleration, knowledge agents help new hires ramp more quickly by surfacing institutional knowledge and reducing reliance on ad hoc mentoring. The financial case for knowledge agents is often cast as high ROI over time, especially when the system is reused across domains and evolves. While many early claims about KMROI were speculative, the key to achieving a strong return is to approach the investment realistically. As the Harvard Business Review notes, success depends on proper planning. This means building in budget contingencies for infrastructure scaling and data quality efforts. Most importantly, view this not as a one-off project, but as a foundational platform investment that will lower the cost and accelerate the deployment of all future AI initiatives across your enterprise. Implementation challenges and success strategies. The stark reality of enterprise RAG implementation demands strategic attention. 72% of deployments fail within the first year, with infrastructure overruns being the primary killer. During periods of market volatility, a finance firm's RAG system suffered infrastructure scaling failures, forcing$200,000 in emergency vector database capacity spend, and 14 hours of downtime and critical trading windows. The incident led to a complete migration toward a hybrid retrieval approach. Accuracy shortfalls can also lead to failures. Systems that skip rigorous document preparation, domain-grounded evaluation, and human review struggle to meet enterprise accuracy expectations. Research on knowledge quality and human-in-the-loop practices highlights the importance of expert curation, periodic human review, and feedback loops in maintaining factuality and minimizing drift. Integration complexity is another challenge that businesses must address. Even the strongest prototypes can stall when connecting to identity, security, and legacy content systems. Consistent guidance from transformation literature, treat integration, access control, auditability, and change management as first-class work streams, not afterthoughts. The organizations that achieve breakthrough business results consistently demonstrate a commitment to four key principles. 1. Executive sponsorship. Successful projects are championed by senior leaders who secure resources, ensure cross-functional alignment, and remove the inevitable organizational roadblocks that arise during implementation. 2. Domain expert involvement. Successful projects are a partnership between AI experts and your company's own subject matter experts. Involving domain experts in curating knowledge sources and validating agent responses is the only way to ensure the system is accurate, relevant, and trusted by users. 3. User-centric design. The best systems are built with, not for, their users. Integrating systematic user feedback loops and treating the agent as an internal product that evolves based on user needs is crucial for driving adoption and ensuring the solution is solving the right problems. 4. Focus on workflow redesign. As McKinsey's research on the state of AI makes clear, enterprise level impact only materializes when you restructure processes around the agent's new capabilities, rather than simply bolting new technology onto old ways of working. Conclusion. Building institutional intelligence. We have evolved our agent from a workflow orchestrator into a knowledge-enabled strategist. While the agent from Chapter 5 provided a foundation for coordinating tasks, a knowledge agent adds access to decades of accumulated expertise. It transcends the limits of old search systems by understanding semantic meaning, creating a system that provides contextual intelligence, not just retrieved information. This capability is a true competitive moat. AI technology may be a commodity, but your organization's proprietary knowledge is not. An effective knowledge agent turns this unique asset into a source of superior decision making and operational speed. A robust knowledge agent is the prerequisite for the next level of AI maturity. In the next chapter, we will explore how multiple specialized agents can collaborate using this shared institutional memory to tackle complex, multifaceted business challenges.