In my previous post, I introduced Microsoft Foundry as a unified platform for building and operating AI applications. The next practical question is usually more important: how do you make a chatbot answer questions about your organization rather than relying only on a model’s general training?
The answer is grounding. By connecting enterprise knowledge sources to a chatbot or agent, Foundry can retrieve relevant company information at the time of a request and use it to produce a more useful, traceable response.
Why grounding matters
A foundation model can explain common concepts, write text, and reason across many domains. It does not automatically know your current product catalog, support procedures, engineering standards, benefits policies, customer contracts, or data warehouse. Even when similar information existed in the model’s training data, that information might be incomplete or outdated.
Without grounding, a chatbot has two bad choices: give a generic answer or confidently invent details. Grounding gives it a third option: retrieve information from sources you control, use that information as context, and return an answer that can include citations back to the source.
This pattern is commonly called retrieval-augmented generation, or RAG. Microsoft Foundry extends the basic RAG pattern with a managed knowledge layer called Foundry IQ.
Knowledge sources and knowledge bases
Foundry IQ separates enterprise content into two useful concepts:
- A knowledge source is a connection to a specific collection of content, such as an Azure Blob container, SharePoint site, search index, OneLake lakehouse, or a remote service.
- A knowledge base groups one or more knowledge sources and defines how retrieval should behave.
An agent connects to the knowledge base instead of being tightly coupled to every data source. That distinction matters in an enterprise environment. A platform team can manage a reusable knowledge base while several agents use it for different experiences. Sources can be added, refreshed, or governed in one place without rebuilding every chatbot.
Two ways to connect enterprise knowledge
Foundry IQ supports both indexed and remote knowledge sources. They solve different problems.
Indexed knowledge sources
With an indexed source, content is processed before a user asks a question. Documents are extracted, divided into useful chunks, enriched with metadata, and placed in an Azure AI Search index. Queries can then use keyword, vector, or hybrid search.
Current options include an existing Azure AI Search index, Azure Blob Storage, OneLake, and—depending on feature availability—Azure SQL, direct file upload, and indexed SharePoint content. For several of these options, Azure AI Search can generate the data source, skillset, indexer, and index needed for ingestion.
Indexed sources are a good fit when you want fast retrieval over a controlled corpus and can tolerate a small delay between a source changing and its index being refreshed.
Remote knowledge sources
A remote source is queried at request time through the source system’s own interface. The content is not copied into an Azure AI Search index first. Current examples include remote SharePoint, Microsoft Fabric data agents and ontologies, MCP servers, Work IQ, and web grounding through Bing. Several of these capabilities are still in preview.
Remote retrieval is useful when information changes frequently, already has a capable query interface, or should remain in its original system. It can also preserve the semantics of a specialized source—for example, letting a Fabric data agent answer questions about business data rather than flattening everything into document chunks.
A single knowledge base can combine indexed and remote sources. Microsoft maintains the current list and availability details in the knowledge source documentation, which is worth checking because portal and API support continue to evolve.
How agentic retrieval works
Traditional RAG often sends one search query to one index. Foundry IQ can use agentic retrieval to do more work around the request. For a complex question, the retrieval engine can:
- Analyze the user’s intent.
- Break the question into smaller subqueries.
- Select appropriate knowledge sources.
- Run searches in parallel.
- Merge and rerank the results.
- Return relevant passages and citations for the agent to use.
This is especially helpful when the answer spans systems. A question such as “Can I work from another country for two weeks, and what equipment can I take?” might require the travel policy, remote-work policy, security guidance, and asset-management rules. The user should not need to know which repository contains each part of the answer.
An enterprise example
Imagine an internal employee-support chatbot. Its knowledge base might connect to:
- HR policies stored in SharePoint.
- IT troubleshooting guides stored in Azure Blob Storage.
- A service catalog in an Azure SQL table.
- A Fabric data agent for approved workforce metrics.
- An internal MCP server that exposes live equipment or ticket information.
When an employee asks a question, the chatbot can retrieve policy text, live operational data, or both. It can cite the relevant document and, if a separate action tool is configured, help the employee complete the next step. Knowledge answers the question; tools perform controlled actions. Keeping those responsibilities clear makes the design easier to reason about and secure.
Permission-aware answers are essential
Connecting more data is not automatically better. A useful enterprise chatbot must return only information the caller is allowed to see.
Foundry IQ can run queries using the caller’s Microsoft Entra identity and enforce source permissions for supported configurations. It can synchronize access-control information for supported sources and can work with Microsoft Purview sensitivity labels in certain indexed scenarios. The exact security behavior varies by source, so permissions must be designed and tested rather than assumed.
The safest architecture keeps authorization close to the data, uses managed identities where appropriate, grants the smallest necessary roles, and verifies access with representative user accounts. A citation is valuable for traceability, but it is not a security boundary by itself.
A practical setup workflow
A small proof of concept can follow this sequence:
- Create or select a Microsoft Foundry project.
- Connect an Azure AI Search service that supports agentic retrieval.
- Create one narrowly scoped knowledge source.
- Add that source to a Foundry IQ knowledge base.
- Create or select an agent and connect the knowledge base.
- Test realistic questions in the playground.
- Evaluate answer quality, citations, permissions, latency, and cost before adding more sources.
Microsoft provides a step-by-step guide for connecting a Foundry IQ knowledge base to Foundry Agent Service. The portal is convenient for experimentation, while production applications should also account for managed identity, networking, deployment automation, monitoring, and the feature’s current preview or general-availability status.
What to evaluate before production
Grounding reduces one major source of hallucination, but it does not eliminate mistakes. Retrieval can miss a relevant document, return an outdated passage, or surface conflicting policies. The model can also misinterpret retrieved text. A production plan should cover:
- Source quality: Are documents authoritative, current, and clearly owned?
- Retrieval quality: Do representative questions return the right passages?
- Permissions: Can users retrieve only what they are authorized to see?
- Citations: Can a user verify important claims at the source?
- Freshness: Does the refresh strategy match how quickly the source changes?
- Latency and cost: Is deeper multi-source retrieval justified for every question?
- Fallback behavior: Does the chatbot admit when the available evidence is insufficient?
Start with a source that has clear ownership and a measurable use case. Ten poorly governed repositories will not create a better chatbot than one accurate and well-maintained knowledge base.
Final thoughts
The most valuable enterprise chatbots are not valuable because they can talk about everything. They are valuable because they can answer specific questions from trusted organizational knowledge while respecting permissions and showing where the answer came from.
Microsoft Foundry provides a path from a general-purpose model to that kind of application. Foundry IQ knowledge bases let teams combine indexed and live sources, apply agentic retrieval across them, and reuse the resulting knowledge layer across agents. The result is a chatbot that is more current, more accountable, and more closely aligned with the way the business actually works.