RAG Best Practices: Improving Knowledge-Based Answers
Retrieval Augmented Generation helps Peers answer from your own datasource content instead of relying only on general model knowledge. The quality of those answers depends on both the content you provide and the way you test the experience.
This guide focuses on product practices for admins and Peer owners. For implementation architecture, APIs, schemas, or provider setup, use the Developer Hub.
Why RAG matters
RAG helps Peers:
- Use current company knowledge.
- Cite or reflect the right source material.
- Reduce unsupported answers.
- Give more specific responses for your business.
- Respect datasource access and scope.
It works best when datasource content is complete, well organized, and tested with real user questions.
Start with the question users actually ask
Before changing settings, collect examples of real questions:
- Frequent support questions.
- Sales or operations questions.
- Policy and process questions.
- Questions that users say the Peer answered poorly.
- Questions with known correct answers.
These examples become your test set. Use them before and after datasource or retrieval changes.
Improve the source content
Poor source content usually produces poor answers. Review whether the datasource has:
- Clear titles and headings.
- One topic per article or section where possible.
- Current information.
- Removed duplicates.
- Consistent terminology.
- Enough context for a user who did not write the document.
If users ask about something that is not in the datasource, the Peer cannot reliably answer it from knowledge.
Use metadata users understand
Helpful metadata can make source review easier. Examples include document owner, product area, region, policy version, customer segment, or last reviewed date.
Keep metadata simple. If admins and business owners cannot understand it, it will be hard to maintain.
Balance precision and recall
Search settings usually involve a tradeoff:
- Higher precision means fewer, more targeted results.
- Higher recall means more possible source material.
For support and policy Peers, start with accuracy and source trust. For research or discovery Peers, broader recall may be useful.
Use strict answers when guessing is risky
Strict knowledge behavior is useful when the Peer should only answer from indexed content. Use it for policies, compliance, support instructions, legal-sensitive material, or other high-risk workflows.
When strict mode is active, make sure the fallback message is helpful. Users should know whether to ask differently, contact a team, or request a datasource update.
Test after every meaningful change
Run checks after:
- Adding a datasource.
- Removing outdated content.
- Changing retrieval settings.
- Changing the Peer prompt.
- Publishing a new Peer version.
- Changing vector storage.
Review not only whether the answer sounds good, but whether it is grounded in the right source.
Common failure patterns
The answer is too generic
Add more specific source content, improve headings, and make sure the Peer knows which datasource to use.
The answer cites the wrong topic
Check for duplicate or overlapping articles. Test whether similar terms are confusing the search.
The Peer says it does not know
Confirm the content exists in the datasource and has finished indexing.
The answer mixes old and new information
Remove stale documents and add ownership for recurring content review.
Admin checklist
- Identify the Peer owner.
- Identify the key datasource owners.
- Build a small test set of real questions.
- Review source content quality.
- Test answers before publishing.
- Track user feedback after rollout.
- Review datasource freshness on a schedule.