Optimizing Peers with AI-Powered Analysis
After you run evaluations, the next question is what to improve. AI-Powered Analysis helps Peer owners review evaluation results and identify changes that may improve answer quality, consistency, or task completion.
What AI-Powered Analysis does
AI Analysis reviews the Peer, its evaluation results, and the patterns behind failed or weak answers. It then suggests changes that a Peer owner can review before applying.
Suggestions may involve:
- Clarifying the system prompt.
- Adjusting instructions.
- Improving datasource coverage.
- Changing how a Peer uses tools.
- Reworking examples or expected answer style.
- Testing a different model behavior.
When to use it
Use AI Analysis when:
- Evaluation scores drop.
- Users report confusing answers.
- The Peer performs well on some topics but poorly on others.
- You changed datasource content.
- You are preparing a Peer for a broader rollout.
- You need a faster way to review many test results.
Review suggestions before applying them
AI suggestions should support human judgment, not replace it. Before applying a suggestion, ask:
- Does this match the Peer purpose?
- Does it improve the user experience?
- Could it conflict with compliance, tone, or policy requirements?
- Does the owner of the datasource agree with the change?
- Can we test the change before publishing?
Apply changes one at a time when possible. This makes it easier to understand what improved or regressed.
Build a quality loop
- Run an evaluation suite.
- Review weak answers.
- Run AI Analysis.
- Choose the suggestions that fit your goals.
- Test the updated Peer.
- Publish only after results are acceptable.
- Monitor user feedback.
This loop keeps Peer improvement grounded in evidence instead of guesswork.
What good suggestions look like
Good suggestions are specific, testable, and tied to a user problem.
Examples:
- Add missing policy context to the datasource.
- Clarify that the Peer should ask a follow-up question when the user omits account type.
- Update the Peer tone for customer-facing responses.
- Add evaluation questions for a weak topic area.
Be cautious with broad suggestions that change the Peer purpose or make answers less predictable.
Admin checklist
- Keep evaluation suites current.
- Assign an owner for each important Peer.
- Review suggestions before applying them.
- Test changes with representative questions.
- Publish changes through version history.
- Track whether user feedback improves after the change.