Prompt Optimizer

The Prompt Optimizer improves a prompt toward a goal. You give it a starting prompt, a goal and a target (a model or an agent). It generates prompt variants, tests every variant against the same set of cases, scores them, keeps the best, and repeats for a number of rounds. You then save the winner as a new version of a managed prompt, or promote it straight to an environment.

Prompt Optimizer is an enterprise module. It is available as Prompt Optimizer when the tenant has an active enterprise licence. See Licensing.

Concepts

Object	What it is
Optimizer	An editable definition: the prompt to optimize, the goal, the target, the models, the test dataset and the shape of the loop. You can run it as often as you like.
Run	One execution of an optimizer. It stores a snapshot of the configuration it ran with, every variant it tried with its score, and the winner.
Variant	One candidate prompt in a run, with its score, pass rate, sample outputs and the judge's reasoning.

What you see on the screen

The Prompt Optimizer page lists optimizers with their name, the prompt source, and the target (model · <key> or agent · <key>). Use New optimizer to create one. Row actions are Edit and Delete; deleting an optimizer also deletes its runs.

Opening an optimizer shows its configuration (goal, prompt source, target, generator and judge models, loop shape, objectives and dataset) and a table of its runs with status (pending, running, completed, failed, cancelled), baseline score, best score and start time. Run starts a new run and opens it.

A run page shows progress while it works, how the run is scored, every variant tested (best first), and the best prompt with a copy button. Click a variant to open a drawer with its full prompt, sample outputs and the judge's reasoning.

Create an optimizer

1. Open Prompt Optimizer and choose New optimizer.

2. Identity. Name the optimizer and optionally describe it.

3. Prompt and goal. Choose where the prompt to optimize comes from:

   • Captured dataset — an evaluation dataset built from production traffic. The live system prompt is lifted out of the captured items. See Dataset from production traffic.
   • Managed prompt — a prompt from the Prompts module.
   • Inline — paste the system prompt. The first time you save the winner, a new managed prompt is created.

   Then write the goal. It drives both the variant generator and the judge's rubric.

4. Target. Choose whether the prompt is tested on a model or an agent, and pick which one.

5. Objectives. Choose one or more:

   • Quality — rewrite toward the goal, judged by an LLM.
   • Length — cut tokens and remove repetition.
   • Cache-friendliness — move dynamic content out of the prompt prefix.

   Length and cache-friendliness are scored deterministically from a prompt lint. Quality is always measured as well, and acts as a regression floor: a candidate that keeps less than the regression floor fraction of the baseline quality score is recorded but can never become the best variant.

6. Optimization. Pick the generator model (proposes variants) and optionally a judge model (defaults to the generator). Set rounds (1–10), variants per round (1–8) and beam width (1–5). Optionally set Stop at score to end the run early once a candidate reaches that score (0 to 1).

7. Test dataset. Every candidate is run against the same set, so scores are comparable between rounds:

   • Evaluation dataset — the strongest option. Items with a reference answer are scored by how close the candidate's output is to it, which needs an embedding model. When items record tool calls, the candidate is given the same tool menu and its calls are scored on selection, order and arguments; tools are never executed. The candidate replaces the system turn stored in the items. Items with neither a reference nor tool calls fall back to an LLM judge scoring the goal. Every candidate runs the whole dataset (hard ceiling 500 items) unless you set an item cap. For multi-turn items you choose between replaying the whole conversation or using only the first user turn; the model is called once per item either way.
   • Inline JSON — hand-written cases such as [{"question":"…","expected":"…"}]. An item with expected is scored against it; one without falls back to the judge.
   • Auto-probe from goal — the generator invents a few test questions from the goal and a judge scores answers against the same goal. This has no ground truth, so use it to explore a new prompt rather than to tune one already in production.

8. Save the optimizer.

Run it and read the result

1. Open the optimizer and choose Run. The run page refreshes itself while the run is active.
2. Watch Progress (round and variant counters). Stop asks the run to cancel; it stops after the current variant.
3. When the run completes, read How this run is scored (it explains how the dataset was graded) and the Iterations table. The best prompt is shown with its score.
4. To keep the winner, choose one of:
   • Save best as version — writes the best prompt as a new version of the managed prompt. A run that started from a captured dataset or an inline prompt creates a new managed prompt the first time you save; later saves from the same run update that prompt.
   • Promote → dev, staging or prod — saves the version and then deploys it to that prompt environment.

Both actions need a completed run that has a best variant.

Permissions and requirements

Topic	Behaviour
Plan	Enterprise licence required.
Permission	The prompts service permission covers the Prompt Optimizer routes (/api/prompt-optimizer).
Scope	Optimizers and runs are project-scoped.
Models	The generator, judge and target models (or agent) must exist in the project. Reference-based scoring needs an embedding model.
Cost	Every variant is tested on every dataset item, so cost grows with rounds, variants per round and dataset size. Use the item cap or Stop at score to bound it.
Where runs execute	Runs go through the cluster queue layer, so they can run on a worker node.

Related

• Prompts
• Dataset from production traffic
• Evaluation and analysis