How do I review a dispute as an evaluator?
Required Feature Flags
The following feature flags and permissions are required to use this feature:
Feature Flag | Description |
Evaluation disputes | Lets agents and evaluators raise and review disputes (also called queries) on a published evaluation |
Required Permissions:
Process Evaluation Queries (
quality.disputes.process) — to review and resolve disputesView Completed Evaluations (
quality.evaluations.view-completed) — to access the disputed evaluations
When an agent disputes an evaluation, a reviewer (usually an evaluator or team lead) decides whether to keep the original scores or update them. This guide walks through that process.
Step 1: Find the dispute
You'll receive a notification when a dispute is assigned to you. To open it:
Go to Evaluate > click Completed evaluations.
Filter the list by dispute status to see evaluations with pending disputes, then open the evaluation you need to review.
Step 2: Read the agent's reasoning
Open the dispute panel on the evaluation.
For each disputed item, you'll see:
The original score and outcome
The score the agent thinks is correct
The agent's written reasoning
Read every disputed item carefully. The agent's reasoning is your starting point — it tells you what to look for in the conversation.
Step 3: Re-check the conversation
Listen to the call or read the transcript again with the disputed items in mind.
Compare what happened in the conversation against the scorecard guidelines for each disputed line item. Stay objective. The point of a dispute is to confirm whether the original score was right, not to defend it.
Step 4: Decide on each disputed item
For each disputed line item, choose one of:
Keep the original score — the original evaluation stands for this item
Change to the agent's proposed score — the agent was right
Set a different score — neither the original nor the proposed score reflects what happened
If the agent disputed the overall outcome, decide whether the new outcome they proposed is correct.
Step 5: Add your reasoning
Use the reviewer verbatim field to explain your decision. This is important whether you keep the original score or change it.
Be clear about:
What you found in the conversation
How it maps to the scorecard guideline
Why you've kept or changed the score
Strong reviewer notes help agents understand the outcome and reduce repeat disputes.
Step 6: Set the dispute status
Mark the dispute as one of:
Status | Use when |
Changed | You've adjusted one or more scores or the outcome |
Unchanged | The original scores and outcome stand |
Step 7: Submit your review
Click Submit to finalise the review.
When you submit, the system notifies:
The agent (or whoever raised the dispute) of the outcome
The original evaluator that the dispute has been resolved
If you changed scores, the evaluation now shows the updated scores and outcome. The dispute history on the evaluation records both the original and the revised values.
Tips for handling disputes well
Treat each dispute on its own merits — don't carry over feelings from previous interactions
Lean on the scorecard guidelines, not personal preference
Document your reasoning even when you're keeping the original score. Agents need to know they were heard
Process disputes promptly. Long delays erode trust in the QA process
If you spot a pattern in disputes (repeated challenges on the same line item, for example), flag it as a potential calibration issue