How can I set a distribution of human oversight and auto publishing?
Required Feature Flags
The following feature flags and permissions are required to use this feature:
Feature Flag | Technical Name | Description |
Auto Work Queues |
| Enables Auto Work Queue templates |
Auto-Publish Evaluations |
| Enables the human-oversight vs auto-publish distribution slider |
SmartScore |
| Required for the automated scorecard the queue uses |
Required Permissions:
Manage auto-work queues (
admin.assignmentbuilders.configure) — to create or edit Work Queue templates and configure the distribution
Introduction
When setting up a work queue using an automated scorecard, you have the option to:
Automatically publish evaluations, allowing for 100% automation, or
Assign auto-scored evaluations for human review, which are then manually published.
This article explains how to use the percentage slider to set a distribution between human oversight and auto-publishing, giving you more control over how your evaluations are managed.
Settings
Go to Evaluate > click Auto Work Queues > click Create assignment rules.
When setting up or editing a work queue where the auto-publish feature is available, you’ll see options to determine whether human oversight is required:
Yes, publish without oversight:
100% of your evaluations will be automatically published by our copilot.
No, require human oversight:
Allows you to set a percentage of evaluations to be assigned to evaluators for review.
When you choose to require human oversight, a % slider will appear.
Use this slider to specify the proportion of evaluations to assign to evaluators for manual review. The remaining percentage will be auto-published by our copilot.
For example, setting the slider to 30% means that 30% of evaluations will require human review, while the remaining 70% will be auto-published.
Add evaluators when oversight is on
Human oversight needs people to do the reviewing. When No, require human oversight is selected — even on a queue that evaluates every contact — you must add at least one evaluator under Add your evaluators before you can save. Without an evaluator, the queue has no one to assign the human-review share to and will stop assigning contacts.
If you already run a queue on "evaluate all" with full auto-publish and then switch it to require oversight, the How many evaluations do you want to complete? section and the evaluator picker will appear. Pick your evaluators, set their queue shares to total 100%, and save.
Assigning evaluators and setting queue share
After you've set the human oversight percentage, you'll choose which evaluators handle the reviewed evaluations. Each evaluator is given a queue share — the percentage of the human-reviewed pool that goes to them.
How quotas are calculated
The platform turns each evaluator's queue share into a whole-number quota. It applies the share to the total number of evaluations that need review, takes the whole-number portion for each evaluator, and then distributes any remaining contacts across the evaluators so nothing is left unassigned.
Worked example. You've set human oversight to 30% and the queue will produce 100 evaluations, so 30 need human review.
Evaluator | Queue share | Quota |
Evaluator A | 50% | 15 |
Evaluator B | 30% | 9 |
Evaluator C | 20% | 6 |
Total | 100% | 30 |
If the totals don't divide cleanly (for example, 25 contacts split across three evaluators on equal shares), the platform spreads the leftover contacts across evaluators so the full pool is allocated.
What to set in practice
Aim for queue shares that total 100%. That way every flagged evaluation is assigned. Shares totalling less than 100% will leave some evaluations unallocated.
Set a share of 0% to temporarily exclude an evaluator from the rotation without removing them from the queue.
Add as many evaluators as you need. If you add more evaluators than there are flagged evaluations in a cycle, some evaluators will simply receive zero contacts that cycle.
Changing the distribution later
You can adjust the oversight percentage and the queue share for each evaluator from the Work Queue template settings at any time. Changes take effect from the next replenishment cycle.