A supervisor changes one quality question from “Was the customer helped?” to “Did the response include every action the customer must take?” The next run produces a very different score. That does not automatically mean service got worse. The rule changed.
The safe way to test Dynamics 365 quality criteria is to keep the new criteria in draft, run them against a deliberately varied set of real cases or conversations, compare each predicted answer with an independent human reference, revise one ambiguity at a time, and rerun the same evidence. Publish only when the rule behaves consistently on both ordinary and difficult records.
Start with one decision the rule must make
Do not begin with a target score. Begin with a decision a reviewer should be able to defend from the record.
Suppose the question is whether the representative explained the next step. Define which message contains that evidence, what a complete next step needs, what must be ignored, and what answer applies when the record is incomplete. A question such as “Was the case handled well?” asks the evaluator to invent its own standard. “Did the final reply name the action, owner, and expected timing?” is testable.
Microsoft’s current evaluation criteria guide recommends specific questions, direct answer conditions, explicit exclusions, and named evidence sources. It also lets a team mark a question as critical when one failed answer must fail the evaluation. Use that power carefully. A vague critical question turns ordinary uncertainty into an automatic failure.
The owner of the support operating model should help define what representatives can actually observe and control.
Build a test set that can expose the rule
A test set full of clean, successful cases proves very little. Include records that force the rule to show its limits.
Microsoft says a simulation can run on draft or published criteria using real case or conversation data. Conditions select the records, and the default uses the 25 most recent matches. Choose conditions deliberately and record why each case is present. Case attachments cannot be included, so do not depend on evidence the evaluator cannot receive.
| Test record | Why it belongs | Expected behavior | Publish blocker |
|---|---|---|---|
| Routine complete case | Confirms the basic rule | Clear supported answer | Wrong answer on visible evidence |
| Missing field or message | Tests evidence limits | Unknown or defined fallback | Confident inference from absent data |
| Transfer or handoff | Tests ownership boundaries | Uses only the named evidence source | Blames the wrong representative |
| Ambiguous wording | Exposes instruction gaps | Reviewers reach the same interpretation | Reasonable reviewers disagree |
| Critical failure | Tests the stop condition | Fails only when the required evidence is present | False critical failure |
Pull the evidence from the same place the live evaluation will use. If conversations move through a shared inbox, preserve the relevant message order, ownership change, and customer reply instead of reconstructing a cleaner story later.
Compare predictions with a human reference
Before the simulation, have two knowledgeable reviewers independently label the records with the written rule. They are creating a reference answer, not voting on whether they liked the response.
Then compare the agent’s predicted answer, reasoning, and critical result with that reference. Separate three problems:
- the evidence was unavailable;
- the instruction allowed more than one reasonable reading;
- the prediction contradicted clear evidence.
Those problems need different fixes. More instruction cannot restore missing data, and a new score cannot repair an ambiguous question. A disagreement log shows where the rule stops being defensible.
If an AI workflow later uses the result to recommend coaching or escalation, test that downstream decision separately. A correct evaluation answer can still feed a badly designed action.
Change one thing and rerun the same cases
Fix the smallest cause you can identify. Rewrite one question. Move one exclusion rule before the evidence instruction. Split a double question into two. Add a fallback answer for missing information.
Do not change the rule, sample, weights, and expected answers together. You will not know which change improved the result.
Dynamics 365 keeps published criteria running while edits are saved as a draft. Its version history records each edit and publish action, and a previous version can be restored. Simulation results do not alter customer records or organizational quality metrics, but each run consumes Copilot credits. Keep the test set compact enough to rerun and broad enough to catch the failures that matter.
Set a publish gate and a rollback owner
No universal agreement percentage makes a criterion safe. Match the gate to the decision risk. Coaching can tolerate more uncertainty than compliance or access decisions.
At minimum, require no unexplained critical failures, no confident answer when required evidence is absent, documented treatment of every reviewer disagreement, and a named owner who can withdraw the version. When the draft passes, publish it with its test set, expected answers, simulation date, and known limitations.
New evaluations use the latest published version. That makes release monitoring part of the test. Watch whether disagreement, unknown answers, or critical failures shift after publication. Keep the previous version available and define the signal that triggers a rollback.
The security model should also cover who may edit criteria, run simulations, inspect evaluations, and publish a version. Microsoft’s quality evaluation setup guide lists distinct Quality Administrator, Quality Manager, and Quality Evaluator roles, plus connection, credit, AI-agent, and data-movement prerequisites. Passing a simulation does not replace those controls.
Keep the result connected to the customer record
Keep the evaluation linked to the customer record, reviewed evidence, criteria version, and action taken. Then a coach can explain the result and an operator can correct it.
If your process spans messaging, AI, and human review, map those boundaries before automating more of the decision. A DripTell demo can help a team inspect how customer context, ownership, and handoff evidence move through the wider workflow. The evaluation rule should still stand on its own and remain understandable outside any one platform.
Frequently Asked Questions
Can I simulate draft criteria without changing live quality scores?
Yes. Microsoft says simulations can use draft or published criteria, and their results do not affect customer records or organizational quality metrics. Confirm the required simulation flow and credits first.
How many cases should I test?
Use enough records to cover routine work, missing evidence, handoffs, ambiguity, and critical failures. The default recent set is convenient, but risk coverage matters more than a universal number.
Should a critical question be tested differently?
Yes. Include clear passes, clear failures, and incomplete evidence. A critical rule should fail only when the required evidence supports that result.
What should stop publication?
Stop when reviewers cannot apply the rule consistently, required evidence is unavailable, critical failures are false, or nobody owns rollback and correction.
Want help applying this guide?Ask the DripTell team
From reading to action
Want help applying this to your business?
Share your details. We will read the article you are on and start with a useful answer for your team.
DripTell Editorial
Practical guidance reviewed by the DripTell product and customer workflow team.
See how DripTell checks product claims, uses primary sources and handles corrections.
Editorial and source policy
