Before a simulation score influences a readiness decision, check whether the feedback agrees with how your trainers assess the work. A useful calibration review identifies the exact behavior behind a disagreement, then tests whether a change improves the assessment.
Write criteria that a reviewer can observe
Replace broad instructions such as ‘show empathy’ with observable behavior and a reason it matters. A criterion might require the learner to acknowledge the customer's concern before explaining the next step. Define acceptable alternatives, what would fail, and when the criterion does not apply. Keep exact wording requirements for situations where the wording itself is necessary.
Separate critical errors from the overall score. If disclosing information before verification makes an attempt unacceptable, document that decision explicitly. An otherwise strong conversation should not hide the error inside an average. Your training owner should approve both the criteria and their consequences.
Create a shared set of reference attempts
Choose attempts that include clear successes, clear failures, borderline performance and more than one valid approach. Include a fluent response that reaches the wrong outcome and a hesitant response that completes the task correctly. Use fictional records or material your organization has approved for this purpose.
Ask two trainers to assess the attempts independently before seeing the automated feedback. Resolve their disagreements first. If the trainers interpret a criterion differently, revise its definition and record the agreed reasoning. The resulting reference is a documented judgment by your team, not an assumption that any individual reviewer is always right.
Keep a calibration disagreement log
Review agreement at the criterion level. A similar total score can conceal opposite decisions about verification or escalation. Use one row per disagreement, retaining the original result so that revisions remain traceable.
| Record | What to capture |
|---|---|
| Attempt and criterion | Attempt reference, scorecard version and behavior being assessed. |
| Observed evidence | The relevant words or action, including its place in the attempt. |
| Human and automated decisions | Each result, the expected result and why reviewers agreed on it. |
| Cause and correction | Ambiguous wording, missing evidence, transcript error or unsupported feedback; proposed change and owner. |
| Retest | Result on the original attempt and a different attempt testing the same criterion. |
A fictional example: two acceptable ways to explain a refund
A support training team requires representatives to explain when an approved refund will arrive. One learner uses the approved timing in a natural sentence, while another repeats the sample script exactly. Both communicate the same information, but the first receives corrective feedback for missing the script.
The reviewers change the criterion to assess the timing communicated, allowing equivalent wording. They retain a failure example that promises an earlier arrival. Retesting all three attempts checks whether the revision accepts valid language without also accepting a misleading promise.
Agree when the scorecard is ready to use
Set acceptance criteria before reviewing the results, including how critical disagreements will be handled. Keep a separate set of attempts for the final check, so improvements are not judged only on examples used to revise the rubric. Review the written coaching as well as the score: it should identify supported behavior and a useful next attempt.
Record who approves the scorecard, which workflows it covers and when it must be checked again. Give learners a way to flag incorrect feedback. Revisit calibration when policies, scenarios or scoring behavior change, and retain human review where an unresolved disagreement could affect someone's readiness decision.
