Give each vendor the same representative task and ask your own team to complete it. A shared evaluation brief makes it easier to see which differences affect training, how much setup your team will own and what still needs to be demonstrated.
Prepare one evaluation brief
Choose a workflow that includes the main reason you are buying. Describe the learner's role, starting information, customer situation, tools involved and expected outcome. Provide the same fictional records and approved policy excerpt to each vendor. Specify which mistakes should be possible and which mistakes should affect the result.
Agree who will evaluate the learner experience, who will review assessment and who will operate the program. Give vendors the same preparation window and record any extra assistance they receive. Allow different implementations, but keep the task and acceptance criteria consistent so that presentation polish does not determine the comparison.
Observe the whole exercise
Have a learner attempt the task, review the feedback and try again. Then ask a trainer to make a small change, such as updating a policy detail or assigning the exercise to a new cohort. Record what they can do independently and what requires vendor support. Include the time spent resolving configuration problems.
Ask an experienced representative to take a valid alternative path and deliberately make a relevant mistake. Check whether the conversation, system behavior and feedback still make sense. Write down any simplified behavior. A simplification may be acceptable when it preserves the skill being practiced, but your team should make that decision explicitly.
Complete the comparison worksheet
Use the same rows for every vendor, with a link to observed evidence. Mark each item demonstrated, partly demonstrated, not demonstrated or outside the agreed scope. A proposed capability remains unverified until you can test it.
| Area | Evidence to record |
|---|---|
| Workflow fit | Required actions completed, alternative paths and behavior after an error. |
| Assessment | Criteria evaluated, evidence shown and disagreements with trainer judgment. |
| Learner experience | Access, clarity of instructions, feedback and repeat-attempt experience. |
| Administration | Who sets up, assigns, edits and maintains the exercise; observed effort and help needed. |
| Commercial scope | A quote for the same learner pattern, services, support and assumptions. |
| Open questions | Missing evidence, proposed workaround, owner and date for verification. |
A fictional example: a duplicate charge
A support team asks two vendors to simulate a duplicate-charge complaint. The learner must verify the customer, compare transaction records, explain the permitted resolution and leave an accurate note. Both demonstrations sound plausible, but only the team's hands-on test reveals whether the record change is assessed.
The evaluators also change the policy's escalation condition. One setup requires vendor assistance and the other allows their trainer to make the change. They record both the assessment difference and maintenance responsibility, then decide how much each matters for their intended program. They do not treat either result as a claim about every workflow the vendors offer.
Make the decision criteria visible
Separate requirements that must pass from preferences you can trade off. Set any scoring weights before reviewing the demonstrations, and keep unresolved critical requirements outside an overall average. Review the completed worksheet with training, operations and procurement. If the comparison remains unclear, request a focused retest of the disputed behavior with the same inputs, rather than another general product tour.
