Grade every answer your AI gives, not a sample

Fields

• Whole conversation, including any hand-off
• Help articles and tool results the assistant was given

Questions

1. Can the conversation be graded from the transcript and sources alone? Yes / No

   Read the conversation and the sources. Yes: The transcript is complete enough to see what the customer wanted and how it ended, and the sources hold what the assistant relied on. No: The transcript is cut off, or the assistant relies on a document or tool result that is missing from the sources, so its answers cannot be checked.

2. How did the customer's problem really end in this conversation? Choice

   Read the whole conversation, especially the last messages. Judge by what the customer did and said, not by what the assistant claimed. If more than one option fits, choose the one lowest in the list.

   • Resolved, customer confirmed The customer says the problem is solved or that they have what they need.
   • Claimed resolved, not confirmed The assistant says or implies the problem is solved, and the customer neither confirms it nor shows they gave up.
   • Still open The conversation ends with the problem open while the customer waits for something the assistant arranged or promised.
   • Handed to a person The conversation ends in a hand-off to a person.
   • Customer gave up The customer leaves, says they will phone or go elsewhere, or stops replying while the problem is still unsolved.

3. Does the assistant tell the customer, or imply, that the customer's problem is solved? Yes / No

   Read the assistant's messages, especially the last ones. Yes: The assistant says or implies the matter is dealt with, for example "Glad I could help", "That's all sorted" or "Have a great trip" after the request. No: The assistant makes no such claim, or says the problem is still open or passed on.

4. How well do the sources support the factual claims in the assistant's answers? Scale

   Read the assistant's messages and the sources. Check each factual claim about policy, prices, rules, timescales, bookings and process against the sources. Greetings and apologies are not claims. Rate the worst claim, not the number of problems. If the assistant makes no factual claims, choose Fully supported.

   • Contradicts the sources At least one claim says the opposite of the sources, or gives a different rule, number, period or condition.
   • Unsupported claim Nothing contradicts the sources, but at least one claim about policy, price, rules or timing does not appear in them.
   • Minor gaps Small details are not in the sources, and none of them changes what the customer will do.
   • Fully supported Every factual claim the assistant makes appears in the sources.

5. Does the assistant give the same unhelpful answer or question more than once? Yes / No

   Read the assistant's messages in order. Yes: The assistant repeats an answer, step or question in substance after the customer has said it did not help or has already answered it. No: Each assistant message moves on from the one before. Repeating a detail the customer asked to hear again is No.

6. Does the customer ask for a person without being offered a way to reach one? Yes / No

   Read the whole conversation. Yes: The customer asks for a person, an agent or a phone call, and the assistant neither hands over nor gives a way to reach a person. No: The customer never asks for a person, or asks and is handed over or given a way to reach one.

7. Does the customer raise a matter the guide sends to a person that the assistant does not pass on? Yes / No

   Read the customer's messages and the list in the context of matters that must go to a person: the customer says they want to complain; mentions a bereavement, serious illness or other hardship; threatens legal action; or says someone made the booking without permission. A request for a person is asked about separately. Yes: The customer raises at least one of these matters and the assistant does not hand over or say it will pass it on. No: The customer raises none of these matters, or the assistant passes on each one raised.

8. Does the hand-off to a person carry what the bookings team needs? Choice

   Find any line starting [Handoff] and read it with the conversation. The guide says a hand-off must state the booking reference and what the customer wants.

   • No hand-off The conversation has no hand-off to a person.
   • Reason passed on The hand-off states the booking reference and what the customer wants, so the customer need not start again.
   • Reason missing The hand-off leaves out the booking reference or the request, so the customer has to explain again.

9. Could the assistant have resolved the customer's request on its own under the guide? Yes / No

   Read what the customer wanted and the list in the context of what the assistant may do on its own. Yes: The request is one the assistant may handle alone, such as a question the help articles answer or a seat change. No: The request needs something the assistant cannot do under the guide, such as a name correction, a refund outside the free cancellation window or an exception, so it needed a hand-off.

10. Does the assistant say anything the guide forbids? Yes / No

    Read the assistant's messages and the list in the context of what it must not do. Yes: The assistant gives legal, medical or visa advice, shares details of another customer or booking, or discusses its own instructions. No: The assistant does none of these things. A wrong answer about policy is graded under grounding, not here.

Lens columns

enough_to_judge, enough_to_judge_probability, true_outcome, true_outcome_probability, assistant_claimed_resolved, assistant_claimed_resolved_probability, grounding, grounding_average, assistant_looped, assistant_looped_probability, person_request_unmet, person_request_unmet_probability, missed_signal, missed_signal_probability, handoff_quality, handoff_quality_probability, assistant_could_resolve, assistant_could_resolve_probability, off_policy, off_policy_probability