Use cases

Check AI call summaries against the call before saving

Compares each AI-written call summary with the transcript: anything invented, a complaint, support need or promise left out, a wrong outcome.

Ask for an invite

Try it on this example

Example · Renewal call: summary says the customer agreed and drops the large print request

Call summary written by the AI model: Call summary Reason for call: Customer called about her home insurance renewal premium, which has increased from £431 to £612. Discussion: Agent explained the increase is due to higher rebuild costs and storm claims in the area. Customer expressed dissatisfaction with the increase as a long-standing customer with no claims. Agent suggested removing accidental damage cover to reduce the premium. Customer advised her husband has passed away and is still named on the policy; agent advised a copy of the death certificate is needed to update the policyholder. Outcome: Customer agreed to renew at £612. Agent has asked underwriting to review the premium. Next steps: Agent to call customer back by Thursday afternoon with a requote. No further action.

Call transcript with speaker labels

Agent: Good morning, Pellbrook Insurance, you're speaking with Dev. Can I take your name and policy number, please? Customer: Hello, yes. It's Irene Castell. The policy number is on the letter somewhere. Here it is, H P 4 4 8 1 2 0. Agent: Thank you, Mrs Castell. For security, can you confirm the first line of your address and your date of birth? Customer: 14 Salter Grove, and the ninth of March 1951. Agent: Thank you, that's all confirmed. How can I help today? Customer: I've had the renewal letter for the house insurance and it's gone up from four hundred and thirty-one pounds to six hundred and twelve. That's nearly two hundred pounds more, and nothing has changed. I've been with you eleven years. I'm not happy about it at all, frankly. Agent: I'm sorry, I can see why that's a shock. Let me look at what's changed. Bear with me a moment. Customer: Take your time. Agent: So the main reason is that the rebuild cost for your house has gone up. That affects buildings cover for all our customers this year. There's also an increase for your area after a lot of storm claims last winter. Customer: But I've never claimed. Not once in eleven years. It doesn't feel very fair, loyal customers paying for everyone else. Agent: I understand, and I'll note that you're unhappy with the increase. Can I ask a couple of questions to see if we can bring it down? Customer: Go on. Agent: At the moment you've got accidental damage cover on both the buildings and the contents. That's about seventy pounds of the price. Some customers take it off if they don't feel they need it. Customer: I'd have to think about that. My husband always dealt with all this. He passed away in June, so this is the first time I've had to do it myself. Agent: I'm very sorry to hear that, Mrs Castell. Customer: Thank you. It's just that I find these letters very hard to follow. The print is so small. And I'd rather not keep ringing up. I get flustered on the phone. Agent: That's completely understandable. Would it help if we sent your letters and documents in large print from now on? Customer: Oh, yes, that would help a lot. Agent: I'll set that up now. I'll also add a note to your record so that anyone you speak to knows about your husband, and that you'd prefer things in writing where we can. Customer: Thank you. Agent: Is the policy in your name only now, or is it still in joint names? Customer: It's still got both our names on it. I didn't know I had to change that. Agent: That's alright, we can sort that out. I'll need a copy of the death certificate, but there's no rush. I'll post you a form and a prepaid envelope, in large print, this week, and you can send it back when you're ready. Customer: Alright. Agent: Now, the price. What I'd like to do is ask our underwriting team to look at it again, as you've been with us eleven years with no claims. I can't promise it will change, but I'll put the request in today, and I'll call you back by Thursday afternoon with their answer and a quote with and without the accidental damage cover. Is this number the best one to reach you on? Customer: Yes, this is my home number. I'm usually in. Agent: Lovely. Your renewal date is the first of October, so there is plenty of time, and nothing will change before I call you back. Customer: So I don't need to do anything yet? Agent: No, don't do anything yet. Let's see what underwriting say first. Customer: Alright, I'll wait for your call then. I'm not agreeing to the six hundred and twelve, mind. Agent: Understood, I've noted that. Is there anything else I can help with today? Customer: No, that's everything. Thank you, Dev, you've been very kind. Agent: You're welcome, Mrs Castell. I'll speak to you by Thursday. Take care. Customer: Bye now.
  1. Can the summary be checked from the transcript alone?Yes97%
  2. How well does the transcript support the statements in the summary?Contradicts the call100%
  3. Does the summary leave out a complaint the customer made on the call?Yes68%
  4. Does the summary leave out circumstances the customer mentioned that may mean they need extra support?Yes94%
  5. Does the summary leave out something the agent promised to do, or the time given for it?Yes96%
  6. Does the summary record how the call ended as the transcript shows it?Different outcome100%
  7. Does the summary attribute something said or decided on the call to the wrong person?Yes84%
  8. How far can the next person rely on this summary as the record of the call?Wrong in a way that matters100%

These are real answers stored from one run on this example.

The prism behind it

Check AI call summaries against the call before saving8 questions

Fields

  • Call transcript with speaker labels
  • Call summary written by the AI model

Context

These are summaries of phone calls with customers of Pellbrook Insurance, a UK home and car insurer. After each call, the contact centre's software writes a summary with a generative AI model, and the summary is saved to the customer's record. The next agent, the complaints team and auditors rely on it as the record of the call. This check runs before a summary is saved. It compares the summary with the call transcript; it does not write or correct a summary. A summary with a problem goes back to the agent who took the call, who corrects it. What every summary must record, when it happened on the call: - Why the customer called. - Any complaint. Our complaints policy counts any expression of dissatisfaction with our service, prices, decisions or staff as a complaint, whether or not the word complaint is used. - Any circumstance the customer mentions that may mean they need extra support, such as a bereavement, an illness or disability, money worries, or difficulty reading, hearing or using the phone, and any support they ask for. - Every commitment the agent makes, such as a call back, a letter, email or form to send, a referral to another team or a change to the policy, with any day or time the agent gives. - How the call ended: what was agreed, changed or left open, and what happens next. Rules for the summary: it states only what was said on the call; it attributes each statement, request and decision to the person who made it; it may shorten what was said but must not change it. The transcript comes from speech recognition, with the speaker labels Agent and Customer, and may contain small recognition errors. Code also checks each amount, date and policy number in the summary against the transcript and the policy system.

Questions

  1. Can the summary be checked from the transcript alone? Yes / No

    Read the transcript and the summary. Yes: The transcript covers the call from greeting to goodbye and is readable enough to see what was said, even with small recognition errors. No: The transcript is cut off, mostly unreadable, or plainly about a different call from the one the summary describes, so the summary cannot be checked.

  2. How well does the transcript support the statements in the summary? Scale

    Check each statement in the summary about why the customer called, what was said, what was agreed and what happens next against the transcript. Something left out of the summary is not judged here; the questions on omissions record it. Rate the worst statement, not the number of problems.

    • Contradicts the call At least one statement says the opposite of the call, or gives a different amount, date, decision or outcome.
    • Unsupported statement Nothing contradicts the call, but at least one statement, such as a reason, a request or an agreement, was never said.
    • Minor gaps Small details are not in the transcript, and none of them changes what the next person reading the record would do.
    • Fully supported Every statement in the summary can be found in the transcript.

  3. Does the summary leave out a complaint the customer made on the call? Yes / No

    Read the customer's lines and the complaints rule in the context, then the summary. Yes: The customer expresses dissatisfaction with our service, prices, decisions or staff, and the summary does not record it as a complaint or as dissatisfaction. No: The summary records each complaint, even in a few words, or the customer expresses no dissatisfaction with us. Frustration with an event itself, such as a storm, with nothing aimed at us, is not a complaint.

  4. Does the summary leave out circumstances the customer mentioned that may mean they need extra support? Yes / No

    Read the customer's lines and the support list in the context, then the summary. Go only on what the customer says, never on accent, fluency or tone of voice. Yes: The customer mentions a circumstance on the support list, or asks for support such as large print or contact in writing, and the summary leaves out that circumstance or that request. No: The summary records each such circumstance and each request for support, or the customer mentions none.

  5. Does the summary leave out something the agent promised to do, or the time given for it? Yes / No

    Read the agent's lines, then the summary. Yes: The agent commits to an action, such as a call back, sending a letter, document or form, a referral, or a change to the policy or the record, and the summary leaves out that action or the day or time the agent gave for it. No: The summary records every commitment with any time given, or the agent makes none. Explaining how something usually works is not a commitment.

  6. Does the summary record how the call ended as the transcript shows it? Choice

    Read the end of the call: what the customer agreed, decided or left open, and what happens next. Compare it with the outcome the summary gives.

    • Outcome recorded correctly The summary states how the call ended, and it matches the transcript.
    • Different outcome The summary states an outcome the call did not reach, such as a renewal, cancellation, change or agreement the customer did not give, or records as open something the call settled.
    • Outcome not stated The summary does not say how the call ended or what happens next.

  7. Does the summary attribute something said or decided on the call to the wrong person? Yes / No

    Compare who says, asks for, proposes and agrees to each thing in the transcript with who the summary says did. Yes: The summary says the customer said, asked for or agreed to something the agent said or proposed, or the other way round. No: Every statement, request and decision in the summary is attributed to the person who made it, or the summary attributes nothing to anyone. A statement that was never made at all is judged under grounding, not here.

  8. How far can the next person rely on this summary as the record of the call? Choice

    Judge the summary as a whole against the transcript and the rules in the context. The answer is a suggestion; the agent who took the call corrects the summary. If more than one option fits, choose the one lowest in the list.

    • Faithful Records everything the context requires that happened on the call, all correct and attributed to the right person.
    • Minor gaps Correct in substance: a small detail is missing or loosely worded, and nothing the context requires is left out or wrong.
    • Wrong in a way that matters A statement contradicts the call or was never said, or the summary leaves out or gets wrong a complaint, a support need, a commitment or the outcome.

Lens columns

enough_to_judge, enough_to_judge_probability, grounding, grounding_average, complaint_omitted, complaint_omitted_probability, vulnerability_omitted, vulnerability_omitted_probability, commitment_omitted, commitment_omitted_probability, outcome_matches, outcome_matches_probability, wrong_speaker, wrong_speaker_probability, summary_fit, summary_fit_probability

Run it on your own text

Add this prism in the app, change any question, and test it on a file of your own.

Ask for an invite