AI execution layer

AI agent vs chatbot: an order-status test for customer service

The clearest AI agent vs chatbot test is a customer asking “where is my order?” A chatbot can explain a published delivery policy. An assistant with authorised access can look up a particular order. Changing its delivery address is a separate action with a different owner and risk.

Two colleagues reviewing documents at an office desk
Photo by Vitaly Gariev on Unsplash

Part of our guide to What is an AI execution layer? A plain-English guide for operators

On this page · 6 sections
  1. Split the customer's sentence into three different jobs
  2. Give the chatbot a fair test
  3. Ask the agent to reconcile two sources that disagree
  4. Draw the approval line at promises and changes
  5. Score the conversation after the handover
  6. Frequently asked questions

A customer asks where a parcel is and whether the address can be changed. The support team has to identify the order, interpret the tracking record and decide whether a change is permitted. A fluent reply can hide the fact that none of those jobs has actually happened. Write the expected outcome for each part before choosing a tool.

Split the customer's sentence into three different jobs

For a general question about delivery areas, a well-maintained answer from a public knowledge base may be all the business needs. Nobody has to identify a customer to explain which suburbs the store serves. For an individual order, the system must establish which order the person is entitled to discuss. For a change, staff may need to consult the warehouse, check dispatch status or perform additional verification under their normal policy.

Use a fictitious order during the test. Give it a reference, an order owner, a carrier event and a date that the customer expected delivery. Include a note that the requested address change is still pending. Ask the candidate to handle the conversation as an ordinary support operator would. Record the moment when it needs access to something beyond the public policy.

Give the chatbot a fair test

A good support chatbot can answer a common question accurately, link to current policy and direct a customer to the right person when the case becomes specific. Evaluate the clarity of that handover. Does it ask a customer to repeat the whole story? Does it tell them what information the human will need? Can they reach a person without trying several variations of the same question?

Try three messages. First ask a general question about standard delivery. Then ask about the made-up order. Finally ask to send it to a different address. The chatbot should be able to explain which part it can answer from approved public material and which part needs the support team. If it invents a dispatch event, the problem is accuracy even if the directions sound polite.

Customer questionSource neededAppropriate completion
“Do you deliver to my suburb?”Current published delivery policyAnswer or refer to the published coverage check
“Where is my parcel?”Authorised order and tracking informationState the latest verified event and its time
“Please change the address”Verified identity and current dispatch stateRoute the requested change for the required approval
“Can you guarantee Friday?”A service commitment the business can actually makeExplain what is confirmed and escalate uncertainty

Ask the agent to reconcile two sources that disagree

Create an order record that says “packed” and a carrier label that has been created but has no collection scan. A claim that the carrier has the parcel would go beyond the evidence. Ask the candidate to identify the latest verified event, explain what remains uncertain and tell the support operator which source to check next. Keep the output short enough to use during a call.

Add a second fictitious customer with the same surname and a similar reference. The agent should apply the business's normal verification process and stay within the authorised customer's records. Do not expose another customer's details in an attempt to confirm identity. The person running the trial should know the expected matches and verify them in the destination independently.

Now give the carrier record a later scan while the original draft is waiting for a support person. Does the operator know when the draft was prepared? Can they refresh the facts before communicating a new delivery expectation? A perfectly accurate answer can go stale quickly when the team is dealing with moving goods. Time-stamped evidence makes that change visible.

Draw the approval line at promises and changes

A support operator may be allowed to read an order and draft a response while a supervisor owns refunds, address changes or replacement shipments. Make those boundaries explicit in the trial. A general right to speak to customers should not silently become permission to redirect a parcel or waive a charge. Each action needs a defined owner and a record of the decision.

Ask the candidate to draft a reply using only the verified events. The reviewer should be able to edit and approve the message without changing the order itself. Then test the reverse case: a supervisor approves a correction in the order system, but the customer has not yet been told. The support record should distinguish the completed update from the pending communication.

If the team evaluates Fulcrum Sessions, use the same separation in the brief given to a specialist. Ask for an evidence-based summary, a draft response and a named person to authorise a change. Verify the tools and permissions available in your actual setup. The existence of an orchestration layer cannot settle which employee may make a customer promise.

Score the conversation after the handover

Measure whether the customer reached a correct answer, whether a support person had to repeat work and whether any order changed prematurely. Track the examples where the candidate asked for clarification. A sensible question can be a successful result when the account identity is ambiguous. Penalising every handover encourages an assistant to guess when uncertainty is exactly what the operator needs to see.

Run the exercise with somebody who covers the support desk occasionally. Give them the transferred conversation and see whether they can identify the customer request, the source checked, the latest event and the action still required. Record any missing detail and the time spent finding it. The value of an agent often lies in a better handover even when the final message remains a human decision.

Choose the least complicated option that handles your actual mix of enquiries. Keep the chatbot on public policy where it works well. Add an agent for a bounded lookup when access, identity checks and recovery are understood. Reserve exceptional commitments for staff. Revisit these boundaries when your fulfilment process changes or the customer support team gains a new responsibility.

Related guides: What is an AI execution layer? A plain-English guide for operators · AI agents vs chatbots for business: what operators actually need · Build your own AI agents vs buy: what a DIY agent stack really needs.

Frequently asked questions

Is a chatbot always unable to look up orders?

Products use the labels differently. Inspect the actual tool access, identity checks and actions. A chat interface with an authorised lookup tool can do more than one limited to a public FAQ.

Should we measure success by fewer support tickets?

Count resolved enquiries and correct handovers as well. A customer who gives up before reaching a person can reduce ticket volume without improving service.

What is the simplest acceptance test for an address change?

Have a verified test customer request a change after the order has reached a stage where it is no longer routine. The system should show the current state, preserve the original address and route the decision to someone with the appropriate authority.