Keywords: Agent Assist, agent copilot, RAG, human-in-the-loop, insurance customer service

For a while I have been building an FAQ chatbot for insurance customers. It handles plenty of common questions, but the longer I worked on it, the clearer one thing became: in a heavily regulated industry like insurance, where wrong answers are expensive, fully automated replies are not the best answer for every conversation. Many conversations, especially those about claims, cancellations or exact benefit amounts, still need a human agent to sign off.

So I flipped the approach. Instead of having the AI answer the customer, let it sit next to the agent. When a customer message arrives, the AI has already prepared a draft reply, the knowledge it relied on, and the facts that still need confirming. The agent glances at it, edits a few words and hits send. This post walks through the design of that “customer service copilot” prototype.

All examples come from a fictional insurer: product names, terms and amounts are made up. Section 3 has an interactive demo you can step through and zoom into, or you can open the prototype in a new tab and explore freely.


1. What already exists

None of this is new. The industry calls it Agent Assist / Agent Copilot:

VendorProductHighlights
AWSAmazon Connect + Amazon Q in ConnectReal-time reply and knowledge-article suggestions during calls/chats
GoogleContact Center AI Agent AssistSmart Reply, Knowledge Assist with FAQ citations
SalesforceEinstein Service Replies / Agentforce for ServiceDrafts plus Knowledge Article citations
MicrosoftDynamics 365 Copilot for Customer ServiceDrafts, summaries, knowledge Q&A with sources
Zendesk / IntercomAgent Copilot / Fin CopilotSide-panel drafts and sources, one-click insert
China marketAlibaba Cloud, NetEase Qiyu, Sobot, etc.Suggested scripts, knowledge recommendations

They share three traits:

  1. AI drafts, a human sends.
  2. Every suggestion has a source the agent can trace.
  3. What the agent changed is recorded and fed back into the knowledge base and models.

If your company already runs one of these contact-center platforms, check whether its built-in copilot can plug into your own knowledge sources first. That is often cheaper than building your own. I built a prototype because the knowledge base and retrieval logic already lived in our own system, and I wanted to validate the interaction before deciding how to integrate it.


2. Design principles

Before drawing any screens, I wrote down five principles. Every later trade-off was judged against them:

  1. AI drafts, humans send. The AI never talks to the customer directly.
  2. Every sentence has a source. Key sentences in the draft carry [1] [2] citation markers; one click shows the original text. Whether agents trust a draft depends on how quickly they can verify it.
  3. Understand the context before answering. “How much do I get for a flight delay?” has completely different answers depending on product and plan. When the context is unclear, ask rather than guess.
  4. When unsure, help the customer narrow it down. A good clarifying question is itself a good reply.
  5. Every interaction is feedback. Each edit, checkbox and thumbs-down is data for improving the knowledge base and evaluating the model.

3. The UI: four columns (interactive demo)

What’s embedded below is the prototype itself, not a screenshot. As you move through the slides, each one switches the prototype to the matching scenario and outlines in red what the slide is about.

① Four columns

Queue · customer context · AI suggested reply · knowledge panel. Every waiting conversation already has a draft prepared in the background.

Prototype screenshot
100% ↗
Double-click or use +/- to zoom, drag to pan; scroll to zoom in fullscreen. Use “Interact” to click around the prototype.
  • Queue: every waiting conversation already has a draft generated in the background, tagged with its intent (answer / clarify / escalate) and product. When the agent switches to it, the draft is ready. No waiting.
  • Customer context: the known facts for this conversation, all editable (see section 5).
  • AI suggested reply: streamed and directly editable, with an intent banner and a collapsible “Why the AI thinks so” section on top.
  • Knowledge panel: three tabs (Cited / Related / Search the whole KB). On each card the agent can include it in the draft, insert its raw text, rate it, or report a problem.

4. Core flow

Customer msg ─┐
              ├─▶ ① Infer context ─▶ ② Retrieve ─▶ ③ Decide intent ─▶ ④ Assemble draft
Agent notes  ─┘    (product/plan)     (filter &       (answer /         (citations +
                                       grade by        clarify /         clarifying
                                       context)        escalate)         question)
                       ▲                                                     │
                       └──────── agent edits context / picks sources ◀───────┘

The key point: this is a loop, not a pipeline. Any change the agent makes to the context or to the selected sources reruns ②③④.


5. Context: make “known facts” explicit

If the context is just a few read-only lines, the gaps show up quickly once you picture how agents actually use it:

  • Upstream systems may send incomplete information. For example, we know it’s travel insurance, but not which plan.
  • Facts the AI “guessed” from the conversation need to be distinguishable and correctable.
  • After looking up the customer, the agent needs a way to attach the policy they found to the conversation.

So the context is an editable panel where every field shows its origin:

FieldPossible origins
ProductUpstream system / AI inference / Agent / Record lookup
PlanSame as above
Policy numberUpstream system / Record lookup
Claim numberUpstream system / Record lookup
Agent notesAdded manually by the agent (also used for retrieval)

A few details:

  • AI inferences should look tentative. Inferred values appear as dashed tags with the evidence (“from the customer’s message: flight, delay”) and a “Confirm” button. Until confirmed, sending triggers a warning (“Plan was inferred by AI, please confirm”) and the confidence score is lowered.
  • The agent overrides the AI. If the agent resets the product to “Unknown”, the AI won’t infer it back on its own. The implementation is simple: store an origin per field and skip inference when origin = agent.
  • Lookup beats typing. Search by name, phone, policy number or claim number. Selecting a policy fills in product and plan. Selecting a claim also brings in the policy it belongs to. If a customer has both a travel and a home policy, the draft answers for whichever one the agent picks.
  • Should a context change refresh the draft? My rule: if the agent hasn’t touched the draft, regenerate it immediately. If they have edited it, don’t overwrite. Show “Context changed — regenerate with new context” instead. Never throw away the agent’s work.

6. Clarification: when unsure, let the customer narrow it down

Each knowledge item is tagged with an applicability scope (which product, which plan), and the current context decides what happens to it: items that clearly don’t match are dropped; items whose product or plan is still unknown are “held”; personal records (policies, claims) are only used when the number matches.

Held items stay out of the draft. They still appear under “Related” with a yellow “Plan unknown” tag, and the agent can include them manually. Intent detection then handles three kinds of clarification:

SituationExampleHow the draft is written
Product unknown“I want to cancel my policy — can I get a refund?”List only the products the knowledge actually matched (e.g. travel, home) for the customer to pick, and mention they can share a policy number instead
Product known, plan unknown“My flight was delayed 7 hours — how much can I claim?”Answer the generic part first (how to make a claim), then list that product’s plans; hold back plan-specific details such as benefit amounts
Not enough knowledgeNothing relevant in the KBAsk the customer for more details, and nudge the agent to “Report missing knowledge”

Two points worth stressing:

  • Narrow the options. Asking “Which type of insurance did you buy?” and then listing every product line is a poor experience. List only the ones the knowledge matched, so the customer can pick at a glance.
  • Answer what you can first. With the plan unknown, “how to make a claim” doesn’t depend on the plan and can be answered now. Only the benefit amount waits for the plan. The exchange doesn’t feel evasive.

Once the customer replies “I have plan X”, the AI infers the plan (marked as an AI inference), re-runs retrieval, and the draft becomes a proper answer citing that plan’s delay terms.

Below the draft is a row of quick-ask buttons (product, plan, policy number, incident date, documents). The one most relevant to the current situation is highlighted, and a click inserts the question at the cursor.


7. Compliance guardrails

Some things an insurance agent must not say. The prototype flags them in real time with simple rules:

  • Commitment wording: “definitely covered”, “guaranteed”, etc.
  • Possible personal data: phone-number or ID-number patterns.
  • Lost grounding: intent is answer but the agent deleted every citation marker.
  • Unconfirmed inferences: the context still contains AI-inferred fields the agent hasn’t confirmed.

Also, when the customer’s message mentions a complaint, a lawyer or a regulator, the intent goes straight to escalate. The draft then leads with an apology and the follow-up arrangement, and suggests handing over to a dedicated colleague.

These rules are crude, and production needs something more robust. At the prototype stage, though, they show business stakeholders where guardrails should appear and what they should look like.


8. Feedback loop: turn every agent action into data

The prototype logs these events and can export them as JSON:

EventPurpose
sent: AI draft vs. what was actually sent, to compute the edit rateCore metric for draft quality
context_change / context_confirmAccuracy of AI inference; how often upstream data is missing
source_added / source_removedSurfaces retrieval-ranking problems
source_rating (👍/👎)Quality of individual knowledge items
kb_report (wrong / outdated / missing)Routed straight to knowledge-base owners
quick_askWhich clarifications are most common, to guide context collection

The edit rate is a normalized edit distance:

edit_rate = levenshtein(ai_draft, final) / max(len(ai_draft), len(final))

0 means sent as-is. The closer to 1, the more was changed. It isn’t perfect, since changing a key number weighs the same as rewording a pleasantry, but it is simple enough to start tracking trends. Each sent event carries the AI draft, the final text, the context and the citations, so it is a ready-made evaluation sample. Over time this builds into a regression test set labeled by frontline agents.


Closing thoughts

My biggest takeaway from this prototype: the hard part of a customer service copilot is not generation. It is context and trust.

  • Without clear context, even the most fluent draft misses the point. So make the context explicit, have the AI narrow things down when it’s unsure, and let the agent correct it at any time.
  • Without the agent’s trust, even a great draft gets deleted and rewritten. So every sentence needs a source, every inference needs to say “this is my guess”, and every guardrail needs to fire before sending.

The AI’s role here isn’t to replace the agent. It’s to let agents spend less time digging through documents and more time making judgment calls.

The company, products, terms, conversations and data in this post are all fictional and do not correspond to any real insurer or product.