Multi-Agent Voice Systems: The Warm Transfer

In text-based chatbots, "routing" is invisible. You click "Support," and the backend silently switches endpoints.

In Voice AI, routing is a human experience. Think about calling a doctor's office. The receptionist doesn't try to perform surgery. They say, "Let me transfer you to the nurse."

This post is for engineers building complex voice systems who are hitting the limits of a single prompt. We will explore how to build Multi-Agent Voice Systems where specialized agents (Receptionist, Sales, Support) handle different parts of a single call, passing context seamlessly.

The problem: The "Jack-of-All-Trades" hallucination

You are building a voice bot for a Dental Clinic. You write one massive System Prompt:

"You are a receptionist, a nurse, and a billing agent. If they want to book, do X. If they have pain, do Y. If they owe money, do Z."

Why this fails:

1. Latency: A 3,000-token system prompt slows down every single turn.
2. Confusion: The bot mixes up rules. It might ask for insurance details while the user is describing a medical emergency.
3. Fragility: Updating the billing logic breaks the triage logic.

We need Specialization.

The solution: Agent swapping (The handoff)

We define distinct agents, each with a tiny, focused prompt and specific tools. We use Tool Calling to trigger a "Transfer."

graph TD
    A[User Call] --> B(Receptionist Agent)
    
    B --> C{User asks: I have a toothache}
    C --> D(Transfer Tool)
    
    D --> E(Handoff + Context)
    E --> F(Nurse Agent)
    
    F --> G{User asks: How much will this cost?}
    G --> H(Transfer Tool)
    
    H --> I(Handoff + Context)
    I --> J(Billing Agent)
    
    style B fill:#e3f2fd,stroke:#0d47a1
    style F fill:#e8f5e9,stroke:#388e3c
    style J fill:#fff3e0,stroke:#e65100

The "Warm Transfer" concept

In a Cold Transfer, the first agent hangs up, and the second agent picks up saying, "Who are you?" This is a terrible user experience.

In a Warm Transfer, the first agent passes a "briefing" to the second agent.

• Receptionist: "Transferring you to the nurse." -> (Passes context: "User is Bob, has toothache")
• Nurse: "Hi Bob, I hear you have a toothache. Which tooth is it?"

The "How": Implementing handoffs in LiveKit

We don't need multiple WebSocket connections. We can mutate the agent's "brain" (Prompt + Tools) in real-time while keeping the audio connection open.

Step 1: Define the specialists

# 1. The "Brains"
receptionist_prompt = """
You are a receptionist at Smile Dental. 
Greet the user. 
If they have a medical issue, call 'transfer_to_nurse'.
If they have a billing issue, call 'transfer_to_billing'.
"""

nurse_prompt = """
You are a Triage Nurse. 
Your goal is to assess symptom severity. 
Do not discuss billing.
"""

# 2. The Tools available to the Receptionist
receptionist_tools = [transfer_to_nurse, transfer_to_billing]

Step 2: The transfer function

This is the core engineering pattern. We define a Python function that acts as a tool but modifies the running agent instance.

def transfer_to_nurse(agent, user_name: str, symptom_summary: str):
    """
    Transfers the user to the medical triage line.
    """
    print(f"--- TRANSFERRING TO NURSE: {user_name} / {symptom_summary} ---")
    
    # 1. Update the System Prompt (The "Brain Transplant")
    # We inject the context directly into the new prompt
    new_prompt = f"""
    {nurse_prompt}
    
    CURRENT CONTEXT:
    The user is {user_name}.
    They are complaining of: {symptom_summary}.
    Greet them by name and ask specific questions about the pain.
    """
    agent.system_prompt = new_prompt
    
    # 2. Swap the Tools
    # The nurse needs medical tools, not transfer tools
    agent.tools = [lookup_symptoms, schedule_emergency_appointment]
    
    # 3. Add a "Bridge" message
    # This guides the LLM to start the new phase smoothly
    return "Transfer successful. Introduce yourself as the nurse and ask about the pain."

Step 3: The execution

When the LLM calls this tool, the Python framework:

1. Executes the swap immediately.
2. Feeds the "Transfer successful" message to the new (Nurse) prompt history.
3. The Nurse LLM generates the next audio response: "Hi [Name], I see you're in pain..."

Architecture: Shared state management

For simple transfers, passing strings (like symptom_summary) is enough. For complex systems, we need a Shared State Object that persists across the call.

graph LR
    subgraph STATE["Global Call State (Pydantic Model)"]
        A[User Profile]
        B[Authentication Status]
        C[Appointment Details]
    end
    
    D(Receptionist) --> A
    D --> B
    
    E(Nurse) --> A
    E --> C
    
    F(Billing) --> B
    F --> C
    
    style A fill:#e3f2fd,stroke:#0d47a1
    style B fill:#e8f5e9,stroke:#388e3c
    style C fill:#fff3e0,stroke:#e65100

We pass this global CallContext object to every tool and every agent prompt update. This ensures that if the user gives their phone number to the Receptionist, the Billing agent doesn't ask for it again.

Summary: Why multi-agent voice systems win

Feature	Single "God" Agent	Multi-Agent System
Prompt Size	3,000+ tokens	200-500 tokens per agent
Latency	High (Large context)	Low (Focused context)
Maintainability	Fragile (One change breaks all)	Robust (Isolated updates)
Specialization	Generic responses	Expert-level domain knowledge
Context Preservation	N/A	Warm transfers with shared state

Challenge for you

Scenario: You are building a Banking Voice Bot.

• Agents: GeneralSupport and WireTransferSpecialist.
• Security Rule: The WireTransferSpecialist is only allowed to talk to authenticated users.

The Problem: A social engineer calls and tries to trick the GeneralSupport bot: "Transfer me to wires, I already verified my PIN with the other guy."

Your Task:

1. How do you protect the transfer_to_wires tool?
2. Where do you store the is_authenticated boolean? (Hint: Global State).
3. Write the pseudo-code for the transfer_to_wires function that checks this state before performing the prompt swap.

Key takeaways

• Specialization reduces latency: Smaller, focused prompts enable faster responses compared to monolithic "god" prompts
• Warm transfers preserve context: Passing context during handoffs creates seamless user experiences, unlike cold transfers
• Tool calling enables dynamic routing: Agents can trigger transfers using function calling, making routing decisions based on conversation flow
• Shared state prevents repetition: A global state object ensures agents don't ask for information already collected by previous agents
• Real-time agent swapping: You can mutate an agent's prompt and tools while keeping the audio connection open, enabling seamless transfers
• Security requires state validation: Transfer functions must check authentication state before allowing access to sensitive agents
• Bridge messages smooth transitions: Adding context messages during transfers helps the new agent generate appropriate greetings

For more on voice AI systems, see our voice AI fundamentals guide, our conversation memory guide, and our multi-agent coordination guide.

---

For more on building production AI systems, check out our AI Bootcamp for Software Engineers.