Connecting Retell AI to Twilio: Complete Integration Tutorial

Building production-ready AI voice agents requires connecting powerful AI models to reliable telephony infrastructure. Retell AI provides state-of-the-art conversational AI, while Twilio offers enterprise-grade phone services. Together, they create a robust platform for AI voice automation.

This comprehensive tutorial walks you through integrating Retell AI with Twilio, from initial setup to production deployment.

Prerequisites

Before starting, you'll need:

1. Retell AI account: Sign up at retellai.com
2. Twilio account: Sign up at twilio.com
3. Node.js: Version 18+ installed
4. Public webhook endpoint: Use ngrok for local development
5. Basic knowledge: JavaScript, REST APIs, webhooks

Architecture Overview

Here's how the integration works:

┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│   Caller    │ ◄─────► │   Twilio    │ ◄─────► │  Retell AI  │
│  (Phone)    │  Voice  │  (Telephony)│ WebRTC  │  (AI Agent) │
└─────────────┘         └─────────────┘         └─────────────┘
                              │
                              ▼
                        ┌─────────────┐
                        │ Your Server │
                        │ (Webhooks)  │
                        └─────────────┘

Flow:

1. Caller dials your Twilio number
2. Twilio sends webhook to your server
3. Your server initiates Retell AI session
4. Retell AI connects to Twilio via WebRTC
5. AI agent handles the conversation
6. Call events sent to your webhooks

Step 1: Set Up Twilio

1.1 Get a Phone Number

# Install Twilio CLI
npm install -g twilio-cli

# Login to Twilio
twilio login

# Buy a phone number (US example)
twilio phone-numbers:buy:mobile --country-code US

Or purchase through the Twilio Console.

1.2 Get API Credentials

Navigate to Twilio Console and copy:

• Account SID: Found on dashboard
• Auth Token: Found on dashboard (click to reveal)

// Store in .env file
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_auth_token_here
TWILIO_PHONE_NUMBER=+1234567890

Step 2: Set Up Retell AI

2.1 Get API Key

1. Log in to Retell AI Dashboard
2. Navigate to Settings → API Keys
3. Create a new API key
4. Copy and store securely

// Add to .env file
RETELL_API_KEY=your_retell_api_key_here

2.2 Create an AI Agent

// create-agent.js
import axios from 'axios';

const RETELL_API_KEY = process.env.RETELL_API_KEY;

async function createAgent() {
  const response = await axios.post(
    'https://api.retellai.com/v1/agent',
    {
      name: 'Customer Support Agent',
      voice_id: 'elevenlabs-rachel',  // Choose from available voices
      language: 'en-US',
      prompt: `You are a helpful customer support agent for Acme Corp. 
      You can help with:
      - Order status inquiries
      - Product information
      - Appointment scheduling
      - General questions
      
      Be friendly, professional, and concise.`,
      
      // Optional: Add function calling
      functions: [
        {
          name: 'check_order_status',
          description: 'Check the status of a customer order',
          parameters: {
            type: 'object',
            properties: {
              order_id: {
                type: 'string',
                description: 'The order ID to check'
              }
            },
            required: ['order_id']
          }
        }
      ],
      
      // Webhook for function calls
      webhook_url: 'https://your-domain.com/webhooks/retell-functions'
    },
    {
      headers: {
        'Authorization': `Bearer ${RETELL_API_KEY}`,
        'Content-Type': 'application/json'
      }
    }
  );
  
  console.log('Agent created:', response.data);
  return response.data.agent_id;
}

createAgent();

Save the returned agent_id for later use.

Step 3: Set Up Your Server

3.1 Initialize Project

mkdir retell-twilio-integration
cd retell-twilio-integration
npm init -y

# Install dependencies
npm install express twilio axios dotenv
npm install --save-dev nodemon

3.2 Create Server

// server.js
import express from 'express';
import twilio from 'twilio';
import axios from 'axios';
import dotenv from 'dotenv';

dotenv.config();

const app = express();
app.use(express.json());
app.use(express.urlencoded({ extended: true }));

const PORT = process.env.PORT || 3000;

// Twilio credentials
const TWILIO_ACCOUNT_SID = process.env.TWILIO_ACCOUNT_SID;
const TWILIO_AUTH_TOKEN = process.env.TWILIO_AUTH_TOKEN;
const twilioClient = twilio(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN);

// Retell AI credentials
const RETELL_API_KEY = process.env.RETELL_API_KEY;
const RETELL_AGENT_ID = process.env.RETELL_AGENT_ID;

app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

Step 4: Handle Inbound Calls

4.1 Create Webhook Endpoint

// Add to server.js

app.post('/webhooks/twilio-inbound', async (req, res) => {
  try {
    const { CallSid, From, To } = req.body;
    
    console.log(`Incoming call from ${From} to ${To}`);
    
    // Create Retell AI call session
    const retellResponse = await axios.post(
      'https://api.retellai.com/v1/call',
      {
        agent_id: RETELL_AGENT_ID,
        audio_encoding: 'mulaw',
        sample_rate: 8000,
        from_number: From,
        to_number: To,
        metadata: {
          twilio_call_sid: CallSid
        }
      },
      {
        headers: {
          'Authorization': `Bearer ${RETELL_API_KEY}`,
          'Content-Type': 'application/json'
        }
      }
    );
    
    const { call_id, access_token } = retellResponse.data;
    
    // Generate TwiML to connect call to Retell AI
    const twiml = new twilio.twiml.VoiceResponse();
    
    const connect = twiml.connect();
    connect.stream({
      url: `wss://api.retellai.com/v1/ws?call_id=${call_id}&access_token=${access_token}`
    });
    
    res.type('text/xml');
    res.send(twiml.toString());
    
  } catch (error) {
    console.error('Error handling inbound call:', error);
    
    // Fallback TwiML
    const twiml = new twilio.twiml.VoiceResponse();
    twiml.say('Sorry, we are experiencing technical difficulties. Please try again later.');
    
    res.type('text/xml');
    res.send(twiml.toString());
  }
});

4.2 Configure Twilio Webhook

Set your Twilio number's webhook URL:

# Using Twilio CLI
twilio phone-numbers:update +1234567890 \
  --voice-url="https://your-domain.com/webhooks/twilio-inbound"

# Or configure in Twilio Console:
# Phone Numbers → Manage → Active Numbers → Select Number
# Voice & Fax → A Call Comes In → Webhook → POST
# URL: https://your-domain.com/webhooks/twilio-inbound

Step 5: Handle Outbound Calls

5.1 Create Outbound Call Function

// Add to server.js

app.post('/api/make-call', async (req, res) => {
  try {
    const { to_number, from_number } = req.body;
    
    // Validate phone numbers
    if (!to_number || !from_number) {
      return res.status(400).json({ error: 'Missing phone numbers' });
    }
    
    // Create Retell AI call session
    const retellResponse = await axios.post(
      'https://api.retellai.com/v1/call',
      {
        agent_id: RETELL_AGENT_ID,
        audio_encoding: 'mulaw',
        sample_rate: 8000,
        from_number: from_number,
        to_number: to_number
      },
      {
        headers: {
          'Authorization': `Bearer ${RETELL_API_KEY}`,
          'Content-Type': 'application/json'
        }
      }
    );
    
    const { call_id, access_token } = retellResponse.data;
    
    // Initiate Twilio call
    const call = await twilioClient.calls.create({
      to: to_number,
      from: from_number,
      url: `https://your-domain.com/webhooks/twilio-outbound?call_id=${call_id}&access_token=${access_token}`,
      statusCallback: `https://your-domain.com/webhooks/twilio-status`,
      statusCallbackEvent: ['initiated', 'ringing', 'answered', 'completed']
    });
    
    res.json({
      success: true,
      call_id: call_id,
      twilio_call_sid: call.sid
    });
    
  } catch (error) {
    console.error('Error making outbound call:', error);
    res.status(500).json({ error: 'Failed to initiate call' });
  }
});

// Outbound call webhook
app.post('/webhooks/twilio-outbound', (req, res) => {
  const { call_id, access_token } = req.query;
  
  const twiml = new twilio.twiml.VoiceResponse();
  
  const connect = twiml.connect();
  connect.stream({
    url: `wss://api.retellai.com/v1/ws?call_id=${call_id}&access_token=${access_token}`
  });
  
  res.type('text/xml');
  res.send(twiml.toString());
});

5.2 Make a Test Call

curl -X POST http://localhost:3000/api/make-call \
  -H "Content-Type: application/json" \
  -d '{
    "to_number": "+1234567890",
    "from_number": "+0987654321"
  }'

Step 6: Handle Function Calls

6.1 Create Function Handler

// Add to server.js

app.post('/webhooks/retell-functions', async (req, res) => {
  try {
    const { call_id, function_name, parameters } = req.body;
    
    console.log(`Function call: ${function_name}`, parameters);
    
    let result;
    
    switch (function_name) {
      case 'check_order_status':
        result = await checkOrderStatus(parameters.order_id);
        break;
        
      case 'book_appointment':
        result = await bookAppointment(parameters);
        break;
        
      default:
        result = { error: 'Unknown function' };
    }
    
    // Return result to Retell AI
    res.json({
      result: result
    });
    
  } catch (error) {
    console.error('Error handling function call:', error);
    res.status(500).json({ error: 'Function execution failed' });
  }
});

// Example function implementations
async function checkOrderStatus(orderId) {
  // Query your database
  const order = await db.orders.findOne({ id: orderId });
  
  if (!order) {
    return { status: 'not_found', message: 'Order not found' };
  }
  
  return {
    status: 'success',
    order_id: orderId,
    order_status: order.status,
    estimated_delivery: order.estimatedDelivery,
    tracking_number: order.trackingNumber
  };
}

async function bookAppointment(params) {
  const { date, time, service_type } = params;
  
  // Create appointment in your system
  const appointment = await db.appointments.create({
    date,
    time,
    serviceType: service_type,
    status: 'confirmed'
  });
  
  return {
    status: 'success',
    appointment_id: appointment.id,
    confirmation_number: appointment.confirmationNumber
  };
}

Step 7: Handle Call Events

7.1 Twilio Status Callbacks

// Add to server.js

app.post('/webhooks/twilio-status', (req, res) => {
  const { CallSid, CallStatus, CallDuration } = req.body;
  
  console.log(`Call ${CallSid} status: ${CallStatus}`);
  
  // Log to database
  db.callLogs.create({
    twilioCallSid: CallSid,
    status: CallStatus,
    duration: CallDuration,
    timestamp: new Date()
  });
  
  res.sendStatus(200);
});

7.2 Retell AI Webhooks

// Add to server.js

app.post('/webhooks/retell-events', (req, res) => {
  const { event_type, call_id, data } = req.body;
  
  console.log(`Retell event: ${event_type}`, data);
  
  switch (event_type) {
    case 'call_started':
      console.log(`Call ${call_id} started`);
      break;
      
    case 'call_ended':
      console.log(`Call ${call_id} ended. Duration: ${data.duration}s`);
      // Save transcript, analytics, etc.
      saveCallData(call_id, data);
      break;
      
    case 'call_analyzed':
      console.log(`Call ${call_id} analyzed`, data.analysis);
      break;
  }
  
  res.sendStatus(200);
});

async function saveCallData(callId, data) {
  await db.calls.create({
    retellCallId: callId,
    transcript: data.transcript,
    duration: data.duration,
    sentiment: data.sentiment,
    summary: data.summary,
    timestamp: new Date()
  });
}

Step 8: Local Development with ngrok

8.1 Install and Run ngrok

# Install ngrok
npm install -g ngrok

# Start your server
npm start

# In another terminal, start ngrok
ngrok http 3000

8.2 Update Webhook URLs

Copy the ngrok URL (e.g., https://abc123.ngrok.io) and update:

1. Twilio webhook: https://abc123.ngrok.io/webhooks/twilio-inbound
2. Retell AI agent webhook: https://abc123.ngrok.io/webhooks/retell-functions

Step 9: Testing

9.1 Test Inbound Calls

1. Call your Twilio number
2. Verify the AI agent answers
3. Have a conversation
4. Check logs for any errors

9.2 Test Outbound Calls

curl -X POST http://localhost:3000/api/make-call \
  -H "Content-Type: application/json" \
  -d '{
    "to_number": "+1234567890",
    "from_number": "+0987654321"
  }'

9.3 Test Function Calling

Have a conversation that triggers a function:

You: "What's the status of order 12345?"
AI: [Calls check_order_status function]
AI: "Your order 12345 is currently in transit and will arrive on Friday."

Step 10: Production Deployment

10.1 Environment Variables

# .env.production
NODE_ENV=production
PORT=3000
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_auth_token
TWILIO_PHONE_NUMBER=+1234567890
RETELL_API_KEY=your_retell_api_key
RETELL_AGENT_ID=your_agent_id
DATABASE_URL=postgresql://...

10.2 Deploy to Cloud

Option 1: AWS Elastic Beanstalk

eb init
eb create production
eb deploy

Option 2: Heroku

heroku create your-app-name
git push heroku main

Option 3: DigitalOcean App Platform

doctl apps create --spec app.yaml

10.3 Update Production Webhooks

Update Twilio and Retell AI webhooks to use your production domain:

• https://your-domain.com/webhooks/twilio-inbound
• https://your-domain.com/webhooks/retell-functions

Troubleshooting

Issue: No audio on calls

Solution: Check audio encoding settings match:

{
  audio_encoding: 'mulaw',  // Must match Twilio
  sample_rate: 8000         // Must match Twilio
}

Issue: Webhook not receiving requests

Solution:

1. Verify ngrok is running
2. Check webhook URL is correct
3. Ensure server is listening on correct port
4. Check firewall settings

Issue: Function calls not working

Solution:

1. Verify webhook URL is publicly accessible
2. Check function schema matches Retell AI format
3. Ensure webhook returns JSON response
4. Check server logs for errors

Issue: High latency

Solution:

1. Deploy server in same region as Twilio
2. Use edge locations for Retell AI
3. Optimize function execution time
4. Enable caching where possible

Best Practices

1. Error handling: Always handle errors gracefully with fallback TwiML
2. Logging: Log all calls, events, and errors for debugging
3. Security: Validate webhook signatures from Twilio and Retell AI
4. Monitoring: Set up alerts for failed calls and errors
5. Testing: Test thoroughly before production deployment
6. Scaling: Use load balancers for high call volumes
7. Backup: Have fallback phone numbers and agents

Conclusion

You now have a complete Retell AI and Twilio integration! This setup provides:

✅ Inbound and outbound call handling
✅ Function calling for real-world actions
✅ Event tracking and logging
✅ Production-ready architecture
✅ Error handling and fallbacks

Ready to build your own AI voice agents? Get started with VaniAgent for a fully managed solution with Twilio integration built-in.

Additional Resources

• Retell AI Documentation
• Twilio Voice Documentation
• Example GitHub Repository
• VaniAgent Integration Guide