Function Calling with LLMs: Building Intelligent AI Voice Agents

Large Language Models (LLMs) are incredible at understanding and generating natural language, but they can't take actions in the real world—at least not without function calling. Function calling (also called tool use) enables LLMs to interact with external systems, query databases, book appointments, and perform real-world tasks.

In this comprehensive guide, we'll explore how to build intelligent AI voice agents that can take actions using function calling with LLMs.

What is Function Calling?

Function calling allows LLMs to invoke external functions or APIs based on user requests. Instead of just generating text, the LLM can:

• Query databases for real-time information
• Book appointments in calendar systems
• Update CRM records
• Process payments
• Send emails or SMS messages
• Control IoT devices
• And much more

How It Works

1. Define functions: You describe available functions to the LLM
2. User request: User asks the AI to perform an action
3. LLM decides: LLM determines which function to call and with what parameters
4. Execute function: Your code executes the function
5. Return result: Function result is sent back to the LLM
6. Generate response: LLM uses the result to respond to the user

Defining Function Schemas

The first step is defining your functions in a schema that the LLM understands.

Basic Function Schema

const functions = [
  {
    name: 'get_weather',
    description: 'Get the current weather for a location',
    parameters: {
      type: 'object',
      properties: {
        location: {
          type: 'string',
          description: 'The city and state, e.g. San Francisco, CA'
        },
        unit: {
          type: 'string',
          enum: ['celsius', 'fahrenheit'],
          description: 'The temperature unit'
        }
      },
      required: ['location']
    }
  }
];

Key Components

• name: Unique identifier for the function
• description: Clear explanation of what the function does (critical for LLM decision-making)
• parameters: JSON Schema defining the function's parameters
• required: Array of required parameter names

Real-World Example: Appointment Booking

Let's build a complete appointment booking system with function calling.

Step 1: Define Functions

const appointmentFunctions = [
  {
    name: 'check_availability',
    description: 'Check available appointment slots for a specific date and service type',
    parameters: {
      type: 'object',
      properties: {
        date: {
          type: 'string',
          description: 'Date in YYYY-MM-DD format'
        },
        serviceType: {
          type: 'string',
          enum: ['consultation', 'checkup', 'procedure'],
          description: 'Type of appointment'
        }
      },
      required: ['date', 'serviceType']
    }
  },
  {
    name: 'book_appointment',
    description: 'Book an appointment at a specific date and time',
    parameters: {
      type: 'object',
      properties: {
        date: {
          type: 'string',
          description: 'Date in YYYY-MM-DD format'
        },
        time: {
          type: 'string',
          description: 'Time in HH:MM format (24-hour)'
        },
        serviceType: {
          type: 'string',
          enum: ['consultation', 'checkup', 'procedure']
        },
        patientName: {
          type: 'string',
          description: 'Full name of the patient'
        },
        phoneNumber: {
          type: 'string',
          description: 'Contact phone number'
        }
      },
      required: ['date', 'time', 'serviceType', 'patientName', 'phoneNumber']
    }
  },
  {
    name: 'cancel_appointment',
    description: 'Cancel an existing appointment',
    parameters: {
      type: 'object',
      properties: {
        appointmentId: {
          type: 'string',
          description: 'Unique appointment identifier'
        }
      },
      required: ['appointmentId']
    }
  }
];

Step 2: Implement Functions

// Function implementations
async function checkAvailability(date, serviceType) {
  // Query your database or calendar system
  const slots = await db.query(
    'SELECT time FROM appointments WHERE date = ? AND service_type = ? AND status = "available"',
    [date, serviceType]
  );
  
  return {
    date,
    serviceType,
    availableSlots: slots.map(s => s.time),
    count: slots.length
  };
}

async function bookAppointment(date, time, serviceType, patientName, phoneNumber) {
  // Create appointment in your system
  const appointment = await db.insert('appointments', {
    date,
    time,
    serviceType,
    patientName,
    phoneNumber,
    status: 'confirmed',
    createdAt: new Date()
  });
  
  // Send confirmation SMS
  await sms.send(phoneNumber, 
    `Your ${serviceType} appointment is confirmed for ${date} at ${time}. Confirmation #${appointment.id}`
  );
  
  return {
    success: true,
    appointmentId: appointment.id,
    confirmationNumber: appointment.id
  };
}

async function cancelAppointment(appointmentId) {
  const appointment = await db.findOne('appointments', { id: appointmentId });
  
  if (!appointment) {
    return { success: false, error: 'Appointment not found' };
  }
  
  await db.update('appointments', 
    { id: appointmentId }, 
    { status: 'cancelled' }
  );
  
  return { success: true, appointmentId };
}

Step 3: LLM Integration

import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function handleConversation(userMessage, conversationHistory) {
  // Add user message to history
  conversationHistory.push({
    role: 'user',
    content: userMessage
  });
  
  // Call LLM with function definitions
  const response = await openai.chat.completions.create({
    model: 'gpt-4-turbo',
    messages: conversationHistory,
    functions: appointmentFunctions,
    function_call: 'auto'  // Let LLM decide when to call functions
  });
  
  const message = response.choices[0].message;
  
  // Check if LLM wants to call a function
  if (message.function_call) {
    const functionName = message.function_call.name;
    const functionArgs = JSON.parse(message.function_call.arguments);
    
    // Execute the function
    let functionResult;
    switch (functionName) {
      case 'check_availability':
        functionResult = await checkAvailability(
          functionArgs.date,
          functionArgs.serviceType
        );
        break;
      case 'book_appointment':
        functionResult = await bookAppointment(
          functionArgs.date,
          functionArgs.time,
          functionArgs.serviceType,
          functionArgs.patientName,
          functionArgs.phoneNumber
        );
        break;
      case 'cancel_appointment':
        functionResult = await cancelAppointment(
          functionArgs.appointmentId
        );
        break;
    }
    
    // Add function call and result to conversation history
    conversationHistory.push(message);
    conversationHistory.push({
      role: 'function',
      name: functionName,
      content: JSON.stringify(functionResult)
    });
    
    // Get LLM's response based on function result
    const finalResponse = await openai.chat.completions.create({
      model: 'gpt-4-turbo',
      messages: conversationHistory
    });
    
    return finalResponse.choices[0].message.content;
  }
  
  // No function call - return LLM's direct response
  return message.content;
}

Step 4: Example Conversation

const history = [
  {
    role: 'system',
    content: 'You are a helpful medical appointment scheduling assistant. Be friendly and efficient.'
  }
];

// User: "I need to book a checkup for next Monday"
let response = await handleConversation(
  "I need to book a checkup for next Monday",
  history
);
// LLM calls check_availability(date: "2024-11-18", serviceType: "checkup")
// Response: "I have availability on Monday, November 18th at 9:00 AM, 11:00 AM, and 2:00 PM. Which time works best for you?"

// User: "2 PM works great"
response = await handleConversation(
  "2 PM works great",
  history
);
// Response: "Perfect! I'll book that for you. Can I get your full name and phone number?"

// User: "John Smith, 555-1234"
response = await handleConversation(
  "John Smith, 555-1234",
  history
);
// LLM calls book_appointment(date: "2024-11-18", time: "14:00", serviceType: "checkup", patientName: "John Smith", phoneNumber: "555-1234")
// Response: "Great! Your checkup is confirmed for Monday, November 18th at 2:00 PM. You'll receive a confirmation text at 555-1234. Your confirmation number is #12345."

Advanced Patterns

Multi-Step Function Calls

Sometimes you need to call multiple functions in sequence:

// User: "Book me the earliest available appointment"

// Step 1: Check availability for multiple dates
const today = new Date();
const dates = [0, 1, 2, 3, 4].map(offset => {
  const date = new Date(today);
  date.setDate(date.getDate() + offset);
  return date.toISOString().split('T')[0];
});

let earliestSlot = null;
for (const date of dates) {
  const availability = await checkAvailability(date, 'checkup');
  if (availability.availableSlots.length > 0) {
    earliestSlot = {
      date,
      time: availability.availableSlots[0]
    };
    break;
  }
}

// Step 2: Book the earliest slot
if (earliestSlot) {
  await bookAppointment(
    earliestSlot.date,
    earliestSlot.time,
    'checkup',
    patientName,
    phoneNumber
  );
}

Error Handling

Always handle errors gracefully:

async function safeExecuteFunction(functionName, args) {
  try {
    // Validate arguments
    if (!validateArgs(functionName, args)) {
      return {
        success: false,
        error: 'Invalid arguments provided'
      };
    }
    
    // Execute function with timeout
    const result = await Promise.race([
      executeFunction(functionName, args),
      new Promise((_, reject) => 
        setTimeout(() => reject(new Error('Timeout')), 5000)
      )
    ]);
    
    return { success: true, data: result };
  } catch (error) {
    console.error(`Function ${functionName} failed:`, error);
    return {
      success: false,
      error: error.message
    };
  }
}

Confirmation Before Actions

For critical actions, always confirm with the user:

const criticalFunctions = ['book_appointment', 'cancel_appointment', 'process_payment'];

if (criticalFunctions.includes(functionName)) {
  // Ask for confirmation
  const confirmation = await askUser(
    `Just to confirm, you want to ${functionName.replace('_', ' ')} with these details: ${JSON.stringify(functionArgs)}. Is that correct?`
  );
  
  if (confirmation.toLowerCase().includes('yes')) {
    // Proceed with function call
    await executeFunction(functionName, functionArgs);
  } else {
    return "Okay, I won't proceed with that action. What would you like to do instead?";
  }
}

CRM Integration Example

Let's build a function that updates CRM records:

const crmFunctions = [
  {
    name: 'update_lead_status',
    description: 'Update the status of a lead in the CRM',
    parameters: {
      type: 'object',
      properties: {
        leadId: {
          type: 'string',
          description: 'Unique lead identifier'
        },
        status: {
          type: 'string',
          enum: ['new', 'contacted', 'qualified', 'unqualified', 'converted'],
          description: 'New status for the lead'
        },
        notes: {
          type: 'string',
          description: 'Notes about the status change'
        }
      },
      required: ['leadId', 'status']
    }
  },
  {
    name: 'create_task',
    description: 'Create a follow-up task for a lead',
    parameters: {
      type: 'object',
      properties: {
        leadId: {
          type: 'string',
          description: 'Lead to create task for'
        },
        taskType: {
          type: 'string',
          enum: ['call', 'email', 'meeting'],
          description: 'Type of follow-up task'
        },
        dueDate: {
          type: 'string',
          description: 'Due date in YYYY-MM-DD format'
        },
        assignedTo: {
          type: 'string',
          description: 'User ID to assign task to'
        }
      },
      required: ['leadId', 'taskType', 'dueDate']
    }
  }
];

// Implementation
async function updateLeadStatus(leadId, status, notes) {
  await crm.updateLead(leadId, {
    status,
    notes,
    lastUpdated: new Date(),
    updatedBy: 'ai_agent'
  });
  
  return { success: true, leadId, newStatus: status };
}

async function createTask(leadId, taskType, dueDate, assignedTo) {
  const task = await crm.createTask({
    leadId,
    type: taskType,
    dueDate,
    assignedTo: assignedTo || 'default_sales_rep',
    createdBy: 'ai_agent',
    status: 'pending'
  });
  
  return { success: true, taskId: task.id };
}

Security Considerations

Function calling introduces security risks. Follow these best practices:

1. Validate All Inputs

function validateFunctionArgs(functionName, args) {
  // Check for SQL injection attempts
  const sqlInjectionPattern = /((SELECT|INSERT|UPDATE|DELETE|DROP|CREATE|ALTER))/i;
  
  for (const value of Object.values(args)) {
    if (typeof value === 'string' && sqlInjectionPattern.test(value)) {
      throw new Error('Invalid input detected');
    }
  }
  
  // Validate data types match schema
  const schema = functions.find(f => f.name === functionName);
  for (const [key, value] of Object.entries(args)) {
    const expectedType = schema.parameters.properties[key].type;
    if (typeof value !== expectedType) {
      throw new Error(`Invalid type for ${key}`);
    }
  }
  
  return true;
}

2. Implement Rate Limiting

const rateLimiter = new Map();

function checkRateLimit(userId, functionName) {
  const key = `${userId}:${functionName}`;
  const now = Date.now();
  const limit = 10; // 10 calls per minute
  const window = 60000; // 1 minute
  
  if (!rateLimiter.has(key)) {
    rateLimiter.set(key, []);
  }
  
  const calls = rateLimiter.get(key).filter(time => now - time < window);
  
  if (calls.length >= limit) {
    throw new Error('Rate limit exceeded');
  }
  
  calls.push(now);
  rateLimiter.set(key, calls);
}

3. Use Least Privilege

Only expose functions that the AI agent needs:

// Don't expose dangerous functions
const allowedFunctions = [
  'check_availability',
  'book_appointment',
  'cancel_appointment'
];

// Never expose these to AI
const forbiddenFunctions = [
  'delete_all_data',
  'update_user_permissions',
  'execute_sql_query'
];

Performance Optimization

1. Cache Function Results

const cache = new Map();

async function cachedCheckAvailability(date, serviceType) {
  const cacheKey = `${date}:${serviceType}`;
  
  if (cache.has(cacheKey)) {
    const cached = cache.get(cacheKey);
    if (Date.now() - cached.timestamp < 60000) { // 1 minute TTL
      return cached.data;
    }
  }
  
  const result = await checkAvailability(date, serviceType);
  cache.set(cacheKey, { data: result, timestamp: Date.now() });
  
  return result;
}

2. Parallel Function Calls

When possible, execute independent functions in parallel:

// Check availability for multiple dates in parallel
const availabilityPromises = dates.map(date => 
  checkAvailability(date, serviceType)
);

const results = await Promise.all(availabilityPromises);

Best Practices

1. Write clear descriptions: The LLM relies on function descriptions to decide when to call them
2. Use specific parameter names: Avoid ambiguous names like "data" or "value"
3. Validate everything: Never trust LLM-generated parameters without validation
4. Handle errors gracefully: Return user-friendly error messages
5. Log all function calls: Track what actions the AI is taking
6. Confirm critical actions: Always confirm before destructive operations
7. Test extensively: Test with edge cases and malicious inputs
8. Monitor in production: Track function call success rates and errors

Conclusion

Function calling transforms LLMs from text generators into intelligent agents that can take real-world actions. By carefully defining functions, implementing robust error handling, and following security best practices, you can build AI voice agents that seamlessly integrate with your business systems.

The result? AI agents that don't just talk—they get things done.

Ready to build intelligent AI agents with function calling? Start with VaniAgent and connect to 8,000+ integrations out of the box.