Skip to main content

Configuring AI voice agents

Integrating AI voice agents into PBXware is available through a service called AI Voice Connector. AI Voice Connector allows communication between users and voice agents from various providers (OpenAI, xAI, ElevenLabs).

Creating voice agents​

Creating voice agents in PBXware GUI requires users to go and fill in multiple fields to ensure that this voice agent will suit their needs. Most of the fields are self-explanatory and easy to set up, so here we will cover the most important ones that will have the most impact on how the voice agent behaves.

Model​

Different providers have different models and choosing the right one will impact performance and cost. For more complicated agents that include calling tools from MCP servers we recommend using the strongest model available, and for simpler agents that don't require tool calling we recommend using lower-end models to reduce cost without impacting performance. Because providers are releasing models almost on a monthly basis, in PBXware GUI users can manually enter what model the voice agent should use. To see what models are available, look up real-time available models on the provider's sites.

Introduction​

The Introduction field is the first message that will be sent to the voice agent. The agent sees this message as if the user said it, even though they didn't. So if you leave this field empty, users that are calling this voice agent will hear silence until they start talking. For simple use cases we found that an introduction field that says Introduce yourself. works just fine and the agent will introduce itself from all the context it got from the prompt field.

Prompt​

The Prompt field is probably the most important one as it shapes how the voice agent behaves. When writing prompts users can use prompting guides (cookbooks), and we recommend using the prompting guide from OpenAI. In our testing we used this guide to create an initial prompt structure that users can use to write voice agents.

# Prompt example

This is our prompt example. Change it however suits your needs.

# Role & Objective

You are a help desk agent. Your task is to answer the user's questions about company.

# Personality & Tone

## Personality

- Friendly, calm and approachable help desk agent.

## Tone

- Concise and confident.

## Length

- 2-3 sentences per turn.

# Context

- Company info page www.bicomsystems.com

# Tools

- Before any tool call, say one short line like "I'm checking that now." Then call the tool immediately.

## universal_transfer_tool(number)

- Use with number parameter "1001" when user wants to talk to support
- Use with number parameter "1002" when user wants to talk to sales

# Unclear audio

- Default to English if the input language is unclear.
- Only respond to clear audio or text.
- If the user's audio is not clear (e.g., ambiguous input/background noise/silent/unintelligible) or if you did not fully hear or understand the user, call `universal_transfer_tool(number)` with number parameter "1003".

Note that this is the default value for the prompt field when creating voice agents.

Fallback​

When a session between a user and a voice agent encounters an unrecoverable error, or when there is no response from the agent for the amount of seconds that is configured, fallback will be triggered. Fallback has 3 options:

  • Not Set - Only on unrecoverable errors will the call be disconnected (hangup)
  • End Call - Call disconnects (hangup)
  • Transfer Call - Call is transferred to configured destination

Silence detection​

To avoid having dangling calls where users call a voice agent and leave the call running without interacting with it, AI Voice Connector implements silence detection. AI Voice Connector detects when audio activity drops below certain configurable treshold and starts silence timeout timer. So when users make a call to a voice agent, if they are not talking and the voice agent is not responding, the call will be disconnected when the configured silence timer hits 0.

AI Voice Connector and MCP integration​

AI Voice Connector allows users to connect external MCP servers to voice agents. It achieves that by acting as an MCP Proxy.

AVC as MCP Proxy​

To provide tools from external MCP servers to the voice agent provider, AVC needs to be an MCP Proxy. When a user configures MCP servers, AVC establishes connections to them and discovers all tools that are available through them. Internally AVC takes care of all MCP connection states and tools that are discovered. This means that when a connection fails AVC will try to reconnect and rediscover tools.

When a user assigns an MCP server to an agent, all tools from the selected MCP servers will be sent to the voice agent provider when the connection is established. This way the provider sends a tool call event to AVC directly and then AVC delegates that tool call to the corresponding tool call from the actual MCP server. Note that tool names that are sent to the provider will be different from the ones that the MCP server provides. This is because AVC sends the generated unique ID as the tool name to avoid issues in situations where there are multiple tools with the same name from different MCP servers, but the description remains the same so tool invocation will not be affected as voice agents use the description to understand which tool to invoke.

How to configure MCP server connection to AVC​

Because AVC is an MCP proxy it acts to connected MCP servers as a client. So to be an MCP client to a server, AVC needs all connection info that is required for a successful connection and that includes:

After MCP configuration is saved, AVC will try to connect to the server and discover all available tools.

How to test MCP server connection with AVC​

When a user configures an MCP Server in the MCP servers list there is status info with the current connection status between AVC and the configured MCP server. There are two states: Connected and Disconnected. Inside the MCP server edit page there is a Connection status section where you can monitor the current connection status and all tools that AVC discovered. Note that this info isn't real-time so a page refresh is required to see the updated status.

All this information describes the connection between AVC and the MCP server. If users have trouble with their MCP servers that requires additional inspection, it's probably best to use some kind of tool for inspecting MCP servers like the official MCP inspector.

How to inspect tool calling from agent provider to AVC​

When a voice agent does a tool call, AVC receives an event that the tool should be called. That event is logged into the AVC Call Log, so users can inspect if the tool has been called or not. These events for OpenAI and xAI providers are in the following format:

{
  "arguments": "{}",
  "call_id": "call_wsEAiIbbzhUSrH6F",
  "event_id": "event_DmFGbwQYogmwRFJH1bia3",
  "item_id": "item_DmFGbqTrxN3CFwAr2Aok6",
  "name": "8e3f4a52-afca-4c24-989c-dbc86b74207c",
  "output_index": 1,
  "response_id": "resp_DmFGajF5RJpVbIf2stw7j",
  "type": "response.function_call_arguments.done"
}

Note that the name for the tool call is a generated unique ID that AVC generates for each tool from the MCP server. To create a link between the real tool name and the generated unique ID, users can look up the table of discovered tools in the MCP edit page.

Default tools​

Besides discovered tools from MCP servers, AI Voice Connector also sends to the provider default tools that are implemented. Those tools include:

  • universal_transfer_tool(number: string) - Transfer call to desired destination
  • end_call_tool() - End call
  • agent_transfer_tool(slug: string, context: object) - Transfer call to desired agent.

Universal transfer tool​

Universal transfer tool will transfer the call to the desired destination. That destination is provided by the number parameter. Example usage when a call needs to be transferred to extension 1001 when the user wants to talk to sales:

...

# Tools

- Before any tool call, say one short line like "I'm checking that now." Then call the tool immediately.

## universal_transfer_tool(number)

- Use with number parameter "1001" when user wants to talk to support

...

End call tool​

End call tool will hang up the call. Example usage:

...

# Tools

- Before any tool call, say one short line like "I'm checking that now." Then call the tool immediately.

## end_call_tool

- Use when user wants to end the call.

...

Agent transfer tool​

Agent transfer tool transfers the call to the desired voice agent with optional context and transcription from the current call with the voice agent. To transfer calls to agents, users need to use the agent slug for agent identification. The slug is a unique name for the agent and is automatically generated when the agent is created, but users can also provide their own slug when creating agents. For the next agent to receive the transcript from the current call, there is an option in the PBXware GUI that needs to be selected when configuring the agent. Example usage when a user wants to collect data with one agent and process that data with another agent:

Collector agent
...

# Role & Objective

You are a data collection agent.
Your task is to collect name and age from user that is talking to you.

...

# Context

- You are collecting data for a marketing company

# Tools

- Before any tool call, say one short line like "I'm executing tool now." Then call the tool immediately.

## agent_transfer_tool(slug, context)

- Use with slug parameter "data-processing-agent" and context parameter in format key value pairs for ex. [name: collected_name, age: collected_age] when you collect name and age.

...

---

Processor agent

# Role & Objective

You are a data processing agent that can read name and age from transfer context.
Your task is to read out name and age to user that is talking to you when asked.

...