Build real-time AI agents that listen and speak on your server
Imagine an AI answering a phone call, taking an order, and responding at the right moment based on the speaker's tone. This repository is a toolkit for running such conversational AI on a server. It bundles speech recognition, language models, and speech synthesis into a single flow, designed so the AI reacts naturally while the user is speaking.
The flow of conversation between voice and text
This framework manages the process of listening to the user, understanding the meaning, and responding with voice. You can freely combine speech recognition (STT), language models (LLM), and speech synthesis (TTS), so you are not dependent on a specific vendor. A 'semantic turn detection' feature determines when the user has finished speaking, reducing instances where the AI interrupts. It also supports MCP (Model Context Protocol), allowing you to connect external tools with a single line of code.
Connections extending to phone and video
You can build agents that participate in actual phone calls or video meetings, going beyond simple chat. Integrated with LiveKit's phone stack, agents can make or receive calls, and they exchange data with apps on various platforms via WebRTC clients. It also supports a 'handoff' pattern where multiple agents divide roles in a conversation; for example, an agent collecting information can naturally pass the conversation to an agent that starts the discussion.
Preparation for testing and deployment
Considering the non-deterministic nature of LLMs, it provides a built-in testing framework. You can use a 'judge' feature to verify that the agent behaves as intended, increasing reliability. During development, you can test voice input and output directly from the terminal or use a development mode that automatically restarts when files change. In production, it runs with optimized settings, and if needed, you can self-host the LiveKit server to operate the entire stack independently.
Things to check before use
This repository is released under the Apache-2.0 license, allowing commercial use. However, speech recognition and synthesis models require external API keys, so you must prepare credentials for services like Deepgram, OpenAI, or Cartesia. You also need to set the URL, API key, and secret key as environment variables to connect to the LiveKit server. Since real-time performance is important, latency may vary depending on the network environment and model choice, so it is recommended to test in your actual usage environment.
