Run a local voice conversation agent with open-source models
You can build an environment on your computer where speaking into a microphone yields an immediate response from the speakers. This pipeline, composed entirely of open-source models, processes every stage locally, from speech recognition to text generation and speech synthesis. It enables real-time voice conversation without API keys or cloud services, making it useful in environments where data privacy is critical or internet connections are unstable.
A four-stage local voice pipeline
Voice input is processed by detecting speech boundaries with Silero VAD, then converting to text with Parakeet TDT. The Qwen3.5-4B language model generates the response, and Qwen3-TTS synthesizes it into audio for output. Each stage runs in an independent thread and is connected via queues to minimize latency. All components can be swapped via CLI flags, allowing you to freely combine models to suit specific hardware or requirements.
OpenAI Realtime protocol-compatible server
This repository supports the OpenAI Realtime protocol via WebSocket and WebRTC. Apps or clients developed for the existing OpenAI Realtime API can connect to this local server without modification. Running the server with the serve command handles real-time voice sessions at the ws://127.0.0.1:8765/v1/realtime address. Compatibility with major frameworks such as the OpenAI Agents SDK and LiveKit Agents has also been tested, allowing you to use existing ecosystem tools as-is.
Platform-optimized model selection
The system automatically selects the most suitable model based on the execution environment. On Apple Silicon Macs, it runs the 4-bit quantized Qwen3.5-4B via MLX LM, with 16GB of unified memory recommended. On Linux or Windows with an NVIDIA GPU, it runs the model via Transformers, with 24GB of VRAM recommended. In other environments, it runs on CPU, though response speed may be slower. Model download takes a few minutes on first run, after which it operates fully offline without an internet connection.
Installation and execution
Install with pip install speech-to-speech in a Python 3.10 or higher environment (3.11 recommended). Ubuntu users must first install the libportaudio2 and libsndfile1 audio libraries. The speech-to-speech local command runs the server and microphone client in a single process, and you can terminate it with Ctrl+C. You should wear headphones or add the --local_audio_block_mic_during_playback option to prevent the microphone from picking up speaker audio.
Hardware requirements to check before use
Since this tool runs all models locally, hardware specifications determine performance. 16GB of unified memory is recommended for Apple Silicon Macs, and 24GB of VRAM for NVIDIA GPU environments, with approximately 8GB of that used by the voice pipeline. It can run on CPU-only environments, but response latency may be longer. Additionally, the default Linux wheel for Qwen3-TTS targets CUDA 12.8 and Ubuntu 24.04, so systems with other CUDA versions or older glibc must separately install the appropriate version from the Hugging Face wheelhouse. If you want fully offline operation where no data leaves the machine, you can block Hugging Face Hub requests by setting the HF_HUB_OFFLINE=1 environment variable after the first run.
