Xiaozhi ESP32
by xinnan-tech
A backend for xiaozhi-esp32 voice hardware that pairs an ASR/LLM/TTS pipeline with an MCP access point, so ESP32 devices can register real-world tool calls an AI can invoke.
Deploy with Docker Compose
docker compose up -d
About
xiaozhi-esp32-server is the backend that turns xiaozhi-esp32 firmware into a talking, tool-using device: it runs voice activity detection, speech recognition, an LLM (with vision model support), and text-to-speech, then routes the conversation over MQTT+UDP or WebSocket. What makes it MCP-relevant is its MCP access point protocol, which lets the ESP32 device itself expose function calls, such as turning on a light, reading a sensor, or controlling an appliance, as tools the connected LLM can invoke mid-conversation.
For teams that want to plug in tools beyond what a single device exposes, the companion mcp-endpoint-server project acts as a WebSocket-based registration hub: it lets you register any number of external MCP-compatible services under one endpoint, merges their tool lists, and routes each call from the device's agent session to the right backend. Both projects ship Docker Compose deployments and a web console for managing devices, users, and configuration.
Key features
- ASR + LLM + TTS voice pipeline with VAD, wake-word, and multi-model LLM routing
- MCP access point protocol for device-side tool/function calls (IoT-style control)
- Companion mcp-endpoint-server for registering and routing multiple external MCP tool providers
- RAGFlow-based knowledge base integration for grounded answers
- Hot-reloadable plugin system and a web dashboard for device/user management
- Docker Compose deployment for both the minimal and full-module setups
Use cases
- Building a voice-controlled smart speaker that can also flip switches or read sensors via tool calls
- Registering third-party MCP tools (weather, search, home automation) behind one endpoint for a fleet of devices
- Running a self-hosted voice assistant backend without sending conversations to a third-party cloud
- Extending a single xiaozhi device's capabilities by wiring in a knowledge base for domain-specific Q&A
Available tools
voice_pipeline
VAD, ASR, and TTS chained together to turn device audio into an LLM conversation and back.
mcp_access_point
Lets the ESP32 device register its own functions (device control, sensor reads) as tools the LLM can call.
mcp_endpoint_server
Separate WebSocket hub that registers external MCP services and routes tool calls from a device session to the right one.
knowledge_base_integration
RAGFlow-backed retrieval so responses can be grounded in a custom knowledge base.
plugin_system
Hot-reloadable plugins for adding new device behaviors without a full redeploy.
web_console
Dashboard for managing connected devices, users, and server configuration.
Frequently asked questions
Is this the same thing as the xiaozhi-esp32 firmware repo?
No. xiaozhi-esp32 (by a different maintainer, 78/xiaozhi-esp32) is the device firmware; xinnan-tech/xiaozhi-esp32-server is the backend service that firmware talks to over MQTT/WebSocket.
How does a device's own functions become an MCP tool an LLM can call?
Through the server's MCP access point protocol: the device registers its available functions at connection time, and the server surfaces them to the LLM as callable tools during the conversation.