Serge
by serge-chat
Serge is a self-hosted, dockerized chat UI and REST API for running local LLMs through llama.cpp — no API keys, no cloud dependency, just a browser and a GGUF model.
Run with Docker
docker run -d --name serge -v weights:/usr/src/app/weights -v datadb:/data/db/ -p 8008:8008 ghcr.io/serge-chat/serge:latest
About
Serge wraps llama.cpp in a FastAPI backend and a SvelteKit frontend so you can chat with a locally running model — originally Alpaca-family weights, now any GGUF model llama.cpp supports — entirely from a browser tab. There's no external API key to configure and no data leaving the machine it runs on; Redis stores chat history and per-conversation parameters so sessions persist across restarts.
It ships as a single Docker image with two mounted volumes — one for model weights, one for the Redis data directory — so standing it up is a docker run away rather than a multi-service install. Every action the UI performs is backed by a documented REST endpoint at /api/docs, which makes Serge usable as a headless local-inference API for scripts as much as it is a chat app.
Key features
- Fully self-hosted chat UI with no external API key required
- FastAPI + LangChain backend wrapping llama.cpp via its Python bindings
- SvelteKit frontend, Redis-backed persistent chat history
- Single Docker image, runs on CPU in as little as 4GB of RAM
- Documented REST API at /api/docs for scripting outside the browser
- Model downloading and swapping without editing config files
Use cases
- Running a private, offline chat assistant on a home server or NAS
- Giving a small team browser access to a shared local model without exposing it to the internet
- Prototyping against a local LLM's REST API before committing to a hosted provider
- Testing different GGUF models side by side without reconfiguring a cloud endpoint
Available tools
Chat session management
Create, rename, and delete persistent conversations, each keeping its own model choice and generation parameters.
Model management
Download and switch between GGUF-format models that llama.cpp can load, directly from the UI.
Streaming responses
The FastAPI backend streams generated tokens to the frontend as they're produced rather than waiting for the full reply.
REST API (/api/docs)
Every UI action has a matching documented endpoint, so chats can be scripted or automated outside the browser.
Frequently asked questions
Does Serge itself speak the Model Context Protocol?
No — Serge predates MCP and is a standalone self-hosted chat UI and REST API for llama.cpp. It's useful as a local model backend, but it doesn't expose MCP tools on its own.
What hardware does it need?
It's designed to run on CPU with as little as 4GB of RAM, though larger models and faster responses benefit from more memory and, if configured, a GPU-accelerated llama.cpp build.