Self-host llama.cpp Server with Levelrail
The llama.cpp HTTP server: run quantized GGUF models on plain CPUs with an OpenAI-compatible API.
- Authorggml-org
- LicenceMIT
- Memory4 GiB
- Archarm64
- Services1
How to self-host llama.cpp Server
-
Pull the images
One image is pulled on your server.
Show images
ghcr.io/ggml-org/llama.cpp:server
-
Create the app
One service starts in a single Compose app, with 1 named volume mount for your data, on port 8080.
-
Fill in the secrets
1 setting is preset, you do not have to fill in anything.
-
Open it
Without a domain the app gets an sslip.io address; add --domain SERVICE=HOST to use your own.
llama.cpp Server requirements
- Memory
- About 4 GiB
- Architectures
- amd64 image published, arm64 image published
- Ports
- 8080
- Domain
- Optional, an sslip.io address is used without one
- Settings
- none to fill in
- License
- MIT, permissive license
- Listed. It is in the catalog but not verified. Back up the data volume before changing an image tag.
- Host ports. It publishes port 8080 on the host, so open them in the VPS firewall only if you want that.
Install Levelrail first
On a server with Levelrail running, run the command above, or open the dashboard: Apps, New app, Browse templates, then llama.cpp Server. New to Levelrail? Install it on a fresh VPS first.
docker pull ghcr.io/glincker/levelrail:latest Then levelrail-cli login --device. See Getting started and the llama.cpp Server docs page.
Deploy llama.cpp Server your way
Every format below deploys the same pinned images. Pick the one that fits how you work.
Deploy llama.cpp Server from the CLI
levelrail-cli templates deploy llama-cpp --name my-llama-cpp Deploy llama.cpp Server with the API
curl -X POST -H "Authorization: Bearer $TOKEN" \
https://levelrail.example.com/api/v1/service-templates/llama-cpp/deploy llama.cpp Server configuration
llama.cpp Server services
| Service | Image | Ports | Volumes |
|---|---|---|---|
| llama-cpp | ghcr.io/ggml-org/llama.cpp:server | 8080 | llama_cpp_models -> /models |
llama.cpp Server environment variables
| Variable | Service | Value |
|---|---|---|
| LLAMA_CACHE | llama-cpp | Preset by the template |
Ports and volumes
- Ports
- 8080
- Volumes
- 1 named mount: llama_cpp_models -> /models
llama.cpp Server FAQ
How much memory does llama.cpp Server need?
The catalog lists about 4 GiB of memory for llama.cpp Server.
Does llama.cpp Server run on arm64?
For llama.cpp Server, an arm64 image is published; amd64 is image published.
What licence is llama.cpp Server under?
llama.cpp Server is licensed under MIT, which is permissive, with few conditions on reuse.
Does llama.cpp Server need configuration before it starts?
No. The template fills in its own defaults.
Which ports does llama.cpp Server use?
llama.cpp Server listens on port 8080 inside the stack.
Where does llama.cpp Server keep its data?
In a named volume: llama_cpp_models mounted at /models.
Can I deploy llama.cpp Server on Dokploy or Dokku?
The services and pinned images on this page can be run on any Docker host. The public Dokploy templates list we compared does not have an entry with the same id for llama.cpp Server, so on Dokploy you would use a Docker Compose application with the file from this page. Dokku deploys container images with its own commands, so the pinned image ghcr.io/ggml-org/llama.cpp:server is what you deploy there.