Integrations / llama.cpp
Serve GGUF models with llama.cpp
Start the llama.cpp server from the built-in template to serve a quantised GGUF model.
The steps follow the tool's documented settings. No tested run is recorded for this page yet.
Why use Badgr for llama.cpp
- Badgr interface
- Serve
- Where llama.cpp fits
- Model server
- Why Badgr
- GGUF model endpoint
- Serve a quantised model on a smaller GPU, which costs less per hour than a large one.
- A built-in template starts the server for you.
- A cost cap and a stop command bound the spend.
What you can do
- Serve a quantised model on a smaller, cheaper GPU
- Run a GGUF file you already trust
- Compare quantisation levels for quality and speed
Quickstart: Start from the template
badgr template info llama-cpp badgr serve template llama-cpp --max-cost 5
How it works
- 1.You run one command
- 2.Badgr picks a GPU and starts the service
- 3.You get an address and a key
- 4.Your app calls it until you stop it
What you get
- A service address
- A receipt when it stops
Requirements
- The Badgr CLI installed and logged in (npm install -g badgr-cli)
- A funded Badgr balance
- A GGUF model file or a Hub repository that has one
What it costs
- A service bills for as long as it runs, until you stop it with badgr down <id>.
- Pass --max-cost to bound spend. Badgr shows the rate for the GPU it picked before it starts. This page does not quote a fixed price.
See pricing for how billing works.
Proof
No tested run is recorded for this integration yet. A receipt and example output will be added here after a real run, not before.
Troubleshooting
Model too large for the GPU
Pick a lower quantisation (for example Q4 instead of Q8) or fewer layers on the GPU.
Wrong file chosen
A repository can hold many GGUF files. Name the exact file you want.
Service never becomes ready
Read the logs with badgr logs <id>. The most common causes are a wrong port, a missing model file or too little GPU memory.
Bill keeps growing
Services run until stopped. Run badgr status, then badgr down <id>.
Developer reference
Ready to start with llama.cpp?
Every run or service has a spending limit you set before it starts.