Serve: keep a model available
What is Serve?
Serve starts a model behind an OpenAI-compatible endpoint and keeps it running until you stop it. Use Serve when something needs to keep answering requests, rather than finish and exit.
- Open-source LLMs through vLLM (any Hugging Face model your GPU can hold)
- Embeddings endpoints
- Persistent transcription endpoints
- Your own launch command or script via --cmd
Serve your first model
npm install -g badgr-cli badgr login badgr serve Qwen/Qwen2.5-7B-Instruct --max-cost 10
When the endpoint is ready, badgr prints its URL. Call it with any OpenAI-compatible client.
What happens
- 1.Badgr picks capacity that can hold the model.
- 2.The machine starts and the model loads.
- 3.Badgr prints the endpoint URL when the model is ready, usually in 2 to 5 minutes.
- 4.The endpoint stays up until you run badgr down, or until the --max-cost cap is reached.
Cost control
- --max-cost auto-stops the endpoint when spend reaches that amount. It is required unless you pass --persistent.
- --persistent keeps the endpoint running with no cap, and billing continues until you run badgr down <id>.
- badgr status lists every running endpoint.
Get results
- The endpoint URL is printed when it is ready. Call it with any OpenAI SDK client and your BADGR_API_KEY.
- badgr logs <id> returns server logs.
- badgr receipts shows cost, route, retry and teardown history for serve actions.
What this does not cover
- Autoscaling and automated recovery are not documented as supported, so do not rely on them.
- Receipts cover run and serve actions as documented in the CLI reference, not every request sent to your endpoint.
Next steps
Serve a model
Flags, GPU choice and verification.
vLLM endpoint
Custom vLLM config and version pinning.
Embeddings
A persistent embeddings API.
Transcription endpoint
Persistent Whisper endpoint.
Bring your own GPU
Serve on a machine you own with --target.
Model-serving product
Build a product on top of served models.