Serve: keep a model available

What is Serve?

Serve starts a model behind an OpenAI-compatible endpoint and keeps it running until you stop it. Use Serve when something needs to keep answering requests, rather than finish and exit.

  • Open-source LLMs through vLLM (any Hugging Face model your GPU can hold)
  • Embeddings endpoints
  • Persistent transcription endpoints
  • Your own launch command or script via --cmd

Serve your first model

npm install -g badgr-cli
badgr login
badgr serve Qwen/Qwen2.5-7B-Instruct --max-cost 10

When the endpoint is ready, badgr prints its URL. Call it with any OpenAI-compatible client.

What happens

  1. 1.Badgr picks capacity that can hold the model.
  2. 2.The machine starts and the model loads.
  3. 3.Badgr prints the endpoint URL when the model is ready, usually in 2 to 5 minutes.
  4. 4.The endpoint stays up until you run badgr down, or until the --max-cost cap is reached.

Cost control

  • --max-cost auto-stops the endpoint when spend reaches that amount. It is required unless you pass --persistent.
  • --persistent keeps the endpoint running with no cap, and billing continues until you run badgr down <id>.
  • badgr status lists every running endpoint.

Get results

  • The endpoint URL is printed when it is ready. Call it with any OpenAI SDK client and your BADGR_API_KEY.
  • badgr logs <id> returns server logs.
  • badgr receipts shows cost, route, retry and teardown history for serve actions.

What this does not cover

  • Autoscaling and automated recovery are not documented as supported, so do not rely on them.
  • Receipts cover run and serve actions as documented in the CLI reference, not every request sent to your endpoint.

Next steps