Serve an open-source model

1. Start the endpoint

badgr serve Qwen/Qwen2.5-7B-Instruct --max-cost 10

GPU is selected automatically based on model size. Badgr starts vLLM and prints the endpoint URL when ready (usually 2–5 minutes).--max-cost auto-stops the endpoint when spend reaches that amount. Use --persistent instead to keep it running until you run badgr down.

2. Call the endpoint

When the model is ready, badgr serve prints the endpoint URL. Export it and point any OpenAI SDK client at it:

import os
from openai import OpenAI

# Export BADGR_ENDPOINT from the URL printed by `badgr serve`
client = OpenAI(
    api_key=os.environ["BADGR_API_KEY"],
    base_url=os.environ["BADGR_ENDPOINT"],
)

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

import OpenAI from "openai";

// Export BADGR_ENDPOINT from the URL printed by `badgr serve`
const client = new OpenAI({
  apiKey: process.env.BADGR_API_KEY,
  baseURL: process.env.BADGR_ENDPOINT,
});

const resp = await client.chat.completions.create({
  model: "Qwen/Qwen2.5-7B-Instruct",
  messages: [{ role: "user", content: "Hello" }],
});
console.log(resp.choices[0].message.content);

# Set BADGR_ENDPOINT to the URL printed by `badgr serve`
curl $BADGR_ENDPOINT/v1/chat/completions \
  -H "Authorization: Bearer $BADGR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

3. Stop billing

badgr down <deployment-id>

Terminates the GPU instance. Use badgr receipts to see cost after.

Per-endpoint API key

Every model endpoint (vLLM-served) gets its own API key, generated at launch and returned exactly once in the /v1/serve response as endpoint_api_key. Store it then — Badgr only keeps a hash, so it can't be shown again. Pass it as Authorization: Bearer <key> when calling the endpoint directly.

Not yet exposed as a badgr serve CLI flag — read it from the JSON response of the underlying POST /v1/serve call for now.

Idle timeout

Pass idle_timeout_minutes to POST /v1/serve to auto-terminate an endpoint after that many minutes of inactivity. Badgr doesn't sit in the inference data path, so it needs a signal: call POST /v1/deployments/{id}/heartbeat on each real request (or wire it into your client) to reset the idle clock. Without heartbeats, the endpoint is torn down after the timeout even if it's technically reachable.

Restart an endpoint

curl -X POST $BADGR_API/v1/deployments/<deployment-id>/restart \
  -H "Authorization: Bearer $BADGR_API_KEY"

Tears down the current pod and relaunches with the same GPU, model, price cap, and endpoint API key. Returns a new deployment with a new endpoint_url — the old pod's IP is gone, so update any client pointed at it.

Check status

# See all running deployments and live billing rate
badgr status

# Stream logs from a running endpoint
badgr logs <deployment-id>

Options