GPU Compute / Recipe

Serve an open-source model

Launch a persistent OpenAI-compatible endpoint for any Hugging Face model. Badgr provisions the GPU, starts vLLM, and returns a URL you can use with any OpenAI SDK client.

Prerequisites: npm install -g badgr-cli then badgr login. Setup guide →

1. Start the endpoint

badgr serve Qwen/Qwen2.5-7B-Instruct --max-cost 10

GPU is selected automatically based on model size. Badgr starts vLLM and prints the endpoint URL when ready (usually 2–5 minutes).--max-cost auto-stops the endpoint when spend reaches that amount. Use --persistent instead to keep it running until you run badgr down.

2. Call the endpoint

When the model is ready, badgr serve prints the endpoint URL. Export it and point any OpenAI SDK client at it:

Python

import os
from openai import OpenAI

# Export BADGR_ENDPOINT from the URL printed by `badgr serve`
client = OpenAI(
    api_key=os.environ["BADGR_API_KEY"],
    base_url=os.environ["BADGR_ENDPOINT"],
)

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Node.js

import OpenAI from "openai";

// Export BADGR_ENDPOINT from the URL printed by `badgr serve`
const client = new OpenAI({
  apiKey: process.env.BADGR_API_KEY,
  baseURL: process.env.BADGR_ENDPOINT,
});

const resp = await client.chat.completions.create({
  model: "Qwen/Qwen2.5-7B-Instruct",
  messages: [{ role: "user", content: "Hello" }],
});
console.log(resp.choices[0].message.content);

curl

# Set BADGR_ENDPOINT to the URL printed by `badgr serve`
curl $BADGR_ENDPOINT/v1/chat/completions \
  -H "Authorization: Bearer $BADGR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

3. Stop billing

badgr down <deployment-id>

Terminates the GPU instance. Use badgr receipts to see cost after.

Check status

# See all running deployments and live billing rate
badgr status

# Stream logs from a running endpoint
badgr logs <deployment-id>

Options

--gpu <type>GPU type: RTX_4090, L40S, A6000, A100, A100_80GB, H100. Auto-selected from model size if omitted.

--count <n>Number of GPU instances (1–8) for higher concurrency

--env KEY=VALUEEnvironment variable passed to the container — repeatable

--region US|EU|AURegion preference. Omit for global best-capacity search.

--tier 1|21 = managed providers (default), 2 = marketplace budget providers

--max-price 2.00Hard hourly cap in USD/hr — won't start if no GPU is under this price

--max-cost 20.00Auto-stop when total spend reaches this amount (required unless --persistent)

--persistentRun indefinitely — billing continues until you run badgr down

--no-waitReturn immediately without waiting for health check

--task embedvLLM task override, e.g. embed for embedding models

--runtime llama.cppServe a HF GGUF file via llama.cpp instead of vLLM (requires --hf-repo and --hf-file)