Serve Llama 3.1 8B
Deploy Llama 3.1 8B as an OpenAI-compatible vLLM endpoint — a widely compatible, low-cost general chat model.
Estimated $0.35–$0.60/hr while serving · Long-running endpoint — starts in 2–5 min
What inputs you need
- Model: meta-llama/Llama-3.1-8B-Instruct (or pinned variant)
- Max-cost budget
- Optional: concurrency overrides
What Badgr returns
- OpenAI-compatible endpoint URL + API key
- Health-check status
- Usage receipt on teardown
Recommended GPU routes
Estimated cost: $0.35–$0.60/hr while serving · Estimated runtime: Long-running endpoint — starts in 2–5 min
Example command
badgr serve meta-llama/Llama-3.1-8B-Instruct \
--gpu L40S \
--max-cost 10Common setup failures Badgr avoids
Requests silently dropped under a traffic spike — set max concurrent sequences below the route's tested ceiling instead of assuming unlimited concurrency.
High first-token latency under load — move to a Fast Serving route or reduce max concurrent sequences.
Ready to run this?
Launch from the dashboard, CLI, or Compute API. Max-cost protection included.