Deploy OpenJev's Decision API with Badgr: One GPU, Verified Inference

Prerequisites

• A Badgr account and the CLI, version 1.2.16 or later (for --endpoint-key): npm i -g badgr-cli, then badgr login
• An 80 GB GPU. The fp8 model uses about 28 GiB for weights and the settings below reserve 90% of GPU memory. In our run the H100 cost about $3.50/hour.
• A budget. The cap below is $3.00. Our whole run, including the tests in this guide, cost $0.85.

The launch script

Save this as openjev-serve.sh. It installs the pinned client versions from the model card, downloads the model, checks shim.py against the file we reviewed, starts vLLM with the model card’s flags, then starts the shim with the calibration constants and the key Badgr hands the container as BADGR_ENDPOINT_API_KEY. While the model loads, port 3000 serves a key-protected /log so you can watch progress.

L=/tmp/startup.log
: > $L
stage() { echo "$(date -u +%H:%M:%S) STAGE $1" >> $L; }
cat > /tmp/logsrv.py <<'PY'
import os, http.server
K = os.environ.get("BADGR_ENDPOINT_API_KEY", "")
class H(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_GET(self):
        if self.headers.get("Authorization", "") != "Bearer " + K:
            self.send_response(401); self.end_headers(); return
        if self.path.startswith("/log"):
            d = open("/tmp/startup.log", "rb").read()[-30000:]
            self.send_response(200); self.send_header("Content-Type", "text/plain"); self.end_headers(); self.wfile.write(d); return
        self.send_response(503); self.end_headers()
http.server.ThreadingHTTPServer(("0.0.0.0", 3000), H).serve_forever()
PY
python3 /tmp/logsrv.py &
LOGPID=$!
stage "entrypoint ok: $(python3 --version 2>&1) $(nvidia-smi --query-gpu=name,memory.total --format=csv,noheader 2>&1 | head -1)"
stage "disk: $(df -h /workspace 2>&1 | tail -1)"
pip install -q "openai==3.16.2" "httpx==0.28.1" transformers huggingface_hub >> $L 2>&1; stage "pip exit=$?"
mkdir -p /workspace; cd /workspace
stage "model download start"
( while true; do sleep 30; echo "$(date -u +%H:%M:%S) PROGRESS download_mb=$(du -sm /workspace/openjev 2>/dev/null | cut -f1)" >> $L; done ) &
DLPID=$!
hf download openjev/openjev --local-dir /workspace/openjev >> $L 2>&1; stage "download exit=$?"
kill $DLPID 2>/dev/null
stage "model dir: $(du -sm /workspace/openjev | cut -f1) MB, $(ls /workspace/openjev | wc -l) files"
echo "81a22f1b1b8912a465059207ef9f60b7c6c16b4de6372305d867efbe38a1987a  openjev/helper/shim.py" | sha256sum -c - >> $L 2>&1; stage "shim sha exit=$?"
stage "vllm start (model-card flags, fp8)"
vllm serve ./openjev --host 127.0.0.1 --served-model-name qwen --port 8000 --enable-prefix-caching --max-model-len 16384 --gpu-memory-utilization 0.90 --limit-mm-per-prompt '{"image":1}' --trust-remote-code --max-num-seqs 256 --max-logprobs 64 --gdn-prefill-backend triton --quantization fp8 >> $L 2>&1 &
VPID=$!
( while kill -0 $VPID 2>/dev/null; do sleep 30; echo "$(date -u +%H:%M:%S) PROGRESS gpu_mem=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader 2>&1 | head -1)" >> $L; done ) &
ok=0
while kill -0 $VPID 2>/dev/null; do
  curl -sf http://127.0.0.1:8000/v1/models >/dev/null && { ok=1; break; }
  sleep 5
done
if [ $ok = 1 ]; then stage "vllm ready"; else stage "vllm DIED exit=$(wait $VPID; echo $?)"; sleep 1500; exit 1; fi
kill $LOGPID; sleep 1
stage "shim start"
VLLM=http://127.0.0.1:8000/v1 TOKENIZER=./openjev READOUT_T=0.85 READOUT_NOUL_T=1.829074 READOUT_NOUL_BIAS=0 READOUT_TARGETED=1 READOUT_INSTR_STYLE=pyrepr SHIM_STAGGER=1 SHIM_TOKEN="${BADGR_ENDPOINT_API_KEY:?no key}" python3 openjev/helper/shim.py --host 0.0.0.0 --port 3000

Deploy

badgr serve \
  --image vllm/vllm-openai:v0.29.0-cu129 \
  --gpu H100 --port 3000 \
  --health-path /v1/version --verify-model qwen \
  --endpoint-key --funding paid --min-disk 150 \
  --startup-timeout 45 --max-cost 3 --yes --detach \
  --cmd "$(cat openjev-serve.sh)"

--endpoint-key makes Badgr generate a key, pass it to the container, and send it on its own readiness and verification checks. --startup-timeout 45 gives the model 45 minutes to become ready. --max-cost 3 caps total spend. The command prints the deployment id and the key once:

  Launch accepted. Deployment: dep-xxxxxxxxxx
  API key:      bge_...   (shown once, copy it now)

Watch it start (the log is only readable with the key):

curl -s $ENDPOINT_URL/log -H "Authorization: Bearer $KEY"
# 03:14:52 PROGRESS gpu_mem=4 MiB
# 03:15:22 PROGRESS gpu_mem=13133 MiB      (weights loading)
# 03:16:22 PROGRESS gpu_mem=31489 MiB      (torch.compile)
# (EngineCore) Capturing CUDA graphs (PIECEWISE): ...

In our run the 52 GB download took about 90 seconds, weights loaded in 10 seconds, and Badgr reported the deployment verified after 473 seconds, having spent $0.4586 to that point. Badgr counts a deployment as verified only after a real request returned an answer through the authenticated endpoint.

Check authentication

Find your endpoint URL with badgr status. Without the key, every route is refused:

curl -s -o /dev/null -w '%{http_code}\n' $ENDPOINT_URL/v1/version
# 401
curl -s -o /dev/null -w '%{http_code}\n' $ENDPOINT_URL/v1/version -H "Authorization: Bearer nope"
# 401

With the key, the version route reports the exact calibration the shim is running:

curl -s $ENDPOINT_URL/v1/version -H "Authorization: Bearer $KEY"
# {"model_dir": "openjev", "T": 0.85, "noul_t": 1.829074, "noul_bias": 0.0,
#  "flags": {"targeted": true, "instr_style": "pyrepr", "stagger": true, ...},
#  "shim_sha256": "81a22f1b1b8912a465059207ef9f60b7c6c16b4de6372305d867efbe38a1987a"}

Send a decision request

curl -s $ENDPOINT_URL/v1/systemone \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "openjev",
    "state": "Customer says they were charged twice.",
    "questions": {
      "route": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {"billing": null, "shipping": null, "technical": null}
      }
    }
  }'

Real response from our run (1.6 s):

{
  "id": "shim-1791775542384",
  "answers": {
    "route": {
      "type": "choice",
      "choice": "billing",
      "probabilities": {"billing": 0.9997, "shipping": 0.0001, "technical": 0.0001},
      "confidence": 0.9996
    }
  },
  "usage": {"input_tokens": 60, "output_tokens": 0}
}

The reply carries a typed route decision with a probability for every option and a confidence. Two more checks from the same run: “My package says delivered but nothing arrived at my door” routed to shipping (0.9995, confidence 0.9992), and “The app crashes every time I open the settings page” routed to technical (0.9998, confidence 0.9997). A yes/no question (type: noul) about whether to refund the double charge returned 0.9291. These are smoke tests of a few hand-written cases, not an accuracy measurement; see the model card for Loop AI’s benchmarks.

Stop paying

badgr down <deployment-id>
badgr status        # Nothing running.

The GPU is released and billing stops. Our run was stopped 15 minutes after launch. Final cost was $0.8518, and the provider’s own pod list was empty afterwards.

What to know

• The shim speaks plain HTTP, and its check is a string comparison with no rate limiting. Treat the key as a gate, not as hardened authentication, and put TLS in front of it for anything beyond testing.
• The vLLM port (8000) is bound to 127.0.0.1 inside the container and is never published. Keep it that way: it has no authentication.
• The key is generated for this endpoint and shown once. Keep it in a secret store, not in a file in a repository.
• We ran this once on an H100 80 GB. Other GPUs and larger context lengths are untested.

Next steps

• Point your agent’s decision step at /v1/systemone on your endpoint and compare the probabilities with your current prompts.
• For a cheaper, smaller build, see the GGUF guide.
• Re-read the licence above before any commercial use.