Qwen 3 8B GGUF

✓ Verified by Badgr · llama.cpp · Serve

Last verified 2026-10-10

badgr serve --runtime llama.cpp --hf-repo Qwen/Qwen3-8B-GGUF --hf-file Qwen3-8B-Q4_K_M.gguf --gpu RTX_4090 --max-cost 0.8 --startup-timeout 20 --env LLAMA_ARG_N_GPU_LAYERS=99 --env LLAMA_ARG_N_PARALLEL=8 --env LLAMA_ARG_KV_UNIFIED=1 --env LLAMA_ARG_CTX_SIZE=20480 --env LLAMA_ARG_CACHE_TYPE_K=q8_0 --env LLAMA_ARG_CACHE_TYPE_V=q8_0 --env LLAMA_ARG_FLASH_ATTN=on

GPU

NVIDIA GeForce RTX 4090Reported by the provider, not confirmed inside the container

Time to verified

154s

Deployment

dep-d95b6fbdc9

• ✓ llama-server loaded Qwen3-8B Q4_K_M with 8 parallel slots and a unified KV cache
• ✓ final_state = VERIFIED
• ✓ 7 streams were generating while new requests were timed: the worst new request waited 4.0 s and 2.55 s in two samples (median 0.73 s)
• ✓ Teardown completed

Default configuration for llama.cpp#30252: with idle-slot prompt caching on, a new request can stall for seconds while other streams are generating. Measured on an RTX 4090 over the public endpoint, two samples of 8 requests each.

GitHub issue this run was for: ggml-org/llama.cpp#30252

final_state=VERIFIED failure_class=None providers_tried=1/1 teardown_confirmed=False spend_usd=0.0186

Run from Badgr's local development environment. Documents that this deployment path works end-to-end for this model.

✓ Verified by Badgr · llama.cpp · Serve

Last verified 2026-10-10

badgr serve --runtime llama.cpp --hf-repo Qwen/Qwen3-8B-GGUF --hf-file Qwen3-8B-Q4_K_M.gguf --gpu RTX_4090 --max-cost 0.8 --startup-timeout 20 --env LLAMA_ARG_N_GPU_LAYERS=99 --env LLAMA_ARG_N_PARALLEL=8 --env LLAMA_ARG_KV_UNIFIED=1 --env LLAMA_ARG_CTX_SIZE=20480 --env LLAMA_ARG_CACHE_TYPE_K=q8_0 --env LLAMA_ARG_CACHE_TYPE_V=q8_0 --env LLAMA_ARG_FLASH_ATTN=on --env LLAMA_ARG_CACHE_IDLE_SLOTS=0

GPU

NVIDIA GeForce RTX 4090Reported by the provider, not confirmed inside the container

Time to verified

128s

Deployment

dep-f7c7827944

• ✓ Same model and settings with idle-slot prompt caching switched off (LLAMA_ARG_CACHE_IDLE_SLOTS=0)
• ✓ final_state = VERIFIED
• ✓ With 7 streams generating, the worst new request waited 0.94 s and 0.87 s in two samples (median 0.85 s and 0.48 s)
• ✓ Teardown completed

Workaround for llama.cpp#30252. Same hardware and load as the default run: the worst wait fell from 4.0 s / 2.55 s to 0.94 s / 0.87 s. Two samples each, so this shows the stall disappears, not an exact speed-up.

GitHub issue this run was for: ggml-org/llama.cpp#30252

final_state=VERIFIED failure_class=None providers_tried=1/1 teardown_confirmed=False spend_usd=0.0153

Run from Badgr's local development environment. Documents that this deployment path works end-to-end for this model.