Issue evidence · Serve · sgl-project/sglang

Speculative decoding crashes server on first non-greedy request: flashinfer verify-sampling op JIT fails

GitHub issue: https://github.com/sgl-project/sglang/issues/43412

Not reproduced

With speculative decoding on, the first sampled request kills the server because flashinfer's JIT build fails in a pip-installed bundle.

What Badgr ran

Built the reporter's pip-only environment in a fresh venv on an RTX 4090 through Badgr (sglang 0.5.21, torch 2.13.0, flashinfer 0.6.18, nvidia-cuda-nvcc 13.4.92, system nvcc removed from PATH), launched Qwen2.5-0.5B-Instruct with --speculative-algorithm NGRAM --attention-backend triton, and sent one request at temperature 0 and one at 0.7.

What came back

  • Temperature 0: HTTP 200.
  • Temperature 0.7: HTTP 200, and the server stayed up.

The reporter used an RTX 4070 Laptop (sm_89) on driver 595.99; this was a 4090 (also sm_89) on driver 580.126, so a driver or flashinfer cache difference is possible. A different small model was used.

Checked 2026-10-15 in Badgr’s local development environment.

← All issue evidence