Issue evidence · Serve · vllm-project/vllm

--enable-prompt-embeds sizes OPT-350m embeds at hidden_size 1024, but OPT embeds at word_embed_proj_dim 512

GitHub issue: https://github.com/vllm-project/vllm/issues/61709

Reproduced

vLLM validates and buffers OPT-350m prompt embeds at hidden_size 1024, but the model's project_in expects 512-wide inputs.

What Badgr ran

Ran the issue's script (vLLM 0.32.0rc1.dev190 (cu134 nightly image) on an RTX 4090 through Badgr, with the repository tests at commit 7f730b4 (main, 2026-10-15)): built ModelConfig for facebook/opt-350m with enable_prompt_embeds, validated 512 and 1024 wide embeds, and wrote 512-wide rows into the inputs_embeds buffer.

What came back

  • hidden_size=1024, word_embed_proj_dim=512; get_hidden_size() and get_inputs_embeds_size() both 1024.
  • Width 512: validation rejected it ("hidden_size 512 does not match the model's hidden_size 1024"), though project_in accepts it.
  • Width 1024: validation passed, but project_in failed (4x1024 against 512x1024).
  • Writing 512-wide rows into the buffer raised "shape mismatch: value tensor of shape [2, 512] cannot be broadcast to indexing result of shape [2, 1024]".

The issue's own script, run on GPU hardware but exercising config and validation code only; no OPT-350m generation with prompt embeds was run.

Checked 2026-10-15 in Badgr’s local development environment.

← All issue evidence