Issue evidence · Serve · huggingface/text-generation-inference

"sharded is not supported for AutoModel" error when deploying a SageMaker endpoint for Qwen 2.5 7B

GitHub issue: https://github.com/huggingface/text-generation-inference/issues/2783

Reproduced

TGI 1.4.0 fails to start Qwen 2.5 7B across two shards with NotImplementedError: sharded is not supported for AutoModel.

What Badgr ran

Ran ghcr.io/huggingface/text-generation-inference:1.4.0 on 2x RTX 4090 through Badgr with Qwen/Qwen2.5-7B-Instruct and --num-shard 2, reading the launcher log.

What came back

  • The server raised NotImplementedError: sharded is not supported for AutoModel from get_model(), then "Shard 0 failed to start" and "Error: ShardCannotStart".

The reporter's fine-tuned model was not available, so the base model was used. A working replacement was attempted with vLLM (v0.30.0, Qwen2.5-7B-Instruct, one GPU) but every launch failed on GPU hosts before serving, so no workaround is proven here.

Checked 2026-10-14 in Badgr’s local development environment.

← All issue evidence