Issue evidence · Run · huggingface/trl

DiffusionGemma SFT example evicts sampled-block prefix from sliding-window KV cache

GitHub issue: https://github.com/huggingface/trl/issues/7734

Reproduced

The DiffusionGemma SFT example encodes the whole sequence with the sliding-window cache, so the prefix before the sampled block may be evicted before the decoder mask is applied.

What Badgr ran

Ran the issue's CPU script against TRL at commit 8f39f8f with transformers 5.19.0 on macOS (arm64), using a tiny DiffusionGemma config (sequence 1024, prompt 128, sliding window 512, canvas 256) and a 384-token control.

What came back

  • Length 1024: retained positions 513 to 1023, visible cached prefix 0 tokens (reference: 128); diffusion loss 4.368756 against 4.366831; encoder AR loss 4.609928 on both.
  • Maximum absolute gradient difference 0.14644 (relative L2 77.14%).
  • Length 384 control: prefix intact, gradient difference 2.3e-07.

Matches the issue's table. The effect shown is on a small random-weight model, not a trained one; no GPU or download was needed.

Checked 2026-10-15 in Badgr’s local development environment.

← All issue evidence