Issue evidence · Run · ostris/ai-toolkit

Prodigy 8-bit optimizer state fails to restore due to Auto8bitTensor deserialization error

GitHub issue: https://github.com/ostris/ai-toolkit/issues/1076

Reproduced, workaround proven

Resuming a job restarts the optimizer from scratch because torch rejects the saved Auto8bitTensor under weights_only loading.

What Badgr ran

Used the repository's real Auto8bitTensor (commit f7a1fb9) to save an optimizer state, then loaded it with torch.load(weights_only=True), with and without allowlisting the class. This reproduces the load failure, not a full training resume.

What came back

  • weights_only=True failed: Unsupported global: toolkit.optimizers.optimizer_utils.Auto8bitTensor.
  • With torch.serialization.safe_globals([Auto8bitTensor]) the state loaded, with the learning-rate term (d) and step intact.

The effect on a training run's learning-rate curve was not measured.

Checked 2026-10-11 in Badgr’s local development environment.

← All issue evidence