Integrations / KTransformers
DeployExperimental
KTransformers on Badgr: compatibility guide
A guide for running KTransformers, which splits a large model between GPU and CPU memory, from a custom container.
A setup guide only. Badgr has not tested this path, so expect to adjust it.
Why use Badgr for KTransformers
- Badgr interface
- Serve
- Where KTransformers fits
- Model runtime
- Why Badgr
- Specialised CPU and GPU inference
- Try a CPU and GPU memory split on rented hardware before you buy any.
- A cost cap and a stop command bound the spend.
- Badgr has not tested this path. It is a guide for your own container.
What you can do
- Run a very large mixture-of-experts model with limited GPU memory
- Test a CPU and GPU split before buying hardware
- Serve a model that does not fit on one GPU alone
Quickstart: Run your own container
badgr serve --image <your-registry>/ktransformers:<tag> --cmd "<your start command>" --max-cost 5
How it works
- 1.You give Badgr your container image and start command
- 2.Badgr runs it on a GPU
- 3.You check that it answers
- 4.You stop it when done
What you get
- A service address if your command exposes one
- Logs and a receipt
Requirements
- The Badgr CLI installed and logged in (npm install -g badgr-cli)
- A funded Badgr balance
- A container image with KTransformers and its start command
- A machine with enough system memory for the CPU part of the model
- A CPU and GPU combination KTransformers supports
What it costs
- Badgr has no ready-made setup for this. The command below runs your own container with the generic custom option. It has not been tested by Badgr, so treat this page as a setup guide.
- A service bills for as long as it runs, until you stop it with badgr down <id>.
- Pass --max-cost to bound spend. Badgr shows the rate for the GPU it picked before it starts. This page does not quote a fixed price.
- Machines with a lot of system memory can cost more per hour.
See pricing for how billing works.
Proof
No tested run is recorded for this integration yet. A receipt and example output will be added here after a real run, not before.
Troubleshooting
Crashes at load
The CPU part of the model needs system memory. Ask for a machine with more of it.
Very slow generation
Parts of the model on the CPU are slower. Move more to the GPU if memory allows.
Service never becomes ready
Read the logs with badgr logs <id>. The most common causes are a wrong port, a missing model file or too little GPU memory.
Developer reference
Ready to start with KTransformers?
Every run or service has a spending limit you set before it starts.