Quantization support
A check mark means that the platform class declares the method. NVIDIA CUDA accepts all registered methods, but each method can require a newer compute capability or an optional kernel package. Test your model on the target device before production use.
| Method | Minimum NVIDIA capability | AMD ROCm | Intel XPU |
|---|---|---|---|
awq | SM 7.5 | ✓ | ✓ |
auto_awq | SM 7.5 | ✓ | ✓ |
fp8 | SM 7.5 | ✓ | ✓ |
fbgemm_fp8 | SM 8.0 | ✓ | — |
fp_quant | SM 10.0 | — | — |
modelopt | SM 8.9 | ✓ | ✓ |
modelopt_fp4 | SM 7.5 | ✓ | — |
modelopt_mxfp8 | SM 8.0 | ✓ | — |
modelopt_mixed | SM 7.5 | ✓ | — |
auto_gptq | SM 6.0 | ✓ | ✓ |
gptq | SM 6.0 | ✓ | ✓ |
gptq_marlin | SM 6.0 | — | — |
awq_marlin | SM 7.5 | ✓ | — |
humming | SM 7.5 | — | — |
compressed-tensors | SM 7.0 | ✓ | ✓ |
bitsandbytes | SM 7.0 | ✓ | — |
experts_int8 | SM 8.0 | — | — |
exl3 | SM 8.0 | — | — |
quark | SM 7.0 | ✓ | — |
moe_wna16 | SM 7.0 | — | — |
torchao | SM 7.5 | ✓ | — |
inc | SM 6.0 | — | ✓ |
mxfp4 | SM 8.0 | ✓ | ✓ |
gpt_oss_mxfp4 | SM 8.0 | ✓ | ✓ |
deepseek_v4_fp8 | SM 7.5 | ✓ | — |
online | SM 7.5 | ✓ | ✓ |
fp8_per_tensor | SM 7.5 | ✓ | ✓ |
fp8_per_block | SM 7.5 | ✓ | ✓ |
fp8_per_channel | SM 7.5 | ✓ | — |
int8_per_channel_weight_only | SM 7.5 | — | — |
nvfp4_per_token | SM 7.5 | — | — |
mxfp8 | SM 8.0 | ✓ | ✓ |
mxfp6 | SM 11.0 | — | — |
Read the table
✓means that the platform registry permits the method.—means that the platform does not declare the method.- A permitted method can still require a specific GPU generation.
- CPU, Metal, and TPU do not yet publish a complete method list in their platform classes. Their columns will be added when that metadata becomes available.