Skip to content
项目

intel/neural-compressor

在 GitHub 查看

intel/neural-compressor has 2,734 GitHub stars and 337 forks. GitHot measured +1 stars in 24 hours, +2 in 7 days, and +5 in 15 days. Last checked 2026-11-22 09:29 UTC. Last push 2026-11-20.

这些数字如何统计

实测星标增长

24小时
+1
7天
+2
15天
+5
1个月
正在收集历史

各周期内的星标净变化,周期相互重叠,并非每日新增。缺失的测量保持空白。星标代表关注度,不代表软件质量。

关于

先进的低比特 LLM 量化(INT8/FP8/MXFP8/INT4/MXFP4/NVFP4)与稀疏化技术;面向 PyTorch、TensorFlow 和 ONNX Runtime 的领先模型压缩技术

机器学习

README.md

原文

Intel® Neural Compressor

An open-source Python library supporting popular model compression techniques on mainstream deep learning frameworks (PyTorch and JAX)

python version license coverage Downloads

Architecture   |   Workflow   |   Documentations


Intel® Neural Compressor aims to provide popular model compression techniques such as Static Quantization, Dynamic Quantization, SmoothQuant, Weight-Only Quantization, Quantization-Aware Training, Mixed Precision, etc.

What's New

  • [2026/03] FP8 quantization support for Keras/JAX (experimental)
  • [2026/03] FP8 KV cache/Attention static quantization with AutoRound (experimental)
  • [2025/12] NVFP4 quantization experimental support
  • [2025/10] MXFP8 / MXFP4 quantization experimental support
  • [2025/09] FP8 dynamic quantization, including Linear, FusedMoE on Intel Gaudi AI Accelerators
  • [2025/05] FP8 static quantization of DeepSeek V3/R1 model on Intel Gaudi AI Accelerators
  • [2025/03] VLM quantization in transformers-like API on Intel CPU/GPU

Installation

Choose the necessary framework dependencies to install based on your deploy environment.

Install Framework for PyTorch Backend (on-demand)

Intel Neural Compressor supports PyTorch with CPU, GPU and HPU. Please install the corresponding PyTorch version based on your hardware environment.

Install Neural Compressor from pypi

# Framework extension API + PyTorch dependency
pip install neural-compressor-pt
# Framework extension API + JAX dependency
pip install neural-compressor-jax

Note: Further installation methods can be found under Installation Guide. check out our FAQ for more details.

Getting Started

After successfully installing these packages, try your first quantization program. Following example code demonstrates FP8 Quantization, it is supported by Intel Gaudi2 AI Accelerator.
To try on Intel Gaudi2, docker image with Gaudi Software Stack is recommended, please refer to following script for environment setup. More details can be found in Gaudi Guide.

Run a container with an interactive shell, more info

docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --ipc=host vault.habana.ai/gaudi-docker/1.24.0/ubuntu24.04/habanalabs/pytorch-installer-2.10.0:latest

Note: Since Habana software >= 1.21.0, PT_HPU_LAZY_MODE=0 is the default setting. However, most low-precision functions (such as convert_from_uint4) do not support this setting. Therefore, we recommend setting PT_HPU_LAZY_MODE=1 to maintain compatibility.

Run the example,

from neural_compressor.torch.quantization import (
    FP8Config,
    prepare,
    convert,
)

import torch
import torchvision.models as models

model = models.resnet18()
qconfig = FP8Config(fp8_config="E4M3")
model = prepare(model, qconfig)

# Customer defined calibration. Below is a dummy calibration
model(torch.randn(1, 3, 224, 224).to("hpu"))

model = convert(model)

output = model(torch.randn(1, 3, 224, 224).to("hpu")).to("cpu")
print(output.shape)

More FP8 quantization doc.

Following example code demonstrates weight-only large language model loading on Intel Gaudi2 AI Accelerator.

from neural_compressor.torch.quantization import load

model_name = "TheBloke/Llama-2-7B-GPTQ"
model = load(
    model_name_or_path=model_name,
    format="huggingface",
    device="hpu",
    torch_dtype=torch.bfloat16,
)

Note: Intel Neural Compressor will convert the model format from auto-gptq to hpu format on the first load and save hpu_model.safetensors to the local cache directory for the next load. So it may take a while to load for the first time.

Documentation

Overview
Architecture Workflow APIs Examples
PyTorch Extension APIs
Overview
Dynamic Quantization Static Quantization Smooth Quantization
Weight-Only Quantization FP8 Quantization Mixed Precision
MX Quantization NVFP4 Quantization
Transformers-like APIs
Overview
JAX Extension APIs
Overview
Other Modules
Auto Tune

Selected Publications/Events

Note: View Full Publication List.

Additional Content

Communication

  • GitHub Issues: mainly for bug reports, new feature requests, question asking, etc.
  • Email: welcome to raise any interesting research ideas on model compression techniques by email for collaborations.

探索相关项目

加载中…