@@ -46,9 +46,12 @@ pip install --upgrade pip # enable PEP 660 support
pip install -e .
```
For edge devices like Orin, please modify [pyproject.toml](pyproject.toml) by commenting out [this line](https://github.com/mit-han-lab/llm-awq/blob/3fce69061682fdd528824e5da3d03a8a8b545f2a/pyproject.toml#L17). You will have to manually install precompiled PyTorch binaries from [NVIDIA](https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048).
* For **edge devices** like Orin, before running the commands above, please:
1. Modify [pyproject.toml](pyproject.toml) by commenting out [this line](https://github.com/mit-han-lab/llm-awq/blob/3fce69061682fdd528824e5da3d03a8a8b545f2a/pyproject.toml#L17).
2. Manually install precompiled PyTorch binaries from [NVIDIA](https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048).
3. Set the appropriate Python version for conda environment (e.g., `conda create -n awq python=3.8 -y` for JetPack 5).
3. Install efficient W4A16 (4-bit weight, 16-bit activation) CUDA kernel and optimized FP16 kernels (e.g. layernorm, positional encodings).
