Skip to content

GitLab

Unverified Commit 90f54199 authored by Matthew Douglas's avatar Matthew Douglas Committed by GitHub
Browse files

Documentation updates for v0.48.0 (#1770) 

* Update installation docs

* Update links

* Fix cuda min glibc in doc

* Update header levels

* Update AMD section

* typo
parent b8d1c261
Hide whitespace changes
Inline Side-by-side
Showing with 101 additions and 121 deletions
docs/source/installation.mdx
View file @ 90f54199
# Installation Guide # Installation Guide
Welcome to the installation guide for the `bitsandbytes` library! This document provides step-by-step instructions to install `bitsandbytes` across various platforms and hardware configurations. The library primarily supports CUDA-based GPUs, but the team is actively working on enabling support for additional backends like CPU, AMD ROCm, Intel XPU, and Gaudi HPU. Welcome to the installation guide for the `bitsandbytes` library! This document provides step-by-step instructions to install `bitsandbytes` across various platforms and hardware configurations.
We provide official support for NVIDIA GPUs, CPUs, Intel XPUs, and Intel Gaudi platforms. We also have experimental support for
additional platforms such as AMD ROCm.
## Table of Contents ## Table of Contents
- [CUDA](#cuda) - [System Requirements](#requirements)
- [NVIDIA CUDA](#cuda)
- [Installation via PyPI](#cuda-pip) - [Installation via PyPI](#cuda-pip)
- [Compile from Source](#cuda-compile) - [Compile from Source](#cuda-compile)
- [Preview Wheels from `main`](#cuda-preview) - [Intel XPU](#xpu)
- [Multi-Backend Preview](#multi-backend) - [Installation via PyPI](#xpu-pip)
- [Supported Backends](#multi-backend-supported-backends) - [Intel Gaudi](#gaudi)
- [Pre-requisites](#multi-backend-pre-requisites) - [Installation via PyPI](#gaudi-pip)
- [Installation](#multi-backend-pip) - [CPU](#cpu)
- [Compile from Source](#multi-backend-compile) - [Installation via PyPI](#cpu-pip)
- [Compile from Source](#cpu-compile)
- [AMD ROCm (Preview)](#rocm-preview)
- [Preview Wheels](#preview-wheels)
## System Requirements[[requirements]]
These are the minimum requirements for `bitsandbytes` across all platforms. Please be aware that some compute platforms may impose more strict requirements.
## CUDA[[cuda]] * Python >= 3.9
* PyTorch >= 2.3
## NVIDIA CUDA[[cuda]]
`bitsandbytes` is currently supported on NVIDIA GPUs with [Compute Capability](https://developer.nvidia.com/cuda-gpus) 6.0+. `bitsandbytes` is currently supported on NVIDIA GPUs with [Compute Capability](https://developer.nvidia.com/cuda-gpus) 6.0+.
The library can be built using CUDA Toolkit versions as old as **11.8**. The library can be built using CUDA Toolkit versions as old as **11.8**.
...@@ -25,11 +39,13 @@ The library can be built using CUDA Toolkit versions as old as **11.8**. ...@@ -25,11 +39,13 @@ The library can be built using CUDA Toolkit versions as old as **11.8**.
| 8-bit optimizers/quantization | 6.0+ | Pascal (GTX 10X0 series, P100) or newer GPUs| | 8-bit optimizers/quantization | 6.0+ | Pascal (GTX 10X0 series, P100) or newer GPUs|
| NF4/FP4 quantization | 6.0+ | Pascal (GTX 10X0 series, P100) or newer GPUs| | NF4/FP4 quantization | 6.0+ | Pascal (GTX 10X0 series, P100) or newer GPUs|
> [!WARNING] > [!WARNING]
> Support for Maxwell GPUs is deprecated and will be removed in a future release. > Support for Maxwell GPUs is deprecated and will be removed in a future release.
> Maxwell support is not included in PyPI distributions from `v0.48.0` on and must be built from source. > Maxwell support is not included in PyPI distributions from `v0.48.0` on and must be built from source.
> For the best results, a Turing generation device or newer is recommended. > For the best results, a Turing generation device or newer is recommended.
### Installation via PyPI[[cuda-pip]] ### Installation via PyPI[[cuda-pip]]
This is the most straightforward and recommended installation option. This is the most straightforward and recommended installation option.
...@@ -40,20 +56,22 @@ The currently distributed `bitsandbytes` packages are built with the following c ...@@ -40,20 +56,22 @@ The currently distributed `bitsandbytes` packages are built with the following c
|--------------------|------------------|----------------------|-------------- |--------------------|------------------|----------------------|--------------
| **Linux x86-64** | 11.8 - 12.6 | GCC 11.2 | sm60, sm70, sm75, sm80, sm86, sm89, sm90 | **Linux x86-64** | 11.8 - 12.6 | GCC 11.2 | sm60, sm70, sm75, sm80, sm86, sm89, sm90
| **Linux x86-64** | 12.8 - 12.9 | GCC 11.2 | sm70, sm75, sm80, sm86, sm89, sm90, sm100, sm120 | **Linux x86-64** | 12.8 - 12.9 | GCC 11.2 | sm70, sm75, sm80, sm86, sm89, sm90, sm100, sm120
| **Linux x86-64** | 13.0 | GCC 11.2 | sm75, sm80, sm86, sm89, sm90, sm100, sm120 | **Linux x86-64** | 13.0 | GCC 11.2 | sm75, sm80, sm86, sm89, sm90, sm100, sm110, sm120
| **Linux aarch64** | 11.8 - 12.6 | GCC 11.2 | sm75, sm80, sm90 | **Linux aarch64** | 11.8 - 12.6 | GCC 11.2 | sm75, sm80, sm90
| **Linux aarch64** | 12.8 - 13.0 | GCC 11.2 | sm75, sm80, sm90, sm100, sm120 | **Linux aarch64** | 12.8 - 13.0 | GCC 11.2 | sm75, sm80, sm90, sm100, sm120
| **Windows x86-64** | 11.8 - 12.6 | MSVC 19.43+ (VS2022) | sm50, sm60, sm75, sm80, sm86, sm89, sm90 | **Windows x86-64** | 11.8 - 12.6 | MSVC 19.43+ (VS2022) | sm50, sm60, sm75, sm80, sm86, sm89, sm90
| **Windows x86-64** | 12.8 - 12.9 | MSVC 19.43+ (VS2022) | sm70, sm75, sm80, sm86, sm89, sm90, sm100, sm120 | **Windows x86-64** | 12.8 - 12.9 | MSVC 19.43+ (VS2022) | sm70, sm75, sm80, sm86, sm89, sm90, sm100, sm120
| **Windows x86-64** | 13.0 | MSVC 19.43+ (VS2022) | sm75, sm80, sm86, sm89, sm90, sm100, sm120 | **Windows x86-64** | 13.0 | MSVC 19.43+ (VS2022) | sm75, sm80, sm86, sm89, sm90, sm100, sm120
Use `pip` or `uv` to install: The Linux build has a minimum glibc version of 2.24.
Use `pip` or `uv` to install the latest release:
```bash ```bash
pip install bitsandbytes pip install bitsandbytes
``` ```
### Compile from source[[cuda-compile]] ### Compile from Source[[cuda-compile]]
> [!TIP] > [!TIP]
> Don't hesitate to compile from source! The process is pretty straight forward and resilient. This might be needed for older CUDA Toolkit versions or Linux distributions, or other less common configurations. > Don't hesitate to compile from source! The process is pretty straight forward and resilient. This might be needed for older CUDA Toolkit versions or Linux distributions, or other less common configurations.
...@@ -102,131 +120,102 @@ Big thanks to [wkpark](https://github.com/wkpark), [Jamezo97](https://github.com ...@@ -102,131 +120,102 @@ Big thanks to [wkpark](https://github.com/wkpark), [Jamezo97](https://github.com
</hfoption> </hfoption>
</hfoptions> </hfoptions>
### Preview Wheels from `main`[[cuda-preview]] ## Intel XPU[[xpu]]
If you would like to use new features even before they are officially released and help us test them, feel free to install the wheel directly from our CI (*the wheel links will remain stable!*): * A compatible PyTorch version with Intel XPU support is required. The current minimum is **PyTorch 2.6.0**. It is recommended to use the latest stable release. See [Getting Started on Intel GPU](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) for guidance.
<hfoptions id="OS"> ### Installation via PyPI[[xpu-pip]]
<hfoption id="Linux">
```bash This is the most straightforward and recommended installation option.
# Note: if you don't want to reinstall our dependencies, append the `--no-deps` flag!
# x86_64 (most users) The currently distributed `bitsandbytes` packages are built with the following configurations:
pip install --force-reinstall https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_main/bitsandbytes-1.33.7.preview-py3-none-manylinux_2_24_x86_64.whl
# ARM/aarch64 | **OS** | **oneAPI Toolkit** | **Kernel Implementation** |
pip install --force-reinstall https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_main/bitsandbytes-1.33.7.preview-py3-none-manylinux_2_24_aarch64.whl |--------------------|------------------|----------------------|
``` | **Linux x86-64** | 2025.1.3 | SYCL + Triton |
| **Windows x86-64** | N/A | SYCL |
</hfoption> The Linux build has a minimum glibc version of 2.34.
<hfoption id="Windows">
Use `pip` or `uv` to install the latest release:
```bash ```bash
# Note: if you don't want to reinstall our dependencies, append the `--no-deps` flag! pip install bitsandbytes
pip install --force-reinstall https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_main/bitsandbytes-1.33.7.preview-py3-none-win_amd64.whl
``` ```
</hfoption>
</hfoptions>
## Intel Gaudi[[gaudi]]
## Multi-Backend Preview[[multi-backend]] * A compatible PyTorch version with Intel Gaudi support is required. The current minimum is **Gaudi v1.21** with **PyTorch 2.6.0**. It is recommended to use the latest stable release. See the Gaudi software [installation guide](https://docs.habana.ai/en/latest/Installation_Guide/index.html) for guidance.
> [!WARNING]
> This functionality existed as an early technical preview and is not recommended for production use. We are in the process of upstreaming improved support for AMD and Intel hardware into the main project.
We provide an early preview of support for AMD and Intel hardware as part of a development branch. ### Installation from PyPI[[gaudi-pip]]
### Supported Backends[[multi-backend-supported-backends]] Use `pip` or `uv` to install the latest release:
| **Backend** | **Supported Versions** | **Python versions** | **Architecture Support** | **Status** |
|-------------|------------------------|---------------------------|-------------------------|------------|
| **AMD ROCm** | 6.1+ | 3.10+ | minimum CDNA - `gfx90a`, RDNA - `gfx1100` | Alpha |
| **Intel CPU** | v2.4.0+ | 3.10+ | Intel CPU | Alpha |
| **Intel GPU** | v2.7.0+ | 3.10+ | Intel GPU | Experimental |
| **Ascend NPU** | 2.1.0+ (`torch_npu`) | 3.10+ | Ascend NPU | Experimental |
For each supported backend, follow the respective instructions below:
### Pre-requisites[[multi-backend-pre-requisites]]
To use this preview version of `bitsandbytes` with `transformers`, be sure to install:
```bash ```bash
pip install "transformers>=4.45.1" pip install bitsandbytes
``` ```
<hfoptions id="backend"> ## CPU[[cpu]]
<hfoption id="AMD ROCm">
> [!WARNING] ### Installation from PyPI[[cpu-pip]]
> Pre-compiled binaries are only built for ROCm versions `6.1.2`/`6.2.4`/`6.3.2` and `gfx90a`, `gfx942`, `gfx1100` GPU architectures. [Find the pip install instructions here](#multi-backend-pip).
>
> Other supported versions that don't come with pre-compiled binaries [can be compiled for with these instructions](#multi-backend-compile).
>
> **Windows is not supported for the ROCm backend**
> [!TIP] This is the most straightforward and recommended installation option.
> If you would like to install ROCm and PyTorch on bare metal, skip the Docker steps and refer to ROCm's official guides at [ROCm installation overview](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/install-overview.html#rocm-install-overview) and [Installing PyTorch for ROCm](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/3rd-party/pytorch-install.html#using-wheels-package) (Step 3 of wheels build for quick installation). Special note: please make sure to get the respective ROCm-specific PyTorch wheel for the installed ROCm version, e.g. `https://download.pytorch.org/whl/nightly/rocm6.2/`!
```bash The currently distributed `bitsandbytes` packages are built with the following configurations:
# Create a docker container with the ROCm image, which includes ROCm libraries
docker pull rocm/dev-ubuntu-22.04:6.3.4-complete
docker run -it --device=/dev/kfd --device=/dev/dri --group-add video rocm/dev-ubuntu-22.04:6.3.4-complete
apt-get update && apt-get install -y git && cd home
# Install pytorch compatible with above ROCm version | **OS** | **Host Compiler** | Hardware Minimum
pip install torch --index-url https://download.pytorch.org/whl/rocm6.3/ |--------------------|----------------------|----------------------|
``` | **Linux x86-64** | GCC 11.4 | AVX2 |
| **Linux aarch64** | GCC 11.4 | |
| **Windows x86-64** | MSVC 19.43+ (VS2022) | AVX2 |
</hfoption> The Linux build has a minimum glibc version of 2.24.
<hfoption id="Intel XPU">
* A compatible PyTorch version with Intel XPU support is required. It is recommended to use the latest stable release. See [Getting Started on Intel GPU](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) for guidance. Use `pip` or `uv` to install the latest release:
</hfoption> ```bash
</hfoptions> pip install bitsandbytes
```
### Installation ### Compile from Source[[cpu-compile]]
You can install the pre-built wheels for each backend, or compile from source for custom configurations. To compile from source, simply install the package from source using `pip`. The package will be built for CPU only at this time.
#### Pre-built Wheel Installation (recommended)[[multi-backend-pip]] ```bash
git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git && cd bitsandbytes/
pip install -e .
```
<hfoptions id="platform"> ## AMD ROCm (Preview)[[rocm]]
<hfoption id="Linux">
This wheel provides support for ROCm and Intel XPU platforms.
``` * A compatible PyTorch version with AMD ROCm support is required. It is recommended to use the latest stable release. See [PyTorch on ROCm](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/3rd-party/pytorch-install.html) for guidance.
# Note, if you don't want to reinstall our dependencies, append the `--no-deps` flag! * ROCm support is currently only available in our preview wheels or when building from source.
pip install --force-reinstall 'https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_multi-backend-refactor/bitsandbytes-0.44.1.dev0-py3-none-manylinux_2_24_x86_64.whl'
```
</hfoption> ### Preview Wheels from `main`[[rocm-preview]]
<hfoption id="Windows">
This wheel provides support for the Intel XPU platform.
```bash The currently distributed preview `bitsandbytes` are built with the following configurations:
# Note, if you don't want to reinstall our dependencies, append the `--no-deps` flag!
pip install --force-reinstall 'https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_multi-backend-refactor/bitsandbytes-0.44.1.dev0-py3-none-win_amd64.whl'
```
</hfoption> | **OS** | **ROCm** | **Targets**
</hfoptions> |--------------------|----------|---------------------------|
| **Linux x86-64** | 6.1.2 | gfx90a / gfx942 / gfx1100
| **Linux x86-64** | 6.2.4 | gfx90a / gfx942 / gfx1100
| **Linux x86-64** | 6.3.4 | gfx90a / gfx942 / gfx1100
| **Linux x86-64** | 6.4.4 | gfx90a / gfx942 / gfx1100
| **Linux x86-64** | 7.0.0 | gfx90a / gfx942 / gfx1100
#### Compile from Source[[multi-backend-compile]] **Windows is not currently supported.**
<hfoptions id="backend"> Please see [Preview Wheels](#preview-wheels) for installation instructions.
<hfoption id="AMD ROCm">
#### AMD GPU ### Compile from Source[[rocm-compile]]
bitsandbytes is supported from ROCm 6.1 - ROCm 6.4. bitsandbytes can be compiled from ROCm 6.1 - ROCm 7.0.
```bash ```bash
# Install bitsandbytes from source # Install bitsandbytes from source
# Clone bitsandbytes repo, ROCm backend is currently enabled on multi-backend-refactor branch # Clone bitsandbytes repo
git clone -b multi-backend-refactor https://github.com/bitsandbytes-foundation/bitsandbytes.git && cd bitsandbytes/ git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git && cd bitsandbytes/
# Compile & install # Compile & install
apt-get install -y build-essential cmake # install build tools dependencies, unless present apt-get install -y build-essential cmake # install build tools dependencies, unless present
...@@ -235,38 +224,29 @@ make ...@@ -235,38 +224,29 @@ make
pip install -e . # `-e` for "editable" install, when developing BNB (otherwise leave that out) pip install -e . # `-e` for "editable" install, when developing BNB (otherwise leave that out)
``` ```
</hfoption> ## Preview Wheels[[preview-wheels]]
<hfoption id="Intel CPU + GPU">
#### Intel CPU + GPU(XPU) If you would like to use new features even before they are officially released and help us test them, feel free to install the wheel directly from our CI (*the wheel links will remain stable!*):
CPU needs to build CPU C++ codes, while XPU needs to build sycl codes. <hfoptions id="OS">
Run `export bnb_device=xpu` if you are using xpu, run `export bnb_device=cpu` if you are using cpu. <hfoption id="Linux">
```
git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git && cd bitsandbytes/
cmake -DCOMPUTE_BACKEND=$bnb_device -S .
make
pip install -e .
```
```bash
# Note: if you don't want to reinstall our dependencies, append the `--no-deps` flag!
</hfoption> # x86_64 (most users)
<hfoption id="Ascend NPU"> pip install --force-reinstall https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_main/bitsandbytes-1.33.7.preview-py3-none-manylinux_2_24_x86_64.whl
#### Ascend NPU # ARM/aarch64
pip install --force-reinstall https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_main/bitsandbytes-1.33.7.preview-py3-none-manylinux_2_24_aarch64.whl
```
Please refer to [the official Ascend installations instructions](https://www.hiascend.com/document/detail/zh/Pytorch/60RC3/configandinstg/instg/insg_0001.html) for guidance on how to install the necessary `torch_npu` dependency. </hfoption>
<hfoption id="Windows">
```bash ```bash
# Install bitsandbytes from source # Note: if you don't want to reinstall our dependencies, append the `--no-deps` flag!
# Clone bitsandbytes repo, Ascend NPU backend is currently enabled on multi-backend-refactor branch pip install --force-reinstall https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_main/bitsandbytes-1.33.7.preview-py3-none-win_amd64.whl
git clone -b multi-backend-refactor https://github.com/bitsandbytes-foundation/bitsandbytes.git && cd bitsandbytes/
# Compile & install
apt-get install -y build-essential cmake # install build tools dependencies, unless present
cmake -DCOMPUTE_BACKEND=npu -S .
make
pip install -e . # `-e` for "editable" install, when developing BNB (otherwise leave that out)
``` ```
</hfoption> </hfoption>
</hfoptions> </hfoptions>
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment