Skip to content

GitLab

Commit d9aabca2 authored by sangwz's avatar sangwz
Browse files

update README and version

parent 50b06d25
Pipeline #1037 failed with stages
in 0 seconds
Show whitespace changes
Inline Side-by-side
Showing with 177 additions and 59 deletions
README.md
View file @ d9aabca2
![FairScale Logo](./docs/source/_static/img/fairscale-logo.png)
[![Support Ukraine](https://img.shields.io/badge/Support-Ukraine-FFD500?style=flat&labelColor=005BBB)](https://opensource.facebook.com/support-ukraine)
![PyPI](https://img.shields.io/pypi/v/fairscale)
[![Documentation Status](https://readthedocs.org/projects/fairscale/badge/?version=latest)](https://fairscale.readthedocs.io/en/latest/?badge=latest)
[![CircleCI](https://circleci.com/gh/facebookresearch/fairscale.svg?style=shield)](https://app.circleci.com/pipelines/github/facebookresearch/fairscale/) ![PyPI - License](https://img.shields.io/pypi/l/fairscale) [![Downloads](https://pepy.tech/badge/fairscale)](https://pepy.tech/project/fairscale) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/facebookresearch/fairscale/blob/main/CONTRIBUTING.md)
--------------------------------------------------------------------------------
## Description
FairScale is a PyTorch extension library for high performance and large scale training.
This library extends basic PyTorch capabilities while adding new SOTA scaling techniques.
FairScale makes available the latest distributed training techniques in the form of composable
modules and easy to use APIs. These APIs are a fundamental part of a researcher's toolbox as
they attempt to scale models with limited resources.
# fairscale
FairScale was designed with the following values in mind:
`fairscale`是一个用于高性能和大规模训练的`pytorch`扩展库。 此库扩展了基本的`pytorch`功能，同时添加了新的SOTA扩展技术。 `fairscale`以可组合模块和易于使用的API的形式提供了最新的分布式训练技术。 参考官方文档[README_ORIGIN.md](README_ORIGIN.md)
* **Usability** - Users should be able to understand and use FairScale APIs with minimum cognitive overload.
* **Modularity** - Users should be able to combine multiple FairScale APIs as part of their training loop seamlessly.
* **Performance** - FairScale APIs provide the best performance in terms of scaling and efficiency.
# 安装
组件支持：
## Watch Introductory Video
* dtk-23.10
* pytorch-2.1
[![Explain Like I’m 5: FairScale](https://img.youtube.com/vi/oDt7ebOwWIc/0.jpg)](https://www.youtube.com/watch?v=oDt7ebOwWIc)
## pip安装
## Installation
To install FairScale, please see the following [instructions](https://github.com/facebookresearch/fairscale/blob/main/docs/source/installation_instructions.rst).
You should be able to install a package with pip or conda, or build directly from source.
## Getting Started
The full [documentation](https://fairscale.readthedocs.io/) contains instructions for getting started, deep dives and tutorials about the various FairScale APIs.
## FSDP
FullyShardedDataParallel (FSDP) is the recommended method for scaling to large NN models.
This library has been [upstreamed to PyTorch](https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/).
The version of FSDP here is for historical references as well as for experimenting with
new and crazy ideas in research of scaling techniques. Please see the following blog
for [how to use FairScale FSDP and how does it work](https://engineering.fb.com/2021/07/15/open-source/fsdp/).
## Testing
在[光合开发者社区](https://developer.hpccube.com/tool/#sdk) AI 生态包中获取对应的fairscale安装包。
```
pip install fairscale*
```
We use circleci to test FairScale with the following PyTorch versions (with CUDA 11.2):
* the latest stable release (e.g. 1.10.0)
* the latest LTS release (e.g. 1.8.1)
* a recent nightly release (e.g. 1.11.0.dev20211101+cu111)
## 源码安装
```shell
git clone https://github.com/facebookresearch/fairscale.git # 根据需要切换分支
cd fairscale
# 支持GPU需要添加环境变量
export BUILD_CUDA_EXTENSIONS=1
export HIPCC_COMPILE_FLAGS_APPEND="--gpu-max-threads-per-block=1024"
pip install -r requirements.txt
# -e signified dev mode since e stands for editable
pip install -e .
```
Please create an [issue](https://github.com/facebookresearch/fairscale/issues) if you are having trouble with installation.
如果安装失败，尝试在 `pip install` 命令后，添加 `--no-build-isolation` 选项。
## Contributors
## 其它
We welcome contributions! Please see the [CONTRIBUTING](CONTRIBUTING.md) instructions for how you can contribute to FairScale.
torch安装之后，如有库缺失导致的错误，参考以下库的安装
## License
* 安装 `intel-mkl`库
FairScale is licensed under the [BSD-3-Clause License](LICENSE).
```shell
yum-config-manager --add-repo https://yum.repos.intel.com/mkl/setup/intel-mkl.repo
yum install intel-mkl-2020.0-088 -y --nogpgchec
```
fairscale.nn.pipe is forked from [torchgpipe](https://github.com/kakaobrain/torchgpipe), Copyright 2019, Kakao Brain, licensed under [Apache License](http://www.apache.org/licenses/LICENSE-2.0).
并将库路径添加到环境变量：
fairscale.nn.model_parallel is forked from [Megatron-LM](https://github.com/NVIDIA/Megatron-LM), Copyright 2020, NVIDIA CORPORATION, licensed under [Apache License](http://www.apache.org/licenses/LICENSE-2.0).
```shell
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/mkl/lib/intel64
```
fairscale.optim.adascale is forked from [AdaptDL](https://github.com/petuum/adaptdl), Copyright 2020, Petuum, Inc., licensed under [Apache License](http://www.apache.org/licenses/LICENSE-2.0).
fairscale.nn.misc.flatten_params_wrapper is forked from [PyTorch-Reparam-Module](https://github.com/SsnL/PyTorch-Reparam-Module), Copyright 2018, Tongzhou Wang, licensed under [MIT License](https://github.com/SsnL/PyTorch-Reparam-Module/blob/master/LICENSE).
* 安装 `magma`
## Citing FairScale
```shell
# 默认dtk安装路径为 /opt/dtk
cd /opt/dtk
wget http://10.6.10.68:8000/debug/pytorch/third_party/magma_v2.7.2-hip_nfs3.2_DTK23.10_intel-2020.1.217_07Oct2023.tar.gz
tar -zxf magma_v2.7.2-hip_nfs3.2_DTK23.10_intel-2020.1.217_07Oct2023.tar.gz
mv magma_v2.7.2-hip_nfs3.2_DTK23.10_intel-2020.1.217_07Oct2023 magma
cd magma/lib/
# 添加环境变量
export LD_LIBRARY_PATH=${ROCM_PATH}/magma/lib:$LD_LIBRARY_PATH
```
If you use FairScale in your publication, please cite it by using the following BibTeX entry.
# 验证
查询软件版本号，与官方版本同步。
```
python -c "import fairscale; print(fairscale.__version__)"
```
```BibTeX
@Misc{FairScale2021,
author = {FairScale authors},
title = {FairScale: A general purpose modular PyTorch library for high performance and large scale training},
howpublished = {\url{https://github.com/facebookresearch/fairscale}},
year = {2021}
}
```
README_ORIGIN.md 0 → 100644
View file @ d9aabca2
![FairScale Logo](./docs/source/_static/img/fairscale-logo.png)
[![Support Ukraine](https://img.shields.io/badge/Support-Ukraine-FFD500?style=flat&labelColor=005BBB)](https://opensource.facebook.com/support-ukraine)
![PyPI](https://img.shields.io/pypi/v/fairscale)
[![Documentation Status](https://readthedocs.org/projects/fairscale/badge/?version=latest)](https://fairscale.readthedocs.io/en/latest/?badge=latest)
[![CircleCI](https://circleci.com/gh/facebookresearch/fairscale.svg?style=shield)](https://app.circleci.com/pipelines/github/facebookresearch/fairscale/) ![PyPI - License](https://img.shields.io/pypi/l/fairscale) [![Downloads](https://pepy.tech/badge/fairscale)](https://pepy.tech/project/fairscale) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/facebookresearch/fairscale/blob/main/CONTRIBUTING.md)
--------------------------------------------------------------------------------
## Description
FairScale is a PyTorch extension library for high performance and large scale training.
This library extends basic PyTorch capabilities while adding new SOTA scaling techniques.
FairScale makes available the latest distributed training techniques in the form of composable
modules and easy to use APIs. These APIs are a fundamental part of a researcher's toolbox as
they attempt to scale models with limited resources.
FairScale was designed with the following values in mind:
* **Usability** - Users should be able to understand and use FairScale APIs with minimum cognitive overload.
* **Modularity** - Users should be able to combine multiple FairScale APIs as part of their training loop seamlessly.
* **Performance** - FairScale APIs provide the best performance in terms of scaling and efficiency.
## Watch Introductory Video
[![Explain Like I’m 5: FairScale](https://img.youtube.com/vi/oDt7ebOwWIc/0.jpg)](https://www.youtube.com/watch?v=oDt7ebOwWIc)
## Installation
To install FairScale, please see the following [instructions](https://github.com/facebookresearch/fairscale/blob/main/docs/source/installation_instructions.rst).
You should be able to install a package with pip or conda, or build directly from source.
## Getting Started
The full [documentation](https://fairscale.readthedocs.io/) contains instructions for getting started, deep dives and tutorials about the various FairScale APIs.
## FSDP
FullyShardedDataParallel (FSDP) is the recommended method for scaling to large NN models.
This library has been [upstreamed to PyTorch](https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/).
The version of FSDP here is for historical references as well as for experimenting with
new and crazy ideas in research of scaling techniques. Please see the following blog
for [how to use FairScale FSDP and how does it work](https://engineering.fb.com/2021/07/15/open-source/fsdp/).
## Testing
We use circleci to test FairScale with the following PyTorch versions (with CUDA 11.2):
* the latest stable release (e.g. 1.10.0)
* the latest LTS release (e.g. 1.8.1)
* a recent nightly release (e.g. 1.11.0.dev20211101+cu111)
Please create an [issue](https://github.com/facebookresearch/fairscale/issues) if you are having trouble with installation.
## Contributors
We welcome contributions! Please see the [CONTRIBUTING](CONTRIBUTING.md) instructions for how you can contribute to FairScale.
## License
FairScale is licensed under the [BSD-3-Clause License](LICENSE).
fairscale.nn.pipe is forked from [torchgpipe](https://github.com/kakaobrain/torchgpipe), Copyright 2019, Kakao Brain, licensed under [Apache License](http://www.apache.org/licenses/LICENSE-2.0).
fairscale.nn.model_parallel is forked from [Megatron-LM](https://github.com/NVIDIA/Megatron-LM), Copyright 2020, NVIDIA CORPORATION, licensed under [Apache License](http://www.apache.org/licenses/LICENSE-2.0).
fairscale.optim.adascale is forked from [AdaptDL](https://github.com/petuum/adaptdl), Copyright 2020, Petuum, Inc., licensed under [Apache License](http://www.apache.org/licenses/LICENSE-2.0).
fairscale.nn.misc.flatten_params_wrapper is forked from [PyTorch-Reparam-Module](https://github.com/SsnL/PyTorch-Reparam-Module), Copyright 2018, Tongzhou Wang, licensed under [MIT License](https://github.com/SsnL/PyTorch-Reparam-Module/blob/master/LICENSE).
## Citing FairScale
If you use FairScale in your publication, please cite it by using the following BibTeX entry.
```BibTeX
@Misc{FairScale2021,
author = {FairScale authors},
title = {FairScale: A general purpose modular PyTorch library for high performance and large scale training},
howpublished = {\url{https://github.com/facebookresearch/fairscale}},
year = {2021}
}
```
fairscale/__init__.py
View file @ d9aabca2
......@@ -16,5 +16,10 @@ from typing import List
from . import nn
from .version import __version_tuple__
try:
from .version import __dcu_version__
except ImportError:
pass
__version__ = ".".join([str(x) for x in __version_tuple__])
__all__: List[str] = []
setup.py
View file @ d9aabca2
......@@ -29,6 +29,44 @@ def find_version(version_file_path) -> str:
return ver_str
raise RuntimeError("Unable to find version tuple.")
def get_abi():
try:
command = "echo '#include <string>' | gcc -x c++ -E -dM - | fgrep _GLIBCXX_USE_CXX11_ABI"
result = subprocess.run(command, shell=True, capture_output=True, text=True)
output = result.stdout.strip()
abi = "abi" + output.split(" ")[-1]
return abi
except Exception:
return 'abiUnknown'
tmpversion = find_version("fairscale/version.py")
dcu_version = tmpversion
dcu_version +='+das1.1'
sha = "Unknown"
cwd = os.path.dirname(os.path.abspath(__file__))
import subprocess
try:
sha = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=cwd).decode("ascii").strip()
except Exception:
pass
if sha != 'Unknown':
dcu_version += '.git' + sha[:7]
dcu_version += "." + get_abi()
if os.getenv("ROCM_PATH"):
rocm_path = os.getenv('ROCM_PATH', "")
rocm_version_path = os.path.join(rocm_path, '.info', "rocm_version")
with open(rocm_version_path, 'r',encoding='utf-8') as file:
lines = file.readlines()
rocm_version=lines[0][:-2].replace(".", "")
dcu_version += ".dtk" + rocm_version
# torch version
import torch
dcu_version += ".torch" + torch.__version__[:]
def write_version_file():
cwd = os.path.dirname(os.path.abspath(__file__))
version_path = os.path.join(cwd, "fairscale", "version.py")
with open(version_path, "a") as f:
f.write(f"\n__dcu_version__ = '{dcu_version}'")
extensions = []
cmdclass = {}
......@@ -54,10 +92,11 @@ if os.getenv("BUILD_CUDA_EXTENSIONS", "0") == "1":
if __name__ == "__main__":
write_version_file()
setuptools.setup(
name="fairscale",
description="FairScale: A PyTorch library for large-scale and high-performance training.",
version=find_version("fairscale/version.py"),
version=dcu_version,
setup_requires=["ninja"], # ninja is required to build extensions
install_requires=fetch_requirements(),
include_package_data=True,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment