Skip to content

GitLab

Unverified Commit 719c4d55 authored by Frank Lee's avatar Frank Lee Committed by GitHub
Browse files

[doc] updated readme for CI/CD (#2600)

parent 4d582893
Hide whitespace changes
Inline Side-by-side
Showing with 50 additions and 208 deletions
.github/workflows/README.md
View file @ 719c4d55
...@@ -9,12 +9,12 @@ ...@@ -9,12 +9,12 @@
- [Code Style Check](#code-style-check) - [Code Style Check](#code-style-check)
- [Unit Test](#unit-test) - [Unit Test](#unit-test)
- [Example Test](#example-test) - [Example Test](#example-test)
- [Dispatch Example Test](#dispatch-example-test) - [Example Test on Dispatch](#example-test-on-dispatch)
- [Compatibility Test](#compatibility-test) - [Compatibility Test](#compatibility-test)
- [Compatibility Test](#compatibility-test-1) - [Compatibility Test on Dispatch](#compatibility-test-on-dispatch)
- [Release](#release) - [Release](#release)
- [Release bdist wheel](#release-bdist-wheel)
- [User Friendliness](#user-friendliness) - [User Friendliness](#user-friendliness)
- [Commmunity](#commmunity)
- [Configuration](#configuration) - [Configuration](#configuration)
- [Progress Log](#progress-log) - [Progress Log](#progress-log)
...@@ -30,6 +30,9 @@ In the section below, we will dive into the details of different workflows avail ...@@ -30,6 +30,9 @@ In the section below, we will dive into the details of different workflows avail
Refer to this [documentation](https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow) on how to manually trigger a workflow. Refer to this [documentation](https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow) on how to manually trigger a workflow.
I will provide the details of each workflow below. I will provide the details of each workflow below.
**A PR which changes the `version.txt` is considered as a release PR in the following coontext.**
### Code Style Check ### Code Style Check
| Workflow Name | File name | Description | | Workflow Name | File name | Description |
...@@ -41,35 +44,34 @@ I will provide the details of each workflow below. ...@@ -41,35 +44,34 @@ I will provide the details of each workflow below.
| Workflow Name | File name | Description | | Workflow Name | File name | Description |
| ---------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | ---------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Build` | `build.yml` | This workflow is triggered when the label `Run build and Test` is assigned to a PR. It will run all the unit tests in the repository with 4 GPUs. | | `Build on PR` | `build_on_pr.yml` | This workflow is triggered when the label `Run build and Test` is assigned to a PR. It will run all the unit tests in the repository with 4 GPUs. |
| `Build on 8 GPUs` | `build_gpu_8.yml` | This workflow will run the unit tests everyday with 8 GPUs. | | `Build on Schedule` | `build_on_schedule.yml` | This workflow will run the unit tests everyday with 8 GPUs. The result is sent to Lark. |
| `Report test coverage` | `report_test_coverage.yml` | This PR will put up a comment to report the test coverage results when `Build` is done. | | `Report test coverage` | `report_test_coverage.yml` | This PR will put up a comment to report the test coverage results when `Build` is done. |
### Example Test ### Example Test
| Workflow Name | File name | Description | | Workflow Name | File name | Description |
| -------------------------- | ------------------------------- | --------------------------------------------------------------------------- | | -------------------------- | ------------------------------- | ------------------------------------------------------------------------------ |
| `Test example on PR` | `example_check_on_pr.yml` | The example will be automatically tested if its files are changed in the PR | | `Test example on PR` | `example_check_on_pr.yml` | The example will be automatically tested if its files are changed in the PR |
| `Test example on Schedule` | `example_check_on_schedule.yml` | This workflow will test all examples every Sunday | | `Test example on Schedule` | `example_check_on_schedule.yml` | This workflow will test all examples every Sunday. The result is sent to Lark. |
| `Example Test on Dispatch` | `example_check_on_dispatch.yml` | Manually test a specified example. | | `Example Test on Dispatch` | `example_check_on_dispatch.yml` | Manually test a specified example. |
#### Dispatch Example Test #### Example Test on Dispatch
parameters: This workflow is triggered by manually dispatching the workflow. It has the following input parameters:
- `example_directory`: the example directory to test. Multiple directories are supported and must be separated by comma. For example, language/gpt, images/vit. Simply input language or simply gpt does not work. - `example_directory`: the example directory to test. Multiple directories are supported and must be separated b$$y comma. For example, language/gpt, images/vit. Simply input language or simply gpt does not work.
### Compatibility Test ### Compatibility Test
| Workflow Name | File name | Description | | Workflow Name | File name | Description |
| ---------------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | -------------------------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------- |
| `Compatibility Test` | `auto_compatibility_test.yml` | This workflow will check the compatiblity of Colossal-AI against PyTorch and CUDA specified in `.compatibility` every Sunday. | | `Compatibility Test on PR` | `compatibility_test_on_pr.yml` | Check Colossal-AI's compatiblity when `version.txt` is changed in a PR. |
| `Auto Compatibility Test` | `auto_compatibility_test.yml` | Check Colossal-AI's compatiblity when `version.txt` is changed in a PR. | | `Compatibility Test on Schedule` | `compatibility_test_on_schedule.yml` | This workflow will check the compatiblity of Colossal-AI against PyTorch specified in `.compatibility` every Sunday. |
| `Dispatch Compatiblity Test` | `dispatch_compatiblity_test.yml` | Test PyTorch and Python Compatibility. | | `Compatiblity Test on Dispatch` | `compatibility_test_on_dispatch.yml` | Test PyTorch Compatibility manually. |
#### Compatibility Test
Parameters: #### Compatibility Test on Dispatch
This workflow is triggered by manually dispatching the workflow. It has the following input parameters:
- `torch version`:torch version to test against, multiple versions are supported but must be separated by comma. The default is value is all, which will test all available torch versions listed in this [repository](https://github.com/hpcaitech/public_assets/tree/main/colossalai/torch_build/torch_wheels). - `torch version`:torch version to test against, multiple versions are supported but must be separated by comma. The default is value is all, which will test all available torch versions listed in this [repository](https://github.com/hpcaitech/public_assets/tree/main/colossalai/torch_build/torch_wheels).
- `cuda version`: cuda versions to test against, multiple versions are supported but must be separated by comma. The CUDA versions must be present in our [DockerHub repository](https://hub.docker.com/r/hpcaitech/cuda-conda). - `cuda version`: cuda versions to test against, multiple versions are supported but must be separated by comma. The CUDA versions must be present in our [DockerHub repository](https://hub.docker.com/r/hpcaitech/cuda-conda).
...@@ -78,24 +80,16 @@ Parameters: ...@@ -78,24 +80,16 @@ Parameters:
### Release ### Release
| Workflow Name | File name | Description | | Workflow Name | File name | Description |
| --------------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `Draft GitHub Release Post` | `draft_github_release_post.yml` | Compose a GitHub release post draft based on the commit history. Triggered when the change of `version.txt` is merged. | | `Draft GitHub Release Post` | `draft_github_release_post_after_merge.yml` | Compose a GitHub release post draft based on the commit history when a release PR is merged. |
| `Release to PyPI` | `release_pypi.yml` | Build and release the wheel to PyPI. Triggered when the change of `version.txt` is merged. | | `Publish to PyPI` | `release_pypi_after_merge.yml` | Build and release the wheel to PyPI when a release PR is merged. The result is sent to Lark. |
| `Release Nightly to PyPI` | `release_nightly.yml` | Build and release the nightly wheel to PyPI as `colossalai-nightly`. Automatically executed every Sunday. | | `Publish Nightly Version to PyPI` | `release_nightly_on_schedule.yml` | Build and release the nightly wheel to PyPI as `colossalai-nightly` every Sunday. The result is sent to Lark. |
| `Release Docker` | `release_docker.yml` | Build and release the Docker image to DockerHub. Triggered when the change of `version.txt` is merged. | | `Publish Docker Image to DockerHub after Merge` | `release_docker_after_merge.yml` | Build and release the Docker image to DockerHub when a release PR is merged. The result is sent to Lark. |
| `Release bdist wheel` | `release_bdist.yml` | Build binary wheels with pre-built PyTorch extensions. Manually dispatched. See more details in the next section. | | `Check CUDA Extension Build Before Merge` | `cuda_ext_check_before_merge.yml` | Build CUDA extensions with different CUDA versions when a release PR is created. |
| `Auto Release bdist wheel` | `auto_release_bdist.yml` | Build binary wheels with pre-built PyTorch extensions.Triggered when the change of `version.txt` is merged. Build specificatons are stored in `.bdist.json` | | `Publish to Test-PyPI Before Merge` | `release_test_pypi_before_merge.yml` | Release to test-pypi to simulate user installation when a release PR is created. |
| `Release bdist wheel` | `release_bdist.yml` | Build binary wheels with pre-built PyTorch extensions. |
#### Release bdist wheel
Parameters:
- `torch version`:torch version to test against, multiple versions are supported but must be separated by comma. The default is value is all, which will test all available torch versions listed in this [repository](https://github.com/hpcaitech/public_assets/tree/main/colossalai/torch_build/torch_wheels) which is regularly updated.
- `cuda version`: cuda versions to test against, multiple versions are supported but must be separated by comma. The CUDA versions must be present in our [DockerHub repository](https://hub.docker.com/r/hpcaitech/cuda-conda).
- `ref`: input the branch or tag name to build the wheel for this ref.
### User Friendliness ### User Friendliness
| Workflow Name | File name | Description | | Workflow Name | File name | Description |
...@@ -104,6 +98,11 @@ Parameters: ...@@ -104,6 +98,11 @@ Parameters:
| `Synchronize submodule` | `submodule.yml` | This workflow will check if any git submodule is updated. If so, it will create a PR to update the submodule pointers. | | `Synchronize submodule` | `submodule.yml` | This workflow will check if any git submodule is updated. If so, it will create a PR to update the submodule pointers. |
| `Close inactive issues` | `close_inactive.yml` | This workflow will close issues which are stale for 14 days. | | `Close inactive issues` | `close_inactive.yml` | This workflow will close issues which are stale for 14 days. |
### Commmunity
| Workflow Name | File name | Description |
| -------------------------------------------- | -------------------------------- | -------------------------------------------------------------------------------- |
| `Generate Community Report and Send to Lark` | `report_leaderboard_to_lark.yml` | Collect contribution and user engagement stats and share with Lark every Friday. |
## Configuration ## Configuration
...@@ -113,15 +112,15 @@ This section lists the files used to configure the workflow. ...@@ -113,15 +112,15 @@ This section lists the files used to configure the workflow.
This `.compatibility` file is to tell GitHub Actions which PyTorch and CUDA versions to test against. Each line in the file is in the format `${torch-version}-${cuda-version}`, which is a tag for Docker image. Thus, this tag must be present in the [docker registry](https://hub.docker.com/r/pytorch/conda-cuda) so as to perform the test. This `.compatibility` file is to tell GitHub Actions which PyTorch and CUDA versions to test against. Each line in the file is in the format `${torch-version}-${cuda-version}`, which is a tag for Docker image. Thus, this tag must be present in the [docker registry](https://hub.docker.com/r/pytorch/conda-cuda) so as to perform the test.
2. `.bdist.json` 2. `.cuda_ext.json`
This file controls what pytorch/cuda compatible pre-built releases will be built and published. You can add a new entry according to the json schema below if there is a new wheel that needs to be built with AOT compilation of PyTorch extensions. This file controls which CUDA versions will be checked against CUDA extenson built. You can add a new entry according to the json schema below to check the AOT build of PyTorch extensions before release.
```json ```json
{ {
"build": [ "build": [
{ {
"torch_version": "", "torch_command": "",
"cuda_image": "" "cuda_image": ""
}, },
] ]
...@@ -130,26 +129,30 @@ This file controls what pytorch/cuda compatible pre-built releases will be built ...@@ -130,26 +129,30 @@ This file controls what pytorch/cuda compatible pre-built releases will be built
## Progress Log ## Progress Log
- [x] Code style check
- [x] pre-commit check
- [x] pre-commit failure report
- [x] unit testing - [x] unit testing
- [x] test on PR - [x] test on PR
- [x] report test coverage - [x] report test coverage
- [x] regular test - [x] regular test
- [x] release - [x] release
- [x] official release - [x] pypi release
- [x] test-pypi simulation
- [x] nightly build - [x] nightly build
- [x] binary build
- [x] docker build - [x] docker build
- [x] draft release post - [x] draft release post
- [x] pre-commit
- [x] check on PR
- [x] report failure
- [x] example check - [x] example check
- [x] check on PR - [x] check on PR
- [x] regular check - [x] regular check
- [x] manual dispatch - [x] manual dispatch
- [x] compatiblity check - [x] compatiblity check
- [x] check on PR
- [x] manual dispatch - [x] manual dispatch
- [x] auto test when release - [x] auto test when release
- [x] community
- [x] contribution report
- [x] user engagement report
- [x] helpers - [x] helpers
- [x] comment translation - [x] comment translation
- [x] submodule update - [x] submodule update
......
.github/workflows/example_check_on_pr.yml
View file @ 719c4d55
...@@ -48,7 +48,7 @@ jobs: ...@@ -48,7 +48,7 @@ jobs:
res=`python .github/workflows/scripts/example_checks/detect_changed_example.py --fileNameList $changedFileName` res=`python .github/workflows/scripts/example_checks/detect_changed_example.py --fileNameList $changedFileName`
echo "All changed examples are $res" echo "All changed examples are $res"
if [ "$res" = "[]" ]; then if [ "$res" == "[]" ]; then
echo "anyChanged=false" >> $GITHUB_OUTPUT echo "anyChanged=false" >> $GITHUB_OUTPUT
echo "matrix=null" >> $GITHUB_OUTPUT echo "matrix=null" >> $GITHUB_OUTPUT
else else
......
.github/workflows/report_test_coverage.yml
View file @ 719c4d55
...@@ -42,7 +42,7 @@ jobs: ...@@ -42,7 +42,7 @@ jobs:
fi fi
- name: Make Coverage Report Collapsable - name: Make Coverage Report Collapsable
if: steps.unzip.outputs.hasReport == "true" if: steps.unzip.outputs.hasReport == 'true'
run: | run: |
covNum=$(cat cov_number) covNum=$(cat cov_number)
title="The code coverage for the changed files is ${covNum}%." title="The code coverage for the changed files is ${covNum}%."
......
.github/workflows/scripts/build_colossalai_wheel.py deleted 100644 → 0
View file @ 4d582893
import argparse
import os
import subprocess
from filecmp import cmp
from functools import cmp_to_key
import requests
from bs4 import BeautifulSoup
from packaging import version
WHEEL_TEXT_ROOT_URL = 'https://github.com/hpcaitech/public_assets/tree/main/colossalai/torch_build/torch_wheels'
RAW_TEXT_FILE_PREFIX = 'https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/torch_build/torch_wheels'
CUDA_HOME = os.environ['CUDA_HOME']
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument('--torch_version', type=str)
parser.add_argument(
'--nightly',
action='store_true',
help=
'whether this build is for nightly release, if True, will only build on the latest PyTorch version and Python 3.8'
)
return parser.parse_args()
def get_cuda_bare_metal_version():
raw_output = subprocess.check_output([CUDA_HOME + "/bin/nvcc", "-V"], universal_newlines=True)
output = raw_output.split()
release_idx = output.index("release") + 1
release = output[release_idx].split(".")
bare_metal_major = release[0]
bare_metal_minor = release[1][0]
return bare_metal_major, bare_metal_minor
def all_wheel_info():
page_text = requests.get(WHEEL_TEXT_ROOT_URL).text
soup = BeautifulSoup(page_text)
all_a_links = soup.find_all('a')
wheel_info = dict()
for a_link in all_a_links:
if 'cuda' in a_link.text and '.txt' in a_link.text:
filename = a_link.text
torch_version, cuda_version = filename.rstrip('.txt').split('-')
cuda_version = cuda_version.lstrip('cuda')
if torch_version not in wheel_info:
wheel_info[torch_version] = dict()
wheel_info[torch_version][cuda_version] = dict()
file_text = requests.get(f'{RAW_TEXT_FILE_PREFIX}/{filename}').text
lines = file_text.strip().split('\n')
for line in lines:
parts = line.split('\t')
method, url, python_version = parts[:3]
if len(parts) > 3:
flags = parts[3]
flags = ' '.join(flags.split('+'))
else:
flags = ''
wheel_info[torch_version][cuda_version][python_version] = dict(method=method, url=url, flags=flags)
return wheel_info
def build_colossalai(wheel_info):
cuda_version_major, cuda_version_minor = get_cuda_bare_metal_version()
cuda_version_on_host = f'{cuda_version_major}.{cuda_version_minor}'
for torch_version, cuda_versioned_wheel_info in wheel_info.items():
for cuda_version, python_versioned_wheel_info in cuda_versioned_wheel_info.items():
if cuda_version_on_host == cuda_version:
for python_version, wheel_info in python_versioned_wheel_info.items():
url = wheel_info['url']
method = wheel_info['method']
flags = wheel_info['flags']
filename = url.split('/')[-1].replace('%2B', '+')
cmd = f'bash ./build_colossalai_wheel.sh {method} {url} {filename} {cuda_version} {python_version} {torch_version} {flags}'
os.system(cmd)
def main():
args = parse_args()
wheel_info = all_wheel_info()
# filter wheels on condition
all_torch_versions = list(wheel_info.keys())
def _compare_version(a, b):
if version.parse(a) > version.parse(b):
return 1
else:
return -1
all_torch_versions.sort(key=cmp_to_key(_compare_version))
if args.nightly:
# only keep the latest version
for key in all_torch_versions[:-1]:
wheel_info.pop(key)
elif args.torch_version != 'all':
torch_versions = args.torch_version.split(',')
# only keep the torch versions specified
for key in all_torch_versions:
if key not in torch_versions:
wheel_info.pop(key)
build_colossalai(wheel_info)
if __name__ == '__main__':
main()
.github/workflows/scripts/build_colossalai_wheel.sh deleted 100644 → 0
View file @ 4d582893
#!/usr/bin/env bash
method=${1}
url=${2}
filename=${3}
cuda_version=${4}
python_version=${5}
torch_version=${6}
flags=${@:7}
git reset --hard HEAD
mkdir -p ./all_dist
source activate base
conda create -n $python_version -y python=$python_version
source activate $python_version
if [ $1 == "pip" ]
then
wget -nc -q -O ./pip_wheels/$filename $url
pip install ./pip_wheels/$filename
elif [ $1 == 'conda' ]
then
conda install pytorch==$torch_version cudatoolkit=$cuda_version $flags
else
echo Invalid installation method
exit
fi
if [ $cuda_version == "10.2" ]
then
cp -r cub-1.8.0/cub/ colossalai/kernel/cuda_native/csrc/kernels/include/
fi
python setup.py bdist_wheel
mv ./dist/* ./all_dist
# must remove build to enable compilation for
# cuda extension in the next build
rm -rf ./build
python setup.py clean
conda deactivate
conda env remove -n $python_version
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment