Skip to content

GitLab

"vscode:/vscode.git/clone" did not exist on "e3d1d746f2f1c741ef4be9f8cf9cc55e8276806c"
Commit 42ffaf62 authored by Karl Ostmo's avatar Karl Ostmo Committed by Vincent QB
Browse files

Adopt native-Python code generation convention (#378) 

Closes #304

See rationale writeup: https://github.com/pytorch/vision/pull/1321#issuecomment-531033978
parent 9801caf6
Show whitespace changes
Inline Side-by-side
Showing with 269 additions and 199 deletions
.circleci/config.yml
View file @ 42ffaf62
......@@ -48,7 +48,7 @@ jobs:
- checkout
- run:
command: |
pip install --user --progress-bar off jinja2
pip install --user --progress-bar off jinja2 pyyaml
python .circleci/regenerate.py
git diff --exit-code || (echo ".circleci/config.yml not in sync with config.yml.in! Run .circleci/regenerate.py to update config"; exit 1)
......@@ -227,352 +227,350 @@ workflows:
build:
jobs:
- circleci_consistency
- binary_linux_wheel:
name: binary_linux_wheel_py2.7
python_version: "2.7"
python_version: '2.7'
- binary_linux_wheel:
name: binary_linux_wheel_py2.7_unicode
python_version: "2.7"
unicode_abi: "1"
python_version: '2.7'
unicode_abi: '1'
- binary_linux_wheel:
name: binary_linux_wheel_py3.5
python_version: "3.5"
python_version: '3.5'
- binary_linux_wheel:
name: binary_linux_wheel_py3.6
python_version: "3.6"
python_version: '3.6'
- binary_linux_wheel:
name: binary_linux_wheel_py3.7
python_version: "3.7"
python_version: '3.7'
- binary_macos_wheel:
name: binary_macos_wheel_py2.7
python_version: "2.7"
python_version: '2.7'
- binary_macos_wheel:
name: binary_macos_wheel_py2.7_unicode
python_version: "2.7"
unicode_abi: "1"
python_version: '2.7'
unicode_abi: '1'
- binary_macos_wheel:
name: binary_macos_wheel_py3.5
python_version: "3.5"
python_version: '3.5'
- binary_macos_wheel:
name: binary_macos_wheel_py3.6
python_version: "3.6"
python_version: '3.6'
- binary_macos_wheel:
name: binary_macos_wheel_py3.7
python_version: "3.7"
python_version: '3.7'
- binary_linux_conda:
name: binary_linux_conda_py2.7
python_version: "2.7"
python_version: '2.7'
- binary_linux_conda:
name: binary_linux_conda_py3.5
python_version: "3.5"
python_version: '3.5'
- binary_linux_conda:
name: binary_linux_conda_py3.6
python_version: "3.6"
python_version: '3.6'
- binary_linux_conda:
name: binary_linux_conda_py3.7
python_version: "3.7"
python_version: '3.7'
- binary_macos_conda:
name: binary_macos_conda_py2.7
python_version: "2.7"
python_version: '2.7'
- binary_macos_conda:
name: binary_macos_conda_py3.5
python_version: "3.5"
python_version: '3.5'
- binary_macos_conda:
name: binary_macos_conda_py3.6
python_version: "3.6"
python_version: '3.6'
- binary_macos_conda:
name: binary_macos_conda_py3.7
python_version: "3.7"
python_version: '3.7'
nightly:
jobs:
- circleci_consistency
- binary_linux_wheel:
name: nightly_binary_linux_wheel_py2.7
python_version: "2.7"
filters:
branches:
only: nightly
name: nightly_binary_linux_wheel_py2.7
python_version: '2.7'
- binary_wheel_upload:
name: nightly_binary_linux_wheel_py2.7_upload
context: org-member
requires:
- nightly_binary_linux_wheel_py2.7
filters:
branches:
only: nightly
name: nightly_binary_linux_wheel_py2.7_upload
requires:
- nightly_binary_linux_wheel_py2.7
- binary_linux_wheel:
name: nightly_binary_linux_wheel_py2.7_unicode
python_version: "2.7"
unicode_abi: "1"
filters:
branches:
only: nightly
name: nightly_binary_linux_wheel_py2.7_unicode
python_version: '2.7'
unicode_abi: '1'
- binary_wheel_upload:
name: nightly_binary_linux_wheel_py2.7_unicode_upload
context: org-member
requires:
- nightly_binary_linux_wheel_py2.7_unicode
filters:
branches:
only: nightly
name: nightly_binary_linux_wheel_py2.7_unicode_upload
requires:
- nightly_binary_linux_wheel_py2.7_unicode
- binary_linux_wheel:
name: nightly_binary_linux_wheel_py3.5
python_version: "3.5"
filters:
branches:
only: nightly
name: nightly_binary_linux_wheel_py3.5
python_version: '3.5'
- binary_wheel_upload:
name: nightly_binary_linux_wheel_py3.5_upload
context: org-member
name: nightly_binary_linux_wheel_py3.5_upload
requires:
- nightly_binary_linux_wheel_py3.5
- smoke_test_linux_pip:
name: nightly_binary_linux_wheel_py3.5_smoke_test_pip
python_version: "3.5"
requires:
- nightly_binary_linux_wheel_py3.5_upload
filters:
branches:
only: nightly
name: nightly_binary_linux_wheel_py3.5_smoke_test_pip
python_version: '3.5'
requires:
- nightly_binary_linux_wheel_py3.5_upload
- binary_linux_wheel:
name: nightly_binary_linux_wheel_py3.6
python_version: "3.6"
filters:
branches:
only: nightly
name: nightly_binary_linux_wheel_py3.6
python_version: '3.6'
- binary_wheel_upload:
name: nightly_binary_linux_wheel_py3.6_upload
context: org-member
name: nightly_binary_linux_wheel_py3.6_upload
requires:
- nightly_binary_linux_wheel_py3.6
- smoke_test_linux_pip:
name: nightly_binary_linux_wheel_py3.6_smoke_test_pip
python_version: "3.6"
requires:
- nightly_binary_linux_wheel_py3.6_upload
filters:
branches:
only: nightly
name: nightly_binary_linux_wheel_py3.6_smoke_test_pip
python_version: '3.6'
requires:
- nightly_binary_linux_wheel_py3.6_upload
- binary_linux_wheel:
name: nightly_binary_linux_wheel_py3.7
python_version: "3.7"
filters:
branches:
only: nightly
name: nightly_binary_linux_wheel_py3.7
python_version: '3.7'
- binary_wheel_upload:
name: nightly_binary_linux_wheel_py3.7_upload
context: org-member
name: nightly_binary_linux_wheel_py3.7_upload
requires:
- nightly_binary_linux_wheel_py3.7
- smoke_test_linux_pip:
filters:
branches:
only: nightly
name: nightly_binary_linux_wheel_py3.7_smoke_test_pip
python_version: "3.7"
python_version: '3.7'
requires:
- nightly_binary_linux_wheel_py3.7_upload
- binary_macos_wheel:
filters:
branches:
only: nightly
- binary_macos_wheel:
name: nightly_binary_macos_wheel_py2.7
python_version: "2.7"
python_version: '2.7'
- binary_wheel_upload:
context: org-member
filters:
branches:
only: nightly
- binary_wheel_upload:
name: nightly_binary_macos_wheel_py2.7_upload
context: org-member
requires:
- nightly_binary_macos_wheel_py2.7
- binary_macos_wheel:
filters:
branches:
only: nightly
- binary_macos_wheel:
name: nightly_binary_macos_wheel_py2.7_unicode
python_version: "2.7"
unicode_abi: "1"
python_version: '2.7'
unicode_abi: '1'
- binary_wheel_upload:
context: org-member
filters:
branches:
only: nightly
- binary_wheel_upload:
name: nightly_binary_macos_wheel_py2.7_unicode_upload
context: org-member
requires:
- nightly_binary_macos_wheel_py2.7_unicode
- binary_macos_wheel:
filters:
branches:
only: nightly
- binary_macos_wheel:
name: nightly_binary_macos_wheel_py3.5
python_version: "3.5"
python_version: '3.5'
- binary_wheel_upload:
context: org-member
filters:
branches:
only: nightly
- binary_wheel_upload:
name: nightly_binary_macos_wheel_py3.5_upload
context: org-member
requires:
- nightly_binary_macos_wheel_py3.5
- binary_macos_wheel:
filters:
branches:
only: nightly
- binary_macos_wheel:
name: nightly_binary_macos_wheel_py3.6
python_version: "3.6"
python_version: '3.6'
- binary_wheel_upload:
context: org-member
filters:
branches:
only: nightly
- binary_wheel_upload:
name: nightly_binary_macos_wheel_py3.6_upload
context: org-member
requires:
- nightly_binary_macos_wheel_py3.6
- binary_macos_wheel:
filters:
branches:
only: nightly
- binary_macos_wheel:
name: nightly_binary_macos_wheel_py3.7
python_version: "3.7"
python_version: '3.7'
- binary_wheel_upload:
context: org-member
filters:
branches:
only: nightly
- binary_wheel_upload:
name: nightly_binary_macos_wheel_py3.7_upload
context: org-member
requires:
- nightly_binary_macos_wheel_py3.7
- binary_linux_conda:
filters:
branches:
only: nightly
- binary_linux_conda:
name: nightly_binary_linux_conda_py2.7
python_version: "2.7"
python_version: '2.7'
- binary_conda_upload:
context: org-member
filters:
branches:
only: nightly
- binary_conda_upload:
name: nightly_binary_linux_conda_py2.7_upload
context: org-member
requires:
- nightly_binary_linux_conda_py2.7
filters:
branches:
only: nightly
- binary_linux_conda:
name: nightly_binary_linux_conda_py3.5
python_version: "3.5"
filters:
branches:
only: nightly
name: nightly_binary_linux_conda_py3.5
python_version: '3.5'
- binary_conda_upload:
name: nightly_binary_linux_conda_py3.5_upload
context: org-member
name: nightly_binary_linux_conda_py3.5_upload
requires:
- nightly_binary_linux_conda_py3.5
- smoke_test_linux_conda:
name: nightly_binary_linux_conda_py3.5_smoke_test_conda
python_version: "3.5"
requires:
- nightly_binary_linux_conda_py3.5_upload
filters:
branches:
only: nightly
name: nightly_binary_linux_conda_py3.5_smoke_test_conda
python_version: '3.5'
requires:
- nightly_binary_linux_conda_py3.5_upload
- binary_linux_conda:
name: nightly_binary_linux_conda_py3.6
python_version: "3.6"
filters:
branches:
only: nightly
name: nightly_binary_linux_conda_py3.6
python_version: '3.6'
- binary_conda_upload:
name: nightly_binary_linux_conda_py3.6_upload
context: org-member
name: nightly_binary_linux_conda_py3.6_upload
requires:
- nightly_binary_linux_conda_py3.6
- smoke_test_linux_conda:
name: nightly_binary_linux_conda_py3.6_smoke_test_conda
python_version: "3.6"
requires:
- nightly_binary_linux_conda_py3.6_upload
filters:
branches:
only: nightly
name: nightly_binary_linux_conda_py3.6_smoke_test_conda
python_version: '3.6'
requires:
- nightly_binary_linux_conda_py3.6_upload
- binary_linux_conda:
name: nightly_binary_linux_conda_py3.7
python_version: "3.7"
filters:
branches:
only: nightly
name: nightly_binary_linux_conda_py3.7
python_version: '3.7'
- binary_conda_upload:
name: nightly_binary_linux_conda_py3.7_upload
context: org-member
name: nightly_binary_linux_conda_py3.7_upload
requires:
- nightly_binary_linux_conda_py3.7
- smoke_test_linux_conda:
filters:
branches:
only: nightly
name: nightly_binary_linux_conda_py3.7_smoke_test_conda
python_version: "3.7"
python_version: '3.7'
requires:
- nightly_binary_linux_conda_py3.7_upload
- binary_macos_conda:
filters:
branches:
only: nightly
- binary_macos_conda:
name: nightly_binary_macos_conda_py2.7
python_version: "2.7"
python_version: '2.7'
- binary_conda_upload:
context: org-member
filters:
branches:
only: nightly
- binary_conda_upload:
name: nightly_binary_macos_conda_py2.7_upload
context: org-member
requires:
- nightly_binary_macos_conda_py2.7
- binary_macos_conda:
filters:
branches:
only: nightly
- binary_macos_conda:
name: nightly_binary_macos_conda_py3.5
python_version: "3.5"
python_version: '3.5'
- binary_conda_upload:
context: org-member
filters:
branches:
only: nightly
- binary_conda_upload:
name: nightly_binary_macos_conda_py3.5_upload
context: org-member
requires:
- nightly_binary_macos_conda_py3.5
- binary_macos_conda:
filters:
branches:
only: nightly
- binary_macos_conda:
name: nightly_binary_macos_conda_py3.6
python_version: "3.6"
python_version: '3.6'
- binary_conda_upload:
context: org-member
filters:
branches:
only: nightly
- binary_conda_upload:
name: nightly_binary_macos_conda_py3.6_upload
context: org-member
requires:
- nightly_binary_macos_conda_py3.6
- binary_macos_conda:
filters:
branches:
only: nightly
- binary_macos_conda:
name: nightly_binary_macos_conda_py3.7
python_version: "3.7"
python_version: '3.7'
- binary_conda_upload:
context: org-member
filters:
branches:
only: nightly
- binary_conda_upload:
name: nightly_binary_macos_conda_py3.7_upload
context: org-member
requires:
- nightly_binary_macos_conda_py3.7
filters:
branches:
only: nightly
docker_build:
triggers:
- schedule:
......
.circleci/config.yml.in
View file @ 42ffaf62
......@@ -48,7 +48,7 @@ jobs:
- checkout
- run:
command: |
pip install --user --progress-bar off jinja2
pip install --user --progress-bar off jinja2 pyyaml
python .circleci/regenerate.py
git diff --exit-code || (echo ".circleci/config.yml not in sync with config.yml.in! Run .circleci/regenerate.py to update config"; exit 1)
......@@ -223,62 +223,6 @@ jobs:
docker tag ${image_name}:${CIRCLE_WORKFLOW_ID} 308535385114.dkr.ecr.us-east-1.amazonaws.com/${image_name}:${CIRCLE_WORKFLOW_ID}
docker push 308535385114.dkr.ecr.us-east-1.amazonaws.com/${image_name}:${CIRCLE_WORKFLOW_ID}
{%- macro workflow(btype, os, python_version, unicode, prefix='', filter_branch=None, upload=False) %}
- binary_{{os}}_{{btype}}:
name: {{prefix}}binary_{{os}}_{{btype}}_py{{python_version}}{{ "_unicode" if unicode }}
python_version: "{{python_version}}"
{%- if unicode %}
unicode_abi: "1"
{%- endif %}
{%- if filter_branch %}
filters:
branches:
only: {{filter_branch}}
{%- endif %}
{%- if upload %}
- binary_{{btype}}_upload:
name: {{prefix}}binary_{{os}}_{{btype}}_py{{python_version}}{{ "_unicode" if unicode }}_upload
context: org-member
requires:
- {{prefix}}binary_{{os}}_{{btype}}_py{{python_version}}{{ "_unicode" if unicode }}
{%- if filter_branch == 'nightly' %}
{%- if os == 'linux' and not python_version.startswith("2.") %}
{%- if btype == 'wheel' %}
- smoke_test_linux_pip:
name: {{prefix}}binary_{{os}}_{{btype}}_py{{python_version}}{{ "_unicode" if unicode }}_smoke_test_pip
python_version: "{{python_version}}"
requires:
- {{prefix}}binary_{{os}}_{{btype}}_py{{python_version}}{{ "_unicode" if unicode }}_upload
{%- else %}
- smoke_test_linux_conda:
name: {{prefix}}binary_{{os}}_{{btype}}_py{{python_version}}{{ "_unicode" if unicode }}_smoke_test_conda
python_version: "{{python_version}}"
requires:
- {{prefix}}binary_{{os}}_{{btype}}_py{{python_version}}{{ "_unicode" if unicode }}_upload
{%- endif %}
{%- endif %}
{%- endif %}
{%- if filter_branch %}
filters:
branches:
only: {{filter_branch}}
{%- endif %}
{%- endif %}
{%- endmacro %}
{%- macro workflows(prefix='', filter_branch=None, upload=False) %}
{%- for btype in ["wheel", "conda"] -%}
{%- for os in ["linux", "macos"] -%}
{%- for python_version in ["2.7", "3.5", "3.6", "3.7"] -%}
{%- for unicode in ([False, True] if btype == "wheel" and python_version == "2.7" else [False]) -%}
{{ workflow(btype, os, python_version, unicode, prefix=prefix, filter_branch=filter_branch, upload=upload) }}
{%- endfor -%}
{%- endfor -%}
{%- endfor -%}
{%- endfor %}
{%- endmacro %}
workflows:
build:
{%- if True %}
......
.circleci/regenerate.py
View file @ 42ffaf62
#!/usr/bin/env python
#!/usr/bin/env python3
"""
This script should use a very simple, functional programming style.
Avoid Jinja macros in favor of native Python functions.
Don't go overboard on code generation; use Python only to generate
content that can't be easily declared statically using CircleCI's YAML API.
Data declarations (e.g. the nested loops for defining the configuration matrix)
should be at the top of the file for easy updating.
See this comment for design rationale:
https://github.com/pytorch/vision/pull/1321#issuecomment-531033978
"""
import jinja2
import yaml
import os.path
d = os.path.dirname(__file__)
env = jinja2.Environment(
def workflows(prefix='', upload=False, filter_branch=None, indentation=6):
w = []
for btype in ["wheel", "conda"]:
for os_type in ["linux", "macos"]:
for python_version in ["2.7", "3.5", "3.6", "3.7"]:
for unicode in ([False, True] if btype == "wheel" and python_version == "2.7" else [False]):
w += workflow_pair(btype, os_type, python_version, unicode, filter_branch, prefix, upload)
return indent(indentation, w)
def workflow_pair(btype, os_type, python_version, unicode, filter_branch, prefix='', upload=False):
w = []
unicode_suffix = "_unicode" if unicode else ""
base_workflow_name = "{prefix}binary_{os_type}_{btype}_py{python_version}{unicode_suffix}".format(
prefix=prefix,
os_type=os_type,
btype=btype,
python_version=python_version,
unicode_suffix=unicode_suffix)
w.append(generate_base_workflow(base_workflow_name, python_version, unicode, filter_branch, os_type, btype))
if upload:
is_py3_linux = os_type == 'linux' and not python_version.startswith("2.")
# XXX This logic is suspect...
upload_job_filter_branch = not is_py3_linux and filter_branch
w.append(generate_upload_workflow(base_workflow_name, upload_job_filter_branch, btype))
if filter_branch == 'nightly' and is_py3_linux:
pydistro = 'pip' if btype == 'wheel' else 'conda'
w.append(generate_smoketest_workflow(pydistro, base_workflow_name, filter_branch, python_version))
return w
def generate_base_workflow(base_workflow_name, python_version, unicode, filter_branch, os_type, btype):
d = {
"name": base_workflow_name,
"python_version": python_version,
}
if unicode:
d["unicode_abi"] = '1'
if filter_branch:
d["filters"] = gen_filter_branch_tree(filter_branch)
return {"binary_{os_type}_{btype}".format(os_type=os_type, btype=btype): d}
def gen_filter_branch_tree(branch_name):
return {"branches": {"only": branch_name}}
def generate_upload_workflow(base_workflow_name, filter_branch, btype):
d = {
"name": "{base_workflow_name}_upload".format(base_workflow_name=base_workflow_name),
"context": "org-member",
"requires": [base_workflow_name],
}
if filter_branch:
d["filters"] = gen_filter_branch_tree(filter_branch)
return {"binary_{btype}_upload".format(btype=btype): d}
def generate_smoketest_workflow(pydistro, base_workflow_name, filter_branch, python_version):
required_build_suffix = "_upload"
required_build_name = base_workflow_name + required_build_suffix
smoke_suffix = "smoke_test_{pydistro}".format(pydistro=pydistro)
d = {
"name": "{base_workflow_name}_{smoke_suffix}".format(
base_workflow_name=base_workflow_name, smoke_suffix=smoke_suffix),
"requires": [required_build_name],
"python_version": python_version,
}
if filter_branch:
d["filters"] = gen_filter_branch_tree(filter_branch)
return {"smoke_test_linux_{pydistro}".format(pydistro=pydistro): d}
def indent(indentation, data_list):
return ("\n" + " " * indentation).join(yaml.dump(data_list).splitlines())
if __name__ == "__main__":
d = os.path.dirname(__file__)
env = jinja2.Environment(
loader=jinja2.FileSystemLoader(d),
lstrip_blocks=True,
autoescape=False,
)
with open(os.path.join(d, 'config.yml'), 'w') as f:
f.write(env.get_template('config.yml.in').render())
)
with open(os.path.join(d, 'config.yml'), 'w') as f:
f.write(env.get_template('config.yml.in').render(workflows=workflows))
.circleci/test/test_sort_yaml.py 0 → 100755
View file @ 42ffaf62
#!/usr/bin/env python3
"""
To compare new version with previous:
./regenerate.sh
meld <(git show HEAD:./config.yml | ./sort-yaml.py) <(cat config.yml | ./sort-yaml.py)
"""
import sys
import yaml
sys.stdout.write(yaml.dump(yaml.load(sys.stdin, Loader=yaml.FullLoader), sort_keys=True))
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment