Migrate CI to GH Actions (#2964)

* Add draft GHA-based CI for Linux

* Fix events

* Dup key

* Fix path

* Update env vars

* More fixes

* Fix env vars again

* add doxygen

* add sudo

* Typo

* upgrade CUDA

* escape newlines

* more backslashes

* fix CUDA_APT

* more env vars fixes

* fix missing file

* acumulate env_vars

* build python wrappers only if requested

* add pytest

* use $GITHUB_ENV

https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-commands-for-github-actions#environment-files

* use ${GITHUB_WORKSPACE}

* Pin older gromacs

* add windows

* Provide default values for unbound vars

* fixes

* this is powershell not cmd

* update envs

* win fix

* make windows use cmd

* Change install prefix

* fixes

* better env files

* Increase timeout threshold

* set xcode to 10.x

* specify action version

* fix sdk in macos

* split jobs a bit more

* build wrappers even if tests failed for core

* revert & resort build/test

* fix run logic?

* install envs in D:\

* add more cudas

* allow longer test times

* sudo that

* fix cuda ver checks

* another little fix

* one more

* missing package

* missing dev packages

* missing backslash

* add nvprof

* factor scripts out

* export CUDA_PATH

* no quotes in env var

* add more cudas

* fix ci name

* fix flags

* typo

* missing parenthesis

* add cuda 11.2 urls

* add retry loops for online installations

* add library existence tests

* verbose

* fix sets

* CPU and PME are not built in GPU variants; do not test for those

* quote?

* fix windows checks

* add macos opencl

* disable opencl tests on macos (but build anyway)

* add docs

* cd into build for docs!

* install then cd

* pin sphinxcontrib-bibtex

* we need tex in the system

* split docs into a separate job

* simplify retrying

* simplify retrying 2

* do install

* fix tlmgr installation

* more tex packages

* one more

* one more

* add textcomp to docs

* usepackage[utf8]{inputenc}

* switch to xelatex?

* add xetex

* more fonts

* do not use xindy

* tables can't contain blocks and use tabularcolumns at the same time

https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html?highlight=tabular#directive-tabularcolumns

* build libs in docs too

* update docs deps; bring in pdfs

* kill server after 404 checks

* chage 404 checker

* explicit locations needed

* cumulative exit codes

* override set -e

* update README badges

* add ppc / arm

* missing vbar

* not it

* one too many extensions

* do not test gromacs

* manage workspace

* source conda.sh

* add python version to matrix vars

* source first thing

* disable unbound checks

* add compilers

* some more cuda stuff

* add CPU only ppc64le

* link through stubs

* One more way to obtain HOST_USER_ID?

* let cmake find cf's opencl

* Try with GCC7

* add timeouts for docker based runs

* add several attempts for stochastic failures

* add tests with conda forge compilers

* do not parallelize pytest in docker runs

* exclude some known slow tests

* enable ccache

* forgot conda shell

* No need for macos sdk retries

* \  -> ^

* fix timestampt windows

* export env vars for current step too

* unneeded quotes windows?

* disable compression on windows?

* add ccache in docker too

* group commands

* fix syntax error

* fix version spec

* Increase timeout in Docker runs

* heh, it's double colon

* fix cache timestamp in windows

* escape % with %%

* ccache in docs too

* don't use wrapper package in windows; call vcvarsall directly

* More docker variants

* Handle ccache env vars in yaml

* Re-enable parallel pytest in Docker runs

* Delete unwanted azure ymls that I didn't end up using

* add some comments

* Update badge URL

* is that path messing with windows ccache?

* add CI-README

* Reduce CI matrix

* increase timeouts

* Add GCC7 on PPC QEmu again (temporarily, just for debugging)

* add docker instructions for local debugging

* fix start_docker_locally

* skip some url checks in docs

* Use new package name

* update link

README.md

-[![Build Status](https://travis-ci.org/openmm/openmm.svg?branch=master)](https://travis-ci.org/openmm/openmm?branch=master)
+[![GH Actions Status](https://github.com/openmm/openmm/workflows/CI/badge.svg)](https://github.com/openmm/openmm/actions?query=branch%3Amaster+workflow%3ACI)
+[![Conda](https://img.shields.io/conda/v/conda-forge/openmm.svg)](https://anaconda.org/conda-forge/openmm)
 [![Anaconda Cloud Badge](https://anaconda.org/conda-forge/openmm/badges/downloads.svg)](https://anaconda.org/conda-forge/openmm)
 
 ## OpenMM: A High Performance Molecular Dynamics Library

devtools/CI-README.md

+<!-- Authored by Jaime Rodríguez-Guerra, Chodera Lab. December 2020 -->
+
+# Our Continuous Integration setup
+
+OpenMM can be described as a C++ library with wrappers available in different programming languages (Python, C, Fortran). The heavy lifting is performed by the backend platforms, which can be based on CPU, CUDA and/or OpenCL (and possibly more in the future). All of this is supported for different operating systems and architectures. As a result, the CI setup can get a bit involved, but this document will try to clarify how it works and what we support.
+
+## Implementation overview
+
+OpenMM's CI runs mainly on GitHub Actions, with one separate Jenkins box running the GPU tests (generously provided by Jason Swails).
+
+The build matrix covers:
+
+- Operating systems and architecture:
+  - Linux x64
+  - MacOS Intel
+  - Windows
+  - Linux ppc64le (PowerPC)
+  - Linux aarch64 (ARM)
+- Python
+  - CPython 3.6, 3.7, 3.8, 3.9
+- CUDA versions
+  - 10.0 and above (Linux x64, Linux ppc64le, Windows)
+- OpenCL implementations
+  - Nvidia (tested along CUDA)
+  - AMD 3.0
+- Sysroots and C++ Compilers
+  - Linux: System's GCC 7 and whatever conda-forge is pinning (GCC 9 as of writing)
+  - MacOS: System's, targetting 10.9 SDK
+  - Windows: VS2019
+
+Before I describe the pipelines, I will clarify some concepts and idiosyncrasies in GitHub Actions
+
+- The configuration file lives on `.github/workflows/CI.yml`. This directory can host more than one YML _workflow_, each describing a set of event that will trigger a run.
+- The workflow specifies a set of triggers (key `on`) and a list of `jobs` to run. We run the `CI` workflow for:
+  - Pushes to `master`
+  - Pull requests targetting `master`
+  - Nightlies
+- Currently, the workflow contains four jobs: `unix`, `windows`, `docker`, `docs`. Each job can be run several times, depending on the configuration of `jobs.*.strategy.matrix`. All those jobs replicas will run in parallel and individually. The [`Actions > Summary`](https://github.com/openmm/openmm/actions/runs/451301350) overview can help visualize this.
+- Within each job, you find `steps`. A step can either run a script on a `shell` or use a GitHub `action` to perform a task.
+  - For example, cloning the repo or setting up Miniconda are both independent GitHub _actions_. You will recognize this because they contain the keyword `uses:`.
+  - Running CMake is a shell step, which uses `run:`.
+  - Note 1: Each step is run a new shell session. Environment variables won't survive across steps, unless you add them to the `$GITHUB_ENV` file: `echo "VARIABLE=VALUE" >> ${GITHUB_ENV}`. You can also use step `outputs` but that's more involved and rarely needed.
+  - Note 2: Due to the design of `conda-incubator/setup-miniconda`, all subsequent steps that rely on a conda environment require us to specify an OS-dependent custom shell. Do remember this if you need to add more steps in the job!
+- Steps can be run or skipped based on conditions expressed inside an `if:` key. This is how we control whether we need to install CUDA or not, for example. Jobs can have `if` check, if needed.
+- Steps can define environment variables in their `env:` key, but they will only be available in that step. A `job` can do it too, and these will be available for all steps.
+
+## Details per operating system
+
+The different implementations are very similar to what we do on Linux x64, so I will explain this one on detail and the rest will only comment on the relevant differences.
+
+### Linux x64
+
+- Part of the `unix` pipeline.
+- Runs on `ubuntu-latest`, as provided by GitHub Actions.
+- Uses `conda-incubator/setup-miniconda` to setup the bundled Miniconda and install a conda environment available that provides the building and testing dependencies (CMake, Swig, the adequate Python version, etc). These environment files are located under `devtools/ci/gh-actions/conda-envs`, per operating system.
+- Depending on the matrix configuration, we also install CUDA and/or AMD's OpenCL. These conditional steps are evaluated using GHA's builtin `if` mechanism. Ideally we would install this within the conda environment, but sometimes they are not available (licensing issues, etc(), so we delegate that to the system packages or vendor installers.
+  - For CUDA, we check whether `cuda-version` is not empty, and pass it to `devtools/ci/gh-actions/scripts/install_cuda.sh` as an environment variable.
+  - For OpenCL, we check whether `OPENCL` is `true` and run `devtools/ci/gh-actions/scripts/install_amd_opencl.sh`. This relies on a installer located in a S3 bucket. This could be refactored to install different OpenCL implementations (ROCm, Intel, etc).
+- Some matrix entries require us to install the conda forge compilers, which are used instead of the system's if present.
+- Now we need to configure and download the CCache contents. The keys are built off the matrix name, and a `YYYYDDMM-HHMMSS` timestamp. A secret `CACHE_VERSION` is also included so one can bump the cache by modifying this secret in the repository settings. The configuration is done through environment variables defined at the beginning of the job (key `jobs.unix.env`).
+- CMake is finally invoked, targetting the conda environment as destination (`CONDA_PREFIX`). Additional flags are passed from the matrix configuration. This is how we enable or disable features per matrix entry.
+- CCache performance is assessed.
+- Then we build the C++ libraries and Python wrappers, but separately. This way we can visually check which part failed more easily. Tests are also run separately for the same reason. Whether Python is built and/or tested is checked through the contents of `CMAKE_FLAGS`.
+
+### MacOS Intel
+
+- Part of the `unix` pipeline.
+- Runs on `macos-latest`.
+- Uses `conda-incubator/setup-miniconda`, pointing to the relevant environment file.
+- Neither CUDA nor OpenCL installation scripts are run. Instead, we download and install the 10.9 SDK using `devtools/ci/gh-actions/scripts/install_macos_sdk.sh`. This is done so we can mimic what Conda Forge does in their feedstocks. Check the scripts comments for more info.
+- Everything else is the same.
+
+### Windows
+
+- Sole member of the `windows` pipeline.
+- Runs on `windows-latest`.
+- Uses `conda-incubator/setup-miniconda`, pointing to the relevant environment file.
+- Installs CUDA with the Nvidia installers using `devtools/ci/gh-actions/scripts/install_cuda.bat`, which requires an environment variable `CUDA_VERSION`, exported from the corresponding matrix entry. Again, this only runs if `matrix.cuda-version` is not empty.
+- Everything else is the same.
+
+### PowerPC & ARM
+
+- Part of the `docker` pipeline.
+- These run on a Docker image on top of `ubuntu-latest`. The Docker image itself depends on the architecture chosen (ppc64le, aarch64) and what CUDA version we want. These are provided by Conda Forge, so they have `conda` preinstalled and ready to go.
+- Since it's a different architecture, we need to configure QEMU first. This is done automatically with a Docker image, mimicking what Conda Forge does.
+- We start the Docker image. The working directory (`$GITHUB_WORKSPACE`) is mounted with read/write permissions on `/home/conda/workspace`, so we can communicate back with the host using files, and also use CCache.
+- The Docker image will run `devtools/ci/gh-actions/scripts/run_steps_inside_docker_image.sh`. This script mostly does what you saw for Linux x64, with some differences:
+  - We don't need to install CUDA or setup Miniconda, because they are preinstalled in the Docker image.
+  - We patch some dependencies from the environment file because they are not available for this architecture. To save one conda environment solve, we also patch the Python version in the environment file.
+  - These images don't come with a system compiler, so we specify one in the matrix configuration:
+    - If `compilers` contains a value that starts with `devtoolset-`, we understand we want a CentOS devtoolse. So far, we specify `devtoolset-7`.
+    - If `compilers` is any other thing, we understand that's a (space-separated series of) Conda packages. Since Conda Forge provides a metapackage named `compilers` that will install all of them for the current platform, we use that one. That's why some entries have a `compilers: compilers` entry.
+  - Everything else runs as usual.
+- Do note that the whole Docker run is a single GitHub Actions step, so it's not as visually appealing. I tried my best to group the commands with the `::group::` syntax so it's easier to follow, but it's not the same.
+- If the script runs successfully, it will create an empty file. We test for existence after the Docker run to make sure.
+
+> Note: Since these use software emulation, they are really slow. Still, they can run successfully within the 6h GHA provides. If GHA upgrades to better CI machines with hardware based virtualization, they might be able to run with close-to-native performance.
+
+### Docs
+
+This is a Linux-x64 pipeline optimized for building the documentation only. It's provided as a separate entry because I didn't want to overcomplicate the `if:` logic in the `unix` pipeline. It's essentially the same, but:
+
+- It uses a different environment file in `setup-miniconda`.
+- It only builds the docs, and their dependencies. No tests, for example.
+- It contains a deployment step, which will copy the contents to the S3 bucket _only_ when run on `master`, ignoring cron jobs. The required secrets must be defined in the repository settings with the following exact key names. Just copy paste the values there. GitHub will encrypt and mask them.
+  - `AWS_S3_BUCKET`
+  - `AWS_ACCESS_KEY_ID`
+  - `AWS_SECRET_ACCESS_KEY`
+- It will also check for dead links using a Node package. This is run _after_ deployment so it won't prevent that, but it will still signal the job as failed if the docs contain broken links.
+
+## Shortcomings
+
+There are some limitations when compared to other CI services, but I guess this list will be shorter over time:
+
+- Cache cannot be invalidated directly. Instead, I included a secret `CACHE_VERSION` that is part of the cache key. If you change the value of this secret, it will functionally prevent access to the previous cache. It also expires every 7 days. Note that since this trick uses a secret, the value of `CACHE_VERSION` will be masked in the log output. As a result, make sure to use something short but meaningless and difficult to find in the wild (e.g. `pqgbhl` instead of `0`).
+- There's no `ci skip` functionality (yet).
+
+## Extra content
+
+### How to debug PowerPC / ARM locally
+
+From the root of the repository, run the following script. There are
+some variables you might want to edit (PPC vs ARM, Python version, etc).
+Take a look to the script first in that case.
+
+```bash
+bash devtools/ci/gh-actions/start_docker_locally.sh
+```
+
+You will be inside the Docker image after a few moments. The repo root has
+been mounted to `/home/conda/workspace`.
+
+Run this other script to reproduce the CI steps exactly. Do NOT `source` scripts,
+since a failure will exit Docker altogether. Always use new `bash` processes
+to avoid starting from scratch.
+
+```bash
+bash /home/conda/workspace/devtools/ci/gh-actions/scripts/run_steps_inside_docker_image.sh
+```

devtools/ci/gh-actions/conda-envs/build-macos-latest.yml

+name: build
+channels:
+- conda-forge
+- bioconda
+dependencies:
+# build
+- cmake
+- ccache
+# host
+- python
+- cython
+- swig
+- fftw
+- numpy
+- doxygen 1.8.14
+# test
+- pytest
+- pytest-xdist
+- pytest-timeout
+- gromacs 2018.*

devtools/ci/gh-actions/conda-envs/build-ubuntu-latest.yml

+name: build
+channels:
+- conda-forge
+- bioconda
+dependencies:
+# build
+- cmake
+- make
+- ccache
+# host
+- python
+- cython
+- swig
+- fftw
+- numpy
+- ocl-icd-system
+- doxygen 1.8.14
+# test
+- pytest
+- pytest-xdist
+- pytest-timeout
+- gromacs 2018.*

devtools/ci/gh-actions/conda-envs/build-windows-latest.yml

+name: build
+channels:
+- conda-forge
+- defaults
+dependencies:
+# build
+- jom
+- cmake
+- ccache
+- m2-coreutils
+# host
+- python
+- cython
+- swig
+- fftw
+- numpy
+- doxygen 1.8.14
+- khronos-opencl-icd-loader
+# test
+- pytest
+- pytest-xdist
+- pytest-timeout

devtools/ci/gh-actions/conda-envs/docs.yml

+name: build
+channels:
+- conda-forge
+dependencies:
+# build
+- cmake
+- ccache
+# host
+- python
+- pip
+- numpy
+- cython
+- swig
+- doxygen 1.8.14
+- pip:
+  - sphinx==2.3.1
+  - sphinxcontrib-bibtex<2.0.0
+  - sphinxcontrib-lunrsearch
+  - sphinxcontrib-autodoc_doxygen
\ No newline at end of file

devtools/ci/gh-actions/scripts/install_amd_opencl.sh

+# This script installs AMD's SDK 3.0 to provide their OpenCL implementation
+# * Installation path will be ${GITHUB_WORKSPACE}/AMDAPPSDK
+
+set -euxo pipefail
+
+
+wget -q --retry-connrefused --waitretry=1 --read-timeout=20 --timeout=15 --tries 5 \
+    http://s3.amazonaws.com/omnia-ci/AMD-APP-SDKInstaller-v3.0.130.135-GA-linux64.tar.bz2
+tar -xjf AMD-APP-SDK*.tar.bz2
+
+AMDAPPSDK=${GITHUB_WORKSPACE}/AMDAPPSDK
+export OPENCL_VENDOR_PATH=${AMDAPPSDK}/etc/OpenCL/vendors
+
+mkdir -p ${OPENCL_VENDOR_PATH}
+sh AMD-APP-SDK*.sh --tar -xf -C ${AMDAPPSDK}
+echo libamdocl64.so > ${OPENCL_VENDOR_PATH}/amdocl64.icd
+
+export LD_LIBRARY_PATH=${AMDAPPSDK}/lib/x86_64:${LD_LIBRARY_PATH:-}
+chmod +x ${AMDAPPSDK}/bin/x86_64/clinfo
+${AMDAPPSDK}/bin/x86_64/clinfo
+sudo apt-get install -y libgl1-mesa-dev
+
+echo "OPENCL_VENDOR_PATH=${OPENCL_VENDOR_PATH}" >> ${GITHUB_ENV}
+echo "LD_LIBRARY_PATH=${LD_LIBRARY_PATH}" >> ${GITHUB_ENV}
\ No newline at end of file

devtools/ci/gh-actions/scripts/install_cuda.bat

+:: This script installs CUDA on Windows.
+:: It downloads the offline installer from the Nvidia servers
+:: and the relevant patches, if applicable.
+:: It uses the default installation path, which is exported as CUDA_PATH
+:: For CMake compatibility, CUDA_TOOLKIT_ROOT_DIR is also exported
+:: It expects a %CUDA_VERSION% environment variable, set to major.minor (e.g. 10.0)
+
+:: We define a default subset of components to be installed for faster installation times
+:: and reduced storage usage (CI is limited to 10GB). Full list of components is available at
+:: https://docs.nvidia.com/cuda/archive/%CUDA_VERSION%/cuda-installation-guide-microsoft-windows/index.html
+set "VAR=nvcc_%CUDA_VERSION% cuobjdump_%CUDA_VERSION% nvprune_%CUDA_VERSION% cupti_%CUDA_VERSION%"
+set "VAR=%VAR% memcheck_%CUDA_VERSION% nvdisasm_%CUDA_VERSION% nvprof_%CUDA_VERSION% cublas_%CUDA_VERSION%"
+set "VAR=%VAR% cublas_dev_%CUDA_VERSION% cudart_%CUDA_VERSION% cufft_%CUDA_VERSION% cufft_dev_%CUDA_VERSION%"
+set "VAR=%VAR% curand_%CUDA_VERSION% curand_dev_%CUDA_VERSION% cusolver_%CUDA_VERSION% cusolver_dev_%CUDA_VERSION%"
+set "VAR=%VAR% cusparse_%CUDA_VERSION% cusparse_dev_%CUDA_VERSION% npp_%CUDA_VERSION% npp_dev_%CUDA_VERSION%"
+set "VAR=%VAR% nvrtc_%CUDA_VERSION% nvrtc_dev_%CUDA_VERSION% nvml_dev_%CUDA_VERSION%"
+set "VAR=%VAR% visual_studio_integration_%CUDA_VERSION%"
+set "CUDA_COMPONENTS=%VAR%"
+
+if "%CUDA_VERSION%" == "9.2"  goto cuda92
+if "%CUDA_VERSION%" == "10.0" goto cuda100
+if "%CUDA_VERSION%" == "10.1" goto cuda101
+if "%CUDA_VERSION%" == "10.2" goto cuda102
+if "%CUDA_VERSION%" == "11.0" goto cuda110
+if "%CUDA_VERSION%" == "11.1" goto cuda111
+if "%CUDA_VERSION%" == "11.2" goto cuda112
+
+echo CUDA '%CUDA_VERSION%' is not supported
+exit /b 1
+
+:: Define URLs per version
+:cuda92
+set "CUDA_NETWORK_INSTALLER_URL=https://developer.nvidia.com/compute/cuda/9.2/Prod2/network_installers2/cuda_9.2.148_win10_network"
+set "CUDA_NETWORK_INSTALLER_CHECKSUM=2bf9ae67016867b68f361bf50d2b9e7b"
+set "CUDA_INSTALLER_URL=https://developer.nvidia.com/compute/cuda/9.2/Prod2/local_installers2/cuda_9.2.148_win10"
+set "CUDA_INSTALLER_CHECKSUM=f6c170a7452098461070dbba3e6e58f1"
+set "CUDA_PATCH_URL=https://developer.nvidia.com/compute/cuda/9.2/Prod2/patches/1/cuda_9.2.148.1_windows"
+set "CUDA_PATCH_CHECKSUM=09e20653f1346d2461a9f8f1a7178ba2"
+set "CUDA_COMPONENTS=%CUDA_COMPONENTS% nvgraph_%CUDA_VERSION% nvgraph_dev_%CUDA_VERSION%"
+goto cuda_common
+
+
+:cuda100
+set "CUDA_NETWORK_INSTALLER_URL=https://developer.nvidia.com/compute/cuda/10.0/Prod/network_installers/cuda_10.0.130_win10_network"
+set "CUDA_NETWORK_INSTALLER_CHECKSUM=3312deac9c939bd78d0e7555606c22fc"
+set "CUDA_INSTALLER_URL=https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_411.31_win10"
+set "CUDA_INSTALLER_CHECKSUM=90fafdfe2167ac25432db95391ca954e"
+set "CUDA_COMPONENTS=%CUDA_COMPONENTS% nvgraph_%CUDA_VERSION% nvgraph_dev_%CUDA_VERSION%"
+goto cuda_common
+
+
+:cuda101
+set "CUDA_NETWORK_INSTALLER_URL=http://developer.download.nvidia.com/compute/cuda/10.1/Prod/network_installers/cuda_10.1.243_win10_network.exe"
+set "CUDA_NETWORK_INSTALLER_CHECKSUM=fae0c958440511576691b825d4599e93"
+set "CUDA_INSTALLER_URL=http://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.243_426.00_win10.exe"
+set "CUDA_INSTALLER_CHECKSUM=b54cf32683f93e787321dcc2e692ff69"
+set "CUDA_COMPONENTS=%CUDA_COMPONENTS% nvgraph_%CUDA_VERSION% nvgraph_dev_%CUDA_VERSION%"
+goto cuda_common
+
+
+:cuda102
+set "CUDA_NETWORK_INSTALLER_URL=http://developer.download.nvidia.com/compute/cuda/10.2/Prod/network_installers/cuda_10.2.89_win10_network.exe"
+set "CUDA_NETWORK_INSTALLER_CHECKSUM=60e0f16845d731b690179606f385041e"
+set "CUDA_INSTALLER_URL=http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_441.22_win10.exe"
+set "CUDA_INSTALLER_CHECKSUM=d9f5b9f24c3d3fc456a3c789f9b43419"
+set "CUDA_PATCH_URL=http://developer.download.nvidia.com/compute/cuda/10.2/Prod/patches/1/cuda_10.2.1_win10.exe"
+set "CUDA_PATCH_CHECKSUM=9d751ae129963deb7202f1d85149c69d"
+set "CUDA_COMPONENTS=%CUDA_COMPONENTS% nvgraph_%CUDA_VERSION% nvgraph_dev_%CUDA_VERSION%"
+goto cuda_common
+
+
+:cuda110
+set "CUDA_NETWORK_INSTALLER_URL=http://developer.download.nvidia.com/compute/cuda/11.0.3/network_installers/cuda_11.0.3_win10_network.exe"
+set "CUDA_NETWORK_INSTALLER_CHECKSUM=1b88bf7bb8e50207bbb53ed2033f93f3"
+set "CUDA_INSTALLER_URL=http://developer.download.nvidia.com/compute/cuda/11.0.3/local_installers/cuda_11.0.3_451.82_win10.exe"
+set "CUDA_INSTALLER_CHECKSUM=80ae0fdbe04759123f3cab81f2aadabd"
+goto cuda_common
+
+
+:cuda111
+set "CUDA_NETWORK_INSTALLER_URL=https://developer.download.nvidia.com/compute/cuda/11.1.1/network_installers/cuda_11.1.1_win10_network.exe"
+set "CUDA_NETWORK_INSTALLER_CHECKSUM=7e36e50ee486a84612adfd85500a9971"
+set "CUDA_INSTALLER_URL=https://developer.download.nvidia.com/compute/cuda/11.1.1/local_installers/cuda_11.1.1_456.81_win10.exe"
+set "CUDA_INSTALLER_CHECKSUM=a89dfad35fc1adf02a848a9c06cfff15"
+goto cuda_common
+
+
+:cuda112
+set "CUDA_NETWORK_INSTALLER_URL=https://developer.download.nvidia.com/compute/cuda/11.2.0/network_installers/cuda_11.2.0_win10_network.exe"
+set "CUDA_NETWORK_INSTALLER_CHECKSUM=ab02a25eed1201cc3e414be943a242df"
+set "CUDA_INSTALLER_URL=https://developer.download.nvidia.com/compute/cuda/11.2.0/local_installers/cuda_11.2.0_460.89_win10.exe"
+set "CUDA_INSTALLER_CHECKSUM=92f38c37ce9c6c11d27c10701b040256"
+goto cuda_common
+
+
+:: The actual installation logic
+:cuda_common
+
+::We expect this CUDA_PATH
+set "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v%CUDA_VERSION%"
+
+echo Downloading CUDA version %CUDA_VERSION% installer from %CUDA_INSTALLER_URL%
+echo Expected MD5: %CUDA_INSTALLER_CHECKSUM%
+
+:: Download installer
+curl --retry 3 -k -L %CUDA_INSTALLER_URL% --output cuda_installer.exe
+if errorlevel 1 (
+    echo Problem downloading installer...
+    exit /b 1
+)
+:: Check md5
+openssl md5 cuda_installer.exe | findstr %CUDA_INSTALLER_CHECKSUM%
+if errorlevel 1 (
+    echo Checksum does not match!
+    exit /b 1
+)
+:: Run installer
+start /wait cuda_installer.exe -s %CUDA_COMPONENTS%
+if errorlevel 1 (
+    echo Problem installing CUDA toolkit...
+    exit /b 1
+)
+del cuda_installer.exe
+
+:: If patches are needed, download and apply
+if not "%CUDA_PATCH_URL%"=="" (
+    echo This version requires an additional patch
+    curl --retry 3 -k -L %CUDA_PATCH_URL% --output cuda_patch.exe
+    if errorlevel 1 (
+        echo Problem downloading patch installer...
+        exit /b 1
+    )
+    openssl md5 cuda_patch.exe | findstr %CUDA_PATCH_CHECKSUM%
+    if errorlevel 1 (
+        echo Checksum does not match!
+        exit /b 1
+    )
+    start /wait cuda_patch.exe -s
+    if errorlevel 1 (
+        echo Problem running patch installer...
+        exit /b 1
+    )
+    del cuda_patch.exe
+)
+
+:: This should exist by now!
+if not exist "%CUDA_PATH%\bin\nvcc.exe" (
+    echo CUDA toolkit installation failed!
+    exit /b 1
+)
+
+echo CUDA_PATH=%CUDA_PATH% >> %GITHUB_ENV%
+echo CUDA_TOOLKIT_ROOT_DIR=%CUDA_PATH:\=/% >> %GITHUB_ENV%
+
+:: Notes about nvcuda.dll
+:: ----------------------
+:: We should also provide the drivers (nvcuda.dll), but the installer will not
+:: proceed without a physical Nvidia card attached (not the case in the CI).
+:: Expanding `<installer.exe>\Display.Driver\nvcuda.64.dl_` to `C:\Windows\System32`
+:: does not work anymore (.dl_ files are not PE-COFF according to Dependencies.exe).
+:: Forcing this results in a DLL error 193. Basically, there's no way to provide
+:: ncvuda.dll in a GPU-less machine without breaking the EULA (aka zipping nvcuda.dll
+:: from a working installation).
\ No newline at end of file

devtools/ci/gh-actions/scripts/install_cuda.sh

+# This script install CUDA on Ubuntu-based systemws
+# It uses the Nvidia repos for Ubuntu 18.04, which as of Dec 2020
+# includes packages for CUDA 10.0, 10.1, 10.2, 11.0, 11.1, 11.2
+# Future versions might require an updated repo (maybe Ubuntu 20)
+# It expects a $CUDA_VERSION environment variable set to major.minor (e.g. 10.0)
+
+set -euxo pipefail
+
+# Enable retrying
+echo 'APT::Acquire::Retries "5";' | sudo tee /etc/apt/apt.conf.d/80-retries
+
+sudo wget --retry-connrefused --waitretry=1 --read-timeout=20 --timeout=15 --tries 5 \
+    -O /etc/apt/preferences.d/cuda-repository-pin-600 \
+    https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
+sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
+sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
+sudo apt-get update -qq
+
+CUDA_APT=${CUDA_VERSION/./-}
+## cufft changed package names in CUDA 11
+if [[ ${CUDA_VERSION} == 10.* ]]; then CUFFT="cuda-cufft"; else CUFFT="libcufft"; fi
+sudo apt-get install -y \
+    libgl1-mesa-dev cuda-compiler-${CUDA_APT} \
+    cuda-drivers cuda-driver-dev-${CUDA_APT} \
+    cuda-cudart-${CUDA_APT} cuda-cudart-dev-${CUDA_APT} \
+    ${CUFFT}-${CUDA_APT} ${CUFFT}-dev-${CUDA_APT} \
+    cuda-nvprof-${CUDA_APT}
+sudo apt-get clean
+
+export CUDA_HOME=/usr/local/cuda-${CUDA_VERSION}
+export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${LD_LIBRARY_PATH:-}
+export PATH=${CUDA_HOME}/bin:${PATH}
+
+echo "CUDA_HOME=${CUDA_HOME}" >> ${GITHUB_ENV}
+echo "LD_LIBRARY_PATH=${LD_LIBRARY_PATH}" >> ${GITHUB_ENV}
+echo "PATH=${PATH}" >> ${GITHUB_ENV}
\ No newline at end of file

devtools/ci/gh-actions/scripts/install_macos_sdk.sh

+# Install an older MacOS SDK
+# This should guarantee OpenMM builds with extended compatibility across MacOS versions
+# Adapted from conda-forge-ci-setup scripts:
+# * https://github.com/conda-forge/conda-forge-ci-setup-feedstock/blob/dde296e/recipe/run_conda_forge_build_setup_osx
+# * https://github.com/conda-forge/conda-forge-ci-setup-feedstock/blob/dde296e/recipe/download_osx_sdk.sh
+#
+# Some possible updates might involve upgrading the download link to future MacOS releases (10.15 to something else),
+# depending on the version provided by the CI
+
+OSX_SDK_DIR="$(xcode-select -p)/Platforms/MacOSX.platform/Developer/SDKs"
+export MACOSX_DEPLOYMENT_TARGET=10.9
+export MACOSX_SDK_VERSION=10.9
+
+export CMAKE_OSX_SYSROOT="${OSX_SDK_DIR}/MacOSX${MACOSX_SDK_VERSION}.sdk"
+
+if [[ ! -d ${CMAKE_OSX_SYSROOT}} ]]; then
+    echo "Downloading ${MACOSX_SDK_VERSION} sdk"
+    curl -L -O --connect-timeout 5 --max-time 10 --retry 5  --retry-delay 0 --retry-max-time 40 --retry-connrefused --retry-all-errors \
+        https://github.com/phracker/MacOSX-SDKs/releases/download/10.15/MacOSX${MACOSX_SDK_VERSION}.sdk.tar.xz
+    tar -xf MacOSX${MACOSX_SDK_VERSION}.sdk.tar.xz -C "$(dirname ${CMAKE_OSX_SYSROOT})"
+fi
+
+if [[ "$MACOSX_DEPLOYMENT_TARGET" == 10.* ]]; then
+# set minimum sdk version to our target
+plutil -replace MinimumSDKVersion -string ${MACOSX_SDK_VERSION} $(xcode-select -p)/Platforms/MacOSX.platform/Info.plist
+plutil -replace DTSDKName -string macosx${MACOSX_SDK_VERSION}internal $(xcode-select -p)/Platforms/MacOSX.platform/Info.plist
+fi
+
+echo "MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET}" >> ${GITHUB_ENV}
+echo "CMAKE_OSX_SYSROOT=${MACOSX_DEPLOYMENT_TARGET}" >> ${GITHUB_ENV}
+echo "CMAKE_OSX_SYSROOT=${CMAKE_OSX_SYSROOT}" >> ${GITHUB_ENV}
\ No newline at end of file

devtools/ci/gh-actions/scripts/run_steps_inside_docker_image.sh

+source /opt/conda/etc/profile.d/conda.sh
+set -eo pipefail
+
+WORKSPACE="$HOME/workspace"
+
+# This endgroup closes the open tag in CI.yml
+echo "::endgroup::"
+
+echo "::group::Prepare build environment..."
+extra_conda_packages=""
+if [[ ${COMPILERS} == devtoolset* ]]; then
+    sudo yum install -y centos-release-scl
+    sudo yum install -y ${COMPILERS}
+    source /opt/rh/${COMPILERS}/enable
+else
+    extra_conda_packages="${COMPILERS}"
+fi
+
+# Patch environment file
+sed -E -e "s/.*gromacs.*//" \
+       -e "s/^- python$/- python ${PYTHON_VER}.*/" \
+       ${WORKSPACE}/devtools/ci/gh-actions/conda-envs/build-ubuntu-latest.yml > conda-env.yml
+for package in $extra_conda_packages; do
+    if [[ -n ${package// } ]]; then
+        echo "- ${package}" >> conda-env.yml
+    fi
+done
+conda env create -n build -f conda-env.yml
+conda activate build || true
+echo "::endgroup::"
+
+echo "::group::Prepare ccache..."
+export CCACHE_BASEDIR=${WORKSPACE}
+export CCACHE_DIR=${WORKSPACE}/.ccache
+export CCACHE_COMPRESS=true
+export CCACHE_COMPRESSLEVEL=6
+export CCACHE_MAXSIZE=400M
+ccache -p
+ccache -z
+echo "::endgroup::"
+
+echo "::group::Configure with CMake..."
+if [[ -d /usr/local/cuda ]]; then
+    export CUDA_PATH="/usr/local/cuda"
+    export CUDA_LIB_PATH="${CUDA_PATH}/lib64/stubs"
+    export LD_LIBRARY_PATH="${CUDA_PATH}/lib64/stubs:${LD_LIBRARY_PATH:-}"
+    export PATH="${CUDA_PATH}/bin:${PATH}"
+fi
+
+rm -rf build || true
+mkdir -p build
+cd build
+cmake ${WORKSPACE} \
+    -DCMAKE_INSTALL_PREFIX=${CONDA_PREFIX} \
+    -DCMAKE_PREFIX_PATH=${CONDA_PREFIX} \
+    -DCMAKE_C_COMPILER_LAUNCHER=ccache \
+    -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
+    -DOPENMM_BUILD_CUDA_TESTS=OFF \
+    -DOPENMM_BUILD_OPENCL_TESTS=OFF
+echo "::endgroup::"
+
+# Build
+echo "::group::Build with make..."
+make -j2 install PythonInstall
+echo "::endgroup::"
+
+echo "::group::Check ccache performance..."
+ccache -s
+echo "::endgroup::"
+
+# Core tests
+
+echo "::group::Run core tests..."
+python ${WORKSPACE}/devtools/run-ctest.py --parallel 2 --timeout 1500 --job-duration 360 --attempts 3
+test -f ${CONDA_PREFIX}/lib/libOpenMM.so
+test -f ${CONDA_PREFIX}/lib/plugins/libOpenMMCPU.so
+test -f ${CONDA_PREFIX}/lib/plugins/libOpenMMPME.so
+if [[ ! -z ${CUDA_VER} ]]; then
+    test -f ${CONDA_PREFIX}/lib/plugins/libOpenMMCUDA.so
+    test -f ${CONDA_PREFIX}/lib/plugins/libOpenMMOpenCL.so
+fi
+echo "::endgroup::"
+
+# Python tests
+echo "::group::Run Python tests..."
+python -m openmm.testInstallation
+python -c "import openmm as mm; print('---Loaded---', *mm.pluginLoadedLibNames, '---Failed---', *mm.Platform.getPluginLoadFailures(), sep='\n')"
+cd python/tests
+# Gromacs is not available on condaforge for PPC/ARM
+# Membrane an MTS Langevin Integrator tests timeout (>6h!), possibly due to the emulation slowdown
+python -m pytest -v -k "not gromacs and not membrane and not MTSLangevinIntegrator" -n 2
+echo "::endgroup::"
+
+echo "We are done!"
+touch "${WORKSPACE}/docker_steps_run_successfully"

devtools/ci/gh-actions/start_docker_locally.sh

+#!/bin/bash
+
+# This is an example script on how to debug locally with Docker!
+# If it does not work, it might be out of date. In that case,
+# check the steps used in /.github/workflows/CI.yml
+
+set -euxo pipefail
+
+# This is the image for PowerPC + CUDA
+export DOCKER_IMAGE="quay.io/condaforge/linux-anvil-ppc64le-cuda:10.2"
+# # Use this other one for ARM debugging
+# export DOCKER_IMAGE="quay.io/condaforge/linux-anvil-aarch64"
+
+# With Conda Forge compilers (GCC9)
+export COMPILERS="compilers"
+# # With RH devtoolset (GCC7)
+# export COMPILERS="devtoolset-7"
+
+# Choose your Python version
+export PYTHON_VER="3.9"
+
+# Number of CPUs to use
+export CPU_COUNT=2
+
+echo "Preparing Docker..."
+docker run --rm --privileged multiarch/qemu-user-static:register --reset --credential yes
+ls /proc/sys/fs/binfmt_misc/
+
+docker info
+
+# In order for the conda-build process in the container to write to the mounted
+# volumes, we need to run with the same id as the host machine, which is
+# normally the owner of the mounted volumes, or at least has write permission
+export HOST_USER_ID=$(id -u)
+
+# Check if docker-machine is being used (normally on OSX) and get the uid from
+# the VM
+if hash docker-machine 2> /dev/null && docker-machine active > /dev/null; then
+    export HOST_USER_ID=$(docker-machine ssh $(docker-machine active) id -u)
+fi
+
+docker run \
+  -it \
+  -v "$(pwd)":/home/conda/workspace:rw,z \
+  -e HOST_USER_ID \
+  -e CPU_COUNT \
+  -e PYTHON_VER \
+  -e COMPILERS \
+  ${DOCKER_IMAGE} \
+  bash
+
+# Once you are inside the Docker session, you can use this to reproduce the CI steps:
+#
+# bash /home/conda/workspace/devtools/ci/gh-actions/scripts/run_steps_inside_docker_image.sh
\ No newline at end of file

devtools/run-ctest.py

@@ -52,12 +52,20 @@ def main():
         help='Number of processors to use',
         type=int,
         default=1)
+    parser.add_argument(
+        '--attempts',
+        help='Number of times failed tests will be re-run',
+        type=int,
+        default=1
+    )
 
     args, raw_args = parser.parse_known_args()
 
     status = execute_tests(args, raw_args)
-    if status != 0:
+    attempts = 0
+    if status != 0 and attempts < args.attempts:
         status = execute_failed_tests(args, raw_args)
+        attempts += 1
     return status
 
 

docs-source/developerguide/conf.py

@@ -170,6 +170,8 @@ htmlhelp_basename = 'OpenMMDeveloperGuidedoc'
 
 # -- Options for LaTeX output --------------------------------------------------
 
+latex_engine = 'xelatex'
+latex_use_xindy = False
 latex_elements = {
 # The paper size ('letterpaper' or 'a4paper').
 'papersize': 'letterpaper,openany',

docs-source/developerguide/developer.rst

@@ -856,8 +856,6 @@ Writing Device Code
 When compiling kernels with the Common Compute API, the following macros are
 defined.
 
-.. tabularcolumns:: |l|l|L|
-
 +-------------------------------+------------------------------------------------------------+--------------------------------------------+
 |Macro                          |OpenCL Definition                                           |CUDA Definition                             |
 +===============================+============================================================+============================================+

docs-source/developerguide/license.rst

@@ -26,4 +26,4 @@ OpenMM software and all related activities, such as this manual, are funded by
 the Simbios National Center for Biomedical Computing through the National
 Institutes of Health Roadmap for Medical Research, Grant U54 GM072970.
 Information on the National Centers can be found at
-http://nihroadmap.nih.gov/bioinformatics.
+https://commonfund.nih.gov/bioinformatics.

docs-source/usersguide/conf.py

@@ -170,6 +170,8 @@ htmlhelp_basename = 'OpenMMUsersGuidedoc'
 
 # -- Options for LaTeX output --------------------------------------------------
 
+latex_engine = 'xelatex'
+latex_use_xindy = False
 latex_elements = {
 # The paper size ('letterpaper' or 'a4paper').
 'papersize': 'letterpaper,openany',

examples/README.txt

@@ -10,7 +10,7 @@ the Simbios National Center for Physics-based Simulation of
 Biological Structures funded by the National Institutes of Health
 through the NIH Roadmap for Medical Research, Grant U54 GM072970. 
 Information on the National Centers for Biomedical Computing can 
-be obtained here: http://nihroadmap.nih.gov/bioinformatics/.
+be obtained here: https://commonfund.nih.gov/bioinformatics.
 
 OpenMM is developed under the supervision of Simbios P.I. Vijay 
 Pande at Stanford. Any work that uses OpenMM should cite the