Merge pull request #396 from mattip/manylinux_2_24

Support manylinux_2_24

.travis.yml

@@ -20,13 +20,19 @@ matrix:
       dist: xenial
       env:
         - TEST_BUILDS=1
-    # 64-bit builds, manylinux2010
+    # 64-bit builds
     - os: linux
       dist: xenial
       env:
         - TEST_BUILDS=1
         - MB_ML_VER=2010
         - DOCKER_TEST_IMAGE=multibuild/xenial_{PLAT}
+    - os: linux
+      dist: xenial
+      env:
+        - TEST_BUILDS=1
+        - MB_ML_VER=_2_24
+        - DOCKER_TEST_IMAGE=multibuild/xenial_{PLAT}
     # 32-bit builds
     - os: linux
       dist: xenial

README.rst

@@ -41,7 +41,8 @@ The Travis CI scripts are designed to build *and test*:
 
 * 64-bit macOS wheels built for macOS 10.9+
 * 64/32-bit macOS wheels built for macOS 10.6+
-* 64-bit ``manylinuxX_x86_64`` wheels, both narrow and wide Unicode builds, where `X` is any valid Manylinux version, such as `1`, or `2010`
+* 64-bit ``manylinuxX_x86_64`` wheels, both narrow and wide Unicode builds,
+  where `X` is any valid Manylinux version: `1`, `2010`, `2014` or `_2_24`.
 * 32-bit ``manylinuxX_i686`` wheels, both narrow and wide Unicode builds
 
 You can currently build and test against Pythons 2.7, 3.5, 3.6, 3.7, 3.8 and 3.9
@@ -55,7 +56,7 @@ The AppVeyor setup is designed to build *and test*:
 * 64-bit Windows ``win_amd64`` wheels
 * 32-bit Windows ``win32`` wheels
 
-You can currently build and test against Pythons 2.7, 3.5, 3.6, 3.7, 3.8
+You can currently build and test against Pythons 2.7, 3.5, 3.6, 3.7, 3.8, 3.9
 
 *****************
 How does it work?
@@ -91,19 +92,35 @@ shell scripts listed above are available for your build and test.
 Build options are controlled mainly by the following environment
 variables:
 
-* ``MB_PYTHON_VER`` sets the Python version targetted: ``major.minor.patch`` for CPython, or ``pypy-major.minor`` for PyPy.
-* ``MB_PYTHON_OSX_VER`` sets the minimum macOS SDK version for any C extensions. For CPython targets it may be set to 10.6 or 10.9, provided a corresponding Python build is available at `python.org <https://www.python.org/downloads/mac-osx/>`_. It defaults to the highest version available. It's ignored for PyPy targets.
-* ``PLAT`` sets the architectures built for any C extensions: ``x86_64`` or ``intel`` for 64-bit or 64/32-bit respectively. It defaults to the same arches as the target Python version: 64-bit for CPython macOS 10.9 or PyPy, and 64/32-bit for CPython 10.6.
-
-In most cases it's best to rely on the defaults for ``MB_PYTHON_OSX_VER`` and ``PLAT``, rather than setting them explicitly. Examples of exceptions to this guideline include:
-
-* setting ``MB_PYTHON_OSX_VER=10.6`` to build a 10.6 64/32-bit CPython wheel for Python 2.7 (default for 2.7 is 10.9 64-bit)
-* setting ``MB_PYTHON_OSX_VER=10.6 and PLAT=x86_64`` to build a 10.6 64-bit only wheel (10.6 would normally be 64/32-bit). Such a wheel would still have a platform tag of ``macosx_10_6_intel`` , advertising support for both 64 and 32-bit, but wouldnt work in 32-bit mode. This may be OK given how unlikely it is that there is still anyone actually running Python on macOS in 32-bit mode.
+* ``MB_PYTHON_VER`` sets the Python version targetted: ``major.minor.patch``
+  for CPython, or ``pypy-major.minor`` for PyPy.
+* ``MB_PYTHON_OSX_VER`` sets the minimum macOS SDK version for any C
+  extensions. For CPython targets it may be set to 10.6 or 10.9, provided a
+  corresponding Python build is available at `python.org
+  <https://www.python.org/downloads/mac-osx/>`_. It defaults to the highest
+  version available. It's ignored for PyPy targets.
+* ``PLAT`` sets the architectures built for any C extensions: ``x86_64`` or
+  ``intel`` for 64-bit or 64/32-bit respectively. It defaults to the same
+  arches as the target Python version: 64-bit for CPython macOS 10.9 or PyPy,
+  and 64/32-bit for CPython 10.6.
+
+In most cases it's best to rely on the defaults for ``MB_PYTHON_OSX_VER`` and
+``PLAT``, rather than setting them explicitly. Examples of exceptions to this
+guideline include:
+
+* setting ``MB_PYTHON_OSX_VER=10.6`` to build a 10.6 64/32-bit CPython wheel
+  for Python 2.7 (default for 2.7 is 10.9 64-bit)
+* setting ``MB_PYTHON_OSX_VER=10.6 and PLAT=x86_64`` to build a 10.6 64-bit
+  only wheel (10.6 would normally be 64/32-bit). Such a wheel would still have
+  a platform tag of ``macosx_10_6_intel`` , advertising support for both 64 and
+  32-bit, but wouldn't work in 32-bit mode. This may be OK given how unlikely it
+  is that there is still anyone actually running Python on macOS in 32-bit
+  mode.
 
 The ``build_wheel`` function builds the wheel, and ``install_run``
 function installs and tests it.  Look in ``multibuild/common_utils.sh`` for
-default definitions of these functions.  See below for more details, many of which are common
-to macOS and Linux.
+default definitions of these functions.  See below for more details, many of
+which are common to macOS and Linux.
 
 Manylinux
 =========
@@ -122,6 +139,7 @@ variable. The default version is ``1``.  Versions that are currently valid are:
 * ``2010``  corresponding to manylinux2010 (see `PEP 571 <https://www.python.org/dev/peps/pep-0571>`_).
 * ``2014`` corresponding to manylinux2014 and adds more architectures to ``PLAT``
   (see `PEP 599 <https://www.python.org/dev/peps/pep-0599>`_).
+* ``_2_24`` corresponding to manylinux_2_24 (see `PEP 600 <https://www.python.org/dev/peps/pep-0600>`_).
 
 The environment variable specified which Manylinux docker container you are building in.
 

common_utils.sh

@@ -24,7 +24,14 @@ GET_PIP_URL=https://bootstrap.pypa.io/get-pip.py
 # with, so it is passed in when calling "docker run" for tests.
 UNICODE_WIDTH=${UNICODE_WIDTH:-32}
 
-if [ $(uname) == "Darwin" ]; then IS_MACOS=1; IS_OSX=1; fi
+if [ $(uname) == "Darwin" ]; then
+  IS_MACOS=1; IS_OSX=1;
+else
+  # In the manylinux_2_24 image, based on Debian9, "python" is not installed
+  # so link in something for the various system calls before PYTHON_EXE is set
+  which python || export PATH=/opt/python/cp39-cp39/bin:$PATH
+fi
+
 
 # Work round bug in travis xcode image described at
 # https://github.com/direnv/direnv/issues/210
@@ -199,7 +206,15 @@ function untar {
 }
 
 function install_rsync {
-    if [ -z "$IS_MACOS" ]; then
+    # install rsync via package manager
+    if [ -n "$IS_MACOS" ]; then
+        # macOS. The colon in the next line is the null command
+        :
+    elif [[ $MB_ML_VER == "_2_24" ]]; then
+        # debian:9 based distro
+        [[ $(type -P rsync) ]] || apt-get install -y rsync
+    else
+        # centos based distro
         [[ $(type -P rsync) ]] || yum_install rsync
     fi
 }

configure_build.sh

@@ -47,13 +47,19 @@ else
     export CFLAGS="${CFLAGS:-$STRIP_FLAGS}"
     export CXXFLAGS="${CXXFLAGS:-$STRIP_FLAGS}"
     export FFLAGS="${FFLAGS:-$STRIP_FLAGS}"
-    if [[ $MB_ML_VER == "1" ]]; then
+    if [[ $MB_ML_VER == "_2_24" ]]; then
+        # This is the first opportunity to distinguish between manylinuxes
+        apt update
         if [ "${MB_PYTHON_VERSION:0:4}" == "pypy" ]; then
-            yum install -y libtool wget
-        else
-            yum install -y libtool
+            # debian:9 based distro
+            apt install -y wget
         fi
+    elif [[ $MB_ML_VER == "1" ]]; then
+        # Need libtool, and for pypy need wget
+        # centos based distro
+        yum install -y libtool wget
     elif [ "${MB_PYTHON_VERSION:0:4}" == "pypy" ]; then
+        # centos based distro
         yum install -y wget
     fi
 fi

library_builders.sh

@@ -14,7 +14,7 @@ OPENBLAS_VERSION="${OPENBLAS_VERSION:-0.3.10}"
 # We use system zlib by default - see build_new_zlib
 ZLIB_VERSION="${ZLIB_VERSION:-1.2.10}"
 LIBPNG_VERSION="${LIBPNG_VERSION:-1.6.21}"
-BZIP2_VERSION="${BZIP2_VERSION:-1.0.6}"
+BZIP2_VERSION="${BZIP2_VERSION:-1.0.7}"
 FREETYPE_VERSION="${FREETYPE_VERSION:-2.6.3}"
 TIFF_VERSION="${TIFF_VERSION:-4.1.0}"
 JPEG_VERSION="${JPEG_VERSION:-9b}"
@@ -133,7 +133,13 @@ function build_zlib {
     # Gives an old but safe version
     if [ -n "$IS_MACOS" ]; then return; fi  # OSX has zlib already
     if [ -e zlib-stamp ]; then return; fi
-    yum_install zlib-devel
+    if [[ $MB_ML_VER == "_2_24" ]]; then
+        # debian:9 based distro
+        apt-get install -y zlib1g-dev
+    else
+        #centos based distro
+        yum_install zlib-devel
+    fi
     touch zlib-stamp
 }
 
@@ -180,10 +186,14 @@ function get_modern_cmake {
     local cmake=cmake
     if [ -n "$IS_MACOS" ]; then
         brew install cmake > /dev/null
+    elif [[ $MB_ML_VER == "_2_24" ]]; then
+        # debian:9 based distro
+        apt-get install -y cmake
     else
         if [ "`yum search cmake | grep ^cmake28\.`" ]; then
             cmake=cmake28
         fi
+        # centos based distro
         yum_install $cmake > /dev/null
     fi
     echo $cmake
@@ -396,7 +406,11 @@ function build_suitesparse {
     if [ -e suitesparse-stamp ]; then return; fi
     if [ -n "$IS_MACOS" ]; then
         brew install suite-sparse > /dev/null
+    elif [[ $MB_ML_VER == "_2_24" ]]; then
+        # debian:9 based distro
+        apt-get install -y libsuitesparse-dev > /dev/null
     else
+        # centos based distro
         yum_install suitesparse-devel > /dev/null
     fi
     touch suitesparse-stamp

manylinux_utils.sh

 #!/bin/bash
-# Useful utilities common across manylinux1 builds
+# Useful utilities common across manylinux builds
 
 MULTIBUILD_DIR=$(dirname "${BASH_SOURCE[0]}")
 source $MULTIBUILD_DIR/common_utils.sh
@@ -44,7 +44,13 @@ function activate_ccache {
     ln -s /parent-home/.ccache $HOME/.ccache
 
     # Now install ccache
-    suppress yum_install ccache
+    if [[ $MB_ML_VER == "_2_24" ]]; then
+        # debian:9 based distro
+        suppress apt-get install -y ccache
+    else
+        # centos based distro
+        suppress yum_install ccache
+    fi
 
     # Create fake compilers and prepend them to the PATH
     # Note that yum is supposed to create these for us,
@@ -60,6 +66,7 @@ function activate_ccache {
     # Prove to the developer that ccache is activated
     echo "Using C compiler: $(which gcc)"
 }
+
 function yum_install {
     # CentOS 5 yum doesn't fail in some cases, e.g. if package is not found
     # https://serverfault.com/questions/694942/yum-should-error-when-a-package-is-not-available

tests/test_manylinux_utils_docker.sh

@@ -3,8 +3,9 @@
 source manylinux_utils.sh
 source tests/utils.sh
 
-suppress yum_install rsync && suppress yum erase -y rsync \
-    || ingest "yum_install valid"
-suppress bash -c '! yum_install nonexistent' || ingest "yum_install nonexistent"
-
-barf
+if [[ $MB_ML_VER != "_2_24" ]]; then
+    suppress yum_install rsync && suppress yum erase -y rsync \
+        || ingest "yum_install valid"
+    suppress bash -c '! yum_install nonexistent' || ingest "yum_install nonexistent"
+    barf
\ No newline at end of file
+fi