README.rst 17.1 KB
Newer Older
1
################################################
Matthew Brett's avatar
Matthew Brett committed
2
Utilities for building on Travis CI and AppVeyor
3
################################################
4

Hugo's avatar
Hugo committed
5
A set of scripts to automate builds of macOS and Manylinux1 wheels on the
6
`Travis CI <https://travis-ci.org/>`_ infrastructure, and also Windows
7
wheels on the `AppVeyor <https://ci.appveyor.com/>`_ infrastructure.
Matthew Brett's avatar
Matthew Brett committed
8

9
The Travis CI scripts are designed to build *and test*:
Matthew Brett's avatar
Matthew Brett committed
10

11
* 64-bit macOS wheels built for macOS 10.9+
robbuckley's avatar
robbuckley committed
12
* 64/32-bit macOS wheels built for macOS 10.6+
13
14
* 64-bit ``manylinuxX_x86_64`` wheels, both narrow and wide Unicode builds, where `X` is any valid Manylinux version, such as `1`, or `2010`
* 32-bit ``manylinuxX_i686`` wheels, both narrow and wide Unicode builds
Matthew Brett's avatar
Matthew Brett committed
15

Rob Buckley's avatar
Rob Buckley committed
16
You can currently build and test against Pythons 2.7, 3.5, 3.6, 3.7 and 3.8
Matthew Brett's avatar
Matthew Brett committed
17

robbuckley's avatar
robbuckley committed
18
The small innovation here is that you can test against Linux 32-bit builds, both
19
20
wide and narrow Unicode Python 2 builds, which was not easy on the default
Travis CI configurations.
Matthew Brett's avatar
Matthew Brett committed
21

22
23
The AppVeyor setup is designed to build *and test*:

24
25
* 64-bit Windows ``win_amd64`` wheels
* 32-bit Windows ``win32`` wheels
26

Andrew Murray's avatar
Andrew Murray committed
27
You can currently build and test against Pythons 2.7, 3.5, 3.6, 3.7.
28

29
30
31
32
33
34
35
36
37
38
*****************
How does it work?
*****************

Multibuild is a series of bash scripts that define bash functions to build and
test wheels.

Configuration is by overriding the default build function, and defining a test
function.

Rob Buckley's avatar
Rob Buckley committed
39
40
The bash scripts are layered, in the sense that they are composed of a number scripts
which are sourced in sequence, each one potentially overriding previous ones.
41

Hugo's avatar
Hugo committed
42
43
macOS
=====
44

Rob Buckley's avatar
Rob Buckley committed
45
The following bash scripts are sourced in this order::
46

robbuckley's avatar
readme  
robbuckley committed
47
48
49
50
51
52
    multibuild/common_utils.sh
    multibuild/osx_utils.sh
    env_vars.sh
    multibuild/configure_build.sh
    multibuild/library_builders.sh
    config.sh
53

Rob Buckley's avatar
Rob Buckley committed
54
See ``multibuild/travis_osx_steps.sh``
55

Rob Buckley's avatar
Rob Buckley committed
56
The macOS build / test phases run on a macOS VM started by Travis CI.
robbuckley's avatar
readme  
robbuckley committed
57
Therefore any environment variable defined in ``.travis.yml`` or the bash
Matthew Brett's avatar
Matthew Brett committed
58
shell scripts listed above are available for your build and test.
59

robbuckley's avatar
readme  
robbuckley committed
60
61
Build options are controlled mainly by the following environment
variables:
62

robbuckley's avatar
robbuckley committed
63
64
65
* ``MB_PYTHON_VER`` sets the Python version targetted: ``major.minor.patch`` for CPython, or ``pypy-major.minor`` for PyPy.
* ``MB_PYTHON_OSX_VER`` sets the minimum macOS SDK version for any C extensions. For CPython targets it may be set to 10.6 or 10.9, provided a corresponding Python build is available at `python.org <https://www.python.org/downloads/mac-osx/>`_. It defaults to the highest version available. Its ignored for PyPy targets.
* ``PLAT`` sets the architectures built for any C extensions: ``x86_64`` or ``intel`` for 64-bit or 64/32-bit respectively. It defaults to the same arches as the target Python version: 64-bit for CPython macOS 10.9 or PyPy, and 64/32-bit for CPython 10.6.
66

robbuckley's avatar
robbuckley committed
67
68
69
In most cases its best to rely on the defaults for ``MB_PYTHON_OSX_VER`` and ``PLAT``, rather than setting them explicitly. Examples of exceptions to this guideline include: 
* setting ``MB_PYTHON_OSX_VER=10.6`` to build a 10.6 64/32-bit CPython wheel for Python 2.7 (default for 2.7 is 10.9 64-bit)
* setting ``MB_PYTHON_OSX_VER=10.6 and PLAT=x86_64``to build a 10.6 64-bit only wheel (10.6 would normally be 64/32-bit). Such a wheel would still have a platform tag of ``macosx_10_6_intel`, advertisting support for both 64 and 32-bit, but wouldnt work in 32-bit mode. This may be OK given how unlikely it is that there is still anyone actually running Python on macOS in 32-bit mode.
70

Rob Buckley's avatar
Rob Buckley committed
71
72
The ``build_wheel`` function builds the wheel, and ``install_run``
function installs and tests it.  Look in ``multibuild/common_utils.sh`` for
Rob Buckley's avatar
Rob Buckley committed
73
74
default definitions of these functions.  See below for more details, many of which are common
to macOS and Linux.
75

Matthew Brett's avatar
Matthew Brett committed
76
77
Manylinux
=========
78

79
The build phase is in a Manylinux Docker container, but the test phase is in
80
a clean Ubuntu 14.04 container.
81

82

Matthew Brett's avatar
Matthew Brett committed
83
84
Build phase
-----------
85

robbuckley's avatar
readme  
robbuckley committed
86
Specify the Manylinux version to build for with the `MB_ML_VER` environment variable. The default version is `1`.  Versions that are currently valid are:
87
88

* `1` (see [PEP 513](https://www.python.org/dev/peps/pep-0513);
Matthew Brett's avatar
Matthew Brett committed
89
* `2010` (see [PEP
90
91
92
93
94
95
96
97
98
99
100
101
102
103
  571](https://www.python.org/dev/peps/pep-0571).

At some point `2014` will be a valid version - see [PEP
599](https://www.python.org/dev/peps/pep-0599).

The environment variable specified which Manylinux docker container you are building in.

The `PLAT` environment variable can be one of `x86_64` or `i686`, specifying 64-bit and 32-bit builds, respectively.  The default is 64-bit.

At the time of writing, Manylinux2010 only supports 64-bit
builds, so `MB_ML_VER=2010` and `PLAT=i686` is an invalid
combination, and will generate an error when trying to find the
matching Docker image.

Matthew Brett's avatar
Matthew Brett committed
104
``multibuild/travis_linux_steps.sh`` defines the ``build_wheel`` function,
Hugo's avatar
Hugo committed
105
which starts up the Manylinux1 Docker container to run a wrapper script
Matthew Brett's avatar
Matthew Brett committed
106
``multibuild/docker_build_wrap.sh``, that (within the container) sources the
Rob Buckley's avatar
Rob Buckley committed
107
following bash scripts::
Matthew Brett's avatar
Matthew Brett committed
108

Rob Buckley's avatar
Rob Buckley committed
109
110
111
112
113
114
    multibuild/common_utils.sh
    multibuild/manylinux_utils.sh
    env_vars.sh
    multibuild/configure_build.sh
    multibuild/library_builders.sh
    config.sh
Matthew Brett's avatar
Matthew Brett committed
115

116
117
See ``docker_build_wrap.sh`` to review the order of script sourcing.

118
119
See the definition of ``build_multilinux`` in
``multibuild/travis_linux_steps.sh`` for the environment variables passed from
120
Travis CI to the Manylinux1 container.
Matthew Brett's avatar
Matthew Brett committed
121
122
123
124
125
126
127
128

Once in the container, after sourcing the scripts above, the wrapper runs the
real ``build_wheel`` function, which now comes (by default) from
``multibuild/common_utils.sh``.

Test phase
----------

Hugo's avatar
Hugo committed
129
Testing is in an Ubuntu 14.04 Docker container - see
Matthew Brett's avatar
Matthew Brett committed
130
``multibuild/docker_test_wrap.sh``.  ``multibuild/travis_linux_steps.sh``
Hugo's avatar
Hugo committed
131
defines the ``install_run`` function, which starts up the testing Docker
Matthew Brett's avatar
Matthew Brett committed
132
container with a wrapper script ``multibuild/docker_test_wrap.sh``.  The
Rob Buckley's avatar
Rob Buckley committed
133
wrapper script sources the following bash scripts::
Matthew Brett's avatar
Matthew Brett committed
134

Rob Buckley's avatar
Rob Buckley committed
135
136
    multibuild/common_utils.sh
    config.sh
Matthew Brett's avatar
Matthew Brett committed
137

138
139
See ``docker_test_wrap.sh`` for script source order.

Matthew Brett's avatar
Matthew Brett committed
140
141
142
143
144
See ``install_run`` in ``multibuild/travis_linux_steps.sh`` for the
environment variables passed into the container.

It then (in the container) runs the real ``install_run`` command, which comes
(by default) from ``multibuild/common_utils.sh``.
145
146
147
148
149

*********************************
Standard build and test functions
*********************************

Andrew Murray's avatar
Andrew Murray committed
150
The standard build command is ``build_wheel``.  This is a bash function.  By
Hugo's avatar
Hugo committed
151
default the function that is run on macOS, and in the Manylinux container for
Matthew Brett's avatar
Matthew Brett committed
152
153
154
the build phase, is defined in ``multibuild/common_utils.sh``.  You can
override the default function in the project ``config.sh`` file (see below).

robbuckley's avatar
typos  
robbuckley committed
155
If you are building a wheel from PyPI, rather than from a source repository,
156
157
158
159
160
161
162
you can use the ``build_index_wheel`` command, again defined in
``multibuild/common_utils.sh``.

Typically, you can get away with leaving the default ``build_wheel`` /
``build_index_wheel`` functions to do their thing, but you may need to define
a ``pre_build`` function in ``config.sh``.  The default ``build_wheel`` and
``build_index_wheel`` functions will call the ``pre_build`` function, if
163
164
165
defined, before building the wheel, so ``pre_build`` is a good place to build
any required libraries.

Matthew Brett's avatar
Matthew Brett committed
166
The standard test command is the bash function ``install_run``.  The version
Hugo's avatar
Hugo committed
167
run on macOS and in the Linux testing container is also defined in
Matthew Brett's avatar
Matthew Brett committed
168
``multibuild/common_utils.sh``.  Typically, you do not override this function,
169
170
171
172
but you in that case you will need to define a ``run_tests`` function, to run
your tests, returning a non-zero error code for failure.  The default
``install_run`` implementation calls the ``run_tests`` function, which you
will likely define in ``config.sh``.  See the examples below for examples of
Andrew Murray's avatar
Andrew Murray committed
173
less and more complicated builds, where the complicated builds override more
174
of the default implementations.
175
176
177
178

********************
To use these scripts
********************
Matthew Brett's avatar
Matthew Brett committed
179

180
* Make a repository for building wheels on Travis CI - e.g.
Matthew Brett's avatar
Matthew Brett committed
181
182
183
184
185
186
187
188
189
190
191
192
193
194
  https://github.com/MacPython/astropy-wheels - or in your case maybe
  ``https://github.com/your-org/your-project-wheels``;

* Add this (here) repository as a submodule::

    git submodule add https://github.com/matthew-brett/multibuild.git

* Add your own project repository as another submodule::

    git submodule add https://github.com/your-org/your-project.git

* Create a ``.travis.yml`` file, something like this::

    env:
195
196
197
198
199
        global:
            - REPO_DIR=your-project
            # Commit from your-project that you want to build
            - BUILD_COMMIT=v0.1.0
            # pip dependencies to _build_ your project
robbuckley's avatar
typos  
robbuckley committed
200
            - BUILD_DEPENDS="cython numpy"
201
202
203
204
205
206
207
208
209
210
211
212
213
            # pip dependencies to _test_ your project.  Include any dependencies
            # that you need, that are also specified in BUILD_DEPENDS, this will be
            # a separate install.
            - TEST_DEPENDS="numpy scipy pytest"
            - UNICODE_WIDTH=32
            - WHEELHOUSE_UPLOADER_USERNAME=travis-worker
            # Following generated with
            # travis encrypt -r your-org/your-project-wheels WHEELHOUSE_UPLOADER_SECRET=<the api key>
            # This is for Rackspace uploads.  Contact Matthew Brett, or the
            # scikit-learn team, for # permission (and the API key) to upload to
            # the Rackspace account used here, or use your own account.
            - secure:
                "MNKyBWOzu7JAUmC0Y+JhPKfytXxY/ADRmUIMEWZV977FLZPgYctqd+lqel2QIFgdHDO1CIdTSymOOFZckM9ICUXg9Ta+8oBjSvAVWO1ahDcToRM2DLq66fKg+NKimd2OfK7x597h/QmUSl4k8XyvyyXgl5jOiLg/EJxNE2r83IA="
Matthew Brett's avatar
Matthew Brett committed
214

215
216
217
218
219
220
221
    # You will likely prefer "language: generic" for travis configuration,
    # rather than, say "language: python". Multibuild doesn't use
    # Travis-provided Python but rather installs and uses its own, where the
    # Python version is set from the MB_PYTHON_VERSION variable. You can still
    # specify a language here if you need it for some unrelated logic and you
    # can't use Multibuild-provided Python or other software present on a
    # builder.
222
    language: generic
223
224
225

    # For CPython macOS builds only, the minimum supported macOS version and
    # architectures of any C extensions in the wheel are set with the variable
Rob Buckley's avatar
Rob Buckley committed
226
227
228
229
230
    # MB_PYTHON_OSX_VER: 10.9 (64-bit only) or 10.6 (64/32-bit dual arch). By
    # default this is set to the highest available for the Python version selected
    # using MB_PYTHON_VERSION. You should only need to set this explicitly if you
    # are building a 10.6 dual-arch build for a CPython version where both a 10.9 and
    # 10.6 build are available (for example, 2.7 or 3.7).
231
232
    # All PyPy macOS builds are 64-bit only.

Andrew Murray's avatar
Andrew Murray committed
233
    # Required in Linux to invoke `docker` ourselves
Matthew Brett's avatar
Matthew Brett committed
234
235
    services: docker

236
237
238
    # Host distribution.  This is the distribution from which we run the build
    # and test containers, via docker.
    dist: xenial
Matthew Brett's avatar
Matthew Brett committed
239

Matthew Brett's avatar
Matthew Brett committed
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
    matrix:
      include:
        - os: linux
          env: MB_PYTHON_VERSION=2.7
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - UNICODE_WIDTH=16
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - PLAT=i686
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - PLAT=i686
            - UNICODE_WIDTH=16
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.5
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.5
            - PLAT=i686
riddell1's avatar
riddell1 committed
264
265
266
267
268
269
270
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.6
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.6
            - PLAT=i686
Matthew Brett's avatar
Matthew Brett committed
271
272
273
        - os: osx
          env:
            - MB_PYTHON_VERSION=2.7
Rob Buckley's avatar
Rob Buckley committed
274
            - MB_PYTHON_OSX_VER=10.6
robbuckley's avatar
readme  
robbuckley committed
275
276
277
        - os: osx
          env:
            - MB_PYTHON_VERSION=2.7
Matthew Brett's avatar
Matthew Brett committed
278
279
280
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.5
riddell1's avatar
riddell1 committed
281
282
283
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.6
Kyle Stewart's avatar
Kyle Stewart committed
284
        - os: osx
285
          env:
286
            - MB_PYTHON_VERSION=3.7
Rob Buckley's avatar
Rob Buckley committed
287
288
289
290
291
292
293
            - MB_PYTHON_OSX_VER=10.6
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.7
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.8
Kyle Stewart's avatar
Kyle Stewart committed
294
295
296
        - os: osx
          language: generic
          env:
297
            - MB_PYTHON_VERSION=pypy-5.7
Matthew Brett's avatar
Matthew Brett committed
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322

    before_install:
        - source multibuild/common_utils.sh
        - source multibuild/travis_steps.sh
        - before_install

    install:
        # Maybe get and clean and patch source
        - clean_code $REPO_DIR $BUILD_COMMIT
        - build_wheel $REPO_DIR $PLAT

    script:
        - install_run $PLAT

    after_success:
        # Upload wheels to Rackspace container
        - pip install wheelhouse-uploader
        # This uploads the wheels to a Rackspace container owned by the
        # scikit-learn team, available at http://wheels.scipy.org.  See above
        # for information on using this account or choosing another.
        - python -m wheelhouse_uploader upload --local-folder
            ${TRAVIS_BUILD_DIR}/wheelhouse/
            --no-update-index
            wheels

Hugo's avatar
Hugo committed
323
  The example above is for a project building from a Git submodule.  If you
324
325
326
327
328
329
330
331
332
333
334
335
336
337
  aren't building from a submodule, but want to use ``pip`` to build from a
  source archive on https://pypi.org or similar, replace the first few lines
  of the ``.travis.yml`` file with something like::

    env:
        global:
            # Instead of REPO_DIR, BUILD_COMMIT
            - PROJECT_SPEC="tornado==4.1.1"

  then your ``install`` section could look something like this::

    install:
        - build_index_wheel $PROJECT_SPEC

338

Matthew Brett's avatar
Matthew Brett committed
339
340
341
342
343
* Next create a ``config.sh`` for your project, that fills in any steps you
  need to do before building the wheel (such as building required libraries).
  You also need this file to specify how to run your tests::

    # Define custom utilities
Hugo's avatar
Hugo committed
344
    # Test for macOS with [ -n "$IS_OSX" ]
Matthew Brett's avatar
Matthew Brett committed
345
346
347
348
349
350
351
352
353
354
355
356
357

    function pre_build {
        # Any stuff that you need to do before you start building the wheels
        # Runs in the root directory of this repository.
        :
    }

    function run_tests {
        # Runs tests on installed distribution from an empty directory
        python --version
        python -c 'import sys; import yourpackage; sys.exit(yourpackage.test())'
    }

358
359
  Optionally you can specify a different location for ``config.sh`` file with
  the ``$CONFIG_PATH`` environment variable.
360

361
* Optionally, create an ``env_vars.sh`` file to override the defaults for any
362
363
364
  environment variables used by
  ``configure_build.sh``/``library_builders.sh``. In Linux, the environment
  variables used for the build cannot be set in the ``.travis.yml`` file,
365
  because the build processing runs in a Docker container, so the only
366
367
368
369
370
371
372
373
  environment variables that reach the container are those passed in via the
  ``docker run`` command, or those set in ``env_vars.sh``.

  As for the ``config.sh`` file, you can specify a different location for the
  file by setting the ``$ENV_VARS_PATH`` environment variable.  The path in
  ``$ENV_VARS_PATH`` is relative to the repository root directory.  For
  example, if your repository had a subdirectory ``scripts`` with a file
  ``my_env_vars.sh``, you should set ``ENV_VARS_PATH=scripts/my_env_vars.sh``.
374

375
* Make sure your project is set up to build on Travis CI, and you should now
Matthew Brett's avatar
Matthew Brett committed
376
  be ready (to begin the long slow debugging process, probably).
377

378
379
* For the Windows wheels, create an ``appveyor.yml`` file, something like:

Andrew Murray's avatar
Andrew Murray committed
380
  - https://github.com/MacPython/numpy-wheels/blob/master/.appveyor.yml
381
382
383
  - https://github.com/MacPython/astropy-wheels/blob/master/appveyor.yml
  - https://github.com/MacPython/nipy-wheels/blob/master/appveyor.yml
  - https://github.com/MacPython/pytables-wheels/blob/master/appveyor.yml
384

385
386
  Note the Windows test customizations etc are inside ``appveyor.yml``,
  and that ``config.sh`` and ``env_vars.sh`` are only for the
387
  Linux/Mac builds on Travis CI.
388

Matthew Brett's avatar
Matthew Brett committed
389
* Make sure your project is set up to build on AppVeyor, and you should now
390
  be ready (for what could be another round of slow debugging).
Matthew Brett's avatar
Matthew Brett committed
391

Hugo's avatar
Hugo committed
392
393
If your project depends on NumPy, you will want to build against the earliest
NumPy that your project supports - see `forward, backward NumPy compatibility
Andrew Murray's avatar
Andrew Murray committed
394
<https://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312>`_.
395
See the `astropy-wheels Travis file
Matthew Brett's avatar
Matthew Brett committed
396
<https://github.com/MacPython/astropy-wheels/blob/master/.travis.yml>`_ for an
Hugo's avatar
Hugo committed
397
example specifying NumPy build and test dependencies.
Matthew Brett's avatar
Matthew Brett committed
398
399
400
401
402
403
404
405
406

Here are some simple example projects:

* https://github.com/MacPython/astropy-wheels
* https://github.com/scikit-image/scikit-image-wheels
* https://github.com/MacPython/nipy-wheels
* https://github.com/MacPython/dipy-wheels

Less simple projects where there are some serious build dependencies, and / or
Hugo's avatar
Hugo committed
407
macOS / Linux differences:
Matthew Brett's avatar
Matthew Brett committed
408
409
410
411

* https://github.com/MacPython/matplotlib-wheels
* https://github.com/python-pillow/Pillow-wheels
* https://github.com/MacPython/h5py-wheels
412
413
414
415
416
417
418
419

**********************
Multibuild development
**********************

The main multibuild repository is always at
https://github.com/matthew-brett/multibuild

420
421
We try to keep the ``master`` branch stable and do testing and development
in the ``devel`` branch.  From time to time we merge ``devel`` into ``master``.
422

423
424
In practice, you can check out the newest commit from ``devel`` that works
for you, then stay at it until you need newer features.