README.rst 14.5 KB
Newer Older
1
################################################
Matthew Brett's avatar
Matthew Brett committed
2
Utilities for building on Travis CI and AppVeyor
3
################################################
4

Hugo's avatar
Hugo committed
5
A set of scripts to automate builds of macOS and Manylinux1 wheels on the
6
`Travis CI <https://travis-ci.org/>`_ infrastructure, and also Windows
7
wheels on the `AppVeyor <https://ci.appveyor.com/>`_ infrastructure.
Matthew Brett's avatar
Matthew Brett committed
8

9
The Travis CI scripts are designed to build *and test*:
Matthew Brett's avatar
Matthew Brett committed
10

11
12
* Dual 32/64-bit architecture macOS wheels built for macOS 10.6+;
* 64-bit macOS wheels built for macOS 10.9+;
13
14
* 64-bit ``manylinux1_x86_64`` wheels, both narrow and wide Unicode builds;
* 32-bit ``manylinux1_i686`` wheels, both narrow and wide Unicode builds.
Matthew Brett's avatar
Matthew Brett committed
15

Hugo's avatar
Hugo committed
16
You can currently build and test against Pythons 2.7, 3.4, 3.5, 3.6, 3.7.
Matthew Brett's avatar
Matthew Brett committed
17
18

The small innovation here is that you can test against 32-bit builds, and both
19
20
wide and narrow Unicode Python 2 builds, which was not easy on the default
Travis CI configurations.
Matthew Brett's avatar
Matthew Brett committed
21

22
23
24
25
26
The AppVeyor setup is designed to build *and test*:

* 64-bit Windows ``win_amd64`` wheels;
* 32-bit Windows ``win32`` wheels.

27
You can currently build and test against Pythons 2.7, 3.4, 3.5, 3.6, 3.7.
28

29
30
31
32
33
34
35
36
37
38
39
40
41
*****************
How does it work?
*****************

Multibuild is a series of bash scripts that define bash functions to build and
test wheels.

Configuration is by overriding the default build function, and defining a test
function.

The bash scripts are layered, in the sense that they loaded in the following
sequence:

Hugo's avatar
Hugo committed
42
43
macOS
=====
44

Matthew Brett's avatar
Matthew Brett committed
45
See ``multibuild/travis_osx_steps.sh``.
46

Matthew Brett's avatar
Matthew Brett committed
47
48
49
For build and test phases, these bash scripts get sourced one after the other,
so that functions and variables defined in later scripts can overwrite
functions and variables in earlier scripts:
50

Matthew Brett's avatar
Matthew Brett committed
51
52
* multibuild/common_utils.sh
* multibuild/osx_utils.sh
53
* env_vars.sh
54
* multibuild/configure_build.sh
Matthew Brett's avatar
Matthew Brett committed
55
56
* multibuild/library_builders.sh
* config.sh
57

58
59
See ``travis_osx_steps.sh`` to review source order.

60
The macOS build / test phases run on the macOS VM started by Travis CI,
Matthew Brett's avatar
Matthew Brett committed
61
62
Therefore any environment variable defined in the ``.travis.yml`` or bash
shell scripts listed above are available for your build and test.
63

64
65
66
67
68
69
70
Note that macOS builds may be targetted either at macOS 10.6 and above
(dual arch 32/64b) or macOS 10.9 and above (64b only), depending on the
value of the environment variable `MB_PYTHON_OSX_VER`. These depend on the
corresponding build of python from https://www.python.org/downloads/mac-osx/.
Versions of python older than 3.6.5 and 2.7.15 don't have 10.9 / 64-bit only
installers available.

71
72
73
The ``build_wheel`` function builds the wheel, and the ``install_run``
function installs the wheel and tests it.  Look in ``common_utils.sh`` for
default definitions of these functions.  See below for more details.
74

Matthew Brett's avatar
Matthew Brett committed
75
76
Manylinux
=========
77

Hugo's avatar
Hugo committed
78
The build phase is in a Manylinux1 Docker container, but the test phase is in
79
a clean Ubuntu 14.04 container.
80

Matthew Brett's avatar
Matthew Brett committed
81
82
Build phase
-----------
83

Matthew Brett's avatar
Matthew Brett committed
84
``multibuild/travis_linux_steps.sh`` defines the ``build_wheel`` function,
Hugo's avatar
Hugo committed
85
which starts up the Manylinux1 Docker container to run a wrapper script
Matthew Brett's avatar
Matthew Brett committed
86
87
88
89
90
``multibuild/docker_build_wrap.sh``, that (within the container) sources the
following bash scripts:

* multibuild/common_utils.sh
* multibuild/manylinux_utils.sh
91
* env_vars.sh
92
* multibuild/configure_build.sh
Matthew Brett's avatar
Matthew Brett committed
93
94
95
* multibuild/library_builders.sh
* config.sh

96
97
See ``docker_build_wrap.sh`` to review the order of script sourcing.

98
99
See the definition of ``build_multilinux`` in
``multibuild/travis_linux_steps.sh`` for the environment variables passed from
100
Travis CI to the Manylinux1 container.
Matthew Brett's avatar
Matthew Brett committed
101
102
103
104
105
106
107
108

Once in the container, after sourcing the scripts above, the wrapper runs the
real ``build_wheel`` function, which now comes (by default) from
``multibuild/common_utils.sh``.

Test phase
----------

Hugo's avatar
Hugo committed
109
Testing is in an Ubuntu 14.04 Docker container - see
Matthew Brett's avatar
Matthew Brett committed
110
``multibuild/docker_test_wrap.sh``.  ``multibuild/travis_linux_steps.sh``
Hugo's avatar
Hugo committed
111
defines the ``install_run`` function, which starts up the testing Docker
Matthew Brett's avatar
Matthew Brett committed
112
113
114
115
116
117
container with a wrapper script ``multibuild/docker_test_wrap.sh``.  The
wrapper script sources the following bash scripts:

* multibuild/common_utils.sh
* config.sh

118
119
See ``docker_test_wrap.sh`` for script source order.

Matthew Brett's avatar
Matthew Brett committed
120
121
122
123
124
See ``install_run`` in ``multibuild/travis_linux_steps.sh`` for the
environment variables passed into the container.

It then (in the container) runs the real ``install_run`` command, which comes
(by default) from ``multibuild/common_utils.sh``.
125
126
127
128
129

*********************************
Standard build and test functions
*********************************

Andrew Murray's avatar
Andrew Murray committed
130
The standard build command is ``build_wheel``.  This is a bash function.  By
Hugo's avatar
Hugo committed
131
default the function that is run on macOS, and in the Manylinux container for
Matthew Brett's avatar
Matthew Brett committed
132
133
134
the build phase, is defined in ``multibuild/common_utils.sh``.  You can
override the default function in the project ``config.sh`` file (see below).

135
136
137
138
139
140
141
142
If you are building a wheel from pypi, rather than from a source repository,
you can use the ``build_index_wheel`` command, again defined in
``multibuild/common_utils.sh``.

Typically, you can get away with leaving the default ``build_wheel`` /
``build_index_wheel`` functions to do their thing, but you may need to define
a ``pre_build`` function in ``config.sh``.  The default ``build_wheel`` and
``build_index_wheel`` functions will call the ``pre_build`` function, if
143
144
145
defined, before building the wheel, so ``pre_build`` is a good place to build
any required libraries.

Matthew Brett's avatar
Matthew Brett committed
146
The standard test command is the bash function ``install_run``.  The version
Hugo's avatar
Hugo committed
147
run on macOS and in the Linux testing container is also defined in
Matthew Brett's avatar
Matthew Brett committed
148
``multibuild/common_utils.sh``.  Typically, you do not override this function,
149
150
151
152
but you in that case you will need to define a ``run_tests`` function, to run
your tests, returning a non-zero error code for failure.  The default
``install_run`` implementation calls the ``run_tests`` function, which you
will likely define in ``config.sh``.  See the examples below for examples of
Andrew Murray's avatar
Andrew Murray committed
153
less and more complicated builds, where the complicated builds override more
154
of the default implementations.
155
156
157
158

********************
To use these scripts
********************
Matthew Brett's avatar
Matthew Brett committed
159

160
* Make a repository for building wheels on Travis CI - e.g.
Matthew Brett's avatar
Matthew Brett committed
161
162
163
164
165
166
167
168
169
170
171
172
173
174
  https://github.com/MacPython/astropy-wheels - or in your case maybe
  ``https://github.com/your-org/your-project-wheels``;

* Add this (here) repository as a submodule::

    git submodule add https://github.com/matthew-brett/multibuild.git

* Add your own project repository as another submodule::

    git submodule add https://github.com/your-org/your-project.git

* Create a ``.travis.yml`` file, something like this::

    env:
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
        global:
            - REPO_DIR=your-project
            # Commit from your-project that you want to build
            - BUILD_COMMIT=v0.1.0
            # pip dependencies to _build_ your project
            - BUILD_DEPENDS="Cython numpy"
            # pip dependencies to _test_ your project.  Include any dependencies
            # that you need, that are also specified in BUILD_DEPENDS, this will be
            # a separate install.
            - TEST_DEPENDS="numpy scipy pytest"
            - PLAT=x86_64
            - UNICODE_WIDTH=32
            - WHEELHOUSE_UPLOADER_USERNAME=travis-worker
            # Following generated with
            # travis encrypt -r your-org/your-project-wheels WHEELHOUSE_UPLOADER_SECRET=<the api key>
            # This is for Rackspace uploads.  Contact Matthew Brett, or the
            # scikit-learn team, for # permission (and the API key) to upload to
            # the Rackspace account used here, or use your own account.
            - secure:
                "MNKyBWOzu7JAUmC0Y+JhPKfytXxY/ADRmUIMEWZV977FLZPgYctqd+lqel2QIFgdHDO1CIdTSymOOFZckM9ICUXg9Ta+8oBjSvAVWO1ahDcToRM2DLq66fKg+NKimd2OfK7x597h/QmUSl4k8XyvyyXgl5jOiLg/EJxNE2r83IA="
Matthew Brett's avatar
Matthew Brett committed
195
196

    language: python
197
    # The Travis Python version is unrelated to the version we build and test
198
    # with.  This is set with the MB_PYTHON_VERSION variable.
199
    python: 3.5
Matthew Brett's avatar
Matthew Brett committed
200
201
202
203
    sudo: required
    dist: trusty
    services: docker

Matthew Brett's avatar
Matthew Brett committed
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
    matrix:
      exclude:
        # Exclude the default Python 3.5 build
        - python: 3.5
      include:
        - os: linux
          env: MB_PYTHON_VERSION=2.7
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - UNICODE_WIDTH=16
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - PLAT=i686
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - PLAT=i686
            - UNICODE_WIDTH=16
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.4
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.4
            - PLAT=i686
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.5
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.5
            - PLAT=i686
riddell1's avatar
riddell1 committed
238
239
240
241
242
243
244
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.6
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.6
            - PLAT=i686
Matthew Brett's avatar
Matthew Brett committed
245
246
247
248
        - os: osx
          language: generic
          env:
            - MB_PYTHON_VERSION=2.7
249
250
251
252
        - os: osx
          env:
            - MB_PYTHON_VERSION=2.7
            - MB_PYTHON_OSX_VER=10.9
Matthew Brett's avatar
Matthew Brett committed
253
254
255
256
257
258
259
260
        - os: osx
          language: generic
          env:
            - MB_PYTHON_VERSION=3.4
        - os: osx
          language: generic
          env:
            - MB_PYTHON_VERSION=3.5
riddell1's avatar
riddell1 committed
261
262
263
264
        - os: osx
          language: generic
          env:
            - MB_PYTHON_VERSION=3.6
265
266
267
        - os: osx
          language: generic
          env:
268
            - MB_PYTHON_VERSION=3.7
269
            - MB_PYTHON_OSX_VER=10.9
Kyle Stewart's avatar
Kyle Stewart committed
270
271
272
        - os: osx
          language: generic
          env:
273
            - MB_PYTHON_VERSION=pypy-5.7
Matthew Brett's avatar
Matthew Brett committed
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298

    before_install:
        - source multibuild/common_utils.sh
        - source multibuild/travis_steps.sh
        - before_install

    install:
        # Maybe get and clean and patch source
        - clean_code $REPO_DIR $BUILD_COMMIT
        - build_wheel $REPO_DIR $PLAT

    script:
        - install_run $PLAT

    after_success:
        # Upload wheels to Rackspace container
        - pip install wheelhouse-uploader
        # This uploads the wheels to a Rackspace container owned by the
        # scikit-learn team, available at http://wheels.scipy.org.  See above
        # for information on using this account or choosing another.
        - python -m wheelhouse_uploader upload --local-folder
            ${TRAVIS_BUILD_DIR}/wheelhouse/
            --no-update-index
            wheels

Hugo's avatar
Hugo committed
299
  The example above is for a project building from a Git submodule.  If you
300
301
302
303
304
305
306
307
308
309
310
311
312
313
  aren't building from a submodule, but want to use ``pip`` to build from a
  source archive on https://pypi.org or similar, replace the first few lines
  of the ``.travis.yml`` file with something like::

    env:
        global:
            # Instead of REPO_DIR, BUILD_COMMIT
            - PROJECT_SPEC="tornado==4.1.1"

  then your ``install`` section could look something like this::

    install:
        - build_index_wheel $PROJECT_SPEC

314

Matthew Brett's avatar
Matthew Brett committed
315
316
317
318
319
* Next create a ``config.sh`` for your project, that fills in any steps you
  need to do before building the wheel (such as building required libraries).
  You also need this file to specify how to run your tests::

    # Define custom utilities
Hugo's avatar
Hugo committed
320
    # Test for macOS with [ -n "$IS_OSX" ]
Matthew Brett's avatar
Matthew Brett committed
321
322
323
324
325
326
327
328
329
330
331
332
333

    function pre_build {
        # Any stuff that you need to do before you start building the wheels
        # Runs in the root directory of this repository.
        :
    }

    function run_tests {
        # Runs tests on installed distribution from an empty directory
        python --version
        python -c 'import sys; import yourpackage; sys.exit(yourpackage.test())'
    }

334
335
  Optionally you can specify a different location for ``config.sh`` file with
  the ``$CONFIG_PATH`` environment variable.
336

337
* Optionally, create an ``env_vars.sh`` file to override the defaults for any
338
339
340
  environment variables used by
  ``configure_build.sh``/``library_builders.sh``. In Linux, the environment
  variables used for the build cannot be set in the ``.travis.yml`` file,
341
  because the build processing runs in a Docker container, so the only
342
343
344
345
346
347
348
349
  environment variables that reach the container are those passed in via the
  ``docker run`` command, or those set in ``env_vars.sh``.

  As for the ``config.sh`` file, you can specify a different location for the
  file by setting the ``$ENV_VARS_PATH`` environment variable.  The path in
  ``$ENV_VARS_PATH`` is relative to the repository root directory.  For
  example, if your repository had a subdirectory ``scripts`` with a file
  ``my_env_vars.sh``, you should set ``ENV_VARS_PATH=scripts/my_env_vars.sh``.
350

351
* Make sure your project is set up to build on Travis CI, and you should now
Matthew Brett's avatar
Matthew Brett committed
352
  be ready (to begin the long slow debugging process, probably).
353

354
355
* For the Windows wheels, create an ``appveyor.yml`` file, something like:

Andrew Murray's avatar
Andrew Murray committed
356
  - https://github.com/MacPython/numpy-wheels/blob/master/.appveyor.yml
357
358
359
  - https://github.com/MacPython/astropy-wheels/blob/master/appveyor.yml
  - https://github.com/MacPython/nipy-wheels/blob/master/appveyor.yml
  - https://github.com/MacPython/pytables-wheels/blob/master/appveyor.yml
360

361
362
  Note the Windows test customizations etc are inside ``appveyor.yml``,
  and that ``config.sh`` and ``env_vars.sh`` are only for the
363
  Linux/Mac builds on Travis CI.
364

Matthew Brett's avatar
Matthew Brett committed
365
* Make sure your project is set up to build on AppVeyor, and you should now
366
  be ready (for what could be another round of slow debugging).
Matthew Brett's avatar
Matthew Brett committed
367

Hugo's avatar
Hugo committed
368
369
If your project depends on NumPy, you will want to build against the earliest
NumPy that your project supports - see `forward, backward NumPy compatibility
Andrew Murray's avatar
Andrew Murray committed
370
<https://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312>`_.
371
See the `astropy-wheels Travis file
Matthew Brett's avatar
Matthew Brett committed
372
<https://github.com/MacPython/astropy-wheels/blob/master/.travis.yml>`_ for an
Hugo's avatar
Hugo committed
373
example specifying NumPy build and test dependencies.
Matthew Brett's avatar
Matthew Brett committed
374
375
376
377
378
379
380
381
382

Here are some simple example projects:

* https://github.com/MacPython/astropy-wheels
* https://github.com/scikit-image/scikit-image-wheels
* https://github.com/MacPython/nipy-wheels
* https://github.com/MacPython/dipy-wheels

Less simple projects where there are some serious build dependencies, and / or
Hugo's avatar
Hugo committed
383
macOS / Linux differences:
Matthew Brett's avatar
Matthew Brett committed
384
385
386
387

* https://github.com/MacPython/matplotlib-wheels
* https://github.com/python-pillow/Pillow-wheels
* https://github.com/MacPython/h5py-wheels
388
389
390
391
392
393
394
395

**********************
Multibuild development
**********************

The main multibuild repository is always at
https://github.com/matthew-brett/multibuild

396
397
We try to keep the ``master`` branch stable and do testing and development
in the ``devel`` branch.  From time to time we merge ``devel`` into ``master``.
398

399
400
In practice, you can check out the newest commit from ``devel`` that works
for you, then stay at it until you need newer features.