README.rst 24.7 KB
Newer Older
1
################################################
Matthew Brett's avatar
Matthew Brett committed
2
Utilities for building on Travis CI and AppVeyor
3
################################################
4

Matthew Brett's avatar
Matthew Brett committed
5
6
7
************************************
Update: Uploads, Rackspace, Anaconda
************************************
Matthew Brett's avatar
Matthew Brett committed
8
9
10
11
12
13

The original Multibuild default was to upload wheels to a Rackspace container,
where Rackspace kindly donated the hosting to the Scikit-learn team.  We had
a URL pointing to the Rackspace container: `http://wheels.scipy.org`.

Rackspace finally stopped subsidizing this container, and the Rackspace of
Matthew Brett's avatar
Matthew Brett committed
14
15
`http://wheels.scipy.org` is no more. Some projects using Multibuild have moved
to using https://anaconda.org/scipy-wheels-nightly/ for weekly uploads and
Matthew Brett's avatar
Matthew Brett committed
16
17
https://anaconda.org/multibuild-wheels-staging for staging wheels to PyPI.

Matthew Brett's avatar
Matthew Brett committed
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
*******************
Uploads to Anaconda
*******************

If you want to upload to Anaconda, and you don't need the extra storage space for nightly builds that Anaconda kindly donates to Numpy, Scipy etc, then you can do this with your own Anaconda organization.

See `https://github.com/MacPython/nipy-wheels`_ for a simple example.

* Make an account at Anaconda.org.
* Make an organization - for example we have used ``nipy``.
* Navigate to ``https://anaconda.org/nipy`` (but use your organization).
* Go to your account menu towards the top left. This should be labeled with you
  organization name.
* Select "Settings", then "Access".
* Create an access token.  Give it permission: "Allow write access to the API
  site", and (not sure if this is necessary) "Allow uploads to PyPI
  repositories".
* "View" your token.  It may be of form
  ``ni-1234abcd-12ab-34dc-1234-d1e1f3a4b5c6``.
* Encrypt this to your repository, maybe using the ``travis`` command line tool
  (``gem install travis``).  Your command will be something like::

      travis encrypt -r MacPython/nipy-wheels ANACONDA_SECRET=ni-1234abcd-12ab-34dc-1234-d1e1f3a4b5c6

  Note that ``MacPython/nipy-wheels`` is your Github organization/repository.  The encryption only applies to Travis-CI running against this repository.

* Go to your `.travis.yml` file and add the output ``secure`` key.  This will
  look something like::

    env:
      global:
        # Following generated with
        # travis encrypt -r MacPython/nipy-wheels ANACONDA_SECRET=<the api key>
        # where <the api key> has API write access to the anaconda.org
        # organization named in $ANACONDA_ORG
        - secure: "IqN7LjXWVBaijggUoB+ohjzFzH6nU0OyxznXEMgWoNxQJRiYXXKAt/Z5c4ldp9LUynefJO306M8foN4Gm8M8PNDlhjElzdOtIkGYtDKUXx7aXtrg8rPk1mzuM1F27er4Dbi7WFtpPClr8z8JKNNV50yeM1o2cXu4HgrPrRKgKk/2D8EQaPQlcOqul0O63D9AjVoW3EIG0aWEnZQQGfuGAPgyr0OS92LX2h1pcD2lNZHhqYmXmm5U0IwZmWL3Y0N7PO3VXcOCeIbiHAlJzhk4C4+86TT7DN+VhmfGyY/s61fOz47K+lEZLVqqeQki+HV75fti0XwYG7rjcSvDanNx+w2J/ogSLQpiNxZ0FZ+W8psXEaFUgFf7oXzRkW9gQ4KAsItEWHifq061ngr5AWLPLh+01LGP1Xg8wT5WEVUzBfD2uJPsy20DLcP9WGYa6cBNwtpqmUkdVgM3ZCPWlro7+v1kqxsKp91uh8SRKVlkD4mwbf0FnWxbNZ9v4Z9gs0pZoRclzL+/YcIcSTYAwiQRqaX7T0tpxaUZ0VYTMwCgpsufUX1idV1HV5+WKr9FUocoq+1RRW/JeXkisX9FRvem8cSGmnxB/hynlxoqzttCVMwtrKWPwxH4dHD+lavouho68Q7iBql1ZBZEhQy0O9NC1wr4Rg2CeDPZuzqVjmSPuXQ="

  When Travis-CI runs, this causes the ``ANACONDA_SECRET`` environment variable
  to contain the API key above.

  Also add this to your global environment variables::

      - ANACONDA_ORG="nipy"

*  To upload from Travis-CI, add a clause like this to the end of your
   ``.travis.yml``::

     after_success:
         # Upload wheels to Anaconda.org
         - pip install git+https://github.com/Anaconda-Platform/anaconda-project
         - pip install git+https://github.com/Anaconda-Platform/anaconda-client
         - anaconda -t ${ANACONDA_SECRET} upload --force -u ${ANACONDA_ORG} ${TRAVIS_BUILD_DIR}/wheelhouse/*.whl

   ``wheelhouse/*.whl`` defines the files you want to upload.

*   You might also want to build and upload from Appveyor.  To encrypt the API
    key above, go to `https://ci.appveyor.com/account/matthew-brett/settings`_
    (where ``matthew-brett`` is the account from which your Appveyor job runs.
    Click on "Encrypt YaML" on the left.  Type in your API key value (e.g.
    ``ni-1234abcd-12ab-34dc-1234-d1e1f3a4b5c6``) as the value to encrypt. Click "Encrypt" and note the text it suggests.  Now, at the top of your ``appveyor.yml`` file, add something like::

        environment:
          global:
            ANACONDA_ORG: "nipy"
            ANACONDA_SECRET:
              secure: Ds0PkQD0b/QOfoNoiPuFJb01zg0Mq0dkAxIG2jRXocCAereSXdWw6XYaDrutHWme

    where ``secure:`` is the text suggested from "Encrypt" above, and ``nipy`` is your Anaconda organization.

    Finally, add a clause like the following to the end of your ``appveyor.yml`` file::

      on_success:
        # Upload the generated wheel package to Anaconda.org
        - pip install git+https://github.com/Anaconda-Platform/anaconda-project
        - pip install git+https://github.com/Anaconda-Platform/anaconda-client
        - anaconda -t %ANACONDA_SECRET% upload --force -u %ANACONDA_ORG% nipy\dist\*.whl

    where ``nipy\dist\*.whl`` finds the files you want to upload.

There's a simple example of these steps applied at
`https://github.com/MacPython/nipy-wheels`_.

Matthew Brett's avatar
Matthew Brett committed
100
101
Here is the NumPy code (for running on Travis CI) to upload to Anaconda:
https://github.com/MacPython/numpy-wheels/blob/master/.travis.yml#L99
Matthew Brett's avatar
Matthew Brett committed
102

Matthew Brett's avatar
Matthew Brett committed
103
For projects housed under the MacPython GitHub organization, you have access to
Matthew Brett's avatar
Matthew Brett committed
104
105
Anaconda upload tokens via the "Organization Secrets"
https://github.com/MacPython/numexpr-wheels/settings/secrets . You can use
Matthew Brett's avatar
Matthew Brett committed
106
these to move to GitHub Actions (they provide x86 machines for Windows, Linux
Matthew Brett's avatar
Matthew Brett committed
107
and Mac). Otherwise we (please raise an issue here) will need to negotiate
Matthew Brett's avatar
Matthew Brett committed
108
109
110
111
112
113
114
115
116
117
getting you tokens, or you can make your own, as above.

**********************
Use Github for uploads
**********************

Another option is to use GitHub for staging --- as do Cython `for Travis CI
<https://github.com/MacPython/cython-wheels/blob/master/.travis.yml#L144>`_ and
`for AppVeyor
<https://github.com/MacPython/cython-wheels/blob/master/appveyor.yml#L118>`_.
Matthew Brett's avatar
Matthew Brett committed
118
119
120
121
122

************
Introduction
************

Hugo's avatar
Hugo committed
123
A set of scripts to automate builds of macOS and Manylinux1 wheels on the
124
`Travis CI <https://travis-ci.org/>`_ infrastructure, and also Windows
125
wheels on the `AppVeyor <https://ci.appveyor.com/>`_ infrastructure.
Matthew Brett's avatar
Matthew Brett committed
126

127
The Travis CI scripts are designed to build *and test*:
Matthew Brett's avatar
Matthew Brett committed
128

129
* 64-bit macOS wheels built for macOS 10.9+
robbuckley's avatar
robbuckley committed
130
* 64/32-bit macOS wheels built for macOS 10.6+
131
132
* 64-bit ``manylinuxX_x86_64`` wheels, both narrow and wide Unicode builds, where `X` is any valid Manylinux version, such as `1`, or `2010`
* 32-bit ``manylinuxX_i686`` wheels, both narrow and wide Unicode builds
Matthew Brett's avatar
Matthew Brett committed
133

134
You can currently build and test against Pythons 2.7, 3.5, 3.6, 3.7, 3.8 and 3.9
Matthew Brett's avatar
Matthew Brett committed
135

robbuckley's avatar
robbuckley committed
136
The small innovation here is that you can test against Linux 32-bit builds, both
137
138
wide and narrow Unicode Python 2 builds, which was not easy on the default
Travis CI configurations.
Matthew Brett's avatar
Matthew Brett committed
139

140
141
The AppVeyor setup is designed to build *and test*:

142
143
* 64-bit Windows ``win_amd64`` wheels
* 32-bit Windows ``win32`` wheels
144

mattip's avatar
mattip committed
145
You can currently build and test against Pythons 2.7, 3.5, 3.6, 3.7, 3.8
146

147
148
149
150
151
152
153
154
155
156
*****************
How does it work?
*****************

Multibuild is a series of bash scripts that define bash functions to build and
test wheels.

Configuration is by overriding the default build function, and defining a test
function.

robbuckley's avatar
robbuckley committed
157
The bash scripts are layered, in the sense that they are composed of a number of scripts
Rob Buckley's avatar
Rob Buckley committed
158
which are sourced in sequence, each one potentially overriding previous ones.
159

Hugo's avatar
Hugo committed
160
161
macOS
=====
162

Rob Buckley's avatar
Rob Buckley committed
163
The following bash scripts are sourced in this order::
164

robbuckley's avatar
readme  
robbuckley committed
165
166
167
168
169
170
    multibuild/common_utils.sh
    multibuild/osx_utils.sh
    env_vars.sh
    multibuild/configure_build.sh
    multibuild/library_builders.sh
    config.sh
171

Rob Buckley's avatar
Rob Buckley committed
172
See ``multibuild/travis_osx_steps.sh``
173

robbuckley's avatar
robbuckley committed
174
The macOS build / test phases run on the macOS VM started by Travis CI.
robbuckley's avatar
readme  
robbuckley committed
175
Therefore any environment variable defined in ``.travis.yml`` or the bash
Matthew Brett's avatar
Matthew Brett committed
176
shell scripts listed above are available for your build and test.
177

robbuckley's avatar
readme  
robbuckley committed
178
179
Build options are controlled mainly by the following environment
variables:
180

robbuckley's avatar
robbuckley committed
181
* ``MB_PYTHON_VER`` sets the Python version targetted: ``major.minor.patch`` for CPython, or ``pypy-major.minor`` for PyPy.
robbuckley's avatar
robbuckley committed
182
* ``MB_PYTHON_OSX_VER`` sets the minimum macOS SDK version for any C extensions. For CPython targets it may be set to 10.6 or 10.9, provided a corresponding Python build is available at `python.org <https://www.python.org/downloads/mac-osx/>`_. It defaults to the highest version available. It's ignored for PyPy targets.
robbuckley's avatar
robbuckley committed
183
* ``PLAT`` sets the architectures built for any C extensions: ``x86_64`` or ``intel`` for 64-bit or 64/32-bit respectively. It defaults to the same arches as the target Python version: 64-bit for CPython macOS 10.9 or PyPy, and 64/32-bit for CPython 10.6.
184

185
In most cases it's best to rely on the defaults for ``MB_PYTHON_OSX_VER`` and ``PLAT``, rather than setting them explicitly. Examples of exceptions to this guideline include:
robbuckley's avatar
robbuckley committed
186

robbuckley's avatar
robbuckley committed
187
* setting ``MB_PYTHON_OSX_VER=10.6`` to build a 10.6 64/32-bit CPython wheel for Python 2.7 (default for 2.7 is 10.9 64-bit)
robbuckley's avatar
robbuckley committed
188
* setting ``MB_PYTHON_OSX_VER=10.6 and PLAT=x86_64`` to build a 10.6 64-bit only wheel (10.6 would normally be 64/32-bit). Such a wheel would still have a platform tag of ``macosx_10_6_intel`` , advertising support for both 64 and 32-bit, but wouldnt work in 32-bit mode. This may be OK given how unlikely it is that there is still anyone actually running Python on macOS in 32-bit mode.
189

Rob Buckley's avatar
Rob Buckley committed
190
191
The ``build_wheel`` function builds the wheel, and ``install_run``
function installs and tests it.  Look in ``multibuild/common_utils.sh`` for
Rob Buckley's avatar
Rob Buckley committed
192
193
default definitions of these functions.  See below for more details, many of which are common
to macOS and Linux.
194

Matthew Brett's avatar
Matthew Brett committed
195
196
Manylinux
=========
197

198
The build phase is in a Manylinux Docker container, but the test phase is in
mattip's avatar
mattip committed
199
a clean container.
200

201

Matthew Brett's avatar
Matthew Brett committed
202
203
Build phase
-----------
204

205
Specify the Manylinux version to build for with the ``MB_ML_VER`` environment
206
variable. The default version is ``1``.  Versions that are currently valid are:
207

Hugo van Kemenade's avatar
Hugo van Kemenade committed
208
209
* ``1`` corresponding to manylinux1 (see `PEP 513 <https://www.python.org/dev/peps/pep-0513>`_).
* ``2010``  corresponding to manylinux2010 (see `PEP 571 <https://www.python.org/dev/peps/pep-0571>`_).
210
* ``2014`` corresponding to manylinux2014 and adds more architectures to ``PLAT``
Hugo van Kemenade's avatar
Hugo van Kemenade committed
211
  (see `PEP 599 <https://www.python.org/dev/peps/pep-0599>`_).
212
213
214

The environment variable specified which Manylinux docker container you are building in.

215
216
217
218
219
220
221
222
223
224
225
The ``PLAT`` environment variable can be one of

* ``x86_64``, for 64-bit x86
* ``i686``, for 32-bit x86
* ``s390x``, for 64-bit s390x
* ``ppc64le``, for PowerPC
* ``aarch64``, for ARM
* ``arm64``, for Apple silicon
* ``universal2``, for both Apple silicon and 64-bit x86

The default is ``x86_64``. Only ``x86_64`` and ``i686`` are valid on manylinux1 and manylinux2010.
226

Matthew Brett's avatar
Matthew Brett committed
227
``multibuild/travis_linux_steps.sh`` defines the ``build_wheel`` function,
Hugo's avatar
Hugo committed
228
which starts up the Manylinux1 Docker container to run a wrapper script
Matthew Brett's avatar
Matthew Brett committed
229
``multibuild/docker_build_wrap.sh``, that (within the container) sources the
Rob Buckley's avatar
Rob Buckley committed
230
following bash scripts::
Matthew Brett's avatar
Matthew Brett committed
231

Rob Buckley's avatar
Rob Buckley committed
232
233
234
235
236
237
    multibuild/common_utils.sh
    multibuild/manylinux_utils.sh
    env_vars.sh
    multibuild/configure_build.sh
    multibuild/library_builders.sh
    config.sh
Matthew Brett's avatar
Matthew Brett committed
238

239
240
See ``docker_build_wrap.sh`` to review the order of script sourcing.

241
242
See the definition of ``build_multilinux`` in
``multibuild/travis_linux_steps.sh`` for the environment variables passed from
243
Travis CI to the Manylinux1 container.
Matthew Brett's avatar
Matthew Brett committed
244
245
246
247
248
249
250
251

Once in the container, after sourcing the scripts above, the wrapper runs the
real ``build_wheel`` function, which now comes (by default) from
``multibuild/common_utils.sh``.

Test phase
----------

252
253
254
255
256
257
258
259
260
261
262
263
Specify the version to test with the ``DOCKER_TEST_IMAGE`` environment
variable. The default version is dependent on ``PLAT``:

* ``matthewbrett/trusty:64``, for ``x86_64``
* ``matthewbrett/trusty:32`` for ``i686``
* ``multibuild/xenial_arm64v8`` for ``aarch64``
* ``multibuild/xenial_ppc64le`` for ``ppc64le``
* ``mutlibuild/xenial_s390x`` for ``s390x``

Other valid values are any in https://hub.docker.com/orgs/multibuild/repositories,
using the correct platform code. Alternatively, you can use the substitution
pattern ``multibuild/xenial_{PLAT}`` in the ``.travis.yml`` file.
mattip's avatar
mattip committed
264
265
266
267
268
269
270

See ``multibuild/docker_test_wrap.sh``.

``multibuild/travis_linux_steps.sh`` defines the ``install_run`` function,
which starts up the testing Docker container with the wrapper script
``multibuild/docker_test_wrap.sh``. The wrapper script sources the following
bash scripts::
Matthew Brett's avatar
Matthew Brett committed
271

Rob Buckley's avatar
Rob Buckley committed
272
273
    multibuild/common_utils.sh
    config.sh
Matthew Brett's avatar
Matthew Brett committed
274

275
276
See ``docker_test_wrap.sh`` for script source order.

Matthew Brett's avatar
Matthew Brett committed
277
278
279
280
281
See ``install_run`` in ``multibuild/travis_linux_steps.sh`` for the
environment variables passed into the container.

It then (in the container) runs the real ``install_run`` command, which comes
(by default) from ``multibuild/common_utils.sh``.
282
283
284
285
286

*********************************
Standard build and test functions
*********************************

Andrew Murray's avatar
Andrew Murray committed
287
The standard build command is ``build_wheel``.  This is a bash function.  By
Hugo's avatar
Hugo committed
288
default the function that is run on macOS, and in the Manylinux container for
Matthew Brett's avatar
Matthew Brett committed
289
290
291
the build phase, is defined in ``multibuild/common_utils.sh``.  You can
override the default function in the project ``config.sh`` file (see below).

robbuckley's avatar
typos  
robbuckley committed
292
If you are building a wheel from PyPI, rather than from a source repository,
293
294
295
296
297
298
299
you can use the ``build_index_wheel`` command, again defined in
``multibuild/common_utils.sh``.

Typically, you can get away with leaving the default ``build_wheel`` /
``build_index_wheel`` functions to do their thing, but you may need to define
a ``pre_build`` function in ``config.sh``.  The default ``build_wheel`` and
``build_index_wheel`` functions will call the ``pre_build`` function, if
300
301
302
defined, before building the wheel, so ``pre_build`` is a good place to build
any required libraries.

Matthew Brett's avatar
Matthew Brett committed
303
The standard test command is the bash function ``install_run``.  The version
Hugo's avatar
Hugo committed
304
run on macOS and in the Linux testing container is also defined in
Matthew Brett's avatar
Matthew Brett committed
305
``multibuild/common_utils.sh``.  Typically, you do not override this function,
306
307
308
309
but you in that case you will need to define a ``run_tests`` function, to run
your tests, returning a non-zero error code for failure.  The default
``install_run`` implementation calls the ``run_tests`` function, which you
will likely define in ``config.sh``.  See the examples below for examples of
Andrew Murray's avatar
Andrew Murray committed
310
less and more complicated builds, where the complicated builds override more
311
of the default implementations.
312
313
314
315

********************
To use these scripts
********************
Matthew Brett's avatar
Matthew Brett committed
316

317
* Make a repository for building wheels on Travis CI - e.g.
Matthew Brett's avatar
Matthew Brett committed
318
319
320
321
322
323
324
325
326
327
328
329
330
331
  https://github.com/MacPython/astropy-wheels - or in your case maybe
  ``https://github.com/your-org/your-project-wheels``;

* Add this (here) repository as a submodule::

    git submodule add https://github.com/matthew-brett/multibuild.git

* Add your own project repository as another submodule::

    git submodule add https://github.com/your-org/your-project.git

* Create a ``.travis.yml`` file, something like this::

    env:
332
333
334
335
336
        global:
            - REPO_DIR=your-project
            # Commit from your-project that you want to build
            - BUILD_COMMIT=v0.1.0
            # pip dependencies to _build_ your project
robbuckley's avatar
typos  
robbuckley committed
337
            - BUILD_DEPENDS="cython numpy"
338
339
340
            # pip dependencies to _test_ your project.  Include any dependencies
            # that you need, that are also specified in BUILD_DEPENDS, this will be
            # a separate install.
Matthew Brett's avatar
Matthew Brett committed
341
342
            # Now see the Uploads section for the stuff you need to
            # upload your wheels after CI has built them.
Matthew Brett's avatar
Matthew Brett committed
343

344
345
346
347
348
349
350
    # You will likely prefer "language: generic" for travis configuration,
    # rather than, say "language: python". Multibuild doesn't use
    # Travis-provided Python but rather installs and uses its own, where the
    # Python version is set from the MB_PYTHON_VERSION variable. You can still
    # specify a language here if you need it for some unrelated logic and you
    # can't use Multibuild-provided Python or other software present on a
    # builder.
351
    language: generic
352
353
354

    # For CPython macOS builds only, the minimum supported macOS version and
    # architectures of any C extensions in the wheel are set with the variable
Rob Buckley's avatar
Rob Buckley committed
355
356
357
358
359
    # MB_PYTHON_OSX_VER: 10.9 (64-bit only) or 10.6 (64/32-bit dual arch). By
    # default this is set to the highest available for the Python version selected
    # using MB_PYTHON_VERSION. You should only need to set this explicitly if you
    # are building a 10.6 dual-arch build for a CPython version where both a 10.9 and
    # 10.6 build are available (for example, 2.7 or 3.7).
360
361
    # All PyPy macOS builds are 64-bit only.

Andrew Murray's avatar
Andrew Murray committed
362
    # Required in Linux to invoke `docker` ourselves
Matthew Brett's avatar
Matthew Brett committed
363
364
    services: docker

365
366
367
    # Host distribution.  This is the distribution from which we run the build
    # and test containers, via docker.
    dist: xenial
Matthew Brett's avatar
Matthew Brett committed
368

Isuru Fernando's avatar
Isuru Fernando committed
369
370
    # osx image that enables building Apple silicon libraries
    osx_image: xcode12.2
371

Matthew Brett's avatar
Matthew Brett committed
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
    matrix:
      include:
        - os: linux
          env: MB_PYTHON_VERSION=2.7
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - UNICODE_WIDTH=16
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - PLAT=i686
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - PLAT=i686
            - UNICODE_WIDTH=16
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.5
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.5
            - PLAT=i686
riddell1's avatar
riddell1 committed
396
397
398
399
400
401
402
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.6
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.6
            - PLAT=i686
Matthew Brett's avatar
Matthew Brett committed
403
404
405
        - os: osx
          env:
            - MB_PYTHON_VERSION=2.7
Rob Buckley's avatar
Rob Buckley committed
406
            - MB_PYTHON_OSX_VER=10.6
robbuckley's avatar
readme  
robbuckley committed
407
408
409
        - os: osx
          env:
            - MB_PYTHON_VERSION=2.7
Matthew Brett's avatar
Matthew Brett committed
410
411
412
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.5
riddell1's avatar
riddell1 committed
413
414
415
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.6
Kyle Stewart's avatar
Kyle Stewart committed
416
        - os: osx
417
          env:
418
            - MB_PYTHON_VERSION=3.7
Rob Buckley's avatar
Rob Buckley committed
419
420
421
422
423
424
425
            - MB_PYTHON_OSX_VER=10.6
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.7
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.8
426
427
428
429
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.9
            - PLAT="universal2"
430
431
432
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.9
Kyle Stewart's avatar
Kyle Stewart committed
433
434
435
        - os: osx
          language: generic
          env:
436
            - MB_PYTHON_VERSION=pypy-5.7
Matthew Brett's avatar
Matthew Brett committed
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451

    before_install:
        - source multibuild/common_utils.sh
        - source multibuild/travis_steps.sh
        - before_install

    install:
        # Maybe get and clean and patch source
        - clean_code $REPO_DIR $BUILD_COMMIT
        - build_wheel $REPO_DIR $PLAT

    script:
        - install_run $PLAT

    after_success:
Matthew Brett's avatar
Matthew Brett committed
452
453
        # Here you should put the code to upload your wheels
        # See the Uploads section for more details.
Matthew Brett's avatar
Matthew Brett committed
454

Hugo's avatar
Hugo committed
455
  The example above is for a project building from a Git submodule.  If you
456
457
458
459
460
461
462
463
464
465
466
467
468
469
  aren't building from a submodule, but want to use ``pip`` to build from a
  source archive on https://pypi.org or similar, replace the first few lines
  of the ``.travis.yml`` file with something like::

    env:
        global:
            # Instead of REPO_DIR, BUILD_COMMIT
            - PROJECT_SPEC="tornado==4.1.1"

  then your ``install`` section could look something like this::

    install:
        - build_index_wheel $PROJECT_SPEC

470

Matthew Brett's avatar
Matthew Brett committed
471
472
473
474
475
* Next create a ``config.sh`` for your project, that fills in any steps you
  need to do before building the wheel (such as building required libraries).
  You also need this file to specify how to run your tests::

    # Define custom utilities
Andrew Murray's avatar
Andrew Murray committed
476
    # Test for macOS with [ -n "$IS_MACOS" ]
Matthew Brett's avatar
Matthew Brett committed
477
478
479
480
481
482
483
484
485
486
487
488
489

    function pre_build {
        # Any stuff that you need to do before you start building the wheels
        # Runs in the root directory of this repository.
        :
    }

    function run_tests {
        # Runs tests on installed distribution from an empty directory
        python --version
        python -c 'import sys; import yourpackage; sys.exit(yourpackage.test())'
    }

490
491
  Optionally you can specify a different location for ``config.sh`` file with
  the ``$CONFIG_PATH`` environment variable.
492

493
* Optionally, create an ``env_vars.sh`` file to override the defaults for any
494
495
496
  environment variables used by
  ``configure_build.sh``/``library_builders.sh``. In Linux, the environment
  variables used for the build cannot be set in the ``.travis.yml`` file,
497
  because the build processing runs in a Docker container, so the only
498
499
500
501
502
503
504
505
  environment variables that reach the container are those passed in via the
  ``docker run`` command, or those set in ``env_vars.sh``.

  As for the ``config.sh`` file, you can specify a different location for the
  file by setting the ``$ENV_VARS_PATH`` environment variable.  The path in
  ``$ENV_VARS_PATH`` is relative to the repository root directory.  For
  example, if your repository had a subdirectory ``scripts`` with a file
  ``my_env_vars.sh``, you should set ``ENV_VARS_PATH=scripts/my_env_vars.sh``.
506

507
* Make sure your project is set up to build on Travis CI, and you should now
Matthew Brett's avatar
Matthew Brett committed
508
  be ready (to begin the long slow debugging process, probably).
509

510
511
512
513
514
* For the Windows wheels, create an ``appveyor.yml`` file, something like:

  - https://github.com/MacPython/astropy-wheels/blob/master/appveyor.yml
  - https://github.com/MacPython/nipy-wheels/blob/master/appveyor.yml
  - https://github.com/MacPython/pytables-wheels/blob/master/appveyor.yml
515

516
517
  Note the Windows test customizations etc are inside ``appveyor.yml``,
  and that ``config.sh`` and ``env_vars.sh`` are only for the
518
  Linux/Mac builds on Travis CI.
519

Matthew Brett's avatar
Matthew Brett committed
520
* Make sure your project is set up to build on AppVeyor, and you should now
521
  be ready (for what could be another round of slow debugging).
Matthew Brett's avatar
Matthew Brett committed
522

523
524
525
* For Apple silicon support you can either create an ``arm64`` wheel or
  a ``universal2`` wheel by supplying ``PLAT`` env variable.
  ``universal2`` builds work on both ``arm64`` and ``x86_64`` platforms
526
527
  and also make it possible for the wheel code to work when switching the
  architecture on Apple silicon machines where ``x86_64`` can be run
528
529
530
  using Rosetta2 emulation.

  There are two ways to build ``universal2`` builds.
Isuru Fernando's avatar
Isuru Fernando committed
531

532
533
  1. Build with ``-arch x86_64 -arch arm64``.
     These flags instruct the C/C++ compiler to compile twice and create a
534
     fat object/executable/library. This is the easiest, but has several
535
536
537
538
539
540
541
     drawbacks. If you are using C/C++ libraries that are built using
     library_builders, it's highly likely that they don't build correctly
     because most build systems and packages don't support building fat binaries.
     We could possibly build them separately and fuse them, but the headers might
     not be identical which is required when building the wheel as a ``universal2``
     wheel. If you are using Fortran, ``gfortran`` doesn't support fat binaries.

Andrew Murray's avatar
Andrew Murray committed
542
  2. Build ``arm64`` and ``x86_64`` wheels separately and fuse them.
543
544
545
546
547
548
     For this to work, we need to build the C/C++ libraries twice. Therefore,
     the library building is once called with ``BUILD_PREFIX=${BUILD_PREFIX:-/usr/local}``
     for ``x86_64`` and then called again with ``BUILD_PREFIX=/opt/arm64-builds``.
     Once the two wheels are created, these two are merged. Both the
     ``arm64`` and ``universal2`` wheels are outputs for this build.

549
550
  In multibuild we are going with option 2. You can override this behaviour by
  overriding the function ``wrap_wheel_builder``.
551
552
553
  To build Apple silicon builds, you should use a CI service with Xcode 12 with
  universal build support and make sure that xcode is the default.

Hugo's avatar
Hugo committed
554
555
If your project depends on NumPy, you will want to build against the earliest
NumPy that your project supports - see `forward, backward NumPy compatibility
Andrew Murray's avatar
Andrew Murray committed
556
<https://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312>`_.
557
See the `astropy-wheels Travis file
Matthew Brett's avatar
Matthew Brett committed
558
<https://github.com/MacPython/astropy-wheels/blob/master/.travis.yml>`_ for an
Hugo's avatar
Hugo committed
559
example specifying NumPy build and test dependencies.
Matthew Brett's avatar
Matthew Brett committed
560
561
562
563
564
565
566
567
568

Here are some simple example projects:

* https://github.com/MacPython/astropy-wheels
* https://github.com/scikit-image/scikit-image-wheels
* https://github.com/MacPython/nipy-wheels
* https://github.com/MacPython/dipy-wheels

Less simple projects where there are some serious build dependencies, and / or
Hugo's avatar
Hugo committed
569
macOS / Linux differences:
Matthew Brett's avatar
Matthew Brett committed
570
571
572
573

* https://github.com/MacPython/matplotlib-wheels
* https://github.com/python-pillow/Pillow-wheels
* https://github.com/MacPython/h5py-wheels
574
575
576
577
578
579
580
581

**********************
Multibuild development
**********************

The main multibuild repository is always at
https://github.com/matthew-brett/multibuild

582
583
We try to keep the ``master`` branch stable and do testing and development
in the ``devel`` branch.  From time to time we merge ``devel`` into ``master``.
584

585
586
In practice, you can check out the newest commit from ``devel`` that works
for you, then stay at it until you need newer features.