README.rst 24.9 KB
Newer Older
1
################################################
Matthew Brett's avatar
Matthew Brett committed
2
Utilities for building on Travis CI and AppVeyor
3
################################################
4

Matthew Brett's avatar
Matthew Brett committed
5
6
7
************************************
Update: Uploads, Rackspace, Anaconda
************************************
Matthew Brett's avatar
Matthew Brett committed
8
9
10
11
12
13

The original Multibuild default was to upload wheels to a Rackspace container,
where Rackspace kindly donated the hosting to the Scikit-learn team.  We had
a URL pointing to the Rackspace container: `http://wheels.scipy.org`.

Rackspace finally stopped subsidizing this container, and the Rackspace of
Matthew Brett's avatar
Matthew Brett committed
14
15
`http://wheels.scipy.org` is no more. Some projects using Multibuild have moved
to using https://anaconda.org/scipy-wheels-nightly/ for weekly uploads and
Matthew Brett's avatar
Matthew Brett committed
16
17
https://anaconda.org/multibuild-wheels-staging for staging wheels to PyPI.

Matthew Brett's avatar
Matthew Brett committed
18
19
20
21
22
23
*******************
Uploads to Anaconda
*******************

If you want to upload to Anaconda, and you don't need the extra storage space for nightly builds that Anaconda kindly donates to Numpy, Scipy etc, then you can do this with your own Anaconda organization.

Andrew Murray's avatar
Andrew Murray committed
24
See https://github.com/MacPython/nipy-wheels for a simple example.
Matthew Brett's avatar
Matthew Brett committed
25
26
27
28

* Make an account at Anaconda.org.
* Make an organization - for example we have used ``nipy``.
* Navigate to ``https://anaconda.org/nipy`` (but use your organization).
Andrew Murray's avatar
Andrew Murray committed
29
* Go to your account menu towards the top left. This should be labeled with your
Matthew Brett's avatar
Matthew Brett committed
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
  organization name.
* Select "Settings", then "Access".
* Create an access token.  Give it permission: "Allow write access to the API
  site", and (not sure if this is necessary) "Allow uploads to PyPI
  repositories".
* "View" your token.  It may be of form
  ``ni-1234abcd-12ab-34dc-1234-d1e1f3a4b5c6``.
* Encrypt this to your repository, maybe using the ``travis`` command line tool
  (``gem install travis``).  Your command will be something like::

      travis encrypt -r MacPython/nipy-wheels ANACONDA_SECRET=ni-1234abcd-12ab-34dc-1234-d1e1f3a4b5c6

  Note that ``MacPython/nipy-wheels`` is your Github organization/repository.  The encryption only applies to Travis-CI running against this repository.

* Go to your `.travis.yml` file and add the output ``secure`` key.  This will
  look something like::

    env:
      global:
        # Following generated with
        # travis encrypt -r MacPython/nipy-wheels ANACONDA_SECRET=<the api key>
        # where <the api key> has API write access to the anaconda.org
        # organization named in $ANACONDA_ORG
        - secure: "IqN7LjXWVBaijggUoB+ohjzFzH6nU0OyxznXEMgWoNxQJRiYXXKAt/Z5c4ldp9LUynefJO306M8foN4Gm8M8PNDlhjElzdOtIkGYtDKUXx7aXtrg8rPk1mzuM1F27er4Dbi7WFtpPClr8z8JKNNV50yeM1o2cXu4HgrPrRKgKk/2D8EQaPQlcOqul0O63D9AjVoW3EIG0aWEnZQQGfuGAPgyr0OS92LX2h1pcD2lNZHhqYmXmm5U0IwZmWL3Y0N7PO3VXcOCeIbiHAlJzhk4C4+86TT7DN+VhmfGyY/s61fOz47K+lEZLVqqeQki+HV75fti0XwYG7rjcSvDanNx+w2J/ogSLQpiNxZ0FZ+W8psXEaFUgFf7oXzRkW9gQ4KAsItEWHifq061ngr5AWLPLh+01LGP1Xg8wT5WEVUzBfD2uJPsy20DLcP9WGYa6cBNwtpqmUkdVgM3ZCPWlro7+v1kqxsKp91uh8SRKVlkD4mwbf0FnWxbNZ9v4Z9gs0pZoRclzL+/YcIcSTYAwiQRqaX7T0tpxaUZ0VYTMwCgpsufUX1idV1HV5+WKr9FUocoq+1RRW/JeXkisX9FRvem8cSGmnxB/hynlxoqzttCVMwtrKWPwxH4dHD+lavouho68Q7iBql1ZBZEhQy0O9NC1wr4Rg2CeDPZuzqVjmSPuXQ="

  When Travis-CI runs, this causes the ``ANACONDA_SECRET`` environment variable
  to contain the API key above.

  Also add this to your global environment variables::

      - ANACONDA_ORG="nipy"

*  To upload from Travis-CI, add a clause like this to the end of your
   ``.travis.yml``::

     after_success:
         # Upload wheels to Anaconda.org
         - pip install git+https://github.com/Anaconda-Platform/anaconda-project
         - pip install git+https://github.com/Anaconda-Platform/anaconda-client
         - anaconda -t ${ANACONDA_SECRET} upload --force -u ${ANACONDA_ORG} ${TRAVIS_BUILD_DIR}/wheelhouse/*.whl

   ``wheelhouse/*.whl`` defines the files you want to upload.

Hugo van Kemenade's avatar
Hugo van Kemenade committed
73
*   You might also want to build and upload from AppVeyor.  To encrypt the API
Andrew Murray's avatar
Andrew Murray committed
74
    key above, go to https://ci.appveyor.com/account/matthew-brett/settings
Hugo van Kemenade's avatar
Hugo van Kemenade committed
75
    (where ``matthew-brett`` is the account from which your AppVeyor job runs.
Matthew Brett's avatar
Matthew Brett committed
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
    Click on "Encrypt YaML" on the left.  Type in your API key value (e.g.
    ``ni-1234abcd-12ab-34dc-1234-d1e1f3a4b5c6``) as the value to encrypt. Click "Encrypt" and note the text it suggests.  Now, at the top of your ``appveyor.yml`` file, add something like::

        environment:
          global:
            ANACONDA_ORG: "nipy"
            ANACONDA_SECRET:
              secure: Ds0PkQD0b/QOfoNoiPuFJb01zg0Mq0dkAxIG2jRXocCAereSXdWw6XYaDrutHWme

    where ``secure:`` is the text suggested from "Encrypt" above, and ``nipy`` is your Anaconda organization.

    Finally, add a clause like the following to the end of your ``appveyor.yml`` file::

      on_success:
        # Upload the generated wheel package to Anaconda.org
        - pip install git+https://github.com/Anaconda-Platform/anaconda-project
        - pip install git+https://github.com/Anaconda-Platform/anaconda-client
        - anaconda -t %ANACONDA_SECRET% upload --force -u %ANACONDA_ORG% nipy\dist\*.whl

    where ``nipy\dist\*.whl`` finds the files you want to upload.

There's a simple example of these steps applied at
Andrew Murray's avatar
Andrew Murray committed
98
https://github.com/MacPython/nipy-wheels.
Matthew Brett's avatar
Matthew Brett committed
99

Matthew Brett's avatar
Matthew Brett committed
100
101
Here is the NumPy code (for running on Travis CI) to upload to Anaconda:
https://github.com/MacPython/numpy-wheels/blob/master/.travis.yml#L99
Matthew Brett's avatar
Matthew Brett committed
102

Matthew Brett's avatar
Matthew Brett committed
103
For projects housed under the MacPython GitHub organization, you have access to
Matthew Brett's avatar
Matthew Brett committed
104
105
Anaconda upload tokens via the "Organization Secrets"
https://github.com/MacPython/numexpr-wheels/settings/secrets . You can use
Matthew Brett's avatar
Matthew Brett committed
106
these to move to GitHub Actions (they provide x86 machines for Windows, Linux
Matthew Brett's avatar
Matthew Brett committed
107
and Mac). Otherwise we (please raise an issue here) will need to negotiate
Matthew Brett's avatar
Matthew Brett committed
108
109
110
111
112
113
114
115
116
117
getting you tokens, or you can make your own, as above.

**********************
Use Github for uploads
**********************

Another option is to use GitHub for staging --- as do Cython `for Travis CI
<https://github.com/MacPython/cython-wheels/blob/master/.travis.yml#L144>`_ and
`for AppVeyor
<https://github.com/MacPython/cython-wheels/blob/master/appveyor.yml#L118>`_.
Matthew Brett's avatar
Matthew Brett committed
118
119
120
121
122

************
Introduction
************

Hugo's avatar
Hugo committed
123
A set of scripts to automate builds of macOS and Manylinux1 wheels on the
124
`Travis CI <https://travis-ci.org/>`_ infrastructure, and also Windows
125
wheels on the `AppVeyor <https://ci.appveyor.com/>`_ infrastructure.
Matthew Brett's avatar
Matthew Brett committed
126

127
The Travis CI scripts are designed to build *and test*:
Matthew Brett's avatar
Matthew Brett committed
128

129
* 64-bit macOS wheels built for macOS 10.9+
robbuckley's avatar
robbuckley committed
130
* 64/32-bit macOS wheels built for macOS 10.6+
mattip's avatar
mattip committed
131
* 64-bit ``manylinuxX_x86_64`` wheels, both narrow and wide Unicode builds,
132
  where `X` is any valid Manylinux version: `1`, `2010`, `2014`, `_2_24` or `_2_28`.
133
* 32-bit ``manylinuxX_i686`` wheels, both narrow and wide Unicode builds
Matthew Brett's avatar
Matthew Brett committed
134

135
You can currently build and test against Pythons 2.7, 3.5, 3.6, 3.7, 3.8 and 3.9
Matthew Brett's avatar
Matthew Brett committed
136

robbuckley's avatar
robbuckley committed
137
The small innovation here is that you can test against Linux 32-bit builds, both
138
139
wide and narrow Unicode Python 2 builds, which was not easy on the default
Travis CI configurations.
Matthew Brett's avatar
Matthew Brett committed
140

141
142
The AppVeyor setup is designed to build *and test*:

143
144
* 64-bit Windows ``win_amd64`` wheels
* 32-bit Windows ``win32`` wheels
145

mattip's avatar
mattip committed
146
You can currently build and test against Pythons 2.7, 3.5, 3.6, 3.7, 3.8, 3.9
147

148
149
150
151
152
153
154
155
156
157
*****************
How does it work?
*****************

Multibuild is a series of bash scripts that define bash functions to build and
test wheels.

Configuration is by overriding the default build function, and defining a test
function.

robbuckley's avatar
robbuckley committed
158
The bash scripts are layered, in the sense that they are composed of a number of scripts
Rob Buckley's avatar
Rob Buckley committed
159
which are sourced in sequence, each one potentially overriding previous ones.
160

Hugo's avatar
Hugo committed
161
162
macOS
=====
163

Rob Buckley's avatar
Rob Buckley committed
164
The following bash scripts are sourced in this order::
165

robbuckley's avatar
readme  
robbuckley committed
166
167
168
169
170
171
    multibuild/common_utils.sh
    multibuild/osx_utils.sh
    env_vars.sh
    multibuild/configure_build.sh
    multibuild/library_builders.sh
    config.sh
172

Rob Buckley's avatar
Rob Buckley committed
173
See ``multibuild/travis_osx_steps.sh``
174

robbuckley's avatar
robbuckley committed
175
The macOS build / test phases run on the macOS VM started by Travis CI.
robbuckley's avatar
readme  
robbuckley committed
176
Therefore any environment variable defined in ``.travis.yml`` or the bash
Matthew Brett's avatar
Matthew Brett committed
177
shell scripts listed above are available for your build and test.
178

robbuckley's avatar
readme  
robbuckley committed
179
180
Build options are controlled mainly by the following environment
variables:
181

182
* ``MB_PYTHON_VER`` sets the Python version targeted: ``major.minor.patch``
mattip's avatar
mattip committed
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
  for CPython, or ``pypy-major.minor`` for PyPy.
* ``MB_PYTHON_OSX_VER`` sets the minimum macOS SDK version for any C
  extensions. For CPython targets it may be set to 10.6 or 10.9, provided a
  corresponding Python build is available at `python.org
  <https://www.python.org/downloads/mac-osx/>`_. It defaults to the highest
  version available. It's ignored for PyPy targets.
* ``PLAT`` sets the architectures built for any C extensions: ``x86_64`` or
  ``intel`` for 64-bit or 64/32-bit respectively. It defaults to the same
  arches as the target Python version: 64-bit for CPython macOS 10.9 or PyPy,
  and 64/32-bit for CPython 10.6.

In most cases it's best to rely on the defaults for ``MB_PYTHON_OSX_VER`` and
``PLAT``, rather than setting them explicitly. Examples of exceptions to this
guideline include:

* setting ``MB_PYTHON_OSX_VER=10.6`` to build a 10.6 64/32-bit CPython wheel
  for Python 2.7 (default for 2.7 is 10.9 64-bit)
* setting ``MB_PYTHON_OSX_VER=10.6 and PLAT=x86_64`` to build a 10.6 64-bit
  only wheel (10.6 would normally be 64/32-bit). Such a wheel would still have
  a platform tag of ``macosx_10_6_intel`` , advertising support for both 64 and
mattip's avatar
mattip committed
203
  32-bit, but wouldn't work in 32-bit mode. This may be OK given how unlikely it
mattip's avatar
mattip committed
204
205
  is that there is still anyone actually running Python on macOS in 32-bit
  mode.
206

Rob Buckley's avatar
Rob Buckley committed
207
208
The ``build_wheel`` function builds the wheel, and ``install_run``
function installs and tests it.  Look in ``multibuild/common_utils.sh`` for
mattip's avatar
mattip committed
209
210
default definitions of these functions.  See below for more details, many of
which are common to macOS and Linux.
211

Matthew Brett's avatar
Matthew Brett committed
212
213
Manylinux
=========
214

215
The build phase is in a Manylinux Docker container, but the test phase is in
mattip's avatar
mattip committed
216
a clean container.
217

218

Matthew Brett's avatar
Matthew Brett committed
219
220
Build phase
-----------
221

222
Specify the Manylinux version to build for with the ``MB_ML_VER`` environment
223
variable. The default version is ``2014``.  Versions that are currently valid are:
224

Hugo van Kemenade's avatar
Hugo van Kemenade committed
225
226
* ``1`` corresponding to manylinux1 (see `PEP 513 <https://www.python.org/dev/peps/pep-0513>`_).
* ``2010``  corresponding to manylinux2010 (see `PEP 571 <https://www.python.org/dev/peps/pep-0571>`_).
227
* ``2014`` corresponding to manylinux2014 and adds more architectures to ``PLAT``
Hugo van Kemenade's avatar
Hugo van Kemenade committed
228
  (see `PEP 599 <https://www.python.org/dev/peps/pep-0599>`_).
mattip's avatar
mattip committed
229
* ``_2_24`` corresponding to manylinux_2_24 (see `PEP 600 <https://www.python.org/dev/peps/pep-0600>`_).
230
* ``_2_28`` corresponding to manylinux_2_28 (see `PEP 600 <https://www.python.org/dev/peps/pep-0600>`_).
231
232
233

The environment variable specified which Manylinux docker container you are building in.

234
235
236
237
238
239
240
241
242
243
244
The ``PLAT`` environment variable can be one of

* ``x86_64``, for 64-bit x86
* ``i686``, for 32-bit x86
* ``s390x``, for 64-bit s390x
* ``ppc64le``, for PowerPC
* ``aarch64``, for ARM
* ``arm64``, for Apple silicon
* ``universal2``, for both Apple silicon and 64-bit x86

The default is ``x86_64``. Only ``x86_64`` and ``i686`` are valid on manylinux1 and manylinux2010.
245

Matthew Brett's avatar
Matthew Brett committed
246
``multibuild/travis_linux_steps.sh`` defines the ``build_wheel`` function,
Hugo's avatar
Hugo committed
247
which starts up the Manylinux1 Docker container to run a wrapper script
Matthew Brett's avatar
Matthew Brett committed
248
``multibuild/docker_build_wrap.sh``, that (within the container) sources the
Rob Buckley's avatar
Rob Buckley committed
249
following bash scripts::
Matthew Brett's avatar
Matthew Brett committed
250

Rob Buckley's avatar
Rob Buckley committed
251
252
253
254
255
256
    multibuild/common_utils.sh
    multibuild/manylinux_utils.sh
    env_vars.sh
    multibuild/configure_build.sh
    multibuild/library_builders.sh
    config.sh
Matthew Brett's avatar
Matthew Brett committed
257

258
259
See ``docker_build_wrap.sh`` to review the order of script sourcing.

260
261
See the definition of ``build_multilinux`` in
``multibuild/travis_linux_steps.sh`` for the environment variables passed from
262
Travis CI to the Manylinux1 container.
Matthew Brett's avatar
Matthew Brett committed
263
264
265
266
267
268
269
270

Once in the container, after sourcing the scripts above, the wrapper runs the
real ``build_wheel`` function, which now comes (by default) from
``multibuild/common_utils.sh``.

Test phase
----------

271
272
273
Specify the version to test with the ``DOCKER_TEST_IMAGE`` environment
variable. The default version is dependent on ``PLAT``:

274
275
* ``multibuild/focal_x86_64``, for ``x86_64``
* ``matthewbrett/trusty:32`` for ``i686`` (Yes, an older image for 32-bit)
276
277
278
279
280
281
282
* ``multibuild/xenial_arm64v8`` for ``aarch64``
* ``multibuild/xenial_ppc64le`` for ``ppc64le``
* ``mutlibuild/xenial_s390x`` for ``s390x``

Other valid values are any in https://hub.docker.com/orgs/multibuild/repositories,
using the correct platform code. Alternatively, you can use the substitution
pattern ``multibuild/xenial_{PLAT}`` in the ``.travis.yml`` file.
mattip's avatar
mattip committed
283
284
285
286
287
288
289

See ``multibuild/docker_test_wrap.sh``.

``multibuild/travis_linux_steps.sh`` defines the ``install_run`` function,
which starts up the testing Docker container with the wrapper script
``multibuild/docker_test_wrap.sh``. The wrapper script sources the following
bash scripts::
Matthew Brett's avatar
Matthew Brett committed
290

Rob Buckley's avatar
Rob Buckley committed
291
292
    multibuild/common_utils.sh
    config.sh
Matthew Brett's avatar
Matthew Brett committed
293

294
295
See ``docker_test_wrap.sh`` for script source order.

Matthew Brett's avatar
Matthew Brett committed
296
297
298
299
300
See ``install_run`` in ``multibuild/travis_linux_steps.sh`` for the
environment variables passed into the container.

It then (in the container) runs the real ``install_run`` command, which comes
(by default) from ``multibuild/common_utils.sh``.
301
302
303
304
305

*********************************
Standard build and test functions
*********************************

Andrew Murray's avatar
Andrew Murray committed
306
The standard build command is ``build_wheel``.  This is a bash function.  By
Hugo's avatar
Hugo committed
307
default the function that is run on macOS, and in the Manylinux container for
Matthew Brett's avatar
Matthew Brett committed
308
309
310
the build phase, is defined in ``multibuild/common_utils.sh``.  You can
override the default function in the project ``config.sh`` file (see below).

robbuckley's avatar
typos  
robbuckley committed
311
If you are building a wheel from PyPI, rather than from a source repository,
312
313
314
315
316
317
318
you can use the ``build_index_wheel`` command, again defined in
``multibuild/common_utils.sh``.

Typically, you can get away with leaving the default ``build_wheel`` /
``build_index_wheel`` functions to do their thing, but you may need to define
a ``pre_build`` function in ``config.sh``.  The default ``build_wheel`` and
``build_index_wheel`` functions will call the ``pre_build`` function, if
319
320
321
defined, before building the wheel, so ``pre_build`` is a good place to build
any required libraries.

Matthew Brett's avatar
Matthew Brett committed
322
The standard test command is the bash function ``install_run``.  The version
Hugo's avatar
Hugo committed
323
run on macOS and in the Linux testing container is also defined in
Matthew Brett's avatar
Matthew Brett committed
324
``multibuild/common_utils.sh``.  Typically, you do not override this function,
325
326
327
328
but you in that case you will need to define a ``run_tests`` function, to run
your tests, returning a non-zero error code for failure.  The default
``install_run`` implementation calls the ``run_tests`` function, which you
will likely define in ``config.sh``.  See the examples below for examples of
Andrew Murray's avatar
Andrew Murray committed
329
less and more complicated builds, where the complicated builds override more
330
of the default implementations.
331
332
333
334

********************
To use these scripts
********************
Matthew Brett's avatar
Matthew Brett committed
335

336
* Make a repository for building wheels on Travis CI - e.g.
Matthew Brett's avatar
Matthew Brett committed
337
338
339
340
341
  https://github.com/MacPython/astropy-wheels - or in your case maybe
  ``https://github.com/your-org/your-project-wheels``;

* Add this (here) repository as a submodule::

Hugo van Kemenade's avatar
Hugo van Kemenade committed
342
    git submodule add https://github.com/multi-build/multibuild.git
Matthew Brett's avatar
Matthew Brett committed
343
344
345
346
347
348
349
350

* Add your own project repository as another submodule::

    git submodule add https://github.com/your-org/your-project.git

* Create a ``.travis.yml`` file, something like this::

    env:
351
352
353
354
355
        global:
            - REPO_DIR=your-project
            # Commit from your-project that you want to build
            - BUILD_COMMIT=v0.1.0
            # pip dependencies to _build_ your project
robbuckley's avatar
typos  
robbuckley committed
356
            - BUILD_DEPENDS="cython numpy"
357
358
359
            # pip dependencies to _test_ your project.  Include any dependencies
            # that you need, that are also specified in BUILD_DEPENDS, this will be
            # a separate install.
Matthew Brett's avatar
Matthew Brett committed
360
361
            # Now see the Uploads section for the stuff you need to
            # upload your wheels after CI has built them.
Matthew Brett's avatar
Matthew Brett committed
362

363
364
365
366
367
368
369
    # You will likely prefer "language: generic" for travis configuration,
    # rather than, say "language: python". Multibuild doesn't use
    # Travis-provided Python but rather installs and uses its own, where the
    # Python version is set from the MB_PYTHON_VERSION variable. You can still
    # specify a language here if you need it for some unrelated logic and you
    # can't use Multibuild-provided Python or other software present on a
    # builder.
370
    language: generic
371
372
373

    # For CPython macOS builds only, the minimum supported macOS version and
    # architectures of any C extensions in the wheel are set with the variable
Rob Buckley's avatar
Rob Buckley committed
374
375
376
377
378
    # MB_PYTHON_OSX_VER: 10.9 (64-bit only) or 10.6 (64/32-bit dual arch). By
    # default this is set to the highest available for the Python version selected
    # using MB_PYTHON_VERSION. You should only need to set this explicitly if you
    # are building a 10.6 dual-arch build for a CPython version where both a 10.9 and
    # 10.6 build are available (for example, 2.7 or 3.7).
379
380
    # All PyPy macOS builds are 64-bit only.

Andrew Murray's avatar
Andrew Murray committed
381
    # Required in Linux to invoke `docker` ourselves
Matthew Brett's avatar
Matthew Brett committed
382
383
    services: docker

384
385
386
    # Host distribution.  This is the distribution from which we run the build
    # and test containers, via docker.
    dist: xenial
Matthew Brett's avatar
Matthew Brett committed
387

Isuru Fernando's avatar
Isuru Fernando committed
388
389
    # osx image that enables building Apple silicon libraries
    osx_image: xcode12.2
390

Matthew Brett's avatar
Matthew Brett committed
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
    matrix:
      include:
        - os: linux
          env: MB_PYTHON_VERSION=2.7
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - UNICODE_WIDTH=16
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - PLAT=i686
        - os: linux
          env:
            - MB_PYTHON_VERSION=2.7
            - PLAT=i686
            - UNICODE_WIDTH=16
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.5
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.5
            - PLAT=i686
riddell1's avatar
riddell1 committed
415
416
417
418
419
420
421
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.6
        - os: linux
          env:
            - MB_PYTHON_VERSION=3.6
            - PLAT=i686
Matthew Brett's avatar
Matthew Brett committed
422
423
424
        - os: osx
          env:
            - MB_PYTHON_VERSION=2.7
Rob Buckley's avatar
Rob Buckley committed
425
            - MB_PYTHON_OSX_VER=10.6
robbuckley's avatar
readme  
robbuckley committed
426
427
428
        - os: osx
          env:
            - MB_PYTHON_VERSION=2.7
Matthew Brett's avatar
Matthew Brett committed
429
430
431
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.5
riddell1's avatar
riddell1 committed
432
433
434
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.6
Kyle Stewart's avatar
Kyle Stewart committed
435
        - os: osx
436
          env:
437
            - MB_PYTHON_VERSION=3.7
Rob Buckley's avatar
Rob Buckley committed
438
439
440
441
442
443
444
            - MB_PYTHON_OSX_VER=10.6
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.7
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.8
445
446
447
448
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.9
            - PLAT="universal2"
449
450
451
        - os: osx
          env:
            - MB_PYTHON_VERSION=3.9
Kyle Stewart's avatar
Kyle Stewart committed
452
453
454
        - os: osx
          language: generic
          env:
455
            - MB_PYTHON_VERSION=pypy-5.7
Matthew Brett's avatar
Matthew Brett committed
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470

    before_install:
        - source multibuild/common_utils.sh
        - source multibuild/travis_steps.sh
        - before_install

    install:
        # Maybe get and clean and patch source
        - clean_code $REPO_DIR $BUILD_COMMIT
        - build_wheel $REPO_DIR $PLAT

    script:
        - install_run $PLAT

    after_success:
Matthew Brett's avatar
Matthew Brett committed
471
472
        # Here you should put the code to upload your wheels
        # See the Uploads section for more details.
Matthew Brett's avatar
Matthew Brett committed
473

Hugo's avatar
Hugo committed
474
  The example above is for a project building from a Git submodule.  If you
475
476
477
478
479
480
481
482
483
484
485
486
487
488
  aren't building from a submodule, but want to use ``pip`` to build from a
  source archive on https://pypi.org or similar, replace the first few lines
  of the ``.travis.yml`` file with something like::

    env:
        global:
            # Instead of REPO_DIR, BUILD_COMMIT
            - PROJECT_SPEC="tornado==4.1.1"

  then your ``install`` section could look something like this::

    install:
        - build_index_wheel $PROJECT_SPEC

489

Matthew Brett's avatar
Matthew Brett committed
490
491
492
493
494
* Next create a ``config.sh`` for your project, that fills in any steps you
  need to do before building the wheel (such as building required libraries).
  You also need this file to specify how to run your tests::

    # Define custom utilities
Andrew Murray's avatar
Andrew Murray committed
495
    # Test for macOS with [ -n "$IS_MACOS" ]
Matthew Brett's avatar
Matthew Brett committed
496
497
498
499
500
501
502
503
504
505
506
507
508

    function pre_build {
        # Any stuff that you need to do before you start building the wheels
        # Runs in the root directory of this repository.
        :
    }

    function run_tests {
        # Runs tests on installed distribution from an empty directory
        python --version
        python -c 'import sys; import yourpackage; sys.exit(yourpackage.test())'
    }

509
510
  Optionally you can specify a different location for ``config.sh`` file with
  the ``$CONFIG_PATH`` environment variable.
511

512
* Optionally, create an ``env_vars.sh`` file to override the defaults for any
513
514
515
  environment variables used by
  ``configure_build.sh``/``library_builders.sh``. In Linux, the environment
  variables used for the build cannot be set in the ``.travis.yml`` file,
516
  because the build processing runs in a Docker container, so the only
517
518
519
520
521
522
523
524
  environment variables that reach the container are those passed in via the
  ``docker run`` command, or those set in ``env_vars.sh``.

  As for the ``config.sh`` file, you can specify a different location for the
  file by setting the ``$ENV_VARS_PATH`` environment variable.  The path in
  ``$ENV_VARS_PATH`` is relative to the repository root directory.  For
  example, if your repository had a subdirectory ``scripts`` with a file
  ``my_env_vars.sh``, you should set ``ENV_VARS_PATH=scripts/my_env_vars.sh``.
525

526
* Make sure your project is set up to build on Travis CI, and you should now
Matthew Brett's avatar
Matthew Brett committed
527
  be ready (to begin the long slow debugging process, probably).
528

529
530
531
532
533
* For the Windows wheels, create an ``appveyor.yml`` file, something like:

  - https://github.com/MacPython/astropy-wheels/blob/master/appveyor.yml
  - https://github.com/MacPython/nipy-wheels/blob/master/appveyor.yml
  - https://github.com/MacPython/pytables-wheels/blob/master/appveyor.yml
534

535
536
  Note the Windows test customizations etc are inside ``appveyor.yml``,
  and that ``config.sh`` and ``env_vars.sh`` are only for the
537
  Linux/Mac builds on Travis CI.
538

Matthew Brett's avatar
Matthew Brett committed
539
* Make sure your project is set up to build on AppVeyor, and you should now
540
  be ready (for what could be another round of slow debugging).
Matthew Brett's avatar
Matthew Brett committed
541

542
543
544
* For Apple silicon support you can either create an ``arm64`` wheel or
  a ``universal2`` wheel by supplying ``PLAT`` env variable.
  ``universal2`` builds work on both ``arm64`` and ``x86_64`` platforms
545
546
  and also make it possible for the wheel code to work when switching the
  architecture on Apple silicon machines where ``x86_64`` can be run
547
548
549
  using Rosetta2 emulation.

  There are two ways to build ``universal2`` builds.
Isuru Fernando's avatar
Isuru Fernando committed
550

551
552
  1. Build with ``-arch x86_64 -arch arm64``.
     These flags instruct the C/C++ compiler to compile twice and create a
553
     fat object/executable/library. This is the easiest, but has several
554
555
556
557
558
559
560
     drawbacks. If you are using C/C++ libraries that are built using
     library_builders, it's highly likely that they don't build correctly
     because most build systems and packages don't support building fat binaries.
     We could possibly build them separately and fuse them, but the headers might
     not be identical which is required when building the wheel as a ``universal2``
     wheel. If you are using Fortran, ``gfortran`` doesn't support fat binaries.

Andrew Murray's avatar
Andrew Murray committed
561
  2. Build ``arm64`` and ``x86_64`` wheels separately and fuse them.
562
563
564
565
566
567
     For this to work, we need to build the C/C++ libraries twice. Therefore,
     the library building is once called with ``BUILD_PREFIX=${BUILD_PREFIX:-/usr/local}``
     for ``x86_64`` and then called again with ``BUILD_PREFIX=/opt/arm64-builds``.
     Once the two wheels are created, these two are merged. Both the
     ``arm64`` and ``universal2`` wheels are outputs for this build.

568
569
  In multibuild we are going with option 2. You can override this behaviour by
  overriding the function ``wrap_wheel_builder``.
570
571
572
  To build Apple silicon builds, you should use a CI service with Xcode 12 with
  universal build support and make sure that xcode is the default.

Hugo's avatar
Hugo committed
573
574
If your project depends on NumPy, you will want to build against the earliest
NumPy that your project supports - see `forward, backward NumPy compatibility
Andrew Murray's avatar
Andrew Murray committed
575
<https://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312>`_.
576
See the `astropy-wheels Travis file
Matthew Brett's avatar
Matthew Brett committed
577
<https://github.com/MacPython/astropy-wheels/blob/master/.travis.yml>`_ for an
Hugo's avatar
Hugo committed
578
example specifying NumPy build and test dependencies.
Matthew Brett's avatar
Matthew Brett committed
579
580
581
582
583
584
585
586
587

Here are some simple example projects:

* https://github.com/MacPython/astropy-wheels
* https://github.com/scikit-image/scikit-image-wheels
* https://github.com/MacPython/nipy-wheels
* https://github.com/MacPython/dipy-wheels

Less simple projects where there are some serious build dependencies, and / or
Hugo's avatar
Hugo committed
588
macOS / Linux differences:
Matthew Brett's avatar
Matthew Brett committed
589
590
591
592

* https://github.com/MacPython/matplotlib-wheels
* https://github.com/python-pillow/Pillow-wheels
* https://github.com/MacPython/h5py-wheels
593
594
595
596
597
598

**********************
Multibuild development
**********************

The main multibuild repository is always at
Hugo van Kemenade's avatar
Hugo van Kemenade committed
599
https://github.com/multi-build/multibuild
600

601
602
We try to keep the ``master`` branch stable and do testing and development
in the ``devel`` branch.  From time to time we merge ``devel`` into ``master``.
603

604
605
In practice, you can check out the newest commit from ``devel`` that works
for you, then stay at it until you need newer features.