compiling.rst 12.4 KB
Newer Older
1
2
.. _compiling:

Wenzel Jakob's avatar
Wenzel Jakob committed
3
4
5
6
7
8
9
10
11
Build systems
#############

Building with setuptools
========================

For projects on PyPI, building with setuptools is the way to go. Sylvain Corlay
has kindly provided an example project which shows how to set up everything,
including automatic generation of documentation using Sphinx. Please refer to
Wenzel Jakob's avatar
Wenzel Jakob committed
12
the [python_example]_ repository.
Wenzel Jakob's avatar
Wenzel Jakob committed
13

Wenzel Jakob's avatar
Wenzel Jakob committed
14
.. [python_example] https://github.com/pybind/python_example
Wenzel Jakob's avatar
Wenzel Jakob committed
15

16
17
18
Building with cppimport
========================

Dean Moldovan's avatar
Dean Moldovan committed
19
20
21
22
[cppimport]_ is a small Python import hook that determines whether there is a C++
source file whose name matches the requested module. If there is, the file is
compiled as a Python extension using pybind11 and placed in the same folder as
the C++ source file. Python is then able to find the module and load it.
23
24
25

.. [cppimport] https://github.com/tbenthompson/cppimport

26
27
28
29
30
.. _cmake:

Building with CMake
===================

Wenzel Jakob's avatar
Wenzel Jakob committed
31
For C++ codebases that have an existing CMake-based build system, a Python
32
extension module can be created with just a few lines of code:
33
34
35

.. code-block:: cmake

36
    cmake_minimum_required(VERSION 3.7)
37
38
    project(example)

39
40
41
    add_subdirectory(pybind11)
    pybind11_add_module(example example.cpp)

Wenzel Jakob's avatar
Wenzel Jakob committed
42
This assumes that the pybind11 repository is located in a subdirectory named
43
:file:`pybind11` and that the code is located in a file named :file:`example.cpp`.
44
45
46
The CMake command ``add_subdirectory`` will import the pybind11 project which
provides the ``pybind11_add_module`` function. It will take care of all the
details needed to build a Python extension module on any platform.
47
48
49
50

A working sample project, including a way to invoke CMake from :file:`setup.py` for
PyPI integration, can be found in the [cmake_example]_  repository.

Wenzel Jakob's avatar
Wenzel Jakob committed
51
.. [cmake_example] https://github.com/pybind/cmake_example
52

53
54
55
56
57
58
59
60
61
pybind11_add_module
-------------------

To ease the creation of Python extension modules, pybind11 provides a CMake
function with the following signature:

.. code-block:: cmake

    pybind11_add_module(<name> [MODULE | SHARED] [EXCLUDE_FROM_ALL]
62
                        [NO_EXTRAS] [THIN_LTO] source1 [source2 ...])
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

This function behaves very much like CMake's builtin ``add_library`` (in fact,
it's a wrapper function around that command). It will add a library target
called ``<name>`` to be built from the listed source files. In addition, it
will take care of all the Python-specific compiler and linker flags as well
as the OS- and Python-version-specific file extension. The produced target
``<name>`` can be further manipulated with regular CMake commands.

``MODULE`` or ``SHARED`` may be given to specify the type of library. If no
type is given, ``MODULE`` is used by default which ensures the creation of a
Python-exclusive module. Specifying ``SHARED`` will create a more traditional
dynamic library which can also be linked from elsewhere. ``EXCLUDE_FROM_ALL``
removes this target from the default build (see CMake docs for details).

Since pybind11 is a template library, ``pybind11_add_module`` adds compiler
flags to ensure high quality code generation without bloat arising from long
79
80
81
82
83
84
85
86
87
symbol names and duplication of code in different translation units. It
sets default visibility to *hidden*, which is required for some pybind11
features and functionality when attempting to load multiple pybind11 modules
compiled under different pybind11 versions.  It also adds additional flags
enabling LTO (Link Time Optimization) and strip unneeded symbols. See the
:ref:`FAQ entry <faq:symhidden>` for a more detailed explanation. These
latter optimizations are never applied in ``Debug`` mode.  If ``NO_EXTRAS`` is
given, they will always be disabled, even in ``Release`` mode. However, this
will result in code bloat and is generally not recommended.
88
89
90
91
92
93
94
95
96
97
98

As stated above, LTO is enabled by default. Some newer compilers also support
different flavors of LTO such as `ThinLTO`_. Setting ``THIN_LTO`` will cause
the function to prefer this flavor if available. The function falls back to
regular LTO if ``-flto=thin`` is not available.

.. _ThinLTO: http://clang.llvm.org/docs/ThinLTO.html

Configuration variables
-----------------------

99
100
101
102
By default, pybind11 will compile modules with the compiler default or the
minimum standard required by PyBind11, whichever is higher.  You can set the
standard explicitly with
`CMAKE_CXX_STANDARD <https://cmake.org/cmake/help/latest/variable/CMAKE_CXX_STANDARD.html>`_:
103
104
105

.. code-block:: cmake

106
107
108
    set(CMAKE_CXX_STANDARD 14)  # or 11, 14, 17, 20
    set(CMAKE_CXX_STANDARD_REQUIRED ON)  # optional, ensure standard is supported
    set(CMAKE_CXX_EXTENSIONS OFF)  # optional, keep compiler extensionsn off
109

110

111
112
113
114
The variables can also be set when calling CMake from the command line using
the ``-D<variable>=<value>`` flag. You can also manually set ``CXX_STANDARD``
on a target or use ``target_compile_features`` on your targets - anything that
CMake supports.
115
116
117
118

The target Python version can be selected by setting ``PYBIND11_PYTHON_VERSION``
or an exact Python installation can be specified with ``PYTHON_EXECUTABLE``.
For example:
119

120
121
122
.. code-block:: bash

    cmake -DPYBIND11_PYTHON_VERSION=3.6 ..
123
124
125
126
127

    # Another method:
    cmake -DPYTHON_EXECUTABLE=/path/to/python ..

    # You will often see this idiom:
128
    cmake -DPYTHON_EXECUTABLE=$(python3 -c "import sys; print(sys.executable)") ..
129
130
131
132
133
134
135

find_package vs. add_subdirectory
---------------------------------

For CMake-based projects that don't include the pybind11 repository internally,
an external installation can be detected through ``find_package(pybind11)``.
See the `Config file`_ docstring for details of relevant CMake variables.
136
137
138

.. code-block:: cmake

139
    cmake_minimum_required(VERSION 3.7)
140
141
142
143
144
    project(example)

    find_package(pybind11 REQUIRED)
    pybind11_add_module(example example.cpp)

145
146
147
148
149
150
Note that ``find_package(pybind11)`` will only work correctly if pybind11
has been correctly installed on the system, e. g. after downloading or cloning
the pybind11 repository  :

.. code-block:: bash

151
    # Classic CMake
152
153
154
155
156
157
    cd pybind11
    mkdir build
    cd build
    cmake ..
    make install

158
159
160
161
162
163
    # CMake 3.15+
    cd pybind11
    cmake -S . -B build
    cmake --build build -j 2  # Build on 2 cores
    cmake --install build

164
165
166
167
168
169
170
171
172
173
Once detected, the aforementioned ``pybind11_add_module`` can be employed as
before. The function usage and configuration variables are identical no matter
if pybind11 is added as a subdirectory or found as an installed package. You
can refer to the same [cmake_example]_ repository for a full sample project
-- just swap out ``add_subdirectory`` for ``find_package``.

.. _Config file: https://github.com/pybind/pybind11/blob/master/tools/pybind11Config.cmake.in

Advanced: interface library target
----------------------------------
174

175
When using a version of CMake greater than 3.0, pybind11 can additionally
176
be used as a special *interface library* . The target ``pybind11::module``
177
is available with pybind11 headers, Python headers and libraries as needed,
178
and C++ compile features attached. This target is suitable for linking
179
180
to an independently constructed (through ``add_library``, not
``pybind11_add_module``) target in the consuming project.
181
182
183

.. code-block:: cmake

184
    cmake_minimum_required(VERSION 3.7)
185
186
    project(example)

187
    find_package(pybind11 REQUIRED)  # or add_subdirectory(pybind11)
188

189
    add_library(example MODULE main.cpp)
190
    target_link_libraries(example PRIVATE pybind11::module)
191
192
193
194
195
196
197
198
199
200
201
202
203
    set_target_properties(example PROPERTIES PREFIX "${PYTHON_MODULE_PREFIX}"
                                             SUFFIX "${PYTHON_MODULE_EXTENSION}")

.. warning::

    Since pybind11 is a metatemplate library, it is crucial that certain
    compiler flags are provided to ensure high quality code generation. In
    contrast to the ``pybind11_add_module()`` command, the CMake interface
    library only provides the *minimal* set of parameters to ensure that the
    code using pybind11 compiles, but it does **not** pass these extra compiler
    flags (i.e. this is up to you).

    These include Link Time Optimization (``-flto`` on GCC/Clang/ICPC, ``/GL``
204
205
    and ``/LTCG`` on Visual Studio) and .OBJ files with many sections on Visual
    Studio (``/bigobj``).  The :ref:`FAQ <faq:symhidden>` contains an
206
    explanation on why these are needed.
207

208
209
210
211
212
    If you want to add these in yourself, you can use:

    .. code-block:: cmake

        set(CMAKE_CXX_VISIBILITY_PRESET hidden)
213
214
        set(CMAKE_VISIBILITY_INLINES_HIDDEN ON)
        set(CMAKE_INTERPROCEDURAL_OPTIMIZATION ON)  # CMake 3.9+ required
215

216
    or set the corresponding property (without the ``CMAKE_``) on the targets
217
218
    manually.

219
220
221
222
223
224
225
226
227
228
229
230
Embedding the Python interpreter
--------------------------------

In addition to extension modules, pybind11 also supports embedding Python into
a C++ executable or library. In CMake, simply link with the ``pybind11::embed``
target. It provides everything needed to get the interpreter running. The Python
headers and libraries are attached to the target. Unlike ``pybind11::module``,
there is no need to manually set any additional properties here. For more
information about usage in C++, see :doc:`/advanced/embedding`.

.. code-block:: cmake

231
    cmake_minimum_required(VERSION 3.7)
232
233
234
235
    project(example)

    find_package(pybind11 REQUIRED)  # or add_subdirectory(pybind11)

236
    add_executable(example main.cpp)
237
238
    target_link_libraries(example PRIVATE pybind11::embed)

239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
.. _building_manually:

Building manually
=================

pybind11 is a header-only library, hence it is not necessary to link against
any special libraries and there are no intermediate (magic) translation steps.

On Linux, you can compile an example such as the one given in
:ref:`simple_example` using the following command:

.. code-block:: bash

    $ c++ -O3 -Wall -shared -std=c++11 -fPIC `python3 -m pybind11 --includes` example.cpp -o example`python3-config --extension-suffix`

The flags given here assume that you're using Python 3. For Python 2, just
change the executable appropriately (to ``python`` or ``python2``).

The ``python3 -m pybind11 --includes`` command fetches the include paths for
both pybind11 and Python headers. This assumes that pybind11 has been installed
using ``pip`` or ``conda``. If it hasn't, you can also manually specify
``-I <path-to-pybind11>/include`` together with the Python includes path
``python3-config --includes``.

Note that Python 2.7 modules don't use a special suffix, so you should simply
use ``example.so`` instead of ``example`python3-config --extension-suffix```.
Besides, the ``--extension-suffix`` option may or may not be available, depending
on the distribution; in the latter case, the module extension can be manually
set to ``.so``.

On Mac OS: the build command is almost the same but it also requires passing
the ``-undefined dynamic_lookup`` flag so as to ignore missing symbols when
building the module:

.. code-block:: bash

    $ c++ -O3 -Wall -shared -std=c++11 -undefined dynamic_lookup `python3 -m pybind11 --includes` example.cpp -o example`python3-config --extension-suffix`

In general, it is advisable to include several additional build parameters
that can considerably reduce the size of the created binary. Refer to section
:ref:`cmake` for a detailed example of a suitable cross-platform CMake-based
build system that works on all platforms including Windows.

.. note::

    On Linux and macOS, it's better to (intentionally) not link against
    ``libpython``. The symbols will be resolved when the extension library
    is loaded into a Python binary. This is preferable because you might
    have several different installations of a given Python version (e.g. the
    system-provided Python, and one that ships with a piece of commercial
    software). In this way, the plugin will work with both versions, instead
    of possibly importing a second Python library into a process that already
    contains one (which will lead to a segfault).
292

293
294
295
296
297
298
299
300
Generating binding code automatically
=====================================

The ``Binder`` project is a tool for automatic generation of pybind11 binding
code by introspecting existing C++ codebases using LLVM/Clang. See the
[binder]_ documentation for details.

.. [binder] http://cppbinder.readthedocs.io/en/latest/about.html
301
302
303
304
305
306
307
308

[AutoWIG]_ is a Python library that wraps automatically compiled libraries into
high-level languages. It parses C++ code using LLVM/Clang technologies and
generates the wrappers using the Mako templating engine. The approach is automatic,
extensible, and applies to very complex C++ libraries, composed of thousands of
classes or incorporating modern meta-programming constructs.

.. [AutoWIG] https://github.com/StatisKit/AutoWIG