faq.rst 13.7 KB
Newer Older
Wenzel Jakob's avatar
Wenzel Jakob committed
1
2
3
Frequently asked questions
##########################

Wenzel Jakob's avatar
Wenzel Jakob committed
4
5
"ImportError: dynamic module does not define init function"
===========================================================
6

Ahuva Kroizer's avatar
Ahuva Kroizer committed
7
1. Make sure that the name specified in PYBIND11_MODULE is identical to the
8
filename of the extension library (without suffixes such as ``.so``).
Ahuva Kroizer's avatar
Ahuva Kroizer committed
9
10

2. If the above did not fix the issue, you are likely using an incompatible
11
version of Python that does not match what you compiled with.
Wenzel Jakob's avatar
Wenzel Jakob committed
12
13
14
15

"Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``"
========================================================================

16
See the first answer.
Wenzel Jakob's avatar
Wenzel Jakob committed
17

18
19
20
"SystemError: dynamic module not initialized properly"
======================================================

21
See the first answer.
22

Wenzel Jakob's avatar
Wenzel Jakob committed
23
24
25
The Python interpreter immediately crashes when importing my module
===================================================================

26
See the first answer.
27

28
29
.. _faq_reference_arguments:

Wenzel Jakob's avatar
Wenzel Jakob committed
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
Limitations involving reference arguments
=========================================

In C++, it's fairly common to pass arguments using mutable references or
mutable pointers, which allows both read and write access to the value
supplied by the caller. This is sometimes done for efficiency reasons, or to
realize functions that have multiple return values. Here are two very basic
examples:

.. code-block:: cpp

    void increment(int &i) { i++; }
    void increment_ptr(int *i) { (*i)++; }

In Python, all arguments are passed by reference, so there is no general
issue in binding such code from Python.

However, certain basic Python types (like ``str``, ``int``, ``bool``,
``float``, etc.) are **immutable**. This means that the following attempt
to port the function to Python doesn't have the same effect on the value
provided by the caller -- in fact, it does nothing at all.

.. code-block:: python

    def increment(i):
55
        i += 1  # nope..
Wenzel Jakob's avatar
Wenzel Jakob committed
56

Wenzel Jakob's avatar
Wenzel Jakob committed
57
58
59
60
61
pybind11 is also affected by such language-level conventions, which means that
binding ``increment`` or ``increment_ptr`` will also create Python functions
that don't modify their arguments.

Although inconvenient, one workaround is to encapsulate the immutable types in
62
a custom type that does allow modifications.
Wenzel Jakob's avatar
Wenzel Jakob committed
63
64
65
66
67
68
69
70
71
72
73
74
75
76

An other alternative involves binding a small wrapper lambda function that
returns a tuple with all output arguments (see the remainder of the
documentation for examples on binding lambda functions). An example:

.. code-block:: cpp

    int foo(int &i) { i++; return 123; }

and the binding code

.. code-block:: cpp

   m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); });
Wenzel Jakob's avatar
Wenzel Jakob committed
77

78

Wenzel Jakob's avatar
Wenzel Jakob committed
79
80
81
How can I reduce the build time?
================================

82
83
84
85
It's good practice to split binding code over multiple files, as in the
following example:

:file:`example.cpp`:
Wenzel Jakob's avatar
Wenzel Jakob committed
86
87
88

.. code-block:: cpp

89
90
    void init_ex1(py::module_ &);
    void init_ex2(py::module_ &);
Wenzel Jakob's avatar
Wenzel Jakob committed
91
92
    /* ... */

93
    PYBIND11_MODULE(example, m) {
Wenzel Jakob's avatar
Wenzel Jakob committed
94
95
96
97
98
        init_ex1(m);
        init_ex2(m);
        /* ... */
    }

99
100
101
102
:file:`ex1.cpp`:

.. code-block:: cpp

103
    void init_ex1(py::module_ &m) {
104
105
106
107
108
109
110
        m.def("add", [](int a, int b) { return a + b; });
    }

:file:`ex2.cpp`:

.. code-block:: cpp

111
    void init_ex2(py::module_ &m) {
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
        m.def("sub", [](int a, int b) { return a - b; });
    }

:command:`python`:

.. code-block:: pycon

    >>> import example
    >>> example.add(1, 2)
    3
    >>> example.sub(1, 1)
    0

As shown above, the various ``init_ex`` functions should be contained in
separate files that can be compiled independently from one another, and then
linked together into the same final shared object.  Following this approach
will:
Wenzel Jakob's avatar
Wenzel Jakob committed
129

Wenzel Jakob's avatar
Wenzel Jakob committed
130
1. reduce memory requirements per compilation unit.
Wenzel Jakob's avatar
Wenzel Jakob committed
131

Wenzel Jakob's avatar
Wenzel Jakob committed
132
2. enable parallel builds (if desired).
Wenzel Jakob's avatar
Wenzel Jakob committed
133

Wenzel Jakob's avatar
Wenzel Jakob committed
134
3. allow for faster incremental builds. For instance, when a single class
135
   definition is changed, only a subset of the binding code will generally need
Wenzel Jakob's avatar
Wenzel Jakob committed
136
   to be recompiled.
Wenzel Jakob's avatar
Wenzel Jakob committed
137

138
139
140
141
142
143
144
145
"recursive template instantiation exceeded maximum depth of 256"
================================================================

If you receive an error about excessive recursive template evaluation, try
specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
culprit is generally the generation of function signatures at compile time
using C++14 template metaprogramming.

146
147
.. _`faq:hidden_visibility`:

148
"'SomeClass' declared with greater visibility than the type of its field 'SomeClass::member' [-Wattributes]"
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
============================================================================================================

This error typically indicates that you are compiling without the required
``-fvisibility`` flag.  pybind11 code internally forces hidden visibility on
all internal code, but if non-hidden (and thus *exported*) code attempts to
include a pybind type (for example, ``py::object`` or ``py::list``) you can run
into this warning.

To avoid it, make sure you are specifying ``-fvisibility=hidden`` when
compiling pybind code.

As to why ``-fvisibility=hidden`` is necessary, because pybind modules could
have been compiled under different versions of pybind itself, it is also
important that the symbols defined in one module do not clash with the
potentially-incompatible symbols defined in another.  While Python extension
modules are usually loaded with localized symbols (under POSIX systems
typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default
can be changed, but even if it isn't it is not always enough to guarantee
complete independence of the symbols involved when not using
``-fvisibility=hidden``.

jakobjw's avatar
jakobjw committed
170
Additionally, ``-fvisibility=hidden`` can deliver considerably binary size
171
savings. (See the following section for more details.)
172

173

174
175
.. _`faq:symhidden`:

Wenzel Jakob's avatar
Wenzel Jakob committed
176
177
178
179
180
How can I create smaller binaries?
==================================

To do its job, pybind11 extensively relies on a programming technique known as
*template metaprogramming*, which is a way of performing computation at compile
luzpaz's avatar
luzpaz committed
181
time using type information. Template metaprogramming usually instantiates code
Wenzel Jakob's avatar
Wenzel Jakob committed
182
involving significant numbers of deeply nested types that are either completely
183
removed or reduced to just a few instructions during the compiler's optimization
Wenzel Jakob's avatar
Wenzel Jakob committed
184
185
186
187
phase. However, due to the nested nature of these types, the resulting symbol
names in the compiled extension library can be extremely long. For instance,
the included test suite contains the following symbol:

Wenzel Jakob's avatar
Wenzel Jakob committed
188
189
190
191
.. only:: html

    .. code-block:: none

Wenzel Jakob's avatar
Wenzel Jakob committed
192
        _​_​Z​N​8​p​y​b​i​n​d​1​1​1​2​c​p​p​_​f​u​n​c​t​i​o​n​C​1​I​v​8​E​x​a​m​p​l​e​2​J​R​N​S​t​3​_​_​1​6​v​e​c​t​o​r​I​N​S​3​_​1​2​b​a​s​i​c​_​s​t​r​i​n​g​I​w​N​S​3​_​1​1​c​h​a​r​_​t​r​a​i​t​s​I​w​E​E​N​S​3​_​9​a​l​l​o​c​a​t​o​r​I​w​E​E​E​E​N​S​8​_​I​S​A​_​E​E​E​E​E​J​N​S​_​4​n​a​m​e​E​N​S​_​7​s​i​b​l​i​n​g​E​N​S​_​9​i​s​_​m​e​t​h​o​d​E​A​2​8​_​c​E​E​E​M​T​0​_​F​T​_​D​p​T​1​_​E​D​p​R​K​T​2​_
Wenzel Jakob's avatar
Wenzel Jakob committed
193
194
195
196

.. only:: not html

    .. code-block:: cpp
Wenzel Jakob's avatar
Wenzel Jakob committed
197

Wenzel Jakob's avatar
Wenzel Jakob committed
198
        __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
Wenzel Jakob's avatar
Wenzel Jakob committed
199
200
201
202
203
204
205

which is the mangled form of the following function type:

.. code-block:: cpp

    pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28])

Wenzel Jakob's avatar
Wenzel Jakob committed
206
207
208
209
210
211
The memory needed to store just the mangled name of this function (196 bytes)
is larger than the actual piece of code (111 bytes) it represents! On the other
hand, it's silly to even give this function a name -- after all, it's just a
tiny cog in a bigger piece of machinery that is not exposed to the outside
world. So we'll generally only want to export symbols for those functions which
are actually called from the outside.
Wenzel Jakob's avatar
Wenzel Jakob committed
212
213

This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
214
215
216
217
218
219
220
221
and Clang, which sets the default symbol visibility to *hidden*, which has a
tremendous impact on the final binary size of the resulting extension library.
(On Visual Studio, symbols are already hidden by default, so nothing needs to
be done there.)

In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids
potential serious issues when loading multiple modules and is required for
proper pybind operation.  See the previous FAQ entry for more details.
Wenzel Jakob's avatar
Wenzel Jakob committed
222

Marc Schlaich's avatar
Marc Schlaich committed
223
Working with ancient Visual Studio 2008 builds on Windows
Wenzel Jakob's avatar
Wenzel Jakob committed
224
225
226
227
228
229
=========================================================

The official Windows distributions of Python are compiled using truly
ancient versions of Visual Studio that lack good C++11 support. Some users
implicitly assume that it would be impossible to load a plugin built with
Visual Studio 2015 into a Python distribution that was compiled using Visual
Marc Schlaich's avatar
Marc Schlaich committed
230
Studio 2008. However, no such issue exists: it's perfectly legitimate to
Wenzel Jakob's avatar
Wenzel Jakob committed
231
232
233
234
235
interface DLLs that are built with different compilers and/or C libraries.
Common gotchas to watch out for involve not ``free()``-ing memory region
that that were ``malloc()``-ed in another shared library, using data
structures with incompatible ABIs, and so on. pybind11 is very careful not
to make these types of mistakes.
236

237
238
239
240
241
242
243
244
245
How can I properly handle Ctrl-C in long-running functions?
===========================================================

Ctrl-C is received by the Python interpreter, and holds it until the GIL
is released, so a long-running function won't be interrupted.

To interrupt from inside your function, you can use the ``PyErr_CheckSignals()``
function, that will tell if a signal has been raised on the Python side.  This
function merely checks a flag, so its impact is negligible. When a signal has
246
247
248
been received, you must either explicitly interrupt execution by throwing
``py::error_already_set`` (which will propagate the existing
``KeyboardInterrupt``), or clear the error (which you usually will not want):
249
250
251
252
253
254
255
256
257

.. code-block:: cpp

    PYBIND11_MODULE(example, m)
    {
        m.def("long running_func", []()
        {
            for (;;) {
                if (PyErr_CheckSignals() != 0)
258
                    throw py::error_already_set();
259
260
261
262
263
                // Long running iteration
            }
        });
    }

Henry Schreiner's avatar
Henry Schreiner committed
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
CMake doesn't detect the right Python version
=============================================

The CMake-based build system will try to automatically detect the installed
version of Python and link against that. When this fails, or when there are
multiple versions of Python and it finds the wrong one, delete
``CMakeCache.txt`` and then add ``-DPYTHON_EXECUTABLE=$(which python)`` to your
CMake configure line. (Replace ``$(which python)`` with a path to python if
your prefer.)

You can alternatively try ``-DPYBIND11_FINDPYTHON=ON``, which will activate the
new CMake FindPython support instead of pybind11's custom search. Requires
CMake 3.12+, and 3.15+ or 3.18.2+ are even better. You can set this in your
``CMakeLists.txt`` before adding or finding pybind11, as well.

279
280
281
Inconsistent detection of Python version in CMake and pybind11
==============================================================

Henry Schreiner's avatar
Henry Schreiner committed
282
283
284
285
286
287
The functions ``find_package(PythonInterp)`` and ``find_package(PythonLibs)``
provided by CMake for Python version detection are modified by pybind11 due to
unreliability and limitations that make them unsuitable for pybind11's needs.
Instead pybind11 provides its own, more reliable Python detection CMake code.
Conflicts can arise, however, when using pybind11 in a project that *also* uses
the CMake Python detection in a system with several Python versions installed.
288

Henry Schreiner's avatar
Henry Schreiner committed
289
This difference may cause inconsistencies and errors if *both* mechanisms are
290
used in the same project.
Henry Schreiner's avatar
Henry Schreiner committed
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306

There are three possible solutions:

1. Avoid using ``find_package(PythonInterp)`` and ``find_package(PythonLibs)``
   from CMake and rely on pybind11 in detecting Python version. If this is not
   possible, the CMake machinery should be called *before* including pybind11.
2. Set ``PYBIND11_FINDPYTHON`` to ``True`` or use ``find_package(Python
   COMPONENTS Interpreter Development)`` on modern CMake (3.12+, 3.15+ better,
   3.18.2+ best). Pybind11 in these cases uses the new CMake FindPython instead
   of the old, deprecated search tools, and these modules are much better at
   finding the correct Python.
3. Set ``PYBIND11_NOPYTHON`` to ``TRUE``. Pybind11 will not search for Python.
   However, you will have to use the target-based system, and do more setup
   yourself, because it does not know about or include things that depend on
   Python, like ``pybind11_add_module``. This might be ideal for integrating
   into an existing system, like scikit-build's Python helpers.
307

308
309
310
311
312
313
314
315
316
317
318
319
320
321
How to cite this project?
=========================

We suggest the following BibTeX template to cite pybind11 in scientific
discourse:

.. code-block:: bash

    @misc{pybind11,
       author = {Wenzel Jakob and Jason Rhinelander and Dean Moldovan},
       year = {2017},
       note = {https://github.com/pybind/pybind11},
       title = {pybind11 -- Seamless operability between C++11 and Python}
    }