GitLab

* Fix async Python functors invoking from multiple C++ threads (#1587)

Ensure GIL is held during functor destruction.

* Add async Python callbacks test that runs in separate Python thread

This udpates all the remaining tests to the new test suite code and
comment styles started in #898.  For the most part, the test coverage
here is unchanged, with a few minor exceptions as noted below.

- test_constants_and_functions: this adds more overload tests with
  overloads with different number of arguments for more comprehensive
  overload_cast testing.  The test style conversion broke the overload
  tests under MSVC 2015, prompting the additional tests while looking
  for a workaround.

- test_eigen: this dropped the unused functions `get_cm_corners` and
  `get_cm_corners_const`--these same tests were duplicates of the same
  things provided (and used) via ReturnTester methods.

- test_opaque_types: this test had a hidden dependence on ExampleMandA
  which is now fixed by using the global UserType which suffices for the
  relevant test.

- test_methods_and_attributes: this required some additions to UserType
  to make it usable as a replacement for the test's previous SimpleType:
  UserType gained a value mutator, and the `value` property is not
  mutable (it was previously readonly).  Some overload tests were also
  added to better test overload_cast (as described above).

- test_numpy_array: removed the untemplated mutate_data/mutate_data_t:
  the templated versions with an empty parameter pack expand to the same
  thing.

- test_stl: this was already mostly in the new style; this just tweaks
  things a bit, localizing a class, and adding some missing
  `// test_whatever` comments.

- test_virtual_functions: like `test_stl`, this was mostly in the new
  test style already, but needed some `// test_whatever` comments.
  This commit also moves the inherited virtual example code to the end
  of the file, after the main set of tests (since it is less important
  than the other tests, and rather length); it also got renamed to
  `test_inherited_virtuals` (from `test_inheriting_repeat`) because it
  tests both inherited virtual approaches, not just the repeat approach.

If a bound std::function is invoked with a bound method, the implicit
bound self is lost because we use `detail::get_function` to unbox the
function.  This commit amends the code to use py::function and only
unboxes in the special is-really-a-c-function case.  This makes bound
methods stay bound rather than unbinding them by forcing extraction of
the c function.

* Fixed compilation error when defining function accepting some forms of std::function.

The compilation error happens only when the functional.h header is
present, and the build is done in debug mode, with NDEBUG being
undefined.  In addition, the std::function must accept an abstract
base class by reference.

The compilation error occurred in cast.h, when trying to construct a
std::tuple<AbstractBase>, rather than a std::tuple<AbstractBase&>.
This was caused by functional.h using std::move rather than
std::forward, changing the signature of the function being used.

This commit contains the fix, along with a test that exhibits the
issue when compiled in debug mode without the fix applied.

* Moved new std::function tests into test_callbacks, added callback_with_movable test.

A Python function can be called with the syntax:
```python
foo(a1, a2, *args, ka=1, kb=2, **kwargs)
```
This commit adds support for the equivalent syntax in C++:
```c++
foo(a1, a2, *args, "ka"_a=1, "kb"_a=2, **kwargs)
```

In addition, generalized unpacking is implemented, as per PEP 448,
which allows calls with multiple * and ** unpacking:
```python
bar(*args1, 99, *args2, 101, **kwargs1, kz=200, **kwargs2)
```
and
```c++
bar(*args1, 99, *args2, 101, **kwargs1, "kz"_a=200, **kwargs2)
```

Adding or removing tests is a little bit cumbersome currently: the test
needs to be added to CMakeLists.txt, the init function needs to be
predeclared in pybind11_tests.cpp, then called in the plugin
initialization.  While this isn't a big deal for tests that are being
committed, it's more of a hassle when working on some new feature or
test code for which I temporarily only care about building and linking
the test being worked on rather than the entire test suite.

This commit changes tests to self-register their initialization by
having each test initialize a local object (which stores the
initialization function in a static variable).  This makes changing the
set of tests being build easy: one only needs to add or comment out
test names in tests/CMakeLists.txt.

A couple other minor changes that go along with this:

- test_eigen.cpp is now included in the test list, then removed if eigen
  isn't available.  This lets you disable the eigen tests by commenting
  it out, just like all the other tests, but keeps the build working
  without eigen eigen isn't available.  (Also, if it's commented out, we
  don't even bother looking for and reporting the building with/without
  eigen status message).

- pytest is now invoked with all the built test names (with .cpp changed
  to .py) so that it doesn't try to run tests that weren't built.

The C++ part of the test code is modified to achieve this. As a result,
this kind of test:

```python
with capture:
    kw_func1(5, y=10)
assert capture == "kw_func(x=5, y=10)"
```

can be replaced with a simple:

`assert kw_func1(5, y=10) == "x=5, y=10"`

Use simple asserts and pytest's powerful introspection to make testing
simpler. This merges the old .py/.ref file pairs into simple .py files
where the expected values are right next to the code being tested.

This commit does not touch the C++ part of the code and replicates the
Python tests exactly like the old .ref-file-based approach.

This commit rewrites the examples that look for constructor/destructor
calls to do so via static variable tracking rather than output parsing.

The added ConstructorStats class provides methods to keep track of
constructors and destructors, number of default/copy/move constructors,
and number of copy/move assignments.  It also provides a mechanism for
storing values (e.g. for value construction), and then allows all of
this to be checked at the end of a test by getting the statistics for a
C++ (or python mapping) class.

By not relying on the precise pattern of constructions/destructions,
but rather simply ensuring that every construction is matched with a
destruction on the same object, we ensure that everything that gets
created also gets destroyed as expected.

This replaces all of the various "std::cout << whatever" code in
constructors/destructors with
`print_created(this)`/`print_destroyed(this)`/etc. functions which
provide similar output, but now has a unified format across the
different examples, including a new ### prefix that makes mixed example
output and lifecycle events easier to distinguish.

With this change, relaxed mode is no longer needed, which enables
testing for proper destruction under MSVC, and under any other compiler
that generates code calling extra constructors, or optimizes away any
constructors.  GCC/clang are used as the baseline for move
constructors; the tests are adapted to allow more move constructors to
be evoked (but other types are constructors much have matching counts).

This commit also disables output buffering of tests, as the buffering
sometimes results in C++ output ending up in the middle of python
output (or vice versa), depending on the OS/python version.

This renames example files from `exampleN` to `example-description`.

Specifically, the following renaming is applied:

example1 -> example-methods-and-attributes
example2 -> example-python-types
example3 -> example-operator-overloading
example4 -> example-constants-and-functions
example5 -> example-callbacks (*)
example6 -> example-sequence-and-iterators
example7 -> example-buffers
example8 -> example-custom-ref-counting
example9 -> example-modules
example10 -> example-numpy-vectorize
example11 -> example-arg-keywords-and-defaults
example12 -> example-virtual-functions
example13 -> example-keep-alive
example14 -> example-opaque-types
example15 -> example-pickling
example16 -> example-inheritance
example17 -> example-stl-binders
example18 -> example-eval
example19 -> example-custom-exceptions

* the inheritance parts of example5 are moved into example-inheritance
(previously example16), and the remainder is left as example-callbacks.

This commit also renames the internal variables ("Example1",
"Example2", "Example4", etc.) into non-numeric names ("ExampleMandA",
"ExamplePythonTypes", "ExampleWithEnum", etc.) to correspond to the
file renaming.

The order of tests is preserved, but this can easily be changed if
there is some more natural ordering by updating the list in
examples/CMakeLists.txt.

- new pybind11::base<> attribute to indicate a subclass relationship
- unified infrastructure for parsing variadic arguments in class_ and cpp_function
- use 'handle' and 'object' more consistently everywhere