GitLab

* Add option to force the use of the PYPY GIL scoped acquire/release logic to support nested gil access, see https://github.com/pybind/pybind11/issues/1276 and https://github.com/pytorch/pytorch/issues/83101



* Apply suggestions from code review

* Update CMakeLists.txt

* docs: update upgrade guide

* Update docs/upgrade.rst

* All bells & whistles.

* Add Reminder to common.h, so that we will not forget to purge `!WITH_THREAD` branches when dropping Python 3.6

* New sentence instead of semicolon.

* Temporarily pull in snapshot of PR #4246

* Add `test_release_acquire`

* Add more unit tests for nested gil locking

* Add test_report_builtins_internals_keys

* Very minor enhancement: sort list only after filtering.

* Revert change in docs/upgrade.rst

* Add test_multi_acquire_release_cross_module, while also forcing unique PYBIND11_INTERNALS_VERSION for cross_module_gil_utils.cpp

* Hopefully fix apparently new ICC error.

```
2022-10-28T07:57:54.5187728Z -- The CXX compiler identification is Intel 2021.7.0.20220726
...
2022-10-28T07:58:53.6758994Z icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
2022-10-28T07:58:54.5801597Z In file included from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../detail/type_caster_base.h(15),
2022-10-28T07:58:54.5803794Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../cast.h(15),
2022-10-28T07:58:54.5805740Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../attr.h(14),
2022-10-28T07:58:54.5809556Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/detail/class.h(12),
2022-10-28T07:58:54.5812154Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/pybind11.h(13),
2022-10-28T07:58:54.5948523Z                  from /home/runner/work/pybind11/pybind11/tests/cross_module_gil_utils.cpp(13):
2022-10-28T07:58:54.5949009Z /home/runner/work/pybind11/pybind11/include/pybind11/detail/../detail/internals.h(177): error #2282: unrecognized GCC pragma
2022-10-28T07:58:54.5949374Z       PYBIND11_TLS_KEY_INIT(tstate)
2022-10-28T07:58:54.5949579Z       ^
2022-10-28T07:58:54.5949695Z
```

* clang-tidy fixes

* Workaround for PYPY WIN exitcode None

* Revert "Temporarily pull in snapshot of PR #4246"

This reverts commit 23ac16e859150f27fda25ca865cabcb4444e0770.

* Another workaround for PYPY WIN exitcode None

* Clean up how the tests are run "run in process" Part 1: uniformity

* Clean up how the tests are run "run in process" Part 2: use `@pytest.mark.parametrize` and clean up the naming.

* Skip some tests `#if defined(THREAD_SANITIZER)` (tested with TSAN using the Google-internal toolchain).

* Run all tests again but ignore ThreadSanitizer exitcode 66 (this is less likely to mask unrelated ThreadSanitizer issues in the future).

* bug fix: missing common.h include before using `PYBIND11_SIMPLE_GIL_MANAGEMENT`

For the tests in the github CI this does not matter, because
`PYBIND11_SIMPLE_GIL_MANAGEMENT` is always defined from the command line,
but when monkey-patching common.h locally, it matters.

* if process.exitcode is None: assert t_delta > 9.9

* More sophisiticated `_run_in_process()` implementation, clearly reporting `DEADLOCK`, additionally exercised via added `intentional_deadlock()`

* Wrap m.intentional_deadlock in a Python function, for `ForkingPickler` compatibility.

```
>       ForkingPickler(file, protocol).dump(obj)
E       TypeError: cannot pickle 'PyCapsule' object
```

Observed with all Windows builds including mingw but not PyPy, and macos-latest with Python 3.9, 3.10, 3.11 but not 3.6.

* Add link to potential solution for WOULD-BE-NICE-TO-HAVE feature.

* Add `SKIP_IF_DEADLOCK = True` option, to not pollute the CI results with expected `DEADLOCK` failures while we figure out what to do about them.

* Add COPY-PASTE-THIS: gdb ... command (to be used for debugging the detected deadlock)

* style: pre-commit fixes

* Do better than automatic pre-commit fixes.

* Add `PYBIND11_SIMPLE_GIL_MANAGEMENT` to `pytest_report_header()` (so that we can easily know when harvesting deadlock information from the CI logs).
Co-authored-by: Arnim Balzer <arnim@seechange.ai>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Report `C++ Info:` from `pytest_configure()`

* Use pytest_report_header() as suggested by @skylion007

Replace error printing code gated by NDEBUG with a new flag: PYBIND11_DETAILED_ERROR_MESSAGES  (#3913)

* Update cast.h

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Move definition to detail/common, change name, apply everywhere

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename debug_enabled in tests to detailed_error_messages_enabled

* chore: add clang-format

* Removing check-style (Classic check-style)

Ported from @henryiii's https://github.com/pybind/pybind11/pull/3683/commits/53056b1b0eeb4136b0d7362a8261b6b59658e0a7



* Automatic clang-format changes (NO manual changes).
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>

clang-tidy automatic changes. NO manual changes.

* WIP: module -> module_ without typedef

* refactor: allow py::module to work again

* tests: refactor and cleanup

* refactor: more consistent

* tests: vendor six

* tests: more xfails, nicer system

* tests: simplify to info

* tests: suggestions from @YannickJadoul and @bstaletic

* tests: restore some pypy tests that now pass

* tests: rename info to env

* tests: strict False/True

* tests: drop explicit strict=True again

* tests: reduce minimum PyTest to 3.1

This udpates all the remaining tests to the new test suite code and
comment styles started in #898.  For the most part, the test coverage
here is unchanged, with a few minor exceptions as noted below.

- test_constants_and_functions: this adds more overload tests with
  overloads with different number of arguments for more comprehensive
  overload_cast testing.  The test style conversion broke the overload
  tests under MSVC 2015, prompting the additional tests while looking
  for a workaround.

- test_eigen: this dropped the unused functions `get_cm_corners` and
  `get_cm_corners_const`--these same tests were duplicates of the same
  things provided (and used) via ReturnTester methods.

- test_opaque_types: this test had a hidden dependence on ExampleMandA
  which is now fixed by using the global UserType which suffices for the
  relevant test.

- test_methods_and_attributes: this required some additions to UserType
  to make it usable as a replacement for the test's previous SimpleType:
  UserType gained a value mutator, and the `value` property is not
  mutable (it was previously readonly).  Some overload tests were also
  added to better test overload_cast (as described above).

- test_numpy_array: removed the untemplated mutate_data/mutate_data_t:
  the templated versions with an empty parameter pack expand to the same
  thing.

- test_stl: this was already mostly in the new style; this just tweaks
  things a bit, localizing a class, and adding some missing
  `// test_whatever` comments.

- test_virtual_functions: like `test_stl`, this was mostly in the new
  test style already, but needed some `// test_whatever` comments.
  This commit also moves the inherited virtual example code to the end
  of the file, after the main set of tests (since it is less important
  than the other tests, and rather length); it also got renamed to
  `test_inherited_virtuals` (from `test_inheriting_repeat`) because it
  tests both inherited virtual approaches, not just the repeat approach.

This commit also adds `doc()` to `object_api` as a shortcut for the
`attr("__doc__")` accessor.

The module macro changes from:
```c++
PYBIND11_PLUGIN(example) {
    pybind11::module m("example", "pybind11 example plugin");
    m.def("add", [](int a, int b) { return a + b; });
    return m.ptr();
}
```

to:

```c++
PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin";
    m.def("add", [](int a, int b) { return a + b; });
}
```

Using the old macro results in a deprecation warning. The warning
actually points to the `pybind11_init` function (since attributes
don't bind to macros), but the message should be quite clear:
"PYBIND11_PLUGIN is deprecated, use PYBIND11_MODULE".

When we are returned a base class pointer (either directly or via
shared_from_this()) we detect its runtime type (using `typeid`), then
end up essentially reinterpret_casting the pointer to the derived type.
This is invalid when the base class pointer was a non-first base, and we
end up with an invalid pointer.  We could dynamic_cast to the
most-derived type, but if *that* type isn't pybind11-registered, the
resulting pointer given to the base `cast` implementation isn't necessarily valid
to be reinterpret_cast'ed back to the backup type.

This commit removes the "backup" type argument from the many-argument
`cast(...)` and instead does the derived-or-pointer type decision and
type lookup in type_caster_base, where the dynamic_cast has to be to
correctly get the derived pointer, but also has to do the type lookup to
ensure that we don't pass the wrong (derived) pointer when the backup
type (i.e. the type caster intrinsic type) pointer is needed.

Since the lookup is needed before calling the base cast(), this also
changes the input type to a detail::type_info rather than doing a
(second) lookup in cast().

Issue #633 suggests people might be tempted to copy the test scripts
self-binding code, but that's a bad idea for pretty much anything other
than a test suite with self-contained test code.

This commit adds a comment as such with a reference to the
documentation that tells people how to do it instead.

* Minor doc syntax fix

The numpy documentation had a bad :file: reference (was using double
backticks instead of single backticks).

* Changed long-outdated "example" -> "tests" wording

The ConstructorStats internal docs still had "from example import", and
the main testing cpp file still used "example" in the module
description.

This adds automatic casting when assigning to python types like dict,
list, and attributes.  Instead of:

    dict["key"] = py::cast(val);
    m.attr("foo") = py::cast(true);
    list.append(py::cast(42));

you can now simply write:

    dict["key"] = val;
    m.attr("foo") = true;
    list.append(42);

Casts needing extra parameters (e.g. for a non-default rvp) still
require the py::cast() call. set::add() is also supported.

All usage is channeled through a SFINAE implementation which either just returns or casts. 

Combined non-converting handle and autocasting template methods via a
helper method that either just returns (handle) or casts (C++ type).

This also adds the `hasattr` and `getattr` functions which are needed
with the new attribute behavior. The new functions behave exactly like
their Python counterparts.

Similarly `object` gets a `contains` method which calls `__contains__`,
i.e. it's the same as the `in` keyword in Python.

Adding or removing tests is a little bit cumbersome currently: the test
needs to be added to CMakeLists.txt, the init function needs to be
predeclared in pybind11_tests.cpp, then called in the plugin
initialization.  While this isn't a big deal for tests that are being
committed, it's more of a hassle when working on some new feature or
test code for which I temporarily only care about building and linking
the test being worked on rather than the entire test suite.

This commit changes tests to self-register their initialization by
having each test initialize a local object (which stores the
initialization function in a static variable).  This makes changing the
set of tests being build easy: one only needs to add or comment out
test names in tests/CMakeLists.txt.

A couple other minor changes that go along with this:

- test_eigen.cpp is now included in the test list, then removed if eigen
  isn't available.  This lets you disable the eigen tests by commenting
  it out, just like all the other tests, but keeps the build working
  without eigen eigen isn't available.  (Also, if it's commented out, we
  don't even bother looking for and reporting the building with/without
  eigen status message).

- pytest is now invoked with all the built test names (with .cpp changed
  to .py) so that it doesn't try to run tests that weren't built.

- ICPC can't handle the NCVirt trampoline which returns a non-copyable
  type, which is likely due to a constexpr/SFINAE issue. This disables
  the test on that compiler so that at least the rest can be tested.

There are more enum tests than 'constants and functions'.

Use simple asserts and pytest's powerful introspection to make testing
simpler. This merges the old .py/.ref file pairs into simple .py files
where the expected values are right next to the code being tested.

This commit does not touch the C++ part of the code and replicates the
Python tests exactly like the old .ref-file-based approach.

This commit rewrites the examples that look for constructor/destructor
calls to do so via static variable tracking rather than output parsing.

The added ConstructorStats class provides methods to keep track of
constructors and destructors, number of default/copy/move constructors,
and number of copy/move assignments.  It also provides a mechanism for
storing values (e.g. for value construction), and then allows all of
this to be checked at the end of a test by getting the statistics for a
C++ (or python mapping) class.

By not relying on the precise pattern of constructions/destructions,
but rather simply ensuring that every construction is matched with a
destruction on the same object, we ensure that everything that gets
created also gets destroyed as expected.

This replaces all of the various "std::cout << whatever" code in
constructors/destructors with
`print_created(this)`/`print_destroyed(this)`/etc. functions which
provide similar output, but now has a unified format across the
different examples, including a new ### prefix that makes mixed example
output and lifecycle events easier to distinguish.

With this change, relaxed mode is no longer needed, which enables
testing for proper destruction under MSVC, and under any other compiler
that generates code calling extra constructors, or optimizes away any
constructors.  GCC/clang are used as the baseline for move
constructors; the tests are adapted to allow more move constructors to
be evoked (but other types are constructors much have matching counts).

This commit also disables output buffering of tests, as the buffering
sometimes results in C++ output ending up in the middle of python
output (or vice versa), depending on the OS/python version.

This renames example files from `exampleN` to `example-description`.

Specifically, the following renaming is applied:

example1 -> example-methods-and-attributes
example2 -> example-python-types
example3 -> example-operator-overloading
example4 -> example-constants-and-functions
example5 -> example-callbacks (*)
example6 -> example-sequence-and-iterators
example7 -> example-buffers
example8 -> example-custom-ref-counting
example9 -> example-modules
example10 -> example-numpy-vectorize
example11 -> example-arg-keywords-and-defaults
example12 -> example-virtual-functions
example13 -> example-keep-alive
example14 -> example-opaque-types
example15 -> example-pickling
example16 -> example-inheritance
example17 -> example-stl-binders
example18 -> example-eval
example19 -> example-custom-exceptions

* the inheritance parts of example5 are moved into example-inheritance
(previously example16), and the remainder is left as example-callbacks.

This commit also renames the internal variables ("Example1",
"Example2", "Example4", etc.) into non-numeric names ("ExampleMandA",
"ExamplePythonTypes", "ExampleWithEnum", etc.) to correspond to the
file renaming.

The order of tests is preserved, but this can easily be changed if
there is some more natural ordering by updating the list in
examples/CMakeLists.txt.

1. Renamed PYTHON_* to PYBIND_* everywhere

2. Renamed pybind::array_dtype<> to pybind::array_t<>