GitLab

* scoped_interpreter overloaded ctor: PyConfig param

* style: pre-commit fixes

* refact: some logics extracted into funcs (precheck_interpreter, _initialize_interpreter); config_guard

* style: pre-commit fixes

* refact: scoped_config, some funcs hidden in detail ns

* refact: macro PYBIND11_PYCONFIG_SUPPORT_PY_VERSION + undef

* feat: PYBIND11_PYCONFIG_SUPPORT_PY_VERSION set to 3.8

* tests: Custom PyConfig

* ci: python 3.6 -> 3.8

* ci: reverted py 38 back  to 36; refact: initialize_interpreter overloads

* style: pre-commit fixes

* fix: readability-implicit-bool-conversion

* refact: each initialize_interpreter overloads in pybind11 ns

* Move `initialize_interpreter_pre_pyconfig()` into the `detail` namespace.

Move the `PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX` define down to where it is used for the first time, and check if it is defined already, so that it is possible to customize from the compilation command line, just in case there is some unforeseen issue for Python 3.8, 3.9, 3.10.

* tests: Add program dir to path, Custom PyConfig with argv

* refact: clang-formatted

* tests: Add-program-dir-to-path covers both scoped_interpreter overloads

* tests: Add-program-dir-to-path fixed

* tests: Add-program-dir-to-path py_version dependant validation
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>

* Introduce a new warning suppression system

* Switch to better name

* Nits

* Add option to force the use of the PYPY GIL scoped acquire/release logic to support nested gil access, see https://github.com/pybind/pybind11/issues/1276 and https://github.com/pytorch/pytorch/issues/83101



* Apply suggestions from code review

* Update CMakeLists.txt

* docs: update upgrade guide

* Update docs/upgrade.rst

* All bells & whistles.

* Add Reminder to common.h, so that we will not forget to purge `!WITH_THREAD` branches when dropping Python 3.6

* New sentence instead of semicolon.

* Temporarily pull in snapshot of PR #4246

* Add `test_release_acquire`

* Add more unit tests for nested gil locking

* Add test_report_builtins_internals_keys

* Very minor enhancement: sort list only after filtering.

* Revert change in docs/upgrade.rst

* Add test_multi_acquire_release_cross_module, while also forcing unique PYBIND11_INTERNALS_VERSION for cross_module_gil_utils.cpp

* Hopefully fix apparently new ICC error.

```
2022-10-28T07:57:54.5187728Z -- The CXX compiler identification is Intel 2021.7.0.20220726
...
2022-10-28T07:58:53.6758994Z icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
2022-10-28T07:58:54.5801597Z In file included from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../detail/type_caster_base.h(15),
2022-10-28T07:58:54.5803794Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../cast.h(15),
2022-10-28T07:58:54.5805740Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../attr.h(14),
2022-10-28T07:58:54.5809556Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/detail/class.h(12),
2022-10-28T07:58:54.5812154Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/pybind11.h(13),
2022-10-28T07:58:54.5948523Z                  from /home/runner/work/pybind11/pybind11/tests/cross_module_gil_utils.cpp(13):
2022-10-28T07:58:54.5949009Z /home/runner/work/pybind11/pybind11/include/pybind11/detail/../detail/internals.h(177): error #2282: unrecognized GCC pragma
2022-10-28T07:58:54.5949374Z       PYBIND11_TLS_KEY_INIT(tstate)
2022-10-28T07:58:54.5949579Z       ^
2022-10-28T07:58:54.5949695Z
```

* clang-tidy fixes

* Workaround for PYPY WIN exitcode None

* Revert "Temporarily pull in snapshot of PR #4246"

This reverts commit 23ac16e859150f27fda25ca865cabcb4444e0770.

* Another workaround for PYPY WIN exitcode None

* Clean up how the tests are run "run in process" Part 1: uniformity

* Clean up how the tests are run "run in process" Part 2: use `@pytest.mark.parametrize` and clean up the naming.

* Skip some tests `#if defined(THREAD_SANITIZER)` (tested with TSAN using the Google-internal toolchain).

* Run all tests again but ignore ThreadSanitizer exitcode 66 (this is less likely to mask unrelated ThreadSanitizer issues in the future).

* bug fix: missing common.h include before using `PYBIND11_SIMPLE_GIL_MANAGEMENT`

For the tests in the github CI this does not matter, because
`PYBIND11_SIMPLE_GIL_MANAGEMENT` is always defined from the command line,
but when monkey-patching common.h locally, it matters.

* if process.exitcode is None: assert t_delta > 9.9

* More sophisiticated `_run_in_process()` implementation, clearly reporting `DEADLOCK`, additionally exercised via added `intentional_deadlock()`

* Wrap m.intentional_deadlock in a Python function, for `ForkingPickler` compatibility.

```
>       ForkingPickler(file, protocol).dump(obj)
E       TypeError: cannot pickle 'PyCapsule' object
```

Observed with all Windows builds including mingw but not PyPy, and macos-latest with Python 3.9, 3.10, 3.11 but not 3.6.

* Add link to potential solution for WOULD-BE-NICE-TO-HAVE feature.

* Add `SKIP_IF_DEADLOCK = True` option, to not pollute the CI results with expected `DEADLOCK` failures while we figure out what to do about them.

* Add COPY-PASTE-THIS: gdb ... command (to be used for debugging the detected deadlock)

* style: pre-commit fixes

* Do better than automatic pre-commit fixes.

* Add `PYBIND11_SIMPLE_GIL_MANAGEMENT` to `pytest_report_header()` (so that we can easily know when harvesting deadlock information from the CI logs).
Co-authored-by: Arnim Balzer <arnim@seechange.ai>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add debug fprintf to test_interpreter.cpp

* Update `sys.path` from `PYTHONPATH` in Python >= 3.11 branch of `initialize_interpreter()`

* Use `config.isolated = 0; config.use_environment = 1;`

As suggsted by @vstinner here: https://github.com/pybind/pybind11/pull/4119#issuecomment-1219442853



* Add `TEST_CASE("PYTHONPATH is used to update sys.path")`

* Fix clang-tidy error.

* Use `_putenv_s()` under Windows.

* Fix clang-tidy error: argument name ... in comment does not match parameter name

* Remove slash from PYTHONPATH addition, to work around Windows slash-vs-backslash issue.

* Use `py::str(...)` instead of `.attr("__str__")` as suggested by @skylion007
Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>

* Clear local internals after finalizing interpreter

* Add descriptive comments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* `#error BYE_BYE_GOLDEN_SNAKE`

* Removing everything related to 2.7 from ci.yml

* Commenting-out Centos7

* Removing `PYTHON: 27` from .appveyor.yml

* "PY2" removal, mainly from tests. C++ code is not touched.

* Systematic removal of `u` prefix from `u"..."` and `u'...'` literals. Collateral cleanup of a couple minor other things.

* Cleaning up around case-insensitive hits for `[^a-z]py.*2` in tests/.

* Removing obsolete Python 2 mention in compiling.rst

* Proper `#error` for Python 2.

* Using PY_VERSION_HEX to guard `#error "PYTHON 2 IS NO LONGER SUPPORTED.`

* chore: bump pre-commit

* style: run pre-commit for pyupgrade 3+

* tests: use sys.version_info, not PY

* chore: more Python 2 removal

* Uncommenting Centos7 block (PR #3691 showed that it is working again).

* Update pre-commit hooks

* Fix pre-commit hook

* refactor: remove Python 2 from CMake

* refactor: remove Python 2 from setup code

* refactor: simplify, better static typing

* feat: fail with nice messages

* refactor: drop Python 2 C++ code

* docs: cleanup for Python 3

* revert: intree

revert: intree

* docs: minor touchup to py2 statement
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>

* chore: add clang-format

* Removing check-style (Classic check-style)

Ported from @henryiii's https://github.com/pybind/pybind11/pull/3683/commits/53056b1b0eeb4136b0d7362a8261b6b59658e0a7



* Automatic clang-format changes (NO manual changes).
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>

* Adding readability-qualified-auto to .clang-tidy

Ported from @henryiii's https://github.com/pybind/pybind11/pull/3683/commits/287527f705c8badee8ba9f6278d691fe0450ae43



* fix: support Python < 3.6
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>

clang-tidy automatic changes. NO manual changes.

* override: Fix wrong caching of the overrides

There was a problem when the python type, which was stored in override
cache for C++ functions, was destroyed and  the record wasn't removed from the
override cache. Therefor, dangling pointer was stored there. Then when the
memory was reused and new type was allocated at the given address and the
method with the same name (as previously stored in the cache) was actually
overridden in python, it would wrongly find it in the override cache for C++
functions and therefor override from python wouldn't be called.
The fix is to erase the type from the override cache when the type is destroyed.

* test: Pass by const ref instead of by value (clang-tidy)

* test: Rename classes and move to different files

Rename the classes and files so they're no too generic. Also, better place to
test the stuff is in test_virtual_functions.cpp/.py as we're basically testing
the virtual functions/trampolines.

* Add TODO for erasure code

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Disambiguate free() to use std::free()

* Add cstdlib include

* Adding google-explicit-constructor to .clang-tidy

* clang-tidy explicit attr.h (all automatic)

* clang-tidy explicit cast.h (all automatic)

* clang-tidy detail/init.h (1 NOLINT)

* clang-tidy detail/type_caster_base.h (2 NOLINT)

* clang-tidy pybind11.h (7 NOLINT)

* clang-tidy detail/common.h (3 NOLINT)

* clang-tidy detail/descr.h (2 NOLINT)

* clang-tidy pytypes.h (23 NOLINT, only 1 explicit)

* clang-tidy eigen.h (7 NOLINT, 0 explicit)

* Adding 2 explicit in functional.h

* Adding 4 explicit in iostream.h

* clang-tidy numpy.h (1 NOLINT, 1 explicit)

* clang-tidy embed.h (0 NOLINT, 1 explicit)

* clang-tidy tests/local_bindings.h (0 NOLINT, 4 explicit)

* clang-tidy tests/pybind11_cross_module_tests.cpp (0 NOLINT, 1 explicit)

* clang-tidy tests/pybind11_tests.h (0 NOLINT, 2 explicit)

* clang-tidy tests/test_buffers.cpp (0 NOLINT, 2 explicit)

* clang-tidy tests/test_builtin_casters.cpp (0 NOLINT, 4 explicit)

* clang-tidy tests/test_class.cpp (0 NOLINT, 6 explicit)

* clang-tidy tests/test_copy_move.cpp (0 NOLINT, 7 explicit)

* clang-tidy tests/test_embed/external_module.cpp (0 NOLINT, 1 explicit)

* clang-tidy tests/test_embed/test_interpreter.cpp (0 NOLINT, 1 explicit)

* clang-tidy tests/object.h (0 NOLINT, 2 explicit)

* clang-tidy batch of fully automatic fixes.

* Workaround for MSVC 19.16.27045.0 C++17 Python 2 C++ syntax error.

* Call PySys_SetArgv when initializing interpreter.

* Document argc/argv parameters in initialize_interpreter.

* Remove manual memory management from set_interpreter_argv in favor of smart pointers.

* Use size_t for indexers in set_interpreter_argv.

* Minimize macros for flow control in set_interpreter_argv.

* Fix 'unused variable' warning on Py2

* whitespace

* Define wide_char_arg_deleter outside set_interpreter_argv.

* Do sys.path workaround in C++ rather than eval.

* Factor out wchar conversion to a separate function.

* Restore widened_argv variable declaration.

* Fix undeclared widened_arg variable on some paths.

* Use delete[] to match new wchar_t[].

* Fix compiler errors

* Use PY_VERSION_HEX for a cleaner CVE-2008-5983 mode check.

* Fix typo

* Use explicit type for deleter so delete[] works cross-compiler.

* Always use PySys_SetArgvEx because pybind11 doesn't support pythons that don't include it.

* Remove pointless ternary operator.

* Use unique_ptr.reset instead of a second initialization.

* Rename add_program_dir_to_path parameter to clarify intent.

* Add defined() check before evaluating HAVE_BROKEN_MBSTOWCS.

* Apply clang-tidy fixes

* Pre-commit

* refactor: use const for set_interpreter_argv

* Try to fix const issue and allocate vector properly

* fix: copy strings on Python 2

* Applying clang-format-diff relative to master.

The only manual change is an added empty line between pybind11 and system `#include`s.

```
git diff -U0 --no-color master | python3 $HOME/clone/llvm-project/clang/tools/clang-format/clang-format-diff.py -p1 -style=file -i
```
Co-authored-by: Boris Staletic <boris.staletic@gmail.com>
Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>

* Add py::raise_from to enable chaining exceptions on Python 3.3+

* Use 'raise from' in initialization

* Documenting the exact base version of _PyErr_FormatVFromCause, adding back `assert`s.
Co-authored-by: Dustin Spicuzza <dustin@virtualroadside.com>

* maint(clang-tidy) Improve code readability

* Fix minor typos

* Revert optimization that removed test case

* Fix comment formatting

* Revert another optimization to repro an issue

* Remove make_unique since it C++14 and newer only

* eformat comments

* Fix unsignedness of comparison

* Update comment

* maint(clang-tidy) Improve code readability

* Fix minor typos

* Revert optimization that removed test case

* Fix comment formatting

* Revert another optimization to repro an issue

* Remove make_unique since it C++14 and newer only

* eformat comments

* Fix unsignedness of comparison

* Update comment

* Initial fixes

* Whoops

* Finish clang-tidy manual fixes

* Add two missing fixes

* Revert

* Update clang-tidy

* Try to fix unreachable code error

* Move nolint comment

* Apply missing fix

* Don't override clang-tidy config

* Does this fix clang-tidy?

* Make all clang-tidy errors visible

* Add comments about NOLINTs and remove a few

* Fix typo

* WIP: module -> module_ without typedef

* refactor: allow py::module to work again

Resolve empty statement warning when using PYBIND11_OVERLOAD_PURE_NAME and PYBIND11_OVERLOAD_PURE (#2325)

* Wrap PYBIND11_OVERLOAD_NAME and PYBIND11_OVERLOAD_PURE_NAME in do { ... } while (false), and resolve trailing semicolon

* Deprecate PYBIND11_OVERLOAD_* and get_overload in favor of PYBIND11_OVERRIDE_* and get_override

* Correct erroneous usage of 'overload' instead of 'override' in the implementation and internals

* Fix tests to use non-deprecated PYBIND11_OVERRIDE_* macros

* Update docs to use override instead of overload where appropriate, and add warning about deprecated aliases

* Add semicolons to deprecated PYBIND11_OVERLOAD macros to match original behavior

* Remove deprecation of PYBIND11_OVERLOAD_* macros and get_overload

* Add note to changelog and upgrade guide

* Fix segfault when reloading interpreter with external modules

When embedding the interpreter and loading external modules in that
embedded interpreter, the external module correctly shares its
internals_ptr with the one in the embedded interpreter.  When the
interpreter is shut down, however, only the `internals_ptr` local to
the embedded code is actually reset to nullptr: the external module
remains set.

The result is that loading an external pybind11 module, letting the
interpreter go through a finalize/initialize, then attempting to use
something in the external module fails because this external module is
still trying to use the old (destroyed) internals.  This causes
undefined behaviour (typically a segfault).

This commit fixes it by adding a level of indirection in the internals
path, converting the local internals variable to `internals **` instead
of `internals *`.  With this change, we can detect a stale internals
pointer and reload the internals pointer (either from a capsule or by
creating a new internals instance).

(No issue number: this was reported on gitter by @henryiii and @aoloe).

In the latest MSVC in C++17 mode including Eigen causes warnings:

    warning C4996: 'std::unary_negate<_Fn>': warning STL4008: std::not1(),
    std::not2(), std::unary_negate, and std::binary_negate are deprecated in
    C++17. They are superseded by std::not_fn(). You can define
    _SILENCE_CXX17_NEGATORS_DEPRECATION_WARNING or
    _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have
    received this warning.

This disables 4996 for the Eigen includes.

Catch generates a similar warning for std::uncaught_exception, so
disable the warning there, too.

In both cases this is temporary; we can (and should) remove the warnings
disabling once new upstream versions of Eigen and Catch are available
that address the warning. (The Catch one, in particular, looks to be
fixed in upstream master, so will probably be fixed in the next (2.0.2)
release).

Fixes a race condition when multiple threads try to acquire the GIL
before `detail::internals` have been initialized. `gil_scoped_release`
is now tasked with initializing `internals` (guaranteed single-threaded)
to ensure the safety of subsequent `acquire` calls from multiple threads.

Py_Finalize could potentially invoke code that calls `get_internals()`,
which could create a new internals object if one didn't exist.
`finalize_interpreter()` didn't catch this because it only used the
pre-finalize interpreter pointer status; if this happens, it results in
the internals pointer not being properly destroyed with the interpreter,
which leaks, and also causes a `get_internals()` under a future
interpreter to return an internals object that is wrong in various ways.

Fixes #887.

At this point, there is only a single test for interpreter basics.

Apart from embedding itself, having a C++ test framework will also
benefit the C++-side features by allowing them to be tested directly.