Improve documentation of Python and C++ exceptions (#2408)

The main change is to treat error_already_set as a separate category
of exception that arises in different circumstances and needs to be
handled differently. The asymmetry between Python and C++ exceptions
is further emphasized.

.gitignore

@@ -38,3 +38,4 @@ sosize-*.txt
 pybind11Config*.cmake
 pybind11Targets.cmake
 /*env*
+/.vscode

docs/advanced/exceptions.rst

 Exceptions
 ##########
 
-Built-in exception translation
-==============================
+Built-in C++ to Python exception translation
+============================================
+
+When Python calls C++ code through pybind11, pybind11 provides a C++ exception handler
+that will trap C++ exceptions, translate them to the corresponding Python exception,
+and raise them so that Python code can handle them.
 
-When C++ code invoked from Python throws an ``std::exception``, it is
-automatically converted into a Python ``Exception``. pybind11 defines multiple
-special exception classes that will map to different types of Python
-exceptions:
+pybind11 defines translations for ``std::exception`` and its standard
+subclasses, and several special exception classes that translate to specific
+Python exceptions. Note that these are not actually Python exceptions, so they
+cannot be examined using the Python C API. Instead, they are pure C++ objects
+that pybind11 will translate the corresponding Python exception when they arrive
+at its exception handler.
 
 .. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}|
 
 +--------------------------------------+--------------------------------------+
-|  C++ exception type                  |  Python exception type               |
+|  Exception thrown by C++             |  Translated to Python exception type |
 +======================================+======================================+
 | :class:`std::exception`              | ``RuntimeError``                     |
 +--------------------------------------+--------------------------------------+
@@ -46,22 +52,11 @@ exceptions:
 |                                      | ``__setitem__`` in dict-like         |
 |                                      | objects, etc.)                       |
 +--------------------------------------+--------------------------------------+
-| :class:`pybind11::error_already_set` | Indicates that the Python exception  |
-|                                      | flag has already been set via Python |
-|                                      | API calls from C++ code; this C++    |
-|                                      | exception is used to propagate such  |
-|                                      | a Python exception back to Python.   |
-+--------------------------------------+--------------------------------------+
 
-When a Python function invoked from C++ throws an exception, pybind11 will convert
-it into a C++ exception of type :class:`error_already_set` whose string payload
-contains a textual summary. If you call the Python C-API directly, and it
-returns an error, you should ``throw py::error_already_set();``, which allows
-pybind11 to deal with the exception and pass it back to the Python interpreter.
-(Another option is to call ``PyErr_Clear`` in the
-`Python C-API <https://docs.python.org/3/c-api/exceptions.html#c.PyErr_Clear>`_
-to clear the error. The Python error must be thrown or cleared, or Python/pybind11
-will be left in an invalid state.)
+Exception translation is not bidirectional. That is, *catching* the C++
+exceptions defined above above will not trap exceptions that originate from
+Python. For that, catch :class:`pybind11::error_already_set`. See :ref:`below
+<handling_python_exceptions_cpp>` for further details.
 
 There is also a special exception :class:`cast_error` that is thrown by
 :func:`handle::call` when the input arguments cannot be converted to Python
@@ -106,7 +101,6 @@ and use this in the associated exception translator (note: it is often useful
 to make this a static declaration when using it inside a lambda expression
 without requiring capturing).
 
-
 The following example demonstrates this for a hypothetical exception classes
 ``MyCustomException`` and ``OtherException``: the first is translated to a
 custom python exception ``MyCustomError``, while the second is translated to a
@@ -140,7 +134,7 @@ section.
 
 .. note::
 
-    You must call either ``PyErr_SetString`` or a custom exception's call
+    Call either ``PyErr_SetString`` or a custom exception's call
     operator (``exc(string)``) for every exception caught in a custom exception
     translator.  Failure to do so will cause Python to crash with ``SystemError:
     error return without exception set``.
@@ -149,6 +143,103 @@ section.
     may be explicitly (re-)thrown to delegate it to the other,
     previously-declared existing exception translators.
 
+.. _handling_python_exceptions_cpp:
+
+Handling exceptions from Python in C++
+======================================
+
+When C++ calls Python functions, such as in a callback function or when
+manipulating Python objects, and Python raises an ``Exception``, pybind11
+converts the Python exception into a C++ exception of type
+:class:`pybind11::error_already_set` whose payload contains a C++ string textual
+summary and the actual Python exception. ``error_already_set`` is used to
+propagate Python exception back to Python (or possibly, handle them in C++).
+
+.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}|
+
++--------------------------------------+--------------------------------------+
+|  Exception raised in Python          |  Thrown as C++ exception type        |
++======================================+======================================+
+| Any Python ``Exception``             | :class:`pybind11::error_already_set` |
++--------------------------------------+--------------------------------------+
+
+For example:
+
+.. code-block:: cpp
+
+    try {
+        // open("missing.txt", "r")
+        auto file = py::module::import("io").attr("open")("missing.txt", "r");
+        auto text = file.attr("read")();
+        file.attr("close")();
+    } catch (py::error_already_set &e) {
+        if (e.matches(PyExc_FileNotFoundError)) {
+            py::print("missing.txt not found");
+        } else if (e.match(PyExc_PermissionError)) {
+            py::print("missing.txt found but not accessible");
+        } else {
+            throw;
+        }
+    }
+
+Note that C++ to Python exception translation does not apply here, since that is
+a method for translating C++ exceptions to Python, not vice versa. The error raised
+from Python is always ``error_already_set``.
+
+This example illustrates this behavior:
+
+.. code-block:: cpp
+
+    try {
+        py::eval("raise ValueError('The Ring')");
+    } catch (py::value_error &boromir) {
+        // Boromir never gets the ring
+        assert(false);
+    } catch (py::error_already_set &frodo) {
+        // Frodo gets the ring
+        py::print("I will take the ring");
+    }
+
+    try {
+        // py::value_error is a request for pybind11 to raise a Python exception
+        throw py::value_error("The ball");
+    } catch (py::error_already_set &cat) {
+        // cat won't catch the ball since
+        // py::value_error is not a Python exception
+        assert(false);
+    } catch (py::value_error &dog) {
+        // dog will catch the ball
+        py::print("Run Spot run");
+        throw;  // Throw it again (pybind11 will raise ValueError)
+    }
+
+Handling errors from the Python C API
+=====================================
+
+Where possible, use :ref:`pybind11 wrappers <wrappers>` instead of calling
+the Python C API directly. When calling the Python C API directly, in
+addition to manually managing reference counts, one must follow the pybind11
+error protocol, which is outlined here.
+
+After calling the Python C API, if Python returns an error,
+``throw py::error_already_set();``, which allows pybind11 to deal with the
+exception and pass it back to the Python interpreter. This includes calls to
+the error setting functions such as ``PyErr_SetString``.
+
+.. code-block:: cpp
+
+    PyErr_SetString(PyExc_TypeError, "C API type error demo");
+    throw py::error_already_set();
+
+    // But it would be easier to simply...
+    throw py::type_error("pybind11 wrapper type error");
+
+Alternately, to ignore the error, call `PyErr_Clear
+<https://docs.python.org/3/c-api/exceptions.html#c.PyErr_Clear>`_.
+
+Any Python error must be thrown or cleared, or Python/pybind11 will be left in
+an invalid state.
+
 .. _unraisable_exceptions:
 
 Handling unraisable exceptions
@@ -156,20 +247,24 @@ Handling unraisable exceptions
 
 If a Python function invoked from a C++ destructor or any function marked
 ``noexcept(true)`` (collectively, "noexcept functions") throws an exception, there
-is no way to propagate the exception, as such functions may not throw at
-run-time.
+is no way to propagate the exception, as such functions may not throw.
+Should they throw or fail to catch any exceptions in their call graph,
+the C++ runtime calls ``std::terminate()`` to abort immediately.
 
-Neither Python nor C++ allow exceptions raised in a noexcept function to propagate. In
-Python, an exception raised in a class's ``__del__`` method is logged as an
-unraisable error. In Python 3.8+, a system hook is triggered and an auditing
-event is logged. In C++, ``std::terminate()`` is called to abort immediately.
+Similarly, Python exceptions raised in a class's ``__del__`` method do not
+propagate, but are logged by Python as an unraisable error. In Python 3.8+, a
+`system hook is triggered
+<https://docs.python.org/3/library/sys.html#sys.unraisablehook>`_
+and an auditing event is logged.
 
 Any noexcept function should have a try-catch block that traps
-class:`error_already_set` (or any other exception that can occur). Note that pybind11
-wrappers around Python exceptions such as :class:`pybind11::value_error` are *not*
-Python exceptions; they are C++ exceptions that pybind11 catches and converts to
-Python exceptions. Noexcept functions cannot propagate these exceptions either.
-You can convert them to Python exceptions and then discard as unraisable.
+class:`error_already_set` (or any other exception that can occur). Note that
+pybind11 wrappers around Python exceptions such as
+:class:`pybind11::value_error` are *not* Python exceptions; they are C++
+exceptions that pybind11 catches and converts to Python exceptions. Noexcept
+functions cannot propagate these exceptions either. A useful approach is to
+convert them to Python exceptions and then ``discard_as_unraisable`` as shown
+below.
 
 .. code-block:: cpp
 
@@ -183,8 +278,6 @@ You can convert them to Python exceptions and then discard as unraisable.
             eas.discard_as_unraisable(__func__);
         } catch (const std::exception &e) {
             // Log and discard C++ exceptions.
-            // (We cannot use discard_as_unraisable, since we have a generic C++
-            // exception, not an exception that originated from Python.)
             third_party::log(e);
         }
     }

docs/advanced/pycpp/object.rst

 Python types
 ############
 
+.. _wrappers:
+
 Available wrappers
 ==================
 
@@ -168,3 +170,11 @@ Generalized unpacking according to PEP448_ is also supported:
     Python functions from C++, including keywords arguments and unpacking.
 
 .. _PEP448: https://www.python.org/dev/peps/pep-0448/
+
+Handling exceptions
+===================
+
+Python exceptions from wrapper classes will be thrown as a ``py::error_already_set``.
+See :ref:`Handling exceptions from Python in C++
+<handling_python_exceptions_cpp>` for more information on handling exceptions
+raised when calling C++ wrapper classes.