Merge branch 'PyO3:main' into main

README.md

@@ -210,6 +210,7 @@ about this topic.
 - [pyheck](https://github.com/kevinheavey/pyheck) _Fast case conversion library, built by wrapping [heck](https://github.com/withoutboats/heck)._
     - Quite easy to follow as there's not much code.
 - [pyre](https://github.com/Project-Dream-Weaver/pyre-http) _Fast Python HTTP server written in Rust._
+- [pyreqwest_impersonate](https://github.com/deedy5/pyreqwest_impersonate) _The fastest python HTTP client that can impersonate web browsers by mimicking their headers and TLS/JA3/JA4/HTTP2 fingerprints._
 - [ril-py](https://github.com/Cryptex-github/ril-py) _A performant and high-level image processing library for Python written in Rust._
 - [river](https://github.com/online-ml/river) _Online machine learning in python, the computationally heavy statistics algorithms are implemented in Rust._
 - [rust-python-coverage](https://github.com/cjermain/rust-python-coverage) _Example PyO3 project with automated test coverage for Rust and Python._

guide/src/building-and-distribution.md

@@ -232,7 +232,7 @@ not work when compiling for `abi3`. These are:
 
 If you want to embed the Python interpreter inside a Rust program, there are two modes in which this can be done: dynamically and statically. We'll cover each of these modes in the following sections. Each of them affect how you must distribute your program. Instead of learning how to do this yourself, you might want to consider using a project like [PyOxidizer] to ship your application and all of its dependencies in a single file.
 
-PyO3 automatically switches between the two linking modes depending on whether the Python distribution you have configured PyO3 to use ([see above](#configuring-the-python-version)) contains a shared library or a static library. The static library is most often seen in Python distributions compiled from source without the `--enable-shared` configuration option. For example, this is the default for `pyenv` on macOS.
+PyO3 automatically switches between the two linking modes depending on whether the Python distribution you have configured PyO3 to use ([see above](#configuring-the-python-version)) contains a shared library or a static library. The static library is most often seen in Python distributions compiled from source without the `--enable-shared` configuration option.
 
 ### Dynamically embedding the Python interpreter
 

guide/src/python-from-rust/calling-existing-code.md

@@ -2,9 +2,9 @@
 
 If you already have some existing Python code that you need to execute from Rust, the following FAQs can help you select the right PyO3 functionality for your situation:
 
-## Want to access Python APIs? Then use `PyModule::import`.
+## Want to access Python APIs? Then use `PyModule::import_bound`.
 
-[`Pymodule::import`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.import) can
+[`PyModule::import_bound`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.import_bound) can
 be used to get handle to a Python module from Rust. You can use this to import and use any Python
 module available in your environment.
 
@@ -95,9 +95,9 @@ assert userdata.as_tuple() == userdata_as_tuple
 # }
 ```
 
-## You have a Python file or code snippet? Then use `PyModule::from_code`.
+## You have a Python file or code snippet? Then use `PyModule::from_code_bound`.
 
-[`PyModule::from_code`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.from_code)
+[`PyModule::from_code_bound`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.from_code_bound)
 can be used to generate a Python module which can then be used just as if it was imported with
 `PyModule::import`.
 
@@ -171,7 +171,7 @@ fn main() -> PyResult<()> {
 ```
 
 If `append_to_inittab` cannot be used due to constraints in the program,
-an alternative is to create a module using [`PyModule::new`]
+an alternative is to create a module using [`PyModule::new_bound`]
 and insert it manually into `sys.modules`:
 
 ```rust
@@ -394,4 +394,4 @@ Python::with_gil(|py| -> PyResult<()> {
 ```
 
 
-[`PyModule::new`]: {{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.new
+[`PyModule::new_bound`]: {{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.new_bound

newsfragments/4117.fixed.md

@@ -0,0 +1 @@
+Correctly handle `#[pyo3(from_py_with = ...)]` attribute on dunder (`__magic__`) method arguments instead of silently ignoring it.

noxfile.py

@@ -415,10 +415,11 @@ def check_guide(session: nox.Session):
     _run(
         session,
         "lychee",
-        PYO3_DOCS_TARGET,
+        str(PYO3_DOCS_TARGET),
         f"--remap=https://pyo3.rs/main/ file://{PYO3_GUIDE_TARGET}/",
         f"--remap=https://pyo3.rs/latest/ file://{PYO3_GUIDE_TARGET}/",
         f"--exclude=file://{PYO3_DOCS_TARGET}",
+        "--exclude=http://www.adobe.com/",
         *session.posargs,
     )
 

pyo3-build-config/src/lib.rs

@@ -86,6 +86,8 @@ pub fn get() -> &'static InterpreterConfig {
             .map(|path| path.exists())
             .unwrap_or(false);
 
+        // CONFIG_FILE is generated in build.rs, so it's content can vary
+        #[allow(unknown_lints, clippy::const_is_empty)]
         if let Some(interpreter_config) = InterpreterConfig::from_cargo_dep_env() {
             interpreter_config
         } else if !CONFIG_FILE.is_empty() {
@@ -177,6 +179,8 @@ pub mod pyo3_build_script_impl {
     /// correct value for CARGO_CFG_TARGET_OS).
     #[cfg(feature = "resolve-config")]
     pub fn resolve_interpreter_config() -> Result<InterpreterConfig> {
+        // CONFIG_FILE is generated in build.rs, so it's content can vary
+        #[allow(unknown_lints, clippy::const_is_empty)]
         if !CONFIG_FILE.is_empty() {
             let mut interperter_config = InterpreterConfig::from_reader(Cursor::new(CONFIG_FILE))?;
             interperter_config.generate_import_libs()?;

pyo3-macros-backend/src/params.rs

@@ -10,7 +10,7 @@ use syn::spanned::Spanned;
 
 pub struct Holders {
     holders: Vec<syn::Ident>,
-    gil_refs_checkers: Vec<syn::Ident>,
+    gil_refs_checkers: Vec<GilRefChecker>,
 }
 
 impl Holders {
@@ -32,14 +32,28 @@ impl Holders {
             &format!("gil_refs_checker_{}", self.gil_refs_checkers.len()),
             span,
         );
-        self.gil_refs_checkers.push(gil_refs_checker.clone());
+        self.gil_refs_checkers
+            .push(GilRefChecker::FunctionArg(gil_refs_checker.clone()));
+        gil_refs_checker
+    }
+
+    pub fn push_from_py_with_checker(&mut self, span: Span) -> syn::Ident {
+        let gil_refs_checker = syn::Ident::new(
+            &format!("gil_refs_checker_{}", self.gil_refs_checkers.len()),
+            span,
+        );
+        self.gil_refs_checkers
+            .push(GilRefChecker::FromPyWith(gil_refs_checker.clone()));
         gil_refs_checker
     }
 
     pub fn init_holders(&self, ctx: &Ctx) -> TokenStream {
         let Ctx { pyo3_path } = ctx;
         let holders = &self.holders;
-        let gil_refs_checkers = &self.gil_refs_checkers;
+        let gil_refs_checkers = self.gil_refs_checkers.iter().map(|checker| match checker {
+            GilRefChecker::FunctionArg(ident) => ident,
+            GilRefChecker::FromPyWith(ident) => ident,
+        });
         quote! {
             #[allow(clippy::let_unit_value)]
             #(let mut #holders = #pyo3_path::impl_::extract_argument::FunctionArgumentHolder::INIT;)*
@@ -50,11 +64,23 @@ impl Holders {
     pub fn check_gil_refs(&self) -> TokenStream {
         self.gil_refs_checkers
             .iter()
-            .map(|e| quote_spanned! { e.span() => #e.function_arg(); })
+            .map(|checker| match checker {
+                GilRefChecker::FunctionArg(ident) => {
+                    quote_spanned! { ident.span() => #ident.function_arg(); }
+                }
+                GilRefChecker::FromPyWith(ident) => {
+                    quote_spanned! { ident.span() => #ident.from_py_with_arg(); }
+                }
+            })
             .collect()
     }
 }
 
+enum GilRefChecker {
+    FunctionArg(syn::Ident),
+    FromPyWith(syn::Ident),
+}
+
 /// Return true if the argument list is simply (*args, **kwds).
 pub fn is_forwarded_args(signature: &FunctionSignature<'_>) -> bool {
     matches!(

pyo3-macros-backend/src/pymethod.rs

@@ -1083,44 +1083,39 @@ impl Ty {
         ctx: &Ctx,
     ) -> TokenStream {
         let Ctx { pyo3_path } = ctx;
-        let name_str = arg.name().unraw().to_string();
         match self {
             Ty::Object => extract_object(
                 extract_error_mode,
                 holders,
-                &name_str,
+                arg,
                 quote! { #ident },
-                arg.ty().span(),
                 ctx
             ),
             Ty::MaybeNullObject => extract_object(
                 extract_error_mode,
                 holders,
-                &name_str,
+                arg,
                 quote! {
                     if #ident.is_null() {
                         #pyo3_path::ffi::Py_None()
                     } else {
                         #ident
                     }
                 },
-                arg.ty().span(),
                 ctx
             ),
             Ty::NonNullObject => extract_object(
                 extract_error_mode,
                 holders,
-                &name_str,
+                arg,
                 quote! { #ident.as_ptr() },
-                arg.ty().span(),
                 ctx
             ),
             Ty::IPowModulo => extract_object(
                 extract_error_mode,
                 holders,
-                &name_str,
+                arg,
                 quote! { #ident.as_ptr() },
-                arg.ty().span(),
                 ctx
             ),
             Ty::CompareOp => extract_error_mode.handle_error(
@@ -1148,24 +1143,37 @@ impl Ty {
 fn extract_object(
     extract_error_mode: ExtractErrorMode,
     holders: &mut Holders,
-    name: &str,
+    arg: &FnArg<'_>,
     source_ptr: TokenStream,
-    span: Span,
     ctx: &Ctx,
 ) -> TokenStream {
     let Ctx { pyo3_path } = ctx;
-    let holder = holders.push_holder(Span::call_site());
-    let gil_refs_checker = holders.push_gil_refs_checker(span);
-    let extracted = extract_error_mode.handle_error(
+    let gil_refs_checker = holders.push_gil_refs_checker(arg.ty().span());
+    let name = arg.name().unraw().to_string();
+
+    let extract = if let Some(from_py_with) =
+        arg.from_py_with().map(|from_py_with| &from_py_with.value)
+    {
+        let from_py_with_checker = holders.push_from_py_with_checker(from_py_with.span());
+        quote! {
+            #pyo3_path::impl_::extract_argument::from_py_with(
+                #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &#source_ptr).0,
+                #name,
+                #pyo3_path::impl_::deprecations::inspect_fn(#from_py_with, &#from_py_with_checker) as fn(_) -> _,
+            )
+        }
+    } else {
+        let holder = holders.push_holder(Span::call_site());
         quote! {
             #pyo3_path::impl_::extract_argument::extract_argument(
                 #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &#source_ptr).0,
                 &mut #holder,
                 #name
             )
-        },
-        ctx,
-    );
+        }
+    };
+
+    let extracted = extract_error_mode.handle_error(extract, ctx);
     quote! {
         #pyo3_path::impl_::deprecations::inspect_type(#extracted, &#gil_refs_checker)
     }

pytests/src/othermod.rs

@@ -34,8 +34,8 @@ pub fn othermod(m: &Bound<'_, PyModule>) -> PyResult<()> {
 
     m.add_class::<ModClass>()?;
 
-    m.add("USIZE_MIN", usize::min_value())?;
-    m.add("USIZE_MAX", usize::max_value())?;
+    m.add("USIZE_MIN", usize::MIN)?;
+    m.add("USIZE_MAX", usize::MAX)?;
 
     Ok(())
 }

src/marker.rs

@@ -352,7 +352,7 @@ pub use nightly::Ungil;
 /// # Releasing and freeing memory
 ///
 /// The [`Python`] type can be used to create references to variables owned by the Python
-/// interpreter, using functions such as [`Python::eval`] and [`PyModule::import`]. These
+/// interpreter, using functions such as [`Python::eval`] and `PyModule::import`. These
 /// references are tied to a [`GILPool`] whose references are not cleared until it is dropped.
 /// This can cause apparent "memory leaks" if it is kept around for a long time.
 ///

src/marshal.rs

@@ -13,12 +13,10 @@ use std::os::raw::c_int;
 pub const VERSION: i32 = 4;
 
 /// Deprecated form of [`dumps_bound`]
-#[cfg_attr(
-    not(feature = "gil-refs"),
-    deprecated(
-        since = "0.21.0",
-        note = "`dumps` will be replaced by `dumps_bound` in a future PyO3 version"
-    )
+#[cfg(feature = "gil-refs")]
+#[deprecated(
+    since = "0.21.0",
+    note = "`dumps` will be replaced by `dumps_bound` in a future PyO3 version"
 )]
 pub fn dumps<'py>(
     py: Python<'py>,
@@ -61,12 +59,10 @@ pub fn dumps_bound<'py>(
 }
 
 /// Deprecated form of [`loads_bound`]
-#[cfg_attr(
-    not(feature = "gil-refs"),
-    deprecated(
-        since = "0.21.0",
-        note = "`loads` will be replaced by `loads_bound` in a future PyO3 version"
-    )
+#[cfg(feature = "gil-refs")]
+#[deprecated(
+    since = "0.21.0",
+    note = "`loads` will be replaced by `loads_bound` in a future PyO3 version"
 )]
 pub fn loads<'py, B>(py: Python<'py>, data: &B) -> PyResult<&'py PyAny>
 where

src/pyclass/create_type_object.rs

@@ -145,12 +145,14 @@ impl PyTypeBuilder {
             #[cfg(all(not(Py_3_9), not(Py_LIMITED_API)))]
             ffi::Py_bf_getbuffer => {
                 // Safety: slot.pfunc is a valid function pointer
-                self.buffer_procs.bf_getbuffer = Some(std::mem::transmute(pfunc));
+                self.buffer_procs.bf_getbuffer =
+                    Some(std::mem::transmute::<*mut T, ffi::getbufferproc>(pfunc));
             }
             #[cfg(all(not(Py_3_9), not(Py_LIMITED_API)))]
             ffi::Py_bf_releasebuffer => {
                 // Safety: slot.pfunc is a valid function pointer
-                self.buffer_procs.bf_releasebuffer = Some(std::mem::transmute(pfunc));
+                self.buffer_procs.bf_releasebuffer =
+                    Some(std::mem::transmute::<*mut T, ffi::releasebufferproc>(pfunc));
             }
             _ => {}
         }

src/types/function.rs

@@ -1,14 +1,17 @@
+#[cfg(feature = "gil-refs")]
 use crate::derive_utils::PyFunctionArguments;
 use crate::ffi_ptr_ext::FfiPtrExt;
 use crate::py_result_ext::PyResultExt;
 use crate::types::capsule::PyCapsuleMethods;
 use crate::types::module::PyModuleMethods;
+#[cfg(feature = "gil-refs")]
+use crate::PyNativeType;
 use crate::{
     ffi,
     impl_::pymethods::{self, PyMethodDef, PyMethodDefDestructor},
     types::{PyCapsule, PyDict, PyModule, PyString, PyTuple},
 };
-use crate::{Bound, IntoPy, Py, PyAny, PyNativeType, PyResult, Python};
+use crate::{Bound, IntoPy, Py, PyAny, PyResult, Python};
 use std::cell::UnsafeCell;
 use std::ffi::CStr;
 
@@ -20,12 +23,10 @@ pyobject_native_type_core!(PyCFunction, pyobject_native_static_type_object!(ffi:
 
 impl PyCFunction {
     /// Deprecated form of [`PyCFunction::new_with_keywords_bound`]
-    #[cfg_attr(
-        not(feature = "gil-refs"),
-        deprecated(
-            since = "0.21.0",
-            note = "`PyCFunction::new_with_keywords` will be replaced by `PyCFunction::new_with_keywords_bound` in a future PyO3 version"
-        )
+    #[cfg(feature = "gil-refs")]
+    #[deprecated(
+        since = "0.21.0",
+        note = "`PyCFunction::new_with_keywords` will be replaced by `PyCFunction::new_with_keywords_bound` in a future PyO3 version"
     )]
     pub fn new_with_keywords<'a>(
         fun: ffi::PyCFunctionWithKeywords,
@@ -66,12 +67,10 @@ impl PyCFunction {
     }
 
     /// Deprecated form of [`PyCFunction::new`]
-    #[cfg_attr(
-        not(feature = "gil-refs"),
-        deprecated(
-            since = "0.21.0",
-            note = "`PyCFunction::new` will be replaced by `PyCFunction::new_bound` in a future PyO3 version"
-        )
+    #[cfg(feature = "gil-refs")]
+    #[deprecated(
+        since = "0.21.0",
+        note = "`PyCFunction::new` will be replaced by `PyCFunction::new_bound` in a future PyO3 version"
     )]
     pub fn new<'a>(
         fun: ffi::PyCFunction,
@@ -104,12 +103,10 @@ impl PyCFunction {
     }
 
     /// Deprecated form of [`PyCFunction::new_closure`]
-    #[cfg_attr(
-        not(feature = "gil-refs"),
-        deprecated(
-            since = "0.21.0",
-            note = "`PyCFunction::new_closure` will be replaced by `PyCFunction::new_closure_bound` in a future PyO3 version"
-        )
+    #[cfg(feature = "gil-refs")]
+    #[deprecated(
+        since = "0.21.0",
+        note = "`PyCFunction::new_closure` will be replaced by `PyCFunction::new_closure_bound` in a future PyO3 version"
     )]
     pub fn new_closure<'a, F, R>(
         py: Python<'a>,