Merge branch 'main' into drop-parser-sep

Doc/c-api/init_config.rst

@@ -1281,6 +1281,15 @@ PyConfig
 
       Default: ``1`` in Python config and ``0`` in isolated config.
 
+   .. c:member:: int use_system_logger
+
+      If non-zero, ``stdout`` and ``stderr`` will be redirected to the system
+      log.
+
+      Only available on macOS 10.12 and later, and on iOS.
+
+      Default: ``0`` (don't use system log).
+
    .. c:member:: int user_site_directory
 
       If non-zero, add the user site directory to :data:`sys.path`.

Doc/using/ios.rst

@@ -292,10 +292,12 @@ To add Python to an iOS Xcode project:
 10. Add Objective C code to initialize and use a Python interpreter in embedded
     mode. You should ensure that:
 
-   * :c:member:`UTF-8 mode <PyPreConfig.utf8_mode>` is *enabled*;
-   * :c:member:`Buffered stdio <PyConfig.buffered_stdio>` is *disabled*;
-   * :c:member:`Writing bytecode <PyConfig.write_bytecode>` is *disabled*;
-   * :c:member:`Signal handlers <PyConfig.install_signal_handlers>` are *enabled*;
+   * UTF-8 mode (:c:member:`PyPreConfig.utf8_mode`) is *enabled*;
+   * Buffered stdio (:c:member:`PyConfig.buffered_stdio`) is *disabled*;
+   * Writing bytecode (:c:member:`PyConfig.write_bytecode`) is *disabled*;
+   * Signal handlers (:c:member:`PyConfig.install_signal_handlers`) are *enabled*;
+   * System logging (:c:member:`PyConfig.use_system_logger`) is *enabled*
+     (optional, but strongly recommended);
    * ``PYTHONHOME`` for the interpreter is configured to point at the
      ``python`` subfolder of your app's bundle; and
    * The ``PYTHONPATH`` for the interpreter includes:
@@ -324,6 +326,49 @@ modules in your app, some additional steps will be required:
 * If you're using a separate folder for third-party packages, ensure that folder
   is included as part of the ``PYTHONPATH`` configuration in step 10.
 
+Testing a Python package
+------------------------
+
+The CPython source tree contains :source:`a testbed project <iOS/testbed>` that
+is used to run the CPython test suite on the iOS simulator. This testbed can also
+be used as a testbed project for running your Python library's test suite on iOS.
+
+After building or obtaining an iOS XCFramework (See :source:`iOS/README.rst`
+for details), create a clone of the Python iOS testbed project by running:
+
+.. code-block:: bash
+
+    $ python iOS/testbed clone --framework <path/to/Python.xcframework> --app <path/to/module1> --app <path/to/module2> app-testbed
+
+You will need to modify the ``iOS/testbed`` reference to point to that
+directory in the CPython source tree; any folders specified with the ``--app``
+flag will be copied into the cloned testbed project. The resulting testbed will
+be created in the ``app-testbed`` folder. In this example, the ``module1`` and
+``module2`` would be importable modules at runtime. If your project has
+additional dependencies, they can be installed into the
+``app-testbed/iOSTestbed/app_packages`` folder (using ``pip install --target
+app-testbed/iOSTestbed/app_packages`` or similar).
+
+You can then use the ``app-testbed`` folder to run the test suite for your app,
+For example, if ``module1.tests`` was the entry point to your test suite, you
+could run:
+
+.. code-block:: bash
+
+    $ python app-testbed run -- module1.tests
+
+This is the equivalent of running ``python -m module1.tests`` on a desktop
+Python build. Any arguments after the ``--`` will be passed to the testbed as
+if they were arguments to ``python -m`` on a desktop machine.
+
+You can also open the testbed project in Xcode by running:
+
+.. code-block:: bash
+
+    $ open app-testbed/iOSTestbed.xcodeproj
+
+This will allow you to use the full Xcode suite of tools for debugging.
+
 App Store Compliance
 ====================
 

Doc/whatsnew/3.14.rst

@@ -245,6 +245,13 @@ Other language changes
   making it a :term:`generic type`.
   (Contributed by Brian Schubert in :gh:`126012`.)
 
+* iOS and macOS apps can now be configured to redirect ``stdout`` and
+  ``stderr`` content to the system log. (Contributed by Russell Keith-Magee in
+  :gh:`127592`.)
+
+* The iOS testbed is now able to stream test output while the test is running.
+  The testbed can also be used to run the test suite of projects other than
+  CPython itself. (Contributed by Russell Keith-Magee in :gh:`127592`.)
 
 New modules
 ===========

Include/cpython/initconfig.h

@@ -179,6 +179,9 @@ typedef struct PyConfig {
     int use_frozen_modules;
     int safe_path;
     int int_max_str_digits;
+#ifdef __APPLE__
+    int use_system_logger;
+#endif
 
     int cpu_count;
 #ifdef Py_GIL_DISABLED

InternalDocs/garbage_collector.md

@@ -518,7 +518,7 @@ Then the above algorithm is repeated, starting from step 2.
 Determining how much work to do
 -------------------------------
 
-We need to do a certain amount of work to enusre that garbage is collected,
+We need to do a certain amount of work to ensure that garbage is collected,
 but doing too much work slows down execution.
 
 To work out how much work we need to do, consider a heap with `L` live objects

Lib/_apple_support.py

@@ -0,0 +1,66 @@
+import io
+import sys
+
+
+def init_streams(log_write, stdout_level, stderr_level):
+    # Redirect stdout and stderr to the Apple system log. This method is
+    # invoked by init_apple_streams() (initconfig.c) if config->use_system_logger
+    # is enabled.
+    sys.stdout = SystemLog(log_write, stdout_level, errors=sys.stderr.errors)
+    sys.stderr = SystemLog(log_write, stderr_level, errors=sys.stderr.errors)
+
+
+class SystemLog(io.TextIOWrapper):
+    def __init__(self, log_write, level, **kwargs):
+        kwargs.setdefault("encoding", "UTF-8")
+        kwargs.setdefault("line_buffering", True)
+        super().__init__(LogStream(log_write, level), **kwargs)
+
+    def __repr__(self):
+        return f"<SystemLog (level {self.buffer.level})>"
+
+    def write(self, s):
+        if not isinstance(s, str):
+            raise TypeError(
+                f"write() argument must be str, not {type(s).__name__}")
+
+        # In case `s` is a str subclass that writes itself to stdout or stderr
+        # when we call its methods, convert it to an actual str.
+        s = str.__str__(s)
+
+        # We want to emit one log message per line, so split
+        # the string before sending it to the superclass.
+        for line in s.splitlines(keepends=True):
+            super().write(line)
+
+        return len(s)
+
+
+class LogStream(io.RawIOBase):
+    def __init__(self, log_write, level):
+        self.log_write = log_write
+        self.level = level
+
+    def __repr__(self):
+        return f"<LogStream (level {self.level!r})>"
+
+    def writable(self):
+        return True
+
+    def write(self, b):
+        if type(b) is not bytes:
+            try:
+                b = bytes(memoryview(b))
+            except TypeError:
+                raise TypeError(
+                    f"write() argument must be bytes-like, not {type(b).__name__}"
+                ) from None
+
+        # Writing an empty string to the stream should have no effect.
+        if b:
+            # Encode null bytes using "modified UTF-8" to avoid truncating the
+            # message. This should not affect the return value, as the caller
+            # may be expecting it to match the length of the input.
+            self.log_write(self.level, b.replace(b"\x00", b"\xc0\x80"))
+
+        return len(b)

Lib/dis.py

@@ -1115,7 +1115,7 @@ def dis(self):
             return output.getvalue()
 
 
-def main():
+def main(args=None):
     import argparse
 
     parser = argparse.ArgumentParser()
@@ -1128,7 +1128,7 @@ def main():
     parser.add_argument('-S', '--specialized', action='store_true',
                         help='show specialized bytecode')
     parser.add_argument('infile', nargs='?', default='-')
-    args = parser.parse_args()
+    args = parser.parse_args(args=args)
     if args.infile == '-':
         name = '<stdin>'
         source = sys.stdin.buffer.read()

Lib/inspect.py

@@ -2943,11 +2943,19 @@ def __init__(self, parameters=None, *, return_annotation=_empty,
                 params = OrderedDict()
                 top_kind = _POSITIONAL_ONLY
                 seen_default = False
+                seen_var_parameters = set()
 
                 for param in parameters:
                     kind = param.kind
                     name = param.name
 
+                    if kind in (_VAR_POSITIONAL, _VAR_KEYWORD):
+                        if kind in seen_var_parameters:
+                            msg = f'more than one {kind.description} parameter'
+                            raise ValueError(msg)
+
+                        seen_var_parameters.add(kind)
+
                     if kind < top_kind:
                         msg = (
                             'wrong parameter order: {} parameter before {} '

Lib/pathlib/_abc.py

@@ -12,6 +12,7 @@
 """
 
 import operator
+import posixpath
 from errno import EINVAL
 from glob import _GlobberBase, _no_recurse_symlinks
 from stat import S_ISDIR, S_ISLNK, S_ISREG, S_ISSOCK, S_ISBLK, S_ISCHR, S_ISFIFO
@@ -27,53 +28,6 @@ class UnsupportedOperation(NotImplementedError):
     pass
 
 
-class ParserBase:
-    """Base class for path parsers, which do low-level path manipulation.
-
-    Path parsers provide a subset of the os.path API, specifically those
-    functions needed to provide PurePathBase functionality. Each PurePathBase
-    subclass references its path parser via a 'parser' class attribute.
-
-    Every method in this base class raises an UnsupportedOperation exception.
-    """
-
-    @classmethod
-    def _unsupported_msg(cls, attribute):
-        return f"{cls.__name__}.{attribute} is unsupported"
-
-    def join(self, path, *paths):
-        """Join path segments."""
-        raise UnsupportedOperation(self._unsupported_msg('join()'))
-
-    def split(self, path):
-        """Split the path into a pair (head, tail), where *head* is everything
-        before the final path separator, and *tail* is everything after.
-        Either part may be empty.
-        """
-        raise UnsupportedOperation(self._unsupported_msg('split()'))
-
-    def splitdrive(self, path):
-        """Split the path into a 2-item tuple (drive, tail), where *drive* is
-        a device name or mount point, and *tail* is everything after the
-        drive. Either part may be empty."""
-        raise UnsupportedOperation(self._unsupported_msg('splitdrive()'))
-
-    def splitext(self, path):
-        """Split the path into a pair (root, ext), where *ext* is empty or
-        begins with a period and contains at most one period,
-        and *root* is everything before the extension."""
-        raise UnsupportedOperation(self._unsupported_msg('splitext()'))
-
-    def normcase(self, path):
-        """Normalize the case of the path."""
-        raise UnsupportedOperation(self._unsupported_msg('normcase()'))
-
-    def isabs(self, path):
-        """Returns whether the path is absolute, i.e. unaffected by the
-        current directory or drive."""
-        raise UnsupportedOperation(self._unsupported_msg('isabs()'))
-
-
 class PathGlobber(_GlobberBase):
     """
     Class providing shell-style globbing for path objects.
@@ -103,7 +57,7 @@ class PurePathBase:
         # the `__init__()` method.
         '_raw_paths',
     )
-    parser = ParserBase()
+    parser = posixpath
     _sep = '/'
     _case_sensitive = True
     _globber = PathGlobber
@@ -836,6 +790,12 @@ def copy_into(self, target_dir, *, follow_symlinks=True,
                          dirs_exist_ok=dirs_exist_ok,
                          preserve_metadata=preserve_metadata)
 
+    def _delete(self):
+        """
+        Delete this file or directory (including all sub-directories).
+        """
+        raise UnsupportedOperation(self._unsupported_msg('_delete()'))
+
     def move(self, target):
         """
         Recursively move this file or directory tree to the given destination.
@@ -870,43 +830,6 @@ def lchmod(self, mode):
         """
         self.chmod(mode, follow_symlinks=False)
 
-    def unlink(self, missing_ok=False):
-        """
-        Remove this file or link.
-        If the path is a directory, use rmdir() instead.
-        """
-        raise UnsupportedOperation(self._unsupported_msg('unlink()'))
-
-    def rmdir(self):
-        """
-        Remove this directory.  The directory must be empty.
-        """
-        raise UnsupportedOperation(self._unsupported_msg('rmdir()'))
-
-    def _delete(self):
-        """
-        Delete this file or directory (including all sub-directories).
-        """
-        if self.is_symlink() or self.is_junction():
-            self.unlink()
-        elif self.is_dir():
-            self._rmtree()
-        else:
-            self.unlink()
-
-    def _rmtree(self):
-        def on_error(err):
-            raise err
-        results = self.walk(
-            on_error=on_error,
-            top_down=False,  # So we rmdir() empty directories.
-            follow_symlinks=False)
-        for dirpath, _, filenames in results:
-            for filename in filenames:
-                filepath = dirpath / filename
-                filepath.unlink()
-            dirpath.rmdir()
-
     def owner(self, *, follow_symlinks=True):
         """
         Return the login name of the file owner.

Lib/pathlib/_local.py

@@ -846,10 +846,18 @@ def rmdir(self):
         """
         os.rmdir(self)
 
-    def _rmtree(self):
-        # Lazy import to improve module import time
-        import shutil
-        shutil.rmtree(self)
+    def _delete(self):
+        """
+        Delete this file or directory (including all sub-directories).
+        """
+        if self.is_symlink() or self.is_junction():
+            self.unlink()
+        elif self.is_dir():
+            # Lazy import to improve module import time
+            import shutil
+            shutil.rmtree(self)
+        else:
+            self.unlink()
 
     def rename(self, target):
         """

Lib/pathlib/_types.py

@@ -0,0 +1,22 @@
+"""
+Protocols for supporting classes in pathlib.
+"""
+from typing import Protocol, runtime_checkable
+
+
+@runtime_checkable
+class Parser(Protocol):
+    """Protocol for path parsers, which do low-level path manipulation.
+
+    Path parsers provide a subset of the os.path API, specifically those
+    functions needed to provide PurePathBase functionality. Each PurePathBase
+    subclass references its path parser via a 'parser' class attribute.
+    """
+
+    sep: str
+    def join(self, path: str, *paths: str) -> str: ...
+    def split(self, path: str) -> tuple[str, str]: ...
+    def splitdrive(self, path: str) -> tuple[str, str]: ...
+    def splitext(self, path: str) -> tuple[str, str]: ...
+    def normcase(self, path: str) -> str: ...
+    def isabs(self, path: str) -> bool: ...