User-defined virtual path objects #110109

barneygale · 2023-09-29T15:16:35Z

Proposal

Add a PathBase class that can be subclassed by users to implement "virtual paths", e.g. paths within .zip files or on FTP servers, and publish it as an external PyPI package.

Phase 1: Private support in pathlib

Add private _PathBase class
- GH-89812: Add pathlib._PathBase #106337
Merge WalkTests into DummyPathTest
- GH-110109: Simplify pathlib test setup #110308
- GH-110109: Move tests for pathlib.Path.walk() into main test classes. #110655
Sort out __hash__(), __eq__, etc - we shouldn't inherit the PurePath implementation
- GH-110109: Churn pathlib.PurePath methods #112012
- GH-110109: Add pathlib._PurePathBase #110670
Simplify/speed up path walking and globbing
- GH-72904: Add glob.translate() function #106703
Add from_uri() classmethod
- GH-107465: Add pathlib.Path.from_uri() classmethod. #107640

Phase 2: Experimental support in PyPI package

Move ABCs into pathlib._abc
- GH-110109: Move pathlib ABCs to new pathlib._abc module. #112881
- GH-110109: Move tests for pathlib ABCs to new module. #112904
Figure out the default value for pathmod, if any
- GH-110109: pathlib tests: store base directory as test class attribute #113221
- GH-110109: pathlib ABCs: do not vary path syntax by host OS. #113219
Publish PyPI package

Previous Discussion

https://discuss.python.org/t/make-pathlib-extensible/3428

Linked PRs

The text was updated successfully, but these errors were encountered:

Use the class under test to create files, directories and symlinks.

…ther. This is a very soft deprecation of `PurePath.as_uri()`. We instead document it as a `Path` method, and add a couple of sentences mentioning that it's also available in `PurePath`.

Add `PurePathBase` and `PathBase` abstract base classes. `PathBase` can be subclassed to implement *virtual paths*: objects representing paths on virtual filesystems, like archive files or remote storage systems.

- Add fast path to `_split_stack()` - Skip unnecessarily resolution of the current directory when a relative path is given to `resolve()` - Remove stat and target caches, which slow down most `resolve()` calls in practice. - Slightly refactor code for clarity.

Use the class under test to create files, directories and symlinks.

…classes. This adds some test coverage to `pathlib._PathBase.walk()`

Add private `pathlib._PurePathBase` class: a private superclass of both `PurePath` and `_PathBase`. Unlike `PurePath`, it does not define any of these special methods: `__fspath__`, `__bytes__`, `__reduce__`, `__hash__`, `__eq__`, `__lt__`, `__le__`, `__gt__`, `__ge__`. This is important for supporting *virtual paths*: user subclasses of `_PathBase` that provide access to archive files, FTP servers, etc. In these classes, the above methods should be implemented by users only as appropriate, with due consideration for the hash/equality of any backing objects, such as file objects or sockets.

…s. (#110655)

…classes. (python#110655)

Re-arrange `pathlib.PurePath` methods in source code. No other changes. The `PurePath` implementations of certain special methods, such as `__eq__()` and `__hash__()`, are not usually applicable to user subclasses of `_PathBase`. To facilitate their removal, another patch will split the `PurePath` class into `_PurePathBase` and `PurePath`, with the latter providing these special methods. This patch prepares the ground for splitting `PurePath`. It's similar to e8d77b0, which preceded splitting `Path`. By churning the methods here, subsequent patches will be easier to review and less likely to break things.

…` and `group()` (python#112239) Move test methods from `WindowsPathAsPureTest` to `WindowsPathTest` unit. The former test unit is intended to exercise only pure path functionality.

…asses (python#112242) Add `PurePathTest` as a superclass of `PathTest`, and therefore also `PathSubclassTest`. This adds coverage of pure functionality in user subclasses of `pathlib.Path`. Remove `PosixPathAsPureTest` and `WindowsPathAsPureTest`, as they now duplicate `PosixPathTest` and `WindowsPathTest`. This makes the MROs of test unit classes match the MROs of pathlib classes.

Add private `pathlib._PurePathBase` class: a private superclass of both `PurePath` and `_PathBase`. Unlike `PurePath`, it does not define any of these special methods: `__fspath__`, `__bytes__`, `__reduce__`, `__hash__`, `__eq__`, `__lt__`, `__le__`, `__gt__`, `__ge__`. Its initializer and path joining methods accept only strings, not os.PathLike objects more broadly. This is important for supporting *virtual paths*: user subclasses of `_PathBase` that provide access to archive files, FTP servers, etc. In these classes, the above methods should be implemented by users only as appropriate, with due consideration for the hash/equality of any backing objects, such as file objects or sockets.

…hon#112881) Move `_PurePathBase` and `_PathBase` to a new `pathlib._abc` module, and drop the underscores from the class names. Tests are mostly left alone in this commit, but they'll be similarly split in a subsequent commit. The `pathlib._abc` module will be published as an independent PyPI package (similar to how `zipfile._path` is published as `zipp`), to be refined and stabilised prior to its possible addition to the standard library.

…ython#112901)

…12904)

…tribute (python#113221) Store the test base directory as a class attribute named `base` rather than module constants named `BASE`. The base directory is a local file path, and therefore not ideally suited to the pathlib ABC tests. In a future commit we'll change its value in `test_pathlib_abc.py` such that it points to a totally fictitious path, which will help to ensure we're not touching the local filesystem.

…hon#113376) `PurePathBase.__repr__()` produces a string like `MyPath('/foo')`. This repr is incorrect/misleading when a subclass's `__init__()` method is customized, which I expect to be the very common. This commit moves the `__repr__()` method to `PurePath`, leaving `PurePathBase` with the default `object` repr. No user-facing changes because the `pathlib._abc` module remains private.

…ython#113219) Change the value of `pathlib._abc.PurePathBase.pathmod` from `os.path` to `posixpath`. User subclasses of `PurePathBase` and `PathBase` previously used the host OS's path syntax, e.g. backslashes as separators on Windows. This is wrong in most use cases, and likely to catch developers out unless they test on both Windows and non-Windows machines. In this patch we change the default to POSIX syntax, regardless of OS. This is somewhat arguable (why not make all aspects of syntax abstract and individually configurable?) but an improvement all the same. This change has no effect on `PurePath`, `Path`, nor their subclasses. Only private APIs are affected.

python#113411) This very boring patch reduces the number of changes needed in `test_pathlib_abc.py` when backporting to the external `pathlib_abc` package.

…on#113417) Do not use the locale-specific default encoding in `PathBase.read_text()` and `write_text()`. Locale settings shouldn't influence the operation of these base classes, which are intended mostly for implementing rich paths on *nonlocal* filesystems.

…ython#113419) The `pathlib._abc` module will be made available as a PyPI backport supporting Python 3.8+. The `warnings._deprecated()` function was only added last year, and it's private from an external package perspective, so here we switch to `warnings.warn()` instead.

…ther. (python#110312) This is a very soft deprecation of `PurePath.as_uri()`. We instead document it as a `Path` method, and add a couple of sentences mentioning that it's also available in `PurePath`. Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

Use the class under test to create files, directories and symlinks.

…classes. (python#110655)

Re-arrange `pathlib.PurePath` methods in source code. No other changes. The `PurePath` implementations of certain special methods, such as `__eq__()` and `__hash__()`, are not usually applicable to user subclasses of `_PathBase`. To facilitate their removal, another patch will split the `PurePath` class into `_PurePathBase` and `PurePath`, with the latter providing these special methods. This patch prepares the ground for splitting `PurePath`. It's similar to e8d77b0, which preceded splitting `Path`. By churning the methods here, subsequent patches will be easier to review and less likely to break things.

- Add fast path to `_split_stack()` - Skip unnecessarily resolution of the current directory when a relative path is given to `resolve()` - Remove stat and target caches, which slow down most `resolve()` calls in practice. - Slightly refactor code for clarity.

…` and `group()` (python#112239) Move test methods from `WindowsPathAsPureTest` to `WindowsPathTest` unit. The former test unit is intended to exercise only pure path functionality.

…asses (python#112242) Add `PurePathTest` as a superclass of `PathTest`, and therefore also `PathSubclassTest`. This adds coverage of pure functionality in user subclasses of `pathlib.Path`. Remove `PosixPathAsPureTest` and `WindowsPathAsPureTest`, as they now duplicate `PosixPathTest` and `WindowsPathTest`. This makes the MROs of test unit classes match the MROs of pathlib classes.

Add private `pathlib._PurePathBase` class: a private superclass of both `PurePath` and `_PathBase`. Unlike `PurePath`, it does not define any of these special methods: `__fspath__`, `__bytes__`, `__reduce__`, `__hash__`, `__eq__`, `__lt__`, `__le__`, `__gt__`, `__ge__`. Its initializer and path joining methods accept only strings, not os.PathLike objects more broadly. This is important for supporting *virtual paths*: user subclasses of `_PathBase` that provide access to archive files, FTP servers, etc. In these classes, the above methods should be implemented by users only as appropriate, with due consideration for the hash/equality of any backing objects, such as file objects or sockets.

…hon#112881) Move `_PurePathBase` and `_PathBase` to a new `pathlib._abc` module, and drop the underscores from the class names. Tests are mostly left alone in this commit, but they'll be similarly split in a subsequent commit. The `pathlib._abc` module will be published as an independent PyPI package (similar to how `zipfile._path` is published as `zipp`), to be refined and stabilised prior to its possible addition to the standard library.

…ython#112901)

…12904)

…tribute (python#113221) Store the test base directory as a class attribute named `base` rather than module constants named `BASE`. The base directory is a local file path, and therefore not ideally suited to the pathlib ABC tests. In a future commit we'll change its value in `test_pathlib_abc.py` such that it points to a totally fictitious path, which will help to ensure we're not touching the local filesystem.

…hon#113376) `PurePathBase.__repr__()` produces a string like `MyPath('/foo')`. This repr is incorrect/misleading when a subclass's `__init__()` method is customized, which I expect to be the very common. This commit moves the `__repr__()` method to `PurePath`, leaving `PurePathBase` with the default `object` repr. No user-facing changes because the `pathlib._abc` module remains private.

…ython#113219) Change the value of `pathlib._abc.PurePathBase.pathmod` from `os.path` to `posixpath`. User subclasses of `PurePathBase` and `PathBase` previously used the host OS's path syntax, e.g. backslashes as separators on Windows. This is wrong in most use cases, and likely to catch developers out unless they test on both Windows and non-Windows machines. In this patch we change the default to POSIX syntax, regardless of OS. This is somewhat arguable (why not make all aspects of syntax abstract and individually configurable?) but an improvement all the same. This change has no effect on `PurePath`, `Path`, nor their subclasses. Only private APIs are affected.

python#113411) This very boring patch reduces the number of changes needed in `test_pathlib_abc.py` when backporting to the external `pathlib_abc` package.

…on#113417) Do not use the locale-specific default encoding in `PathBase.read_text()` and `write_text()`. Locale settings shouldn't influence the operation of these base classes, which are intended mostly for implementing rich paths on *nonlocal* filesystems.

…ython#113419) The `pathlib._abc` module will be made available as a PyPI backport supporting Python 3.8+. The `warnings._deprecated()` function was only added last year, and it's private from an external package perspective, so here we switch to `warnings.warn()` instead.

…ther. (python#110312) This is a very soft deprecation of `PurePath.as_uri()`. We instead document it as a `Path` method, and add a couple of sentences mentioning that it's also available in `PurePath`. Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

barneygale added type-feature A feature request or enhancement topic-pathlib labels Sep 29, 2023

barneygale self-assigned this Sep 29, 2023

barneygale mentioned this issue Oct 3, 2023

Python 3.12 - AttributeError: module 'pathlib' has no attribute '_Flavour' devopshq/artifactory#430

Open

barneygale added a commit to barneygale/cpython that referenced this issue Oct 3, 2023

pythonGH-110109: Simplify pathlib test setup

957e1bb

Use the class under test to create files, directories and symlinks.

bedevere-app bot mentioned this issue Oct 3, 2023

GH-110109: Simplify pathlib test setup #110308

Merged

bedevere-app bot mentioned this issue Oct 3, 2023

GH-110109: pathlib docs: bring from_uri() and as_uri() together. #110312

Merged

bedevere-app bot mentioned this issue Oct 3, 2023

[WIP] GH-110109: Make pathlib extensible #110321

Closed

barneygale added a commit to barneygale/cpython that referenced this issue Oct 5, 2023

Merge branch 'main' into pythongh-110109-simplify-test-setup

eb0298c

bedevere-app bot mentioned this issue Oct 5, 2023

GH-110109: Speed up pathlib._PathBase.resolve() #110412

Merged

barneygale added a commit to barneygale/cpython that referenced this issue Oct 10, 2023

Merge branch 'main' into pythongh-110109-simplify-test-setup

ab5057a

barneygale added a commit that referenced this issue Oct 10, 2023

GH-110109: Simplify pathlib test setup (#110308)

da0a68a

Use the class under test to create files, directories and symlinks.

barneygale added a commit to barneygale/cpython that referenced this issue Oct 10, 2023

pythonGH-110109: Move tests for pathlib.Path.walk() into main test …

17511b3

…classes. This adds some test coverage to `pathlib._PathBase.walk()`

bedevere-app bot mentioned this issue Oct 10, 2023

GH-110109: Move tests for pathlib.Path.walk() into main test classes. #110655

Merged

bedevere-app bot mentioned this issue Oct 11, 2023

GH-110109: Add pathlib._PurePathBase #110670

Merged

barneygale added a commit that referenced this issue Oct 28, 2023

GH-110109: Move tests for pathlib.Path.walk() into main test classe…

81bc802

…s. (#110655)

barneygale added a commit to barneygale/cpython that referenced this issue Oct 28, 2023

Merge branch 'main' into pythongh-110109-purepathbase

6489e4b

iritkatriel pushed a commit to iritkatriel/cpython that referenced this issue Oct 29, 2023

pythonGH-110109: Move tests for pathlib.Path.walk() into main test …

5c9882e

…classes. (python#110655)

barneygale added a commit to barneygale/cpython that referenced this issue Nov 12, 2023

Merge branch 'main' into pythongh-110109-purepathbase

6c742ed

bedevere-app bot mentioned this issue Nov 13, 2023

GH-110109: Churn pathlib.PurePath methods #112012

Merged

barneygale added a commit to barneygale/cpython that referenced this issue Nov 13, 2023

Merge branch 'main' into pythongh-110109-churn-purepath

7559ca5

barneygale added a commit to barneygale/cpython that referenced this issue Nov 13, 2023

Merge branch 'main' into pythongh-110109-purepathbase

3f56662

barneygale added a commit to barneygale/cpython that referenced this issue Nov 13, 2023

Merge branch 'main' into pythongh-110109-pathbase-resolve-speedups

4346183

barneygale added a commit to barneygale/cpython that referenced this issue Nov 17, 2023

Merge branch 'main' into pythongh-110109-pathlib-uri-docs

cae0964

barneygale added a commit to barneygale/cpython that referenced this issue Nov 17, 2023

Merge branch 'main' into pythongh-110109-pathbase-resolve-speedups

482752f

aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024

pythongh-110109: Fix installed buildbots now pathlib is a package (p…

579e390

…ython#112901)

aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024

pythonGH-110109: Move tests for pathlib ABCs to new module. (python#1…

c972eaf

…12904)

Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024

pythonGH-110109: Simplify pathlib test setup (python#110308)

aa604a6

Use the class under test to create files, directories and symlinks.

Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024

pythonGH-110109: Move tests for pathlib.Path.walk() into main test …

b4afa38

…classes. (python#110655)

Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024

pythongh-110109: Fix installed buildbots now pathlib is a package (p…

648e04b

…ython#112901)

Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024

pythonGH-110109: Move tests for pathlib ABCs to new module. (python#1…

6f2d606

…12904)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

User-defined virtual path objects #110109

User-defined virtual path objects #110109

barneygale commented Sep 29, 2023 •

edited

Loading

User-defined virtual path objects #110109

User-defined virtual path objects #110109

Comments

barneygale commented Sep 29, 2023 • edited Loading

Proposal

Phase 1: Private support in pathlib

Phase 2: Experimental support in PyPI package

Previous Discussion

Linked PRs

barneygale commented Sep 29, 2023 •

edited

Loading