DBUtils implementation for Volumes

[mgyucht]

Changes

UC Volumes has been released for some time, but users are unable to use UC Volumes with dbutils.fs from the SDK. This PR implements support for /Volumes paths in DBUtils in the SDK.

I've done this primarily by extending DbfsExt to work with UC Volumes paths. A new class _VolumePath is supported, implementing the set of base operations defined on the _Path parent abstract class for volume paths. An accompanying _VolumesIO is also implemented to provide a consistent interface for reading from and writing to UC Volumes files, especially for writing: the download API already returns a BinaryIO, but the upload API accepts a BinaryIO, so this adapter allows for a user to "open" a Volumes path for writing.

In order to properly implement ls, I changed the existing implementation of list in _Path subclasses to return a generator of FileInfos. This allows for better reuse of this common functionality between ls, cp and mv.

Open questions:

• Should we do anything to distinguish between a path named /Volumes on disk and UC Volumes? E.g. via a different scheme?

Tests

• make test run locally
• make fmt applied
• relevant integration tests applied

[github-actions]

This PR breaks backwards compatibility for databrickslabs/ucx downstream. See build logs for more details.

_{Running from downstreams #65}

[tanmay-db]

Slicing the list will create a copy so this will be O(N), we can improve this to O(1) using deque and use .popleft() to get the first element.

[tanmay-db]

same here

[tanmay-db]

Should we consider typos here: https://docs.databricks.com/en/connect/unity-catalog/volumes.html

Paths are also reserved for potential typos for these paths from Apache Spark APIs and dbutils, including /volumes, /Volume, /volume, whether or not they are preceded by dbfs:/. The path /dbfs/Volumes is also reserved, but cannot be used to access volumes.

[mgyucht]

I think those paths are reserved but may not be usable themselves for looking up volumes (hopefully). Given that we use the REST API, we may need to be more narrow in what we accept than in DBR, since these paths correspond directly to REST API parameters.