Directory prefix Argo GDAC file system

[gmaze]

In this PR we provide a file system with a directory prefix for any Argo GDAC compliant path.

The goal is to be able to simply create a gdacfs instance with a GDAC path (local or remote) that won't require the GDAC path to be systematically provided to open files.

In other word, before this PR we have to do this:

from argopy.stores import gdacfs
fs = gdacfs("https://data-argo.ifremer.fr")
ds = fs.open_dataset("https://data-argo.ifremer.fr/dac/coriolis/6903091/profiles/R6903091_001.nc")
# or
fs = gdacfs("https://usgodae.org/pub/outgoing/argo")
ds = fs.open_dataset("https://usgodae.org/pub/outgoing/argo/dac/coriolis/6903091/profiles/R6903091_001.nc")

and after the PR we will be able to do this:

from argopy.stores import gdacfs
fs = gdacfs("https://data-argo.ifremer.fr")
# or
fs = gdacfs("https://usgodae.org/pub/outgoing/argo")
# then
ds = fs.open_dataset("dac/coriolis/6903091/profiles/R6903091_001.nc")

This is easily implemented in the current gdacfs but requires a lot of upstream compatibility check for all argopy stores implementations.

The ultimate test will be this to run ok:

from argopy.stores import gdacfs
from argopy.utils import list_gdac_servers

for gdac_path in list_gdac_servers():
  fs = gdacfs(gdac_path)
  ds =  fs.open_dataset("dac/coriolis/6903091/profiles/R6903091_001.nc")

[codecov]

❌ 178 Tests Failed:

Tests completed	Failed	Passed	Skipped
1729	178	1551	546

View the top 3 failed tests by shortest run time

test_fetchers_index_gdac.py::TestBackend::test_fetching[file, float]

Stack Traces | 0.001s run time

No failure message available

test_fetchers_index_gdac.py::TestBackend::test_fetching[file, profile]

Stack Traces | 0.001s run time

No failure message available

test_plot_plot.py::Test_bar_plot::test_with_a_region[with_seaborn=False, by=institution]

Stack Traces | 0.001s run time

self = <argopy.tests.test_plot_plot.Test_bar_plot object at 0x000001DA1E4E9D10>
opts = (False, 'institution')

    @pytest.mark.parametrize("opts", opts, indirect=False, ids=opts_ids)
    def test_with_a_region(self, opts):
>       with argopy.set_options(src=self.src, gdac=self.local_gdac):

argopy\tests\test_plot_plot.py:151: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
argopy\options.py:264: in __init__
    VALIDATE(k, v)
argopy\options.py:144: in VALIDATE
    if not _VALIDATORS[key](val):
argopy\options.py:90: in validate_gdac
    return check_gdac_option(this_path, errors="raise")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = 'C:\\Users\\runneradmin\\.argopy_tutorial_data\\ftp', errors = 'raise'

    def check_gdac_option(path, errors="ignore"):  # noqa: C901
        """Check if a path has the expected GDAC server structure
    
        Expected GDAC structure::
    
            .
            └── dac
                ├── aoml
                ├── ...
                ├── coriolis
                ├── ...
                ├── meds
                └── nmdis
    
        Examples::
    
        >>> check_gdac_option("https://data-argo.ifremer.fr")  # True
        >>> check_gdac_option("ftp://ftp.ifremer.fr/ifremer/argo") # True
        >>> check_gdac_option("ftp://usgodae..../pub/outgoing/argo") # True
        >>> check_gdac_option(".../home/ref-argo/gdac") # True
        >>> check_gdac_path("s3://argo-gdac-sandbox/") # True
        >>> check_gdac_option("https://www.ifremer.fr") # False
        >>> check_gdac_option("ftp://usgodae.org/pub/outgoing") # False
    
        Parameters
        ----------
        path: str
            Path name to check, including access protocol
        errors: str
            "ignore" or "raise" (or "warn")
    
        Returns
        -------
        checked: boolean
        """
        from .stores import gdacfs  # Otherwise raises circular import
    
        try:
            fs = gdacfs(path)
        except GdacPathError:
            if errors == "raise":
                raise
            elif errors == "warn":
                warnings.warn("Can't get address info (GAIerror) on '%s'" % path)
                return False
            else:
                return False
    
        check1 = fs.exists(fs.sep.join([path, "dac"]))
        if check1:
            return True
    
        elif errors == "raise":
>           raise GdacPathError(
                "This path is not GDAC compliant (no legitimate sub-folder `dac`):\n%s"
                % path
            )
E           argopy.errors.GdacPathError: This path is not GDAC compliant (no legitimate sub-folder `dac`):
E           C:\Users\runneradmin\.argopy_tutorial_data\ftp

argopy\options.py:353: GdacPathError

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!