Feature request: dataset over multiple NetCDF files.

[briochemc]

How does one open a dataset spread over multiple files (like with xarray's open_mfdataset function)?

[Balinus]

There is an undocumented function:

YAXArrays.jl/src/DatasetAPI/Datasets.jl

Line 335 in 04aaaad

open_mfdataset(g::AbstractString; kwargs...) = open_mfdataset(_glob(g); kwargs...)

and

YAXArrays.jl/src/DatasetAPI/Datasets.jl

Line 945 in 04aaaad

function merge_datasets(dslist)

I do not remember exactly how to use it, something along the lines of:

YAXArrays.Datasets.open_mfdataset("/path/files") or perhaps YAXArrays.Datasets.open_mfdataset(list_of_files) ? or both

[Balinus]

I had to reuse some code where mfdataset was used. In this case, it worked as expected.

[briochemc]

Yes I've used it successfully too 😃

I think it ought to be documented though, right?

[Balinus]

Is there a preprocessing option that can be added to open_mfdataset? I'd like to add a time dimension based on the filename for a suite of > 300 files that has only 2D longitude-latitude data.

for example, in Python I can do:

import xarray as xr
import numpy as np


def preprocessing(ds): 
    time_str = os.path.basename(ds.encoding['source']).split('_')[0:2]    # get year and month from filename
    time_str.append('01') # add a hardcoded day to list of time
    time_str =  "-".join(time_str) # create a single date string
    ds['time'] = np.datetime64(time_str, 'ns') # assign to dataset ds
    ds = ds.set_coords('time') # specify time as a coordinate
    return ds

ds_crps = xr.open_mfdataset(list_of_files, concat_dim='time', combine='nested', preprocess = preprocessing)