Efficient way to append data to dfs2 / dfs3?

[Snowthe]

Hi there,
Is there an efficient way to append data to an existing dfs2 or dfs3 file? I.e. a way to append data without having to load and read the entire existing file, and then baiscally write a totally new one?
Would be helpful for me; we are often dealing with large dfs files.

The DFSU 2D class seems to offer an append() method: https://dhi.github.io/mikeio/dfsu-2d.html - but I'm not sure if that offers the functionality I am asking about.

[ecomodeller]

It is possible to open Dfs2,3 files in edit mode and append data to the file using the lower level mikecore library https://github.com/DHI/mikecore-python/blob/29fd0e4fe84e9fecd3da1e3725f0b2ed46891723/mikecore/DfsFileFactory.py#L53

This is efficient, and appending to an existing file is easier than creating a new from scratch since you don't need to specify the entire header.

I am thinking about how to make this even easier with MIKE IO.

[JacobGudbjerg]

At GEUS we are going to need this as well. Is there anything in the pipeline or should we go directly to the low-level approach?

[ecomodeller]

Nothing like that in the pipeline.

Out of curiosity, do you need the data to be in a single file?

Or would the approach used by xarray Reading multi-file datasets be appropriate, e.g. splitting by year.

[JacobGudbjerg]

We would like to keep it in a single file (20GB or more). We are currently building the file using a python-library I built but we would like to move to MikeIO instead. My library used ufs.dll so I can use the low level approach.

[ecomodeller]

What do you think of this, this will append a dataset to an existing file, avoiding reading/writing the entire existing file?

class Dfs2

  def append(self, ds: Dataset) -> None:
          """
          Append a Dataset to an existing dfs2 file
  
          Parameters
          ----------
          ds: Dataset
              Dataset to append
  
          Notes
          -----
          The Dataset must have the same items as the dfs2 file.
          The original file is modified.
          """
          # TODO check ds compliant with self
          dfs = DfsFileFactory.Dfs2FileOpenAppend(str(self._filename))
          write_dfs_data(dfs=dfs, ds=ds, n_spatial_dims=2)

[JacobGudbjerg]

I think that would work. We would then write the first timestep in the traditional way and then append the remaining as we process them. Maybe it could be considered to keep the file open and then close it in a dispose step? This is of course only relevant if the opening and closing is significant time-wise.

[ecomodeller]

It is possible to open Dfs2,3 files in append mode and append data to the file using the lower level mikecore library https://github.com/DHI/mikecore-python/blob/29fd0e4fe84e9fecd3da1e3725f0b2ed46891723/mikecore/DfsFileFactory.py#L53

This is efficient, and appending to an existing file is easier than creating a new from scratch since you don't need to specify the entire header.

I am thinking about how to make this even easier with MIKE IO.