Ignore iternums which only have pickup files

[knaughten]

I found that xmitgcm breaks if you point it to a directory where some iternums only have pickup files, and no matching diagnostic files at the same timestep. Here is a small patch to ignore those iternums.

[codecov-io]

Codecov Report

Merging #81 into master will decrease coverage by 0.29%.
The diff coverage is 0%.

@@           Coverage Diff            @@
##           master     #81     +/-   ##
========================================
- Coverage   91.89%   91.6%   -0.3%     
========================================
  Files           4       4             
  Lines         629     631      +2     
  Branches      138     139      +1     
========================================
  Hits          578     578             
- Misses         32      33      +1     
- Partials       19      20      +1

Impacted Files	Coverage Δ
xmitgcm/mds_store.py	91.39% <0%> (-0.47%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f8ae6b3...f4e5ff4. Read the comment docs.

[rabernat]

Thanks so much for this contribution!

I'm wondering if it would be possible to include a very short test for this in the test suite. Could you clarify how exactly "xmitgcm breaks" in this scenario? (i.e. Post the full error message and stack trace that you encounter.) Then we can figure out how to add a test for it.

[knaughten]

Hey Ryan, here is an example:

My workflow is that after each MITgcm run segment (as much as the model can do in a 24-hour queue limit) I convert all the diagnostic .data files to NetCDF using xmitgcm, then put the original files in a special folder so they don't get processed again after the next run segment. But I don't move any pickup files, as sometimes I need them for the restart.

Here's what happens if I point xmitgcm to a directory with some old permanent checkpoint files (eg pickup.0003682080.data) which don't correspond to any diagnostic files, which are starting at timestep 4075200. Here I'm using the commit from 6 April (commit f8ae6b3), before I made any changes.

This is my code:

import sys
from xmitgcm import open_mdsdataset
ds=open_mdsdataset('.',delta_t=60,ref_date='1979-1-1 0:0:0')

and here is the error:

IOError                                   Traceback (most recent call last)
<ipython-input-3-df3d49ffabaf> in <module>()
----> 1 ds=open_mdsdataset('.',delta_t=60,ref_date='1979-1-1 0:0:0')

/work/n02/n02/kaight/python/xmitgcm/xmitgcm/mds_store.py in open_mdsdataset(data_dir, grid_dir, iters, prefix, read_grid, delta_t, ref_date, calendar, geometry, grid_vars_to_coords, swap_dims, endian, chunks, ignore_unknown_vars, default_dtype, nx, ny, nz, llc_method)
    148                                       "found %s)" % (repr(first_prefixes),
    149                                                      iternum,
--> 150                                                      repr(these_prefixes)))
    151 
    152                 # chunk at least by time

IOError: Could not find the expected file prefixes [] at iternum 4.0752e+06. (Instead found ['stateUVEL', 'stateEXF', 'state2D', 'stateWVEL', 'stateSALT', 'stateTHETA', 'stateICE', 'stateVVEL'])

Looks like what's happening in this case is that file prefixes are determined from the first iternum (which only has pickup files so they're ignored), and then the file prefixes from the diagnostic files don't match this (empty) list.