time variables with "nanoseconds" units fail to load

[st-bender]

Hi,
Apologies if this is the wrong place to address this. I want to combine files that were split into yearly chunks back to one big one. A task that is easy to do in python with xarray using mfdataset or concat, but in julia I am failing miserably, or I am missing something obvious. Any pointers would be highly appreciated.
It looks like NCDatasets (or whatever) stumbles upon time variables with units "nanoseconds since ..."

julia> ds1
NCDataset: work/src/data_2006.nc
Group: /

Dimensions
   orbit = 10

Variables
  jd   (10)
    Datatype:    Float64
    Dimensions:  orbit
    Attributes:
     _FillValue           = NaN
     name                 = Julian date

  time   (10)
    Datatype:    Int64
    Dimensions:  orbit
    Attributes:
     units                = nanoseconds since 2006-12-11 19:30:04.881632400
     calendar             = proleptic_gregorian

  orbit   (10)
    Datatype:    Int64
    Dimensions:  orbit

Accessing the "time" variable now raises a warning, as discussed in #173

julia> ds1["time"]
┌ Warning: unknown units nanoseconds
└ @ NCDatasets ~/.julia/packages/NCDatasets/h1epE/src/cfvariable.jl:407
time (10)
  Datatype:    Int64
  Dimensions:  orbit
  Attributes:
   units                = nanoseconds since 2006-12-11 19:30:04.881632400
   calendar             = proleptic_gregorian

However according to the comment #181 (comment), "nanoseconds" should be a valid unit.

Manually converting with "seconds" works fine:

julia> timedecode(ds1["time"] ./ 1e9, "seconds since 2006-12-11 19:30:04.881632400", "proleptic_gregorian")
┌ Warning: unknown units nanoseconds
└ @ NCDatasets ~/.julia/packages/NCDatasets/h1epE/src/cfvariable.jl:407
10-element Vector{Dates.DateTime}:
 2006-12-11T19:30:04.882
 2006-12-11T21:10:13.441
 2006-12-11T22:51:33.205
 2006-12-12T00:33:42.081
 2006-12-12T02:16:29.906
 2006-12-12T03:59:16.903
 2006-12-12T05:41:51.619
 2006-12-12T07:24:24.082
 2006-12-12T09:07:07.771
 2006-12-12T10:49:39.066

However, the same trick fails with a different file where time is float and contains missing values, but that may be a different issue:

julia> ds2["time"]
┌ Warning: unknown units nanoseconds
└ @ NCDatasets ~/.julia/packages/NCDatasets/h1epE/src/cfvariable.jl:407
time (4834)
  Datatype:    Float64
  Dimensions:  orbit
  Attributes:
   _FillValue           = NaN
   units                = nanoseconds since 2007-01-11 18:05:51.068389200
   calendar             = proleptic_gregorian

julia> timedecode(ds2["time"] ./ 1e9, "seconds since 2007-01-11 18:05:51.068389200", "proleptic_gregorian")
┌ Warning: unknown units nanoseconds
└ @ NCDatasets ~/.julia/packages/NCDatasets/h1epE/src/cfvariable.jl:407
ERROR: MissingException: cannot convert a missing value to type Int64: use Union{Int64, Missing} instead
Stacktrace:
  [1] round(::Type{Int64}, ::Missing, ::RoundingMode{:Nearest}) (repeats 2 times)
    @ Base ./missing.jl:135
  [2] (::CFTime.var"#convert#7"{Int64, DateTimeProlepticGregorian})(x::Missing)
    @ CFTime ~/.julia/packages/CFTime/n09Um/src/CFTime.jl:495
  [3] _broadcast_getindex_evalf
    @ ./broadcast.jl:648 [inlined]
  [4] _broadcast_getindex
    @ ./broadcast.jl:621 [inlined]
  [5] getindex
    @ ./broadcast.jl:575 [inlined]
  [6] macro expansion
    @ ./broadcast.jl:984 [inlined]
  [7] macro expansion
    @ ./simdloop.jl:77 [inlined]
  [8] copyto!
    @ ./broadcast.jl:983 [inlined]
  [9] copyto!
    @ ./broadcast.jl:936 [inlined]
 [10] copy
    @ ./broadcast.jl:908 [inlined]
 [11] materialize
    @ ./broadcast.jl:883 [inlined]
 [12] timedecode(#unused#::Type{DateTimeProlepticGregorian}, data::Vector{Union{Missing, Float64}}, units::String)
    @ CFTime ~/.julia/packages/CFTime/n09Um/src/CFTime.jl:496
 [13] timedecode(data::Vector{Union{Missing, Float64}}, units::String, calendar::String; prefer_datetime::Bool)
    @ CFTime ~/.julia/packages/CFTime/n09Um/src/CFTime.jl:545
 [14] timedecode(data::Vector{Union{Missing, Float64}}, units::String, calendar::String)
    @ CFTime ~/.julia/packages/CFTime/n09Um/src/CFTime.jl:544
 [15] top-level scope
    @ REPL[40]:1

Environment

• operating system: Ubuntu 18.04
• CPU architecture Intel/AMD x86_64
• Julia version: 1.6.7
• NCDatasets version: v0.12.9
• CFTime version: v0.1.2

[Alexander-Barth]

Can you open an issue for the nanoseconds support at JuliaGeo/CFTime.jl ? I will handle it there .

Maybe this part is already resolved:

However, the same trick fails with a different file where time is float and contains missing values, but that may be a different issue.

Yes, this one should be easy to address. In fact, I just committed this in CFTime:
JuliaGeo/CFTime.jl@3913b0d

Can you confirm that it also work for you (with the master version of CFTime)?

timedecode(ds2["time"].var[:] ./ 1e9, "seconds since 2007-01-11 18:05:51.068389200", "proleptic_gregorian")

If you can share an example file, that would also be useful. (A file with random data still reproducing the issue is also fine).

[st-bender]

Hi @Alexander-Barth

Thanks for the fix.

Can you open an issue for the nanoseconds support at JuliaGeo/CFTime.jl ? I will handle it there .

Ok, will do that.

Maybe this part is already resolved:

However, the same trick fails with a different file where time is float and contains missing values, but that may be a different issue.

Yes, this one should be easy to address. In fact, I just committed this in CFTime: JuliaGeo/CFTime.jl@3913b0d

Can you confirm that it also work for you (with the master version of CFTime)?

Yes, it works now with floats and missing values.

timedecode(ds2["time"].var[:] ./ 1e9, "seconds since 2007-01-11 18:05:51.068389200", "proleptic_gregorian")

If you can share an example file, that would also be useful. (A file with random data still reproducing the issue is also fine).

I think the issue was just handling the missing values in the input, but I can provide an example file if you still need it.
Where would be the preferred place to upload it?

[Alexander-Barth]

Thanks a lot! In fact, as the missing value issue is solved, I have everything to reproduce the issue with the time units (so no need to upload an example).