update read-in module for ATL11

[JessicaS11]

The read-in module was not working for ATL11. This PR aims to remedy that.

[github-actions]

👈 Launch a binder notebook on this branch for commit 76576ae

I will automatically update this comment whenever this PR is modified

👈 Launch a binder notebook on this branch for commit 31594a9

👈 Launch a binder notebook on this branch for commit 79c4b7c

👈 Launch a binder notebook on this branch for commit 9226737

👈 Launch a binder notebook on this branch for commit 35088ba

👈 Launch a binder notebook on this branch for commit 191851f

👈 Launch a binder notebook on this branch for commit 5b2da3a

👈 Launch a binder notebook on this branch for commit 1b26978

👈 Launch a binder notebook on this branch for commit 8568fae

👈 Launch a binder notebook on this branch for commit 45540ce

👈 Launch a binder notebook on this branch for commit 03befa0

👈 Launch a binder notebook on this branch for commit dfaf2e9

👈 Launch a binder notebook on this branch for commit 775dba1

👈 Launch a binder notebook on this branch for commit bf6e632

[chris-picard]

Hi Jessica, when trying to pull ATL11 using icepyx, I kept getting an error "pop from empty list":

#get a list of granule IDs for the available granules
region_a.avail_granules(ids=True) 

Which returned the error message:

IndexError                                Traceback (most recent call last)
/var/folders/d2/jjjzv8t95113nztdlt260q8r0000gn/T/ipykernel_14830/3270823387.py in <module>
      1 #get a list of granule IDs for the available granules
      2 print(region_a.CMRparams)
----> 3 region_a.avail_granules(ids=True)

~/opt/anaconda3/lib/python3.9/site-packages/icepyx/core/query.py in avail_granules(self, ids, cycles, tracks, s3urls)
    907         if ids or cycles or tracks or s3urls:
    908             # list of outputs in order of ids, cycles, tracks, s3urls
--> 909             return granules.gran_IDs(
    910                 self.granules.avail,
    911                 ids=ids,

~/opt/anaconda3/lib/python3.9/site-packages/icepyx/core/granules.py in gran_IDs(grans, ids, cycles, tracks, dates, s3urls)
    107                 AUX,
    108                 SFX,
--> 109             ) = rx.findall(producer_granule_id).pop()
    110             gran_cycles.append(CYCL)
    111             gran_tracks.append(TRK)

IndexError: pop from empty list

I ran this identical code for ATL06 and ATl12 without any issues. It seems like there is an issue with identifying the ID's of granules for ATL11. I have adjusted the spatial boundaries and maximized the temporal ones to make sure that I am getting enough data -- but there is still this error. Thoughts?

[weiji14]

Hi Jessica, sorry for the late review, just warming up after a long break. I've just left some minor comments for now from glancing through the code, will do a more thorough review once I set up my development environment properly on a new laptop :D

[weiji14]

Won't this while loop code break on L796 once the wanted_groups_list variable is assigned a value? Or does it run to the end of the while-block first before breaking? Maybe I'm confusing Python with some other programming language...

[JessicaS11]

No, because it will either get to this version of the while loop from the elif on L743 (which ends with a return) or the L796 version of the while loop via the else on L775.

[JessicaS11]

@weiji14 I think it's pretty close to being ready to merge!

[weiji14]

Yay, reading ATL11 finally works in icepyx 🥲. So I tried downloading a sample file from https://n5eil01u.ecs.nsidc.org/ATLAS/ATL11.005/2022.03.27/ATL11_006305_0315_005_03.h5 and ran the following code in this atl11 branch:

import icepyx as ipx
import xarray as xr

# Read in ATL11
reader = ipx.Read(
    data_source=".",
    product="ATL11",
    filename_pattern="ATL{product:2}_{rgt:4}{orbitsegment:2}_{cycles:4}_{version:3}_{revision:2}.h5",
)
reader.vars.append(
    beam_list=["pt1", "pt2", "pt3"], var_list=["longitude", "latitude", "h_corr"]
)
ds: xr.Dataset = reader.load()
print(ds)

gives

<xarray.Dataset>
Dimensions:              (gran_idx: 1, cycle_number: 13, photon_idx: 5851,
                          pair_track: 3)
Coordinates:
  * gran_idx             (gran_idx) float64 1e+06
  * cycle_number         (cycle_number) float32 3.0 4.0 5.0 ... 13.0 14.0 15.0
  * photon_idx           (photon_idx) int64 0 1 2 3 4 ... 5847 5848 5849 5850
  * pair_track           (pair_track) <U3 'pt1' 'pt2' 'pt3'
    source_file          (gran_idx) <U29 './ATL11_006305_0315_005_03.h5'
    delta_time           (photon_idx, cycle_number) datetime64[ns] NaT ... 20...
    ref_pt               (photon_idx) float64 6.062e+05 6.062e+05 ... 6.112e+05
Data variables:
    atlas_sdp_gps_epoch  (gran_idx) datetime64[ns] 2018-01-01T00:00:18
    data_start_utc       (gran_idx) datetime64[ns] 2022-03-27T05:32:07.954158
    data_end_utc         (gran_idx) datetime64[ns] 2022-03-27T05:35:11.277237
    longitude            (pair_track, gran_idx, photon_idx) float64 154.3 ......
    latitude             (pair_track, gran_idx, photon_idx) float64 71.07 ......
    h_corr               (pair_track, gran_idx, photon_idx, cycle_number) float32 ...
    gt                   (gran_idx, pair_track) <U3 'pt1' 'pt2' 'pt3'
    rgt                  (pair_track, gran_idx, photon_idx) float64 nan ... 5...
Attributes: (12/24)
    data_product:             ATL11
    ATL06_xover_field_list:   ['delta_time', 'h_li', 'h_li_sigma', 'latitude'...
    L_search_AT:              60
    L_search_XT:              65
    N_coeffs:                 8
    N_poly_coeffs:            8
    ...                       ...
    seg_atc_spacing:          100
    seg_number_skip:          3.0
    seg_sigma_threshold_min:  0.05
    t_scale:                  31557600.0
    xy_scale:                 100.0
    beam_pair:                3

A couple of nitpicks:

1. The cycle_number coordinate has a float32 dtype, but it should really just be uint8 if possible
2. The gt data variable lists pair tracks 'pt1', 'pt2', 'pt3'. Should the data variable name be called 'pt' instead of 'gt' (ground track is for ATL06, pair track is for ATL11)?

Other than that, the filename_pattern of "ATL{product:2}_{rgt:4}{orbitsegment:2}_{cycles:4}_{version:3}_{revision:2}.h5" should probably be documented somewhere (at least in the docstring), and once that's done, it should be almost ready to merge!

[weiji14]

Awesome, thanks again @JessicaS11!