please add some astro examples!

[rabernat]

As discussed in pangeo-data/pangeo#269 (comment), in a few weeks I will be presenting on Pangeo at the EarthCube RCN Workshop: “Towards Integration of Heliophysics Data, Models, and Analysis Tools".

I would love to be able to share astro.pangeo.io. On the old pangeo.pydata.org, I developed the following code to view the SDO AIA data with holoviews:

import intake
import xarray as xr
import numpy as np
import holoviews as hv
from holoviews.operation.datashader import regrid
hv.extension('bokeh')

source = intake.open_fits_array('gcs://pangeo-data/SDO_AIA_Images/094/*.fits')
data = source.to_dask()
# the only trick was first wrapping in xarray...otherwise holoviews doesn't seem to work
xrds = xr.DataArray(data, dims=['time', 'y', 'x'],
                    coords={'time': ('time', np.arange(1800)),
                     'x': ('x', np.arange(4096)),
                     'y': ('y', np.arange(4096))},
                    name='fits_data')
hvds = hv.Dataset(xrds)
im = hvds.to(hv.Image, kdims=['x', 'y'], dynamic=True)

# new cell
%opts Image [width=800, height=800 logz=True] (cmap='magma')
regrid(im, dynamic=True)

I don't pretend to be an expert here. I would prefer if someone else would provide some examples of how to view and process this dataset. Ideally these could be added to the default examples in astro.pangeo.io.

So this is my plea: astonomers / heliophysicists, please add some example notebooks for me!

[martindurant]

@SimonKrughoff , the Gaia image would be a nice start, I assume you already have a notebook for that.

[SimonKrughoff]

I have a couple notebooks. I'll add them to the deploy (but probably not until tomorrow).

[martindurant]

@rabernat , should we add 'gcs://pangeo-data/SDO_AIA_Images/094/*.fits' (and others in this directory) and 'gcs://pangeo-data/gaia_dr2.parquet' to the pangeo catalog, or would you rather these lived in a different, astro-specific one?

[martindurant]

Actually, trying your code, I find that images after timepoint 1200 don't show

[wtbarnes]

Can I access gcs://pangeo-data from astro.pangeo.io? I'm trying to test out an example on the SDO data discussed in pangeo-data/pangeo#269 but don't seem to be able to resolve the path in the same way...

Additionally, is there a way to upload data to the GCS bucket? If not, the solar data already uploaded will also work.

[martindurant]

Yes you can, all the data are public. What, exactly were you trying to do?
e.g.,

import gcsfs
gcs = gcsfs.GCSFileSystem()
gcs.ls('pangeo-data')

but you probably wanted some xarray/dask/intake method?

For uploading, you need credentials. I don't now if the containers of astro.pangeo have suitable credentials or not.

[wtbarnes]

Thanks @martindurant. I was just missing the special gcs.ls. The previously uploaded data set should work fine so I won't worry about upload credentials.

[wtbarnes]

@rabernat Sorry for the long delay! I'm working on an example now that I will push to pangeo-data/pangeo-astro-examples and should run on astro.pangeo.io. What is the format of the presentation? Are you planning to do a live demo or just show a precompiled notebook?

[rabernat]

Fantastic, thanks @wtbarnes! I will be doing a live demo of course! 🎆 It's the only way to really convince people that it actually works.

It would be ideal to binderize pangeo-astro-examples, as has been done for https://github.com/pangeo-data/pangeo-example-notebooks

Then the examples can truly be run by anyone, anywhere

[wtbarnes]

@rabernat Sorry I never followed up on this again. Did the demo go ok? I hope that showing solar physicists the power of tools like Dask, xarray, et al. will convince them all to move to Python!

I was at another conference at the time and had a few people try out the binder version of the timelag notebook and they were able to run it without issue.