Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nest CombineReferences inside WriteCombinedReferences #634

Closed
cisaacstern opened this issue Sep 27, 2023 · 2 comments
Closed

Nest CombineReferences inside WriteCombinedReferences #634

cisaacstern opened this issue Sep 27, 2023 · 2 comments

Comments

@cisaacstern
Copy link
Member

Working on #610 has me thinking about how to easily communicate common recipe styles. Over there, I'm centering on the idea that the basic version of all common pipeline styles should be expressible as: FilePattern | Opener | Writer. StoreToZarr fits this style, but our kerchunk recipes do not:

| OpenWithKerchunk(
file_type=pattern.file_type,
remote_protocol=remote_protocol,
storage_options=storage_options,
kerchunk_open_kwargs={"filter": grib_filters},
)
| CombineReferences(
concat_dims=pattern.concat_dims,
identical_dims=identical_dims,
precombine_inputs=True,
)
| WriteCombinedReference(
store_name="hrrr-concat-step",
)

To align kerchunk recipes with this style, we can nest CombineReferences inside WriteCombinedReferences, making the basic kerchunk recipe expressible as:

recipe = (
    beam.Create(pattern.items())
    | OpenWithKerchunk(...)
    | WriteCombinedReferences(...)
)
@norlandrhagen
Copy link
Contributor

This seems like a good call!

@norlandrhagen
Copy link
Contributor

Closed by #635

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants