Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

new file to s3 even no new data #150

Closed
peterdudfield opened this issue May 20, 2024 · 2 comments
Closed

new file to s3 even no new data #150

peterdudfield opened this issue May 20, 2024 · 2 comments
Assignees
@devsjc
Labels
bug Something isn't working

Comments

@peterdudfield
Copy link
Contributor

peterdudfield commented May 20, 2024 •
edited
Loading

Describe the bug

There seems to be a new file to s3, even though there is no new nwp data

To Reproduce

Steps to reproduce the behavior:

  1. go to s3 bucket, check timestamp
  2. go to ariflow, kick off nwp-consumer
  3. after finished, check s3 new timestamp
  4. repeat, ECMWF should only have new data twice a day

Expected behavior

Only upload to s3 if there is new data, This could be done by

  1. checking timestamps in the nwp, or.
  2. only if new raw data, then upload. Force upload if latest.zarr doesnt exists

Additional context

#openclimatefix/uk-pv-national-gsp-api#337
@peterdudfield peterdudfield added the bug Something isn't working label May 20, 2024
@devsjc devsjc mentioned this issue May 30, 2024
Merged
@peterdudfield
Copy link
Contributor Author

peterdudfield commented Jun 20, 2024 •
edited
Loading

The deployment is blocked by #156

@devsjc
Copy link
Collaborator

devsjc commented Nov 25, 2024

Fixed with #195 also

@devsjc devsjc closed this as completed Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@devsjc devsjc

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@peterdudfield @devsjc