API: Input Data Last Updated Field incorrect

[zakwatts]

When the "include_metadata" tag is true the following information returns:

"inputDataLastUpdated": { "gsp": "2023-10-24T16:22:44.812068Z", "nwp": "2023-08-29T14:22:49.582920Z", "pv": "2023-10-24T16:18:29.611871Z", "satellite": "2023-10-24T16:06:39.083938Z" } }

However these dates look very outdated to today 2024-05-15. Possibly a bug.

[zakwatts]

@peterdudfield Do you think this is a bug?

[peterdudfield]

yea this is a bug. How important is it for the client?

[zakwatts]

No one has mentioned it, I just came across when testing the API.

[peterdudfield]

ok, thanks @zakwatts

[peterdudfield]

So just checked,
It looks like, GSP and satellite is wokring, but NWP isnt. This is becasue of the new nwp-consumer. I dont think the right thing it to add it to the nwp-consumer, as that is nice and stand alone.

Here's some options

1. Add to new nwp-consumer ( not want to do this)
2. Add to airflow, something that hits the database, and updates the input data last updated. This is neat and in ariflow, but has the disadvanate, we dont know if the data consumer receieved any new data, or just ran correctly
3. Add endpoint to api, that goes and checks when the GSP, Satellite and NWP was last updated. Need to hide this and maybe add auth0 with scopes. Airflow can then call this after running the data consumer

[peterdudfield]

There's another fix thats needed. Some ends point load a latest forecast. These forecast get updated everytime a forecast gets saved. We need to update the input_data_lates_updated as well in here

[peterdudfield]

Discuss with @devsjc and @braddf Idea we think we are going to go with is

1. Add enpoint in API that goes and check when the NWP data was last made. Then save to InputDataLastUdpated in the database.
2. Add auth to this route, scope: write: data_updated and create a auth0 role called admin that is allowed to call this endpoint
3. Add dag in airflow that is triggered at the end of the nwp consuer, that hits this new endpoint in the API

[peterdudfield]

This is fixed with

• blend 0.0.29 (done on prod)
• API 1.5.52 (done on prod)
• ocf-infrastructure hash of 72ba38d. - Need to also make sure API has access to nwp and satellite bucket (done on prod)
• GSP Consumer 1.1.14
• Satip 2.11.62

[zakwatts]

Nice, good work!

[peterdudfield]

Had to role back openclimatefix/Satip#273, but this is not a problem