Bug: Odd timestamp

[peterdudfield]

Im currently pulling pvlive/api/v4/gsp/0?extra_fields=installedcapacity_mwp,capacity_mwp,updated_gmt&start=2024-12-06T15:00:00Z&end=2024-12-06T17:30:00Z&period=30
And i get back

{"data":[[0,"2024-12-06T16:December:00",0.0,17804.681,16699.632,"2024-12-06T16:December:20"],[0,"2024-12-06T16:December:00",1.71723,17804.681,16699.632,"2024-12-06T16:December:20"],[0,"2024-12-06T15:December:00",51.0678,17804.681,16699.632,"2024-12-06T16:December:21"],[0,"2024-12-06T15:December:00",244.489,17804.681,16699.632,"2024-12-06T16:December:22"]],"meta":["gsp_id","datetime_gmt","generation_mw","installedcapacity_mwp","capacity_mwp","updated_gmt"]}

which has some odd datestamps. I think i could be a mistake between %m and %M in the timestring

[JulianBriggs]

Resolved
Bug was using Python datetime format string in MySQL query :-(

[gjlipman]

I note that the new API is now returning the index without a 'Z' suffix, while the old index has a 'Z'. (Also, the new API is missing some records and has different numbers for some records).

[JulianBriggs]

Hi, Thanks. I've fixed the missing Z. Pls give more info about the missing records and particularly different numbers for some records. Thanks kind regards Julian Briggs

…

On Fri, 6 Dec 2024 at 21:57, gjlipman ***@***.***> wrote: I note that the new API is now returning the index without a 'Z' suffix, while the old index has a 'Z'. (Also, the new API is missing some records and has different numbers for some records). — Reply to this email directly, view it on GitHub <#43 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APAUAON2UYOPTNKJCJJIFAL2EIMVRAVCNFSM6AAAAABTFAYOJWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRUGI2DQMBVG4> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

-- Julian Briggs Research Software Development Manager Sheffield Solar Physics & Astronomy University of Sheffield I am currently working remotely. My preferred method of contact is email. My normal working hours are 07:00 - 17:00 Mon-Thu

[peterdudfield]

Thanks @JulianBriggs

[gjlipman]

Thanks - it looks like the missing rows and different values that I raised on Friday have corrected themselves (the values from the new api now match the values that the old api was giving us on Friday).

However, if I run it now, for more recent periods (using the following snippet of code), while I get the same number of rows, I get a few different values for today. I'm guessing the new API maybe doesn't have as up to date data as the old api?

import requests

start = "2024-12-07T00:00"
end= "2024-12-08T23:59"

url = (f"https://api.solar.sheffield.ac.uk/pvlive/api/v4/pes/"
            f"0?start={start}&end={end}")
data_old = requests.get(url, allow_redirects=True)
df_old = pd.DataFrame(data_old.json()["data"])[[1, 2]].set_index(1).round(1).rename(columns={2: 'old'})

url = (f"https://api.pvlive.uk/pvlive/api/v4/pes/"
            f"0?start={start}&end={end}")
data_new = requests.get(url, allow_redirects=True)
df_new = pd.DataFrame(data_new.json()["data"])[[1, 2]].set_index(1).round(1).rename(columns={2: 'new'})

print(f"{len(df_old)=}, {len(df_new)=}")

df = pd.concat([df_old['old'], df_new['new']], axis=1)

df[(df.new-df.old).abs()>1]```

                                          old	      new	
2024-12-08T13:00:00Z	1509.6	1507.2
2024-12-08T12:30:00Z	1655.6	1643.8
2024-12-08T10:30:00Z	1298.4	1303.3
2024-12-08T10:00:00Z	952.8	1086.6
2024-12-08T09:30:00Z 540.8	539.4