30-min data vs 1-hour data

[HarryFlatter]

This is probably not a bug, but more likely my lack of understanding as to how you treat the forecasting data returned from Solcast.

This issue came to light when I started using @Webreaper excellent SolisAgile.

I noticed that his daily solar forecast was exactly double what I was receiving from your HA integration, despite both sourcing their forecast data from Solcast using the same API key.

So I did some diagnostic work of my own....

Firstly, I downloaded my PV arrays data from Solcast and just simply totalled the 30-minute slots that it provides and found that they agreed spot on with SolisAgileManager and were double what I was seeing in HA.

I then looked at the 'Attribute' data that arrives with the 'sensor.solcast_pv_forecast_forecast_today' entity and discovered something that might sort of explain what's happening.

When I totalled the "Detailed Forecast" 30-minute slots from the attributes, they agreed spot on with both my Solcast CSV downloads and SolisAgileManager.

However, when I looked further down the attribute data I noticed "Detailed Hourly" 1-hour slots. I then totalled these and discovered that they came to exactly half of what the 30-minute slots added up to and agreed to the penny with what I was seeing in HA.

I have attached a snapshot of my XL spreadsheet which illustrate my findings.

What I need to understand is why the 1-hour slots total to half of the 30-minute slots and more importantly, why do the 1-hour slots give me a much more accurate forecast than the raw 30-minute slots from Solcast????

Sorry in advance if I am being a pain here!

[BJReplay]

Without the source data in your spreadsheet, this is only a guess - but it looks like you are summing power (kW) rather than summing energy (kWh).

Hard to tell from an image of the spreadsheet.

Share more if you would like a more detailed explanation.

But the key point is that the Solcast forecasts are power forecasts.

You can convert them to energy (and the integration does this for the daily and remaining energy sensors).

[HarryFlatter]

Hi, many thanks for the quick response.
I'm attaching my XL sheet and the 2 CSVs from Solcast
I've copied the HA data from the 'sensor.solcast_pv_forecast_forecast_today' entity attribute drop down and pasted it into the XL sheet

Analysis.xlsx
GetRooftopSiteForecast - PV1.csv
GetRooftopSiteForecast-PV2.csv

[autoSteve]

A more detailed eplanation, @HarryFlatter. #38 (comment)

[HarryFlatter]

Many thanks for that. Much clearer now :-)

[BJReplay]

OK, @HarryFlatter, you're definitely trying to sum half hourly or hourly power predictions. That's not how Power works :)

The HA Attributes are the data as returned by Solcast, which are also Power (kW) estimates - even though these attributes are on an energy sensor (such as Forecast Today) which is in kWh.

They represent Solcast's estimate of the average power over either 30 minutes or one hour, depending on which series you're looking at.

You can accidently correctly sum the 60 minute power figures, because 1kW over 1 hour is 1kW/h of energy. But 1kW over 30 minutes is 0.5kW of energy.

Imagine if you had a flat curve that was 1kW from 6am to 6pm. If you figure that out, that's 12kWh, because the 1kW was generating for 12 hours.

Now, if you happen to sum up 1 over 12 hourly buckets, you get 12.

But the power estimates for half-hourly power will still be 1kW - that's the estimate for that particular half hour. But if you sum up 24 half hourly buckets from 6am to 6pm, you'll get 24 now. That's because you're summing power, rather than converting to energy first.

1kW for 30 minutes is 0.5kWh.

See attached spreadsheet with corrected workings (highlighting your values vs corrected values).

Often you might see "convert kW to kWh by dividing by 2" when it talks about 1/2-hourly power forecasts, but the correct way is to multiply by the integration period (30 minutes) divided by an hour (60 minutes) to convert a power figure for a period of time to the energy for that period of time.

Analysis.xlsx

[BJReplay]

Just to confirm that whist the 30-min kW slots in the attributes are lifted from the Solcast data, the hourly kWH slots are generated by the integration?

Sort of: They are calculated as the average of the two half-hourly forecasts from the Solcast data (so yes to generated, but a simple average of two half-hourly figures from Solcast):

ha-solcast-solar/custom_components/solcast_solar/solcastapi.py

Line 1683 in b4f259f

def build_hourly(forecast) -> list[dict[str, Any]]:

A side effect of this is that they are both average kW forecasts for the hour, as well as kWh figures. That's why you can get away with adding them in your example - the combination of two figures, and dividing by 2 both turns them into an hourly average, and also, quite neatly, converts kW to kWh.

[HarryFlatter]

Again, thanks again for such a lucid explanation.

Now I just have to balance the rather meagre 10 call limit to the Solcast API between HA and SolisAgile.

One last question on this. Would I be correct in assuming that the integration keeps it's own tally of API calls made and that there is no function provided to reveal the total calls made by all applications using my Solcast account? Each app will just receive an HTTP 429 (Too many requests) once the limit is blown?

[autoSteve]

Yep. There used to be a non-quota consuming API call to ask how many calls had been made and were allowed. It went away, so just set the Solcast Solar integration max to five calls if you're buddy-breathing with another.

(Not encouraging you to break the Solcast Ts&Cs, but) if only you had two accounts with the same setup, hence ten calls per account...

[BJReplay]

Would I be correct in assuming that the integration keeps it's own tally of API calls

Yes. The API used to provide a call that would tell you, but Solcast discontinued it.

Each app will just receive an HTTP 429 (Too many requests) once the limit is blown?

Yes, that's right.

[HarryFlatter]

@autoSteve re "Not encouraging you to break the Solcast Ts&Cs, but) if only you had two accounts with the same setup, hence ten calls per account..."

That did occur to me, but I'm sure I read somewhere that they might pick this up as two accounts with the same lat/lon. Or was that 2 arrays in a Hobbyist account whose lat/lon references were too far apart?

[Webreaper]

Yes, thanks @BJReplay - this is really useful. Currently I'm calculating forecast power by taking the 30-min forecast reading in kW and dividing by 2. However, I think your approach of taking the current and next reading and using them to make an average across the 30-minute period is liable to be much better, as it'll reflect the delta for that period. Appreciate the explanations!

[Webreaper]

Given that this question came from someone running Home Assistant, depending on how many of your users have HA, and how savvy they are, one thing you could consider is allowing users who have both to specify the path where this integration writes its solcast.json and solcast-undampened.json and you could, potentially load that file and not call the API.

Thanks, that's a neat idea, but the support overhead of managing it (given the many combinations of docker and non-docker installs of both apps) is just going to be painful, tbh.

The more I see the solcast data, the more obvious it is that really doing 2 calls a day is probably more than enough, so unless somebody is running HA, my app and something else, with a 2-string / 2-site Solcast setup, it's not really going to be a problem. And even in that case, I have an option in my app to say "just call solcast once a day". :)

[BJReplay]

but the support overhead of managing it

Fair enough.

the more obvious it is that really doing 2 calls a day is probably more than enough

Yeah, when I look at your project, I tend to agree. Getting the most up to date forecast before you do something expensive (like charging from the grid on more expensive power than ideal), if you're doing it based on the forecast, is a good idea, but if your charging windows are typically overnight for the next day, then no point really getting lots of forecasts overnight - they won't be a lot more accurate until T-4 or thereabouts.

However, due to a bug in handling empty 429 responses, my updates have stalled after the 9-ish in the morning update the last two days.

I fixed it at 1:50pm, and you can see the notch in the forecast - that was because Solcast had been forecasting less sun when I last successfully updated at 9 in the morning, but by mid-afternoon, it was clear that the forecast cloud was going to substantially clear.

So, if you end up with an opportunity to charge during the day (depending on what pricing is doing - for me it is free between 11am and 2pm - so I may as well charge regardless of forecast, and then discharge excess), an update a little bit before decision time will help refine the strategy.

I used to have discount - but not free - power between 10am and 3pm (9c/kWh vs 20c/kWh or 30c/kWh later in the day), so when charging wasn't free, I checked the forecasts regularly between sunrise and 3pm, and kept on updating my expected generation that I'd get from solar prior to the end of the cheap charging period, and the time needed to charge my expected peak / overnight consumption, and delay my charging decision as much as possible. If forecasts improved, I hadn't charged unnecessarily. If they worsened, I had enough buffer time in my calculations to charge as soon the forecast worsened.

The last polled data shows the big gap in updates that caused the forecast to be out for quite a while.

[BJReplay]

@Webreaper if it is of any use for your thoughts on how to use Solcast data - this is what I used to do, back in the day: https://github.com/BJReplay/PowerwallService/blob/master/PowerwallService/PowerwallService.vb#L629

VB.Net - because that's what I used to know, and a windows service and I'm using a few time bands (Super Off Peak, Off Peak, Peak, and Overnight) but I also have a charge buffer (to allow for forecast inaccuracy) that reduces as you get closer to the start of Peak (when your available charge and the available remaining insolation are better known, so the buffer you need to avoid forecasting errors is reduced).

This is part of an ugly piece of code that uses an SQL Server database for historical load (to allow for load forecasting - which was remarkably accurate - basically same time last year, biased with last two weeks was all the AI that was needed 🤣), solar forecasting, and configuration managed in config files.

The vast majority of it is a bad example of what not to do, but it might give you some ideas for your use case.

[gcoan]

The more I see the solcast data, the more obvious it is that really doing 2 calls a day is probably more than enough, so unless somebody is running HA, my app and something else, with a 2-string / 2-site Solcast setup, it's not really going to be a problem. And even in that case, I have an option in my app to say "just call solcast once a day". :)

The approach Predbat uses to coexist with Solcast is either Predbat can call the Solcast API's directly, or you can use the Solcast integration and Predbat reads the Solcast data from the integration's HA entities.
That way there isn't two different things consuming Solcast API calls. Wonder if you could follow the same approach?

I personally use the Solcast integration because of the benefits it brings, but plenty of people just use the Predbat-direct calls to Solcast.

[autoSteve]

overhead of managing it (given the many combinations of docker and non-docker installs of both apps) is just going to be painful, tbh

Probably not. The location of config/ should be trivial to work out given deployment variations. We solved for this quite a while back.

The raw data may be there. Use it if you can find it @Webreaper. If you can't, then go to the API.

[Webreaper]

Solcast integration and Predbat reads the Solcast data from the integration's HA entities.
That way there isn't two different things consuming Solcast API calls. Wonder if you could follow the same approach?

My app is intentionally completely independent from HA, so has absolutely nothing to do with it. So that's not really an option (and I'm trying to get away from HA, not integrate more closely with it, so....

Probably not. The location of config/ should be trivial to work out given deployment variations. We solved for this quite a while back.

I guess my point is that my app can be installed locally, on every platform, or via docker, on every platform. HA is the same. I just don't really want the support overhead of trying to help people match up their different installation etc. Again, one of my goals here is to make HA redundant, because I'm tired of maintaining it for the minimal value it provides. That's one of the primary reasons I'm writing my app.

But... thanks for the info and suggestions. If my stance changes in future, or I find myself with excessive time, I'll certainly consider what you've suggested. But given that I don't run solcast in HA at all (ever since the dev deleted the repo and broke the solcast integration I just deleted it and haven't bothered looking back since) there's not much value in this for me. :)