Dampening

[autoSteve]

I think a beneficial improvement may be interpolated dampening values for inter-hour forecast values.

As we know, these are currently configured by the hour, which makes total sense. All generally lines up nicely on the Energy dashboard for a clear day, yet falls somewhat short when one utilises the half-hourly forecast attribute values for things like the chart above (thank you, @gcoan).

The forecast for each day has the same hourly dampening value applied to both 30-minute intervals, and this strikes me as room for improvement.

Thoughts?

[gcoan]

There are pro's and con's of adding interpolation to the dampening output.

Dampening to take account of say shading from next door's house, that shades the panels by 50% up to 9am then 0% from 10am, an interpolated 25% shading at 9:30am for the half hourly forecast might be correct in that the panels are partially shaded, or it might be wrong that it really is a vertical time line of when the shading stops.

Would depend in part on what type of shading it is, a building giving a much more knife edge to the shading whereas a tree would be softer transition.

(Ignoring of course that the dampening factors should change through the year as the sun elevation changes through the seasons)

Might be the case of deckchairs on the titanic to fiddle with this

[autoSteve]

I have thought long and hard about this over the past few days.

I see issue with using the generation history. Well, I see other issues as well.

For me, my nasty power distributor (not retailer, who are also nasty) requires that a 5kW export limit be in place for the site. This has the effect that of an afternoon once my batteries fill, the PV export is curtailed if it exceeds the limit. Because 12kW AC generation it will be curtailed a lot in Summer. This could be interpreted as shading.

This led me down a rabbit hole of examining the history of the generation sensor vs. site export sensors to try and determine whether curtailing was happening for a given period. My thought was to exclude these curtailed readings to not impact dampening. Turns out that the sensor does not hit a red-line of 5kW. Sometimes 5.4. Sometime 5.1. Sometimes... Even averaged for half-hour periods the number jumps.

This all led me down a further rabbit hole of, "How does one get sensor history data for a sensor that my integration does not own?" Which led to the creation of a user API token, which got messy from an integration standpoint. It would need to be added in integration configuration. Annoyingly if this integration were an HA add-on instead then I could make supervisor proxied calls to HA without needing an API token (assuming a supervisor was part of the install set-up, which sometimes it is not). An integration cannot just blindly make calls to get other integration sensor values and history.

Then I deleted the dev branch. On purpose.

[Nilogax]

Thanks for taking a look. Seems like a good example of why it's so much easier to build a home grown solution that fits your (my?) own needs than to build a solution that will support all the wierd and wonderful configurations out in the wild.

Playing devil's advocate, does this just mean that your setup isn't suitable for automated damping? So if you had calculated damping values available you could determine which ones are valid (shading etc) and which are artefacts of network rules.

[autoSteve]

That was the plan. I have failed for now.

I would love to get automated dampening running factoring network limit, esp. in Winter when I do get some shading at the tail end of a day because of a bloody big tree a couple houses down. Spring/Summer/Autumn is epic. Ugh. Back to HA automations for now, bad as they are for me.

I will stow this in the back of my mind to work on an automated integration solution some day.

[autoSteve]

BTW, ChatGPT did a super half-arsed job just now, if I'm honest, totally avoiding Python.

built an automated home assistant solar forecast dampening calculation for BJRaplay Solcast integration that takes into account transmission limit

The result soooo won't work. And I'm so not surprised. Any wonder SkyNet got it so wrong in Terminator with everyone dying.

The "build it in Python" version was even more hystrical, even though our overlord was confident in its response.

I still have not used ChatGPT for anything at all. And doubt I ever will.

[autoSteve]

One last one, sorry. This nearly broke its head, despite prior questioning.

Build an automated home assistant solar forecast 24-hour dampening calculation in Python, adding it to the Ozzie Solcast integration, taking into account feed-in limit, but also factoring battery state of charge. The dampening needs to take into account the prior five days of actual production.

It literally stalled for ten seconds before anything came back. Then there was spewed unsurprisingly gibberish-esq Python.

This is a fun game.

But leave it to the grown-ups, ChatGPT, yeah?

[froxic]

I have another idea. What about to adjust dampening curve based on historic PV generation from PV HA sensor? Algo then adjust dampening proactively based on e.g. last 7 days generation with correlation to previous SolCast forecast. This could be nice also during whole year since PV generation could differ during autumn / winter period for instance and on my installation i need to readjust dampening based on hours since there is sunset 6 hours sooner.

[CTRDevelopments]

OK, it seems to have worked this morning which is super. However I got an error when the update script ran

Due to my data issues, I had previously copied the same data to all of the logs. (I have disabled the write function while I get this working)

The plots of the damping factors etc are also blank as if it stopped running the script before it got to that stage

[CTRDevelopments]

I think I have fixed it. I had previously tried to define a sensor named 'adaptivedamping'. I removed this and then manually ran the code again. It now seems to complete and the graphs update. I will leave it for a few days to collect sensible data now and then update :) thanks for all the help, I have been wanting to do this for a while

[CTRDevelopments]

Does it matter if the log file seems to log 1am at the end?

[Nilogax]

No the log file records times as UTC then the damping update script adjusts for the current time offset

[CTRDevelopments]

Had this running now for a few days and it seems to be working nicely. Many thanks for support and a great bit of code! Will continue to see how it goes over the next few weeks

[Nilogax]

In the unlikely event that anyone is brave/daft enough to try to use my MacGyver soluton I've updated it so it will now work with Solcast 4.1.9.

[Nilogax]

@gcoan I saw your comment in the 4.2.0 discussion re granular damping. I'm still using hourly damping factors at the moment because of a couple of things.

First off, there's the combination of my relatively modest generation combined with GivTCP reporting generation in 0.1Kwh steps. I think breaking down to 30 min intervals should be a pretty simple tweak but I am concerned it could lead to some spurious results. Maybe using a longer history would help here or maybe adding something like @autoSteve's spliney goodness to the generation figures, maybe it's all in my head and I should just try it anyway.

Secondly, as the saying goes "It's only a forecast". Hourly dampening figures are giving results that I feel are acceptable. On days where the weather follows the forecast I'm getting +/- 10% of the predicted Solcast generation. Days with bigger variance usually correlate with days where it's unexpectedly sunny (or cloudy) at my peak generation times. Obviously with higher generation that %age variance translates into more KWh so it could be more important to try to narrow it down with half hourly dampening figures, but the weather will always have the final say.

Also FWIW I have Solcast's efficiency turned right up so I start with an unfiltered forecast and the dampening calcs do all the work. I didn't see the point in massaging figures that had already been massaged.

[gcoan]

Makes sense about turning off the Solcast efficiency rating so you are starting with raw Solcast figures.

I had my efficiency as 98 or 99%, reflecting what I thought would be the case for 18 month old panels. The main roof is completely unshaded so I expected to get near perfect alignment of forecast to generation, but I didn't. Part of it was due to having a large number of panels that I push the grid voltage up too high and two of my three inverters will stop generating and then restart a short time later (seconds not minutes). But this throttling off and on knocks the generation back.

Dampening might be a route to getting a more accurate forecast, but given the throttling off and on is power output related, it might just be a different blunt instrument!

For clarification, does your script just operate at the total PV output level or does it work on a per-site basis? I definitely get more throttling on my main East facing array (site).

But as solar generation is dropping off this is much less of an issue now than it was in the summer.

[Nilogax]

Ah sounds like you need some sort of threshold damping so have one value if the forecast PV for the period is above X and another if its below. That's beyond my simple solution for sure but would obviously be more sensitive with 30 min segments.

Anyhow yes it's built around a single site as that's what I have, so uses total PV generation. Extending to multiple sites is left as an exercise for the reader 😁

[erepeo]

Are there any plans for this project to have automatic hourly damping calculation implemented?

[autoSteve]

TL>DR: There are no plans, @erepeo.

Attempting to do this in-integration would require getting generation actuals from outside of the integration to measure against expected forecasts, and then building a weekly/seasonal model and adjusting it over time. We do forecasts. Inverters do actuals. Getting inverter data is possible from the integration, but unweildy and impractical.

The focus of this integration for now is on being rock-solid, providing as much transparency as possible, and as much raw forecast detail as needed to support automations that folks can think up. Actions are available to set dampening via automations / other things at both overall and per rooftop site level. Actions are available to get forecasts, again both overall and per-site. Hourly dampening is a thing. Half-hourly is a thing.

An horizon obstruction model is another thought. If one could define that a local obstruction existed of a certain height, and at a certain distance, width, density and azimuth from "here" it could be possible to determine its impact based on a calculated position of the sun at any given hour of the year, right? Nope. PV generation doesn't work that way. Reflective generation would need to be approximated base on cloud cover, which we don't have, so we would get the answer horribly wrong on mostly cloudy days when diffuse light is king and obstructive trees mean nothing. But Solcast have that answered, just not for us.

Solcast do model irradiance far better than this integration ever could, and for paying customers they do have horizon modelling available. We just consume the more-than-good-enough data produced by their data scientist's typical models. For free. Which is more than good enough for hobbyist applications.

Bottom line, there are no plans from me.

[erepeo]

It's a pity but I understand. Thank you for your work.