Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EV charging support/integration #121

Open
mergwyn opened this issue Feb 13, 2024 · 186 comments
Open

EV charging support/integration #121

mergwyn opened this issue Feb 13, 2024 · 186 comments
Labels
enhancement New feature or request long term Good idea but not a priority

Comments

@mergwyn
Copy link

mergwyn commented Feb 13, 2024

Is your feature request related to a problem? Please describe.
I'm all electric at home with ASHP, Solar, storage and and an EV charger. PV Opt does a good job of managing my solar and storage but I also need to be able schedule EV charging

Describe the solution you'd like
I'd really like pvopt to ensure that charging the EV is synchronised such that ir does not discharge the storage battery. At its simplest level the EV is just another battery which needs be charged to a target SOC in the cheapest slots so hopefully there is an overlap in the algorithm. This could either be done fully within PV Opt or via hooks that allow users to build their own automation using the results (eg publishing x cheapest slots)

Describe alternatives you've considered
At the moment, I am using zappi 'agile knowledge' to set a schedule that only charges when the tariff is below a certain level, but obviously this may or may not cause the storage battery to discharge. (I also have to change schedule manually depending on the latest tariff)

I've tried using Predbat, but this is soo complicated and I've not been able to understand the charging plans it puts together and am struggling with the amount of 'hold' and 'reserve' charging that does on.

Additional context
I understand that this might not be where you want or need to take PV Opt and appreciate all that you have done so far. I wish I could program in python to help out but that it beyond me! If I can help in any other way please let me know.

@SzosszeNET
Copy link
Contributor

SzosszeNET commented Feb 13, 2024

Like this idea. While you get a response as an alternative not sure if you looked into it, but zappi seems to have its own integration.
https://github.com/CJNE/ha-myenergi

Could imagine to have both, and create an automation (under settings in HA) if not already one in the myenergi repo, that would watch the myenergi integration and if it's scheduled to charge, at the beginning of the charge window:
Stop AppDaemon (or maybe less drastic to set pv_opt to read only)
Set a static charge window to the inverter (to hold charge
Just to be sure 00 the discharge window and current

And potentially a second one that would start AppDaemon when the myenergi charge is finished/is outside the charge window?

@fboundy
Copy link
Owner

fboundy commented Feb 13, 2024

Keen to support this as an idea but I don't want to get as complicated as PredBat!

One starting point would be the entity that already exists called switch.pvopt_charge_active - this is On when pv_opt is charging so is a good starting point. If I knew how much charge your needed for Zappi in a given window and at what rate I could probably call the zappi integration too.

It sounds like installing the Zappi integration would be a start. I would then need to know what sensors and services it exposes.

@mergwyn
Copy link
Author

mergwyn commented Feb 14, 2024 via email

@fboundy fboundy added the long term Good idea but not a priority label Mar 20, 2024
@stevebuk1
Copy link
Contributor

stevebuk1 commented Mar 23, 2024

Seconded for an EV integration. I'm on IOG so all I'd be looking for is a "hold SOC" whilst the EV is charging. I'd prefer a charge current = 0 to do the hold function rather than setting Backup/Reserve, as in Backup/Reserve my inverter doesnt respect Bit 6 in the Energy Storage register to prevent grid charging, causing oscillations between full rate charging and full rate discharging during EV charging. This rules me out of Pv-opt for now.

I'm currently running Predbat and am contributing to Solis code changes for IOG, but I can see that the fundamental way it works is never going to work properly for Agile due to GivEnergy functions Solis inverters just don't have (like Freeze Charging) - and I figure one day I'll be moving to Agile.

I have an ID4 and Zappi so similar to [mergwyn] and will happily assist with Alpha/Beta testing.

@fboundy
Copy link
Owner

fboundy commented Mar 25, 2024

I think that setting charge current to zero for the Solis is a better method of holding SOC anyway so I'm about to start testing that.

How would you flag up the EV charging slots with IOG?

@mergwyn
Copy link
Author

mergwyn commented Mar 25, 2024

BottlecapDave's integration has a sensor: binary_sensor.octopus_energy_{{ACCOUNT_ID}}_intelligent_dispatching which is used to determine if you're currently in a planned dispatch period (i.e. "smart-charge" determined by Octopus Energy) or are within the standard off peak period. This sensor will not come on during a bump charge which I think is an OK limitation.

There are some limitations: https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/blob/b06499c86548847b5c75c17a487a87178f59b64a/_docs/entities/intelligent.md?plain=1#L9

This doesn't cover how to make EV charging work with Agile rather than IOG which is a harder problem I suspect. I'm probably going to switch to IOG shortly until the Autumn, so again this is a limitation I can live with for a while at least!

Thanks again for your great work on pv_opt.

@fboundy
Copy link
Owner

fboundy commented Mar 25, 2024

OK - so looking at the attributes for this would it be reasonable to assume that pv_opt should simply "hold SOC" for all entries in planned_dispatches?

@stevebuk1 - I'm a bit confused by your comment regarding Bit 6 for Energy Storage as this is the Feed In Priority bit. Do you mean this or Bit 5 which is Grid Charging? How does this interact with setting Backup Mode (Bit 4) and Backup SOC? One of the current limitations with the Solax integration is that the control switch isn't set bit-wise or by number but using a select. That said I've successfully added modes to the integration before.

@mergwyn
Copy link
Author

mergwyn commented Mar 25, 2024

For me, the ideal situation would be for pv_opt to do its thing regarding whether the solar batteries need to be charged. If this is at the same time as the EV needs to be charged (indicated by intelligent_dispatching for IOG) is it possible just to set the discharge current to zero (or some equivalent) such that both the EV and solar batteries can be charged at the same time?

@stevebuk1
Copy link
Contributor

I'm a bit confused by your comment regarding Bit 6 for Energy Storage as this is the Feed In Priority bit. Do you mean this or Bit 5 which is Grid Charging?

Apologies, I was responding to an Issue in the Solis Inverters Facebook group where people were labelling the 8 bits as Bit1 to Bit 8, and I forgot to reset my language to the traditional LSB = 0, MSB = 7 nomenclature. It is indeed Bit 5.

Regarding the Solax integration, it currently has two options for Backup/Reserve:

"Backup/Reserve"
"Backup/Reserve - No Grid Charging".

and I imagine the 2nd one clears Bit 5.

What I was trying to say in my previous post is that there is no change in behavior between these modes (both charge from grid when battery SOC is below backup SOC), so I'd prefer using charge/discharge slots and current (amps) setting to do it instead.

@stevebuk1
Copy link
Contributor

OK - so looking at the attributes for this would it be reasonable to assume that pv_opt should simply "hold SOC" for all entries in planned_dispatches?

Yes, this is the way Predbat works, or rather it does now in my local copy since the addition of setting charge slots and current = zero for Solis inverters. Note: I'd toyed with using discharge start/end times and discharge current = 0, but Predbat doesnt clear out old charge start/end times anyway, so I just used the charge start/end times. I guess either would work as long as there are no conflicts between charge and discharge times.

@fboundy
Copy link
Owner

fboundy commented Mar 25, 2024 via email

@fboundy
Copy link
Owner

fboundy commented Mar 25, 2024 via email

@stevebuk1
Copy link
Contributor

I wrote some of the initial Solis code for PredBat but I gave up on it because I found it too heavily predicated on GivEnergy behaviour. I also couldn’t get my head around the optimiser which often didn’t seem very optimal for Agile.

I remember you saying in the Solis Facebook group you authored some of the code in Predbat. I've carried on making some changes and additions for getting it working in IOG (e.g. I've disabled it using Backup/Reserve mode to fix low rate charging) but the target SOC changes wildly through the 6 hours cheap rate period for what looks like no input changes (solcast, power forecast etc), resulting in a mess of high rate and low rate charging with constant writes to the inverter all night. For this reason I've just started trying to figure out the optimizer too but not encouraged by your experience. With EV hold functionality PV_opt will do the job, although getting in an electrician to add a Henley block for the Zappi is the other way ;-)

@stevebuk1
Copy link
Contributor

Regarding consumption data, I'm not sure if the current algorithm just uses what the Solis supplied to the house as energy, or takes account of the house load coming from grid during charging slots and hold slots. If one load shifts to co-inside with charging slots to avoid battery losses (as I do) then its likely direct use of grid energy will be a significant portion of consumption, so I assume grid energy forms part of the consumption data calculations?

If the assumption is correct, then car charging is going to add to (and corrupt) any existing method of grid use being factored into consumption data.
This means the energy consumed from the grid use for car charging will need discounting from any consumption data, so it doesn't influence the charging plans for the house battery. Ultimately, we are compensating for the fact that the Solis CT clamp sees both the EV and the house as consumption.

I see 3 different ways of doing this:

  1. Assume any grid draw above 6kW will mean the EV is charging, so subtract 7kw (the actual EV charge rate) from the consumption figures
  2. Utilise a Zappi entity that tracks consumption "sensor.myenergi_zappi_{Serial number}_charge_added_session", and subtract this from the consumption figures. Note this resets to zero when the car is plugged in.
  3. Utilise the "completed dispatches" attribute within the "_intelligent_dispatching" entity mentioned above and subtract it from the consumption figures.

Apologies if this was already obvious.

@fboundy
Copy link
Owner

fboundy commented Mar 26, 2024 via email

@stevebuk1
Copy link
Contributor

stevebuk1 commented Mar 26, 2024

Make sense?

Not really. Looking at the log file, consumption is loaded from "sensor.solis_house_load_today". The plot of that sensor is below, and it includes the 26kWh ish of car charging that I did last night. It can't be blind to it, otherwise the we wouldn't have the problem of the house battery discharging into the EV in the first place?

image

@fboundy
Copy link
Owner

fboundy commented Mar 27, 2024

OK - got it.

I think all 3 of your suggestions would work. I'd probably prioritise them in terms of what is available on a per case basis:

  1. Zappi (or other EV charger entity)
  2. IO dispatches (would only work for IO I presume)
  3. Charge threshold power which is the crudest and might pick up
    cooking Sunday Roast in error

@fboundy
Copy link
Owner

fboundy commented Mar 27, 2024

I'm hoping to have something out to test later today as a pre-release I don't think it will work fully but hopefully it will log enough stuff to gather the necessary info.

@fboundy
Copy link
Owner

fboundy commented Mar 27, 2024

v3.13.0-ev-beta-1 should now be available as a pre-release via HACS. It adds:

If you can load it up and check it doesn;t bust anything. If it doesn't then could you run it for at least 24 hours and then send me the pv_opt.log and error.log (if any errors occur)?

If this works OK then next step is to isolate the IO charging intervals from the optimisation.

Thanks

@fboundy fboundy added the enhancement New feature or request label Mar 27, 2024
@stevebuk1
Copy link
Contributor

Great - I've now installed it and its up and running. Its scheduled a house battery charge for this evening (the weather in Scotland is currently awful solar-wise) and I'll plug in the car.

I assume there aren't any changes needed to apps.yaml?

@stevebuk1
Copy link
Contributor

stevebuk1 commented Mar 27, 2024

I think all 3 of your suggestions would work. I'd probably prioritise them in terms of what is available on a per case basis:

  1. Zappi (or other EV charger entity)
  2. IO dispatches (would only work for IO I presume)
  3. Charge threshold power which is the crudest and might pick up
    cooking Sunday Roast in error

Agree that Option1 is the best, and should easily be configurable for other chargers.
2) agree thats only useful for IO, so no use for Agile or the other tariffs
3) is crude and it would probably work for me as during the day as I try and keep all consumption below 3kW (max battery discharge rate in my 3.6kW inverter) so I don't draw from grid. Overnight is similar (I offset all appliances), but I doubt it would work for those having G99 inverters and bigger battery banks.

@fboundy
Copy link
Owner

fboundy commented Mar 27, 2024

Great - I've now installed it and its up and running. Its scheduled a house battery charge for this evening (the weather in Scotland is currently awful solar-wise) and I'll plug in the car.

I assume there aren't any changes needed to apps.yaml?

Shouldn't be any changes needed. It should just check for the io and zappi entities and then spit out their contents to the log file.

@stevebuk1
Copy link
Contributor

Apologies, I spoke too soon. I flicked the toggle on Consumption to go fixed, and have since realised that it did a restart of PV_opt ok but the first run had hung. I've now restarted AppDeamon and Pv_Opt is stuck in "Loading Tarrifs".

Error log code is:

15:58:03 WARNING pv_opt: ------------------------------------------------------------
15:58:03 WARNING pv_opt: Unexpected error running initialize() for pv_opt
15:58:03 WARNING pv_opt: ------------------------------------------------------------
15:58:03 WARNING pv_opt: Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/appdaemon/app_management.py", line 162, in initialize_app
await utils.run_in_executor(self, init)
File "/usr/lib/python3.11/site-packages/appdaemon/utils.py", line 304, in run_in_executor
response = future.result()
^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/homeassistant/appdaemon/apps/pv_opt/pv_opt.py", line 407, in initialize
self._check_for_zappi()
File "/homeassistant/appdaemon/apps/pv_opt/pv_opt.py", line 472, in _check_for_zappi
for entity_id in self.self.io:
^^^^^^^^^
AttributeError: 'PVOpt' object has no attribute 'self'

15:58:03 WARNING pv_opt: ------------------------------------------------------------

I thought it might be a duplicate "self" at line 472, but removing it didnt make any difference:

16:03:22 WARNING pv_opt: ------------------------------------------------------------
16:05:16 WARNING pv_opt: ------------------------------------------------------------
16:05:16 WARNING pv_opt: Unexpected error in worker for App pv_opt:
16:05:16 WARNING pv_opt: Worker Ags: {}
16:05:16 WARNING pv_opt: ------------------------------------------------------------
16:05:16 WARNING pv_opt: Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/appdaemon/app_management.py", line 162, in initialize_app
await utils.run_in_executor(self, init)
File "/usr/lib/python3.11/site-packages/appdaemon/utils.py", line 304, in run_in_executor
response = future.result()
^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/homeassistant/appdaemon/apps/pv_opt/pv_opt.py", line 407, in initialize
self._check_for_zappi()
File "/homeassistant/appdaemon/apps/pv_opt/pv_opt.py", line 472, in _check_for_zappi
for entity_id in self.io:
TypeError: 'bool' object is not iterable

@fboundy
Copy link
Owner

fboundy commented Mar 27, 2024

If you are happy to edit it yourself it's just a simple typo. Line 472 should be:

        for entity_id in self.self.io_entities:

Didn't flag in my system because it was an empty list for me and the prior if caught it

@stevebuk1
Copy link
Contributor

stevebuk1 commented Mar 27, 2024 via email

@fboundy
Copy link
Owner

fboundy commented Mar 27, 2024

Any changes to the .py files or config.yaml should cause AD to restart it

@mergwyn
Copy link
Author

mergwyn commented Mar 27, 2024

I've also installed this version and made the edit to py_opt.py, but I get a different error:

16:45:14 WARNING pv_opt: ------------------------------------------------------------
16:46:15 WARNING pv_opt: ------------------------------------------------------------
16:46:15 WARNING pv_opt: Unexpected error running initialize() for pv_opt
16:46:15 WARNING pv_opt: ------------------------------------------------------------
16:46:15 WARNING pv_opt: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/appdaemon/app_management.py", line 162, in initialize_app
    await utils.run_in_executor(self, init)
  File "/usr/local/lib/python3.10/site-packages/appdaemon/utils.py", line 304, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home-assistant/appdaemon/apps/pv_opt/pv_opt.py", line 407, in initialize
    self._check_for_zappi()
  File "/home-assistant/appdaemon/apps/pv_opt/pv_opt.py", line 472, in _check_for_zappi
    for entity_id in self.self.io_entities:
AttributeError: 'PVOpt' object has no attribute 'self'

I'm also seeing something I've not noticed before:

16:45:14 WARNING pv_opt: ------------------------------------------------------------
16:45:14 WARNING pv_opt: Unexpected error running initialize() for pv_opt
16:45:14 WARNING pv_opt: ------------------------------------------------------------
16:45:14 WARNING pv_opt: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/appdaemon/app_management.py", line 162, in initialize_app
    await utils.run_in_executor(self, init)
  File "/usr/local/lib/python3.10/site-packages/appdaemon/utils.py", line 304, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home-assistant/appdaemon/apps/pv_opt/pv_opt.py", line 372, in initialize
    while (not self.inverter.is_online()) and (retry_count < ONLINE_RETRIES):
AttributeError: 'InverterController' object has no attribute 'is_online'

@fboundy
Copy link
Owner

fboundy commented Mar 27, 2024

OK hopefully 3.14.0-ev-beta-2 fixes all these issues. Too many beta versions flying around...

@stevebuk1
Copy link
Contributor

I think the only thing outstanding for IOG might be stopping the EV charging on a forced discharge of the battery. I think this is an issue because IOG puts the Zappi charger into Eco++ mode so that it only charges when export is greater than 1.7kW. So on a forced discharge the EV will be charged form the solar battery which doesn't make sense when there is a differential of 7.5p between cheap rate and export.

Apologies but I think I completely misunderstood the point you made above. I thought you were looking for Pv_opt to suppress a discharge if the car was charging because it was in an scheduled IOG slot. what you really meant was that if a discharge is taking place, the Zappi should not divert this to the car if the car is plugged in, it should instead go to grid as that pays better?

If so, the most usual way to fix that is to set an "export margin" in the Zappi to a level above what the inverter is capable of. This ensures all export goes to grid and not to car. This is a front screen Zappi setting, rather than anything thats available via the myEnerrgi API or myEnergi App. Theres a MyEnergi article about it here:

https://support.myenergi.com/hc/en-gb/articles/18468235308433-How-to-stop-menergi-products-diverting-exported-power

@fboundy
Copy link
Owner

fboundy commented Jul 3, 2024

@stevebuk1 - a im testing beta-4 testing with no car/io:

  File "/homeassistant/appdaemon/apps/Github/pv_opt/apps/pv_opt/pv_opt.py", line 1958, in optimise
    self.log(self.io_slots.to_string())
             ^^^^^^^^^^^^^
AttributeError: 'PVOpt' object has no attribute 'io_slots'

I think you need to set a default for self.io_slots. I simply added:

527      self.io_slots = pd.DataFrame()

to pv_opt.py and that has fixed it.

Seems to now be running OK so I'll test for a couple of days.

@stevebuk1
Copy link
Contributor

I think you need to set a default for self.io_slots. I simply added:

527      self.io_slots = pd.DataFrame()

to pv_opt.py and that has fixed it.

Thanks - I thought I'd caught all the variables that wouldn't be initialised on a non-EV setup but missed this one. I've added it into my local code.

Seems to now be running OK so I'll test for a couple of days.

Great, let me know of any issues.

I'm still working on the odd negative power consumption value after substracting EV consumption from Total consumption. I think its due to the interpolation going on in _get_hass_power_from_daily_kwh when it requantises to half hour slots. The Zappi power data is kWh to one decimal place, so when the car stops charging the entries can be hours apart as the consumption is either zero or a very low value.

I think _get_hass_power_from_daily_kwh interpolates beyond the last entry it downloads from HA but if it does this with the Zappi it will be wrong - the interpolation feature needs to be off for the Zappi and any times beyond the last entry from HA need to be equal to that last entry. Will either add a switch to the function call or just write a new function for EV chargers in general.

I'll also do an updated dashboard based on the clean version in the repo (this is mostly done) and write the release notes.

Your testing aside, I think it will be another week before I have a production ready version.

@fboundy
Copy link
Owner

fboundy commented Jul 5, 2024

I've made some of your logging on self.debug - probably worth you syncing your fork with my dev branch which also has the 3.15.4 patch revs

@stevebuk1
Copy link
Contributor

I've made some of your logging on self.debug - probably worth you syncing your fork with my dev branch which also has the 3.15.4 patch revs

I synced forks last night to include the 3.15.4 changes - no impacts as far as I can see.

I've since also added two commits to suppress any negative values in consumption which was the last bug I was working on. As I've recorded in stevebuk1#1, there's either something in the interpolation that doesn't work well with the typical consumption of an EV charger (mostly 7kW or nothing) or its an artefact of doing a power subtraction as opposed to an energy subtraction. With the negative suppression the error is a single 1/2 hour of background house load which for me is around the 150Wh mark so I think its an error that can ignored. If I find time I think permanent fix would be to quantitize to 1/2 hour slots using energy, substract EV from Total and then convert to power, but its low priority and to be honest, probably beyond my abilities to unpick that single line of .interpolate/.resample/.asfreq etc!

No problem on the logging, I was just leaving it place for your test runs to aid debug (self.opt being the useful one) and was going to remove it after your testing. Agree all of it should now be behind self.debug.

I've uploaded an update to the dashboard that includes extra views for IOG and Zappi stuff - these should all self hide if EV_charger is set to none.

Suggested release notes are as follows:

  • Enhanced functionality for the Intelligent Octopus Go tariff and Zappi EV chargers:
    • Prevention of house battery discharge during car charging, if required
    • Removal of EV consumption from the consumption history
    • Cheap charging slots allocated outside of the 6 hour off-peak period taken into account in the house battery charge/discharge plan
      - Updated dashboard to:
      - Display the car charging plan generated on car plugin
      - Display when house battery discharge is being prevented
      Indicated by Power = 1W, Hold SOC = "<=IOG", as below:

image

I think the README.md "Consumption Parameters" needs entries for: "EV Part of House Load", and the need for the MyEnergi HA integration should be added - is that best to with a PR?

@stevebuk1
Copy link
Contributor

stevebuk1 commented Jul 7, 2024

I think the README.md "Consumption Parameters" needs entries for: "EV Part of House Load", and the need for the MyEnergi HA integration should be added - is that best to with a PR?

@fboundy I've updated README.md via a patch to my Dev so it now appears as e581d25 as part of the main PR.

@mergwyn
Copy link
Author

mergwyn commented Jul 8, 2024

I've downloaded 3.16.0-Beta-6 this morning but seems to be getting stuck INFO: Getting yesterday's solar generation (07-Jul 00:00 UTC - 08-Jul 00:00 UTC). Not sure if this is an error at my end or not, any pointers greatly appreciated!

pv_opt.log
error.log

@stevebuk1
Copy link
Contributor

Hi @mergwyn, I had a quick look.

The log shows various HA entities going unavailable, then shows that attempted loads of consumption data from "sensor.solax_house_load" fail with "ERROR: No data returned from HASS entity sensor.solax_house_load"

Looking at your previous logs when it was all working, reading of power from sensor.solax_house_load was working ok.

I don't think any of this area (loading the total consumption from the Solax integration) has changed recently, either in the EV changes I've made or elsewhere.

Also, the restarts should show the following banner
******************* PV Opt v3.16.0-Beta-6 *******************

but they are all showing
******************* PV Opt v3.15.5 *******************

In looking at this I've discovered Pv_opt will attempt to use power sensors first and then try energy sensors ("house_load_today"), and my setup has only ever used energy sensors for some reason, but I don't think there is an untested element here as yours was working previously with the power sensors.

I think ensure you've got the files from the dev branch at https://github.com/stevebuk1/pv_opt/tree/dev/apps/pv_opt and then try an HA restart.

@dolce08
Copy link

dolce08 commented Jul 9, 2024

Hi @stevebuk1 thanks for your work on this. I'm running v3.16.0-Beta-6 since yesterday with no issues and a successful IOG charge overnight and one outside of the usual 23.30-05.30 window. In your pvopt_dashboard.yaml there are several script entities for changing the charge current. Are these necessary for the integration to function or to you use them for something else?

@mergwyn
Copy link
Author

mergwyn commented Jul 9, 2024

Thanks for your response. The versions I was using was from before my holiday (maybe June 21) and I don't think the version was set correctly in the code at that time.

I've re-download the code at 1337 today, and it looks like I am getting the right tag this time. After restarting HA, I started seeing ERROR: 1014: 'PVOpt' object has no attribute 'self' accompanied by 'PVOpt' object has no attribute 'contract_last_loaded' in the error log.

When I tried to load the information direct from octopus I got the following error Unable to load Octopus Account details using API Key: 'Contract' object has no attribute 'mpans'

Here are the logs:
pv_opt.01.log
pv_opt.log
error.log

@stevebuk1
Copy link
Contributor

Hi @stevebuk1 thanks for your work on this. I'm running v3.16.0-Beta-6 since yesterday with no issues and a successful IOG charge overnight and one outside of the usual 23.30-05.30 window. In your pvopt_dashboard.yaml there are several script entities for changing the charge current. Are these necessary for the integration to function or to you use them for something else?

Hi @dolce08, thats great, pleased its working well for you.

Re the script entries - are you refering to these?

image

If so, then no, these arent necessary for Pv_opt to function and aren't part of Pv_opt. These are effectively manual overrides to Pv_opt; when run they set Pv_opt to read only, set a fixed charge current and charge between 23:30 and 05:30. As I used these frequently during development of the EV stuff if it got to late at night and things weren't working, I'd placed these on the Pv_opt dashboard for convenience. I've now removed these from the production candidate dashboard, which is here:

https://github.com/stevebuk1/pv_opt/blob/dev/pvopt_control_card.yaml

Conversely, if you do want the scripts for your own use let me know and I'll upload them!

@stevebuk1
Copy link
Contributor

stevebuk1 commented Jul 9, 2024

After restarting HA, I started seeing ERROR: 1014: 'PVOpt' object has no attribute 'self' accompanied by 'PVOpt' object has no attribute 'contract_last_loaded' in the error log.

@mergwyn, this error is due to contract_last_loaded being uninitialised if the contract fails to load after 5 attempts. I've fixed this but I have a bad feeling that the problem is that the contract is not loaded, as follows:

Pv opt has 3 ways of loading the contract - it tries first via the Octopus Energy Integration, then via the website using your account ID and API, and then finally via manual tariffs in config.yaml. If it can't complete these then Pv_opt will stop with a fatal error.

The log file includes multiple entries for both the contract last loaded and the contract fatal error, but as they are all identically timestamped with 13:44:56 I can't see whats actually occuring first.

Try this code for pv_opt.py (remove the .txt extension) which has a fix for the initialized variable and post a log please.

pv_opt.py.txt

If it doesn't work I'll add some debugging in to trace the problem.

@mergwyn
Copy link
Author

mergwyn commented Jul 10, 2024

Thanks for the quick response - here are the logs:
pv_opt.log
error.log

@stevebuk1
Copy link
Contributor

@mergwyn I think I've found the cause of the problem - try Beta-7 at https://github.com/stevebuk1/pv_opt/tree/dev/apps/pv_opt

@mergwyn
Copy link
Author

mergwyn commented Jul 11, 2024

@stevebuk1 that looks like it has done to trick - many thanks. Installed and appdeamon restarted at 12:52:
pv_opt.log
error.log

@stevebuk1
Copy link
Contributor

stevebuk1 commented Jul 11, 2024

@stevebuk1 that looks like it has done to trick - many thanks.

@mergwyn Good news. It took me a while to figure out you've got debug text switched on, which is where the bug was. I don't tend to use it as I find it generates far too much to find anything, instead just adding debug to the code directly. I'd switch it off by setting Debug to False in config.yaml ( debug: false at line 23)

@fboundy, any objection if I add in a debug_level setting alongside the debug switch with value 1 to 5 for controlling the amount of logging? Level 5 would be the verbose setting with everything as is, with the other four levels with less and less. I'll probably run my system at Level 1 permanently to include the power flows and charging plans to catch the odd instability in charge current variation I see from time to time, and have Level 2 for the EV work, which I do intend to continue for the Agile tariff (but probably not start in the next few months).

@mergwyn
Copy link
Author

mergwyn commented Jul 17, 2024

@stevebuk1 On the face of it things are working, but I seem to have stopped force discharging since installing the latest version. Looking at the logs I see Ignoring export pricing because Use Export is turned off though include export is set to true (I think this info message has always said this). I then see lines similar to:

21:40:13     INFO: 43 slots have an export price greater than the min import price
21:40:14     INFO:
21:40:14     INFO: Discharge net cost delta: -0.0p: < Discharge threshold (5.0p) => Slots excluded
21:40:14     INFO: Iteration  2: Slots added:   0
21:40:14     INFO:
21:40:14     INFO: Removing cyclic charge/discharge
21:40:14     INFO:   Net cost revised from 3.8p to 3.8p

It seems weird that the cost delta is 0.

Here are the logs - I'll turn debug back on, let me know if you want me to upload those logs:
pv_opt.log
error.log

@stevebuk1
Copy link
Contributor

Here are the logs - I'll turn debug back on, let me know if you want me to upload those logs:

Yes please to logs with debug logging on. Also if you can upload config.yaml at the same time.
I agree about the use export message, my logs have it too. The logs from when your discharge was working don't include it so I suspect it is around this area.

@mergwyn
Copy link
Author

mergwyn commented Jul 18, 2024

@stevebuk1
Copy link
Contributor

stevebuk1 commented Jul 19, 2024

@mergwyn, thanks for the logs, they didn't contain any further detail around the area that calculates the cost delta.

I forced an export tariff into my setup and enabled force discharging and I think I have repeated the same behaviour you've got, in that there is no forced discharge due to a cost delta of 0. I'm also getting no Low cost charging plan. You did get one of these in your logs you posted when you first raised this but not in the logs with debug switched on.

I then reverted to the latest full production release (3.15.4 from https://github.com/fboundy/pv_opt), did the same tweak to force an export tariff and have found exactly the same behaviour.

I also tried 3.14.0 and then 3.13.2, exactly the same.
(3.13.2 predates the start of any EV integration work)

It might be worth repeating the same reversion testing to see if it makes a difference, otherwise its a case of opening a new issue.

I'm afraid I can't explain why there isn't any charging and discharging going on. Uncommenting some extra logging lines in the code illicts this, which I can't explain: (@fboundy there is a fix for the * being an hour out within the existing PR, its a UTC v GMT thing)

21:10:22     INFO: 44 slots have an export price greater than the min import price
21:10:22     INFO: 43 Max export price 15.00p/kWh at 19/07 20:00   SOC:  16.0%-> 16.0% Net: 164.8 
21:10:22     INFO: 42 Max export price 15.00p/kWh at 19/07 20:30   SOC:  16.0%-> 16.0% Net: 164.8 
21:10:23     INFO: 41 Max export price 15.00p/kWh at 19/07 21:00 * SOC:  16.0%-> 16.0% Net: 164.8 
21:10:23     INFO: 40 Max export price 15.00p/kWh at 19/07 21:30   SOC:  16.0%-> 16.0% Net: 164.8 
21:10:23     INFO: 39 Max export price 15.00p/kWh at 19/07 22:00   SOC:  16.0%-> 16.0% Net: 164.8 
21:10:23     INFO: 38 Max export price 15.00p/kWh at 20/07 04:30   SOC:  25.0%-> 24.0% Net: 171.8 
21:10:23     INFO: 37 Max export price 15.00p/kWh at 20/07 05:00   SOC:  24.0%-> 23.0% Net: 170.9 
21:10:23     INFO: 36 Max export price 15.00p/kWh at 20/07 05:30   SOC:  23.0%-> 22.1% Net: 170.2 
21:10:23     INFO: 35 Max export price 15.00p/kWh at 20/07 06:00   SOC:  22.1%-> 21.5% Net: 169.6 
21:10:23     INFO: 34 Max export price 15.00p/kWh at 20/07 06:30   SOC:  21.5%-> 21.6% Net: 169.7 
21:10:24     INFO: 33 Max export price 15.00p/kWh at 20/07 07:00   SOC:  21.6%-> 22.3% Net: 170.2 
21:10:24     INFO: 32 Max export price 15.00p/kWh at 20/07 07:30   SOC:  22.3%-> 23.5% Net: 171.1 
21:10:24     INFO: 31 Max export price 15.00p/kWh at 20/07 08:00   SOC:  23.5%-> 24.5% Net: 172.1 
21:10:24     INFO: 30 Max export price 15.00p/kWh at 20/07 08:30   SOC:  24.5%-> 26.4% Net: 173.5 
21:10:24     INFO: 29 Max export price 15.00p/kWh at 20/07 09:00   SOC:  26.4%-> 28.4% Net: 175.3 
21:10:24     INFO: 28 Max export price 15.00p/kWh at 20/07 09:30   SOC:  28.4%-> 31.3% Net: 177.7 
21:10:24     INFO: 27 Max export price 15.00p/kWh at 20/07 10:00   SOC:  31.3%-> 34.3% Net: 180.3 
21:10:24     INFO: 26 Max export price 15.00p/kWh at 20/07 10:30   SOC:  34.3%-> 37.2% Net: 181.0 
21:10:25     INFO: 25 Max export price 15.00p/kWh at 20/07 11:00   SOC:  37.2%-> 40.0% Net: 181.0 
21:10:25     INFO: 24 Max export price 15.00p/kWh at 20/07 11:30   SOC:  40.0%-> 43.2% Net: 181.2 
21:10:25     INFO: 23 Max export price 15.00p/kWh at 20/07 12:00   SOC:  43.2%-> 46.5% Net: 181.3 
21:10:25     INFO: 22 Max export price 15.00p/kWh at 20/07 12:30   SOC:  46.5%-> 49.7% Net: 181.3 
21:10:25     INFO: 21 Max export price 15.00p/kWh at 20/07 13:00   SOC:  49.7%-> 52.2% Net: 180.8 
21:10:25     INFO: 20 Max export price 15.00p/kWh at 20/07 13:30   SOC:  52.2%-> 54.6% Net: 180.7 
21:10:25     INFO: 19 Max export price 15.00p/kWh at 20/07 14:00   SOC:  54.6%-> 57.0% Net: 180.7 
21:10:25     INFO: 18 Max export price 15.00p/kWh at 20/07 14:30   SOC:  57.0%-> 59.4% Net: 180.7 
21:10:26     INFO: 17 Max export price 15.00p/kWh at 20/07 15:00   SOC:  59.4%-> 60.7% Net: 180.0 
21:10:26     INFO: 16 Max export price 15.00p/kWh at 20/07 15:30   SOC:  60.7%-> 60.4% Net: 178.9 
21:10:26     INFO: 15 Max export price 15.00p/kWh at 20/07 16:00   SOC:  60.4%-> 58.0% Net: 177.0 
21:10:26     INFO: 14 Max export price 15.00p/kWh at 20/07 16:30   SOC:  58.0%-> 56.3% Net: 177.6 
21:10:26     INFO: 13 Max export price 15.00p/kWh at 20/07 17:00   SOC:  56.3%-> 55.1% Net: 178.1 
21:10:26     INFO: 12 Max export price 15.00p/kWh at 20/07 17:30   SOC:  55.1%-> 53.9% Net: 178.1 
21:10:26     INFO: 11 Max export price 15.00p/kWh at 20/07 18:00   SOC:  53.9%-> 52.7% Net: 178.1 
21:10:26     INFO: 10 Max export price 15.00p/kWh at 20/07 18:30   SOC:  52.7%-> 52.4% Net: 178.8 
21:10:27     INFO:  9 Max export price 15.00p/kWh at 20/07 19:00   SOC:  52.4%-> 51.2% Net: 178.1 
21:10:27     INFO:  8 Max export price 15.00p/kWh at 20/07 19:30   SOC:  51.2%-> 44.0% Net: 172.8 
21:10:27     INFO:  7 Max export price 15.00p/kWh at 20/07 20:00   SOC:  44.0%-> 36.7% Net: 172.7 
21:10:27     INFO:  6 Max export price 15.00p/kWh at 20/07 20:30   SOC:  36.7%-> 30.7% Net: 173.9 
21:10:27     INFO:  5 Max export price 15.00p/kWh at 20/07 21:00   SOC:  30.7%-> 25.6% Net: 173.2 
21:10:27     INFO:  4 Max export price 15.00p/kWh at 20/07 21:30   SOC:  25.6%-> 18.1% Net: 166.6 
21:10:27     INFO:  3 Max export price 15.00p/kWh at 20/07 22:00   SOC:  18.1%-> 16.0% Net: 164.8 
21:10:27     INFO:  2 Max export price 15.00p/kWh at 20/07 22:30   SOC:  16.0%-> 16.0% Net: 164.8 
21:10:28     INFO:  1 Max export price 15.00p/kWh at 20/07 23:00   SOC:  16.0%-> 16.0% Net: 164.8 
21:10:28     INFO:  0 Max export price 15.00p/kWh at 20/07 23:30   SOC:  16.0%-> 16.0% Net: 164.8 
21:10:28     INFO: 
21:10:28     INFO: Discharge net cost delta: -0.0p: > Discharge Threshold (0.0p) => Slots included
21:10:28     INFO: Iteration  1: Slots added:   0

@stevebuk1
Copy link
Contributor

stevebuk1 commented Jul 30, 2024

Hi @mergwyn,I'm about to start the EV support under the Agile tariff. I've released Beta-8 which allows me to turn off Octopus Auto and set manual tariffs so I can progress the development.

I do however have a question though, mainly for my own benefit - I remember you saying that you tend to use IOG in spring/summer and then swap back to Agile for late Autumn/Winter.

On IOG, I can pretty much average 7p per kWh as I have sufficient battery capacity to power the house for all consumption. Looking at months of Agile, I cannot see how its possible to approach that 7p, even the regular cheapest Agile slots seem to exceed that. Granted I have seen negative pricing and plunge events, but I assume you think it is is possible to achieve 7p or less using the Agile tariff over winter. Am I correct in my assumptions? Whilst I don't have export, the price for export on IOG and Agile is the same (15p)?

@mergwyn
Copy link
Author

mergwyn commented Jul 31, 2024

@stevebuk1 Unfortunately I don't have the battery capacity to bridge a whole day. We have an ASHP which in the depths of winter can consume over 800 kWh in a month on its own (vs less than 50 kWh in the lowest summer month)! My 10kWh batteries are usually enough to bridge the 3 hour agile peak periods in the morning and afternoon.

It doesn't like you have had chance to add the fix for discharge issue in Beta-8 as per your email:

Ok I think I know what the issue is here. Its either been caused by the price change on IOG that happened the 1st of July or the change of numpy library we all had to force to a particular version, or both combined.

I think prior to the 1st of July the IOG cheap rate price was a nice round number, now it is to something like four decimal places.

Both the low cost charging and the discharging algorithm search for the cheapest slots by looking at all of them, finding the minimum, matching that slot, utilising it then removing it from the list. (Whilst IOG slots are all exactly the same price the theory is still the same). The problem is that the calculation to find the minimum isn't reporting the actual minimum, so when the matching part runs - it finds none of them, hence no forced charging/discharging.

Is that still something you are thinking of taking a look at?

@stevebuk1
Copy link
Contributor

stevebuk1 commented Jul 31, 2024

Unfortunately I don't have the battery capacity to bridge a whole day. We have an ASHP which in the depths of winter can consume over 800 kWh in a month on its own (vs less than 50 kWh in the lowest summer month)! My 10kWh batteries are usually enough to bridge the 3 hour agile peak periods in the morning and afternoon.

Thanks for your answer - makes perfect sense. I'm planning an ASHP, but aware that for the cold months I'm averaging around 110kWh a day on gas, so even with better insulation (WIP) and a decent SCOP I'll have the same problem (I have 12kWh of batteries).

@stevebuk1
Copy link
Contributor

Ok I think I know what the issue is here. Its either been caused by the price change on IOG that happened the 1st of July or the change of numpy library we all had to force to a particular version, or both combined.
I think prior to the 1st of July the IOG cheap rate price was a nice round number, now it is to something like four decimal places.
Both the low cost charging and the discharging algorithm search for the cheapest slots by looking at all of them, finding the minimum, matching that slot, utilising it then removing it from the list. (Whilst IOG slots are all exactly the same price the theory is still the same). The problem is that the calculation to find the minimum isn't reporting the actual minimum, so when the matching part runs - it finds none of them, hence no forced charging/discharging.

Is that still something you are thinking of taking a look at?

Ok, the reason you'll no longer find that post in the thread when viewed on the Github website is because I deleted it. Basically I'd written nonsense: although the figures reported by my debug log were very slightly different, it didnt actually affect the if/then/else statement that used them.

I'd gone on a bit further after that and I'm still not sure I understand what "Low cost charging" actually does. I think it adds a single slot of full rate charging and then sees if it saves you money, but this cost saving is calculated prior to running the forced discharge part of the algorithm. Therefore, the only way it can work is making the assumption that any spare charge added is over and above needs and can be exported. It should always be over and above needs, as the "High cost swaps" that runs prior to that deals with all your actual needs, i.e ensures the battery finishes the day at just above max DOD. So, then doing a full rate charge in the cheapest slot should mean that excess can be exported, thus netting a cost saving. When I ran it I found the same as you. It may be because the cheapest slot for IOG is all 12 overnight slots (as they are all the same price), but as it appears to do one at a time I don't think its that. Its a case of setting up some detailed logging of the power flows to get a better understanding of whats actually happenning, to ensure the algorithm that was developed for the Agile tariff works equally as well for a dual rate tariff. I feel it should, but the devil is somewhere in the detail.....

@stevebuk1
Copy link
Contributor

stevebuk1 commented Jul 31, 2024

@mergwyn to carry on from the previous post I found this write-up on what the algorithm does from Francis in the Solax development thread.

#188 (comment)

"Low Cost Charging" is point 4, and should result in a cost saving because of "Max Export". As stated in #121 (comment), the issue we are both finding in Beta-7 and that I'm finding in all historic versions that I've tried is that Net Cost Delta is zero. Did you try out 3.14.0 and then 3.13.2?

@mergwyn
Copy link
Author

mergwyn commented Aug 2, 2024

@stevebuk1 I could not get either 3.14.0 or 3.13.2 to work for me - both got stuck at "Checking tariff prices vs Octopus Energy Integration" so I gave up on both. I did try 13.15.4 which did work but did not produce any discharge slots (I can supply logs if it helps) At the moment my export rate is 15p and overnight import is 7p so I would have expected discharge slots.

@stevebuk1
Copy link
Contributor

stevebuk1 commented Aug 5, 2024

Regarding Discharge slots, I think I've found the source of the problem , it was introduced in 3.15.4 (3.15.3 is ok) which I keep the Dev branch up to date with. I've raised this as a separate issue (#254) so it can get fixed on the production release. I'll do another Beta release with this code fix and additional logging, and post again when its done.

Regarding low cost charging, I think I now understand how this works.

As mentioned in #188 (comment) , the algorithm runs in 3 parts:

  1. High Cost usage swaps. Basically adds charging to cover your usage

  2. Low cost charging. Looks for any slots where the import price is < export price and see if there is a reduction in net cost by charging during these Low Cost Charging slots.

  3. Force Discharging. Do the same as low cost charging but for discharging.

On a time of use tariff , 1) will basically aim to leave your battery at max DOD at 11.30pm.

The key part I've noticed is that 2) runs on each slot at a time, by simulating a maximum rate charge for 1/2 hour and see if that saves you any money. The only way this can reduce cost prior to 3) being run is that the battery reaches 100% so there is then an export that will go to grid. If the available solar in that day doesn't get your battery anywhere near 100% then adding a full charge slot isnt going to get the battery to 100%, so no export and no cost saving. All that happens for each simulation is that you end the day with a battery that it is now above max DOD. If each slot simulation in 2) doesn't save you any money then no charge is actually added, so when 3) runs the battery is sitting at max DOD at 11.30pm, so there isnt anything additional to discharge in any slot (any discharge means you'll be importing at peak rate later on).

If the algorithm instead tried one slot of low rate charging and then ran through all of the slots of discharging to see if that saved you any money, then it would find a battery that had spare charge in it at 11pm and discharge it. Both of these slots would then be locked in, so the next low rate charging slot added would give more energy in the battery that could be discharged in another discharge slot, and so on and so forth. I guess doing that might be theoretically possible, but wasn't chosen when Pv Opt was originally written and at first glance, would be an architectural rewrite from scratch.

Francis may well be able to chime in and explain more? I'm aware that he always thought that Predbat was not all that optimum for Agile, and I know Trefor (the author of Predbat) is on IOG, so its certainly possible we may well be looking at the fact that the optimal algorithm for these tariffs differs, i.e an Agile based algorithm only considers everything a slot at a time but a time of use tariff would be better considering the 6 hour cheap rate period as a full block.

@stevebuk1
Copy link
Contributor

I'll do another Beta release with this code fix and additional logging, and post again when its done

3.16.0-Beta-9 released at:
https://github.com/stevebuk1/pv_opt/tree/3.16.0-Beta-9

Copy all of pv_opt.py, pvpy.py and solis.py.

In your config.yaml, add the following lines:

  #=======================================
  #Logging Category Control
  #=======================================
  #Defines Logging subjects to add to logfile
  #Ignored if "debug" is set to False above
  #
  # S = Startup/Initialisation Logging
  # T = Tariff loading Logging
  # P = Power consumption history Logging
  # C = Charge algorithm Logging
  # D = Discharge algorithm Logging
  # W = Charge/Discharge Windows Logging
  # F = Power Flows Logging
  # V = Power Flows debugging (extra verbose)
  # I = inverter control/commands Logging
  # E = EV debugging

  # Letters can be added to "debug_categories" in any order

  debug_categories: CDFW

And when you see charging/discharging that doesn't look right (late evening is best), change
debug: False
to
debug: True
(or add "debug: True" if not present in config.yaml

and post the log from a single optimser run.

@mergwyn
Copy link
Author

mergwyn commented Aug 6, 2024

@stevebuk1 Thanks so much for tracking this down and supplying a fix. I've installed Beta-9 this morning and discharge slots have returned! However, as you suggest above, the timing of the slots doesn't look right: I have discharge slots at 16:30 when it makes more sense to discharge immediately before the cheap rate at 23:30.

Also, given there is good solar today and the battery looks like it will be at 100% from around 10:30 this morning, there are opportunities to discharge and then recharge from solar at at 15p per kWh nominal gain.

Here are the logs for an optimiser run starting at 10:213:23 with debug on:
pv_opt.06.log
pv_opt.05.log
pv_opt.04.log
pv_opt.03.log
pv_opt.02.log
pv_opt.01.log
pv_opt.log
error.log

@mergwyn
Copy link
Author

mergwyn commented Aug 6, 2024

@stevebuk1

If the algorithm instead tried one slot of low rate charging and then ran through all of the slots of discharging to see if that saved you any money, then it would find a battery that had spare charge in it at 11pm and discharge it. Both of these slots would then be locked in, so the next low rate charging slot added would give more energy in the battery that could be discharged in another discharge slot, and so on and so forth. I guess doing that might be theoretically possible, but wasn't chosen when Pv Opt was originally written and at first glance, would be an architectural rewrite from scratch.

I don't pretend to fully understand the subtleties of what you are saying, but I think my expectation is roughly what you describe. At a high level the goal is the same irrespective of Agile or IOG, ie to use cheap rate slots to charge the battery such that battery + solar can bridge higher rate slots. As I think you explain above the devil is in the detail!

Taking the simple case of IOG, I would hope that the algorithm can can determine whether to force discharge before the cheap rate slot, as well as not charge during the cheap rate slot if there is enough battery capacity to meet the forecast demand until solar generation can take over and start filling the battery.

Whilst I don't claim to understand Predbat (and find it hugely complicated), it does produce differer very different results for the next day or so:
Screenshot 2024-08-06 at 17 32 27

Screenshot 2024-08-06 at 17 31 28

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request long term Good idea but not a priority
Projects
None yet
Development

No branches or pull requests

5 participants