CheapPower module

[dr-m]

Description:

This fetches electricity prices and chooses the cheapest future time slot. Currently, the only data sources are the Nordpool prices, as provided by ENTSO-E and https://sahkotin.fi (FI) and https://mgrey.se/espot (price zones SE1 through SE4). To use:

• copy cheap_power.tapp to the file system
• Invoke the Tasmota command CheapPower1 FI, CheapPower2 SE2, … to
• download prices for the next 24 to 48 hours
• automatically choose the cheapest future time slot
• to schedule Power1 ON, Power2 ON, … at the chosen slot
• to install a Web UI in the main menu
• For a full installation, you will want something like the following:

Backlog0 Timezone 99; TimeStd 0,0,10,1,4,120; TimeDst 0,0,3,1,3,180

Backlog0 SwitchMode1 15; SwitchTopic1 0
Backlog0 WebButton1 boiler; WebButton2 heat
PulseTime1 3700

Rule1 ON Clock#Timer DO CheapPower1 ENDON
Timer {"Enable":1,"Mode":0,"Time":"18:00","Window":0,"Days":"1111111","Repeat":1,"Output":1,"Action":3}
Rule1 1
Timers 1

The download schedule can be adjusted in the timer configuration menu. The prices for the next day will typically be updated in the afternoon or evening of the previous day.

For the SE data source, prices are currently fetched only for one day (the current day) at a time and they are assumed to be in the local time zone.

In case the prices cannot be downloaded, the download will be retried in 1, 2, 4, 8, 16, 32, 64, 64, 64, … minutes until it succeeds.

The user interface in the main menu consists of 4 buttons: ⏮ moves to the previous time slot (or wraps from the first to the last) ⏯ pauses (switches off) or chooses the optimal slot 🔄 requests the prices to be downloaded and the optimal slot to be chosen ⏭ moves to the next time slot (or wraps from the last to the first)

The status output above the buttons may also indicate that the output is paused until further command or price update:
⭘
It may also indicate the start time and the price of the slot:
⭙ 2024-11-22 21:00 12.8 ¢

I am using this for controlling a 3×2kW warm water boiler. For my usage, 1 hour every 24 or 48 hours is sufficient.

Checklist:

• The pull request is done against the latest development branch
• Only relevant files were touched
• Only one feature/fix was added per PR and the code change compiles without warnings
• The code change is tested and works with Tasmota core ESP8266 V.2.7.8
• The code change is tested and works with Tasmota core ESP32 V.3.1.0.241117
• I accept the CLA.

[s-hadinger]

Thanks. Is it valid only for Finland? If so, I believe it should show in the name of the file like "cheap_power_Finland"

[dr-m]

Thanks. Is it valid only for Finland? If so, I believe it should show in the name of the file like "cheap_power_Finland"

The current version is useful only for Finland, but it could be fairly easily extended to cover at least the entire Nord pool area. The data is produced by ENTSO-E, which based on https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/show seems to cover most of Europe, including Türkiye and Ukraine. Some of these countries might not implement dynamic pricing for end consumers yet, but I believe that it will come.

Ideally, someone would run a public service that is based on some code like https://github.com/oysteinjakobsen/fetch-day-ahead-price or https://github.com/JaccoR/hass-entso-e so that end users can save themselves the trouble of registering and configuring an API key with ENTSO-E. Besides, this API could be too resource intensive to implement in Tasmota.

The hypothetical service would deliver the known prices starting from the currently active slot in a uniform format. The command would take a parameter to the command to specify the price area (such as an area of Norway):

CheapPower2 NO4

This could be translated into a simple URL like http://e-prices.example.com/no4 or http://no4.e-prices.example.com if such a service existed, or it could use country specific services (see below). I’d prefer HTTP instead of HTTPS, because SSL and TLS are constantly evolving, which could require frequent firmware updates.

The second parameter could also be a full URL, for example pointing to server in the LAN, which could run an ENTSO-E interface and cache to serve multiple devices. This would in no means be limited to ENTSO-E or Europe.

Another thinkable enhancement would be a third parameter to specify the desired number of slots to choose per day, like this:

CheapPower1 DE 5

Some heating could need to run for multiple hours per day. There are some plans to narrow the price slots from 60 to 15 minutes in the future. In that case, even my deployment would require 2 to 4 such slots per day.

A quick search turned up some further open JSON data sources:

• Sweden: https://mgrey.se/espot/api
• Norway: https://www.hvakosterstrommen.no/strompris-api
• Denmark: https://www.elprisenligenu.dk/elpris-api

These could be implemented fairly easily, after asking the operators if this kind of automated access is okay with them. The format of the URL and the data would likely vary between any area-specific JSON data sources. I realize that to reduce the memory footprint, it could make sense to split this interface into separate modules that would be loaded on demand.

For Estonia, I only found https://elektrihind.ee/borsihind/ which does not seem to include any public JSON based interface. The German Fraunhofer-Institut für Solare Energiesysteme is running https://energy-charts.info/charts/price_spot_market/chart.htm with a nice country selection, but apparently without any raw data interface that is suitable for this kind of use. There is a CSV export function that seems to spit out data for the current week.

I think that this needs to start somewhere. If it helps, I can make the country parameter mandatory in the first version, and reject anything else than FI. Possibly I could refactor the parsing and implement dynamically loaded parser for two data sources, one of them being the FI data source.

[dr-m]

I was in contact with the provider of the Swedish price data, and now there is a simpler URL https://mgrey.se/espot?format=json&domain=SE1&date=2024-12-13 that will return the price for a single zone for the given day. Unfortunately, when I tried accessing this HTTPS server in the Berry console of Tasmota 14.1.0, I got an error, I suppose due to some TLS or SSL incompatibility. It is not possible to enable plain HTTP support on this server.

I could make the URL pattern configurable, so that this service could be reached via (say) http://router.lan/espot which would be a proxy for the external HTTPS server, for example by nginx reverse proxy. In that way, it could be claimed that this feature is not specific to a particular country.

[s-hadinger]

Hmmm. I probed mgrey.se:443 with openssl and they support only the following ciphers:

Testing ECDHE-ECDSA-AES256-GCM-SHA384... YES
Testing ECDHE-ECDSA-CHACHA20-POLY1305... YES
Testing ECDHE-ECDSA-AES128-GCM-SHA256... YES

I will evaluate the impact of supporting ECDHE-ECDSA-AES128-GCM-SHA256 in addition to ECDHE-RSA-AES128-GCM-SHA256

[s-hadinger]

Please try with the latest version which includes #22649

I have now enabled ECDSA:

wc = webclient()
print(wc.begin('https://mgrey.se/espot?format=json&domain=SE1&date=2024-12-13'))
print(wc.GET())
print(wc.get_string())
print(wc.close())

Output shows:

<instance: webclient()>
200
{"date":"2024-12-13","SE1":[{"hour":0,"price_eur":1.35,"price_sek":15.54,"kmeans":0},{"hour":1,"price_eur":1.35,"price_sek":15.54,"kmeans":0},{"hour":2,"price_eur":1.35,"price_sek":15.52,"kmeans":0},{"hour":3,"price_eur":1.36,"price_sek":15.6,"kmeans":0},{"hour":4,"price_eur":1.4,"price_sek":16.07,"kmeans":0},{"hour":5,"price_eur":1.38,"price_sek":15.87,"kmeans":0},{"hour":6,"price_eur":1.31,"price_sek":15.13,"kmeans":0},{"hour":7,"price_eur":1.38,"price_sek":15.87,"kmeans":0},{"hour":8,"price_eur":1.37,"price_sek":15.8,"kmeans":0},{"hour":9,"price_eur":1.5,"price_sek":17.21,"kmeans":1},{"hour":10,"price_eur":1.37,"price_sek":15.76,"kmeans":0},{"hour":11,"price_eur":1.43,"price_sek":16.43,"
nil

[dr-m]

Thank you a lot, @s-hadinger! I upgraded to a development snapshot (b3b9699 is 1 commit ahead of the merge 615c676 of #22649):

Version 14.4.0.1(b3b9699-tasmota32)-3_1_0(2024-12-14T23:37:22)

I am glad to see that also sahkotin.fi now is accessible with https. My upcoming update will revise that URL as well.

[sfromis]

FTR, most example projects with Tasmota+Berry are published on Github repositories owned by the creator. I've collected a list of repositories including Berry code:
http://sfromis.strangled.net/tasmota/berry/github-repositories

[dr-m]

That URL is timing out for me. Yes, I was uncertain if this would be the appropriate place. I was expecting to a link to a package directory at https://tasmota.github.io/docs/Tasmota-Application/.

I spent quite a bit of time debugging today, trying to figure out what I am doing wrong when implementing support for a second data source. It turns out that the problem is directly caused by switching http to https. 555dc04 is a simple rebase of the original implementation, only replacing the .bec file inside the .tapp file with straight .be. That works fine.

As soon as I change the code to use https instead of http (adding just one s) and try to execute CheapPower FI, there will be no response to the Tasmota console; it seemingly hangs. Much of the time, after a few seconds, the device will recover, and the console will show some output indicating that it was reset. Twice, I had to reset it by pressing the button on the device. I’m running a tasmota32.bin of Tasmota 14.4.0.1 (b3b9699-tasmota32).

Unfortunately, there is no serial console connection to my only Tasmota equipped device, so I’m afraid I am unable to debug this deeper. When I was experimenting with the Berry console, it seemed that

      var data = json.load(wc.get_string())
      wc.close()

could run out of memory (end up with data=nil) while

      var data = wc.get_string()
      wc.close()
      data = json.load(data)

might allow the Berry garbage collector to free some memory earlier. However, when I tried to revise the program like this, it would still not work (cause the device to be reset or to lose the WLAN connection).

I would appreciate it if you could check if the anomaly is reproducible for you.

[s-hadinger]

With the URL above https://mgrey.se/espot?format=json&domain=SE1&date=2024-12-13, the payload is 1404 bytes.

Once loaded the JSON takes 3.5KB which is not huge but still significant

[sfromis]

Well, of course using https will have to use more memory than unencrypted http, but with a string size of "only" 1395 bytes, that should not be a pain point.

I had no trouble getting past the point of fetching the data, using:
import cheap_power cheap_power.update()
Unsurprisingly, it failed later (when not having a channel where to turn on power), but I suppose that the test got "far enough"? I added a couple of print to see the used URL, and size of data retrieved.

My test was using a recent build of tasmota 14.4.1.1 (a bit newer than yours), on an ESP32-S3 with PSRAM available. Switching to an ESP32-C3 with less RAM (and no PSRAM) made no difference, it still had no trouble https-fetching the data.

While deferring the json.load till after closing the connection would in principle allow a bit earlier garbage collection, I'd not expect the still open webclient instance to take up a lot of RAM, and with the moderate amount of data, I'd not even expect RAM to be "the issue", as confirmed by it also working on an ESP32-C3 with lower resources.

Of course, not knowing or replicating your test case, I can't be sure if my test got far enough for whatever issue you had.

In general, I like to reduce scope of test cases to "zoom in". As long as I can prune to code to be shorter I know that errors still occurring has to be within what's left.

[dr-m]

I had tried to add several print statements as well, but the problem was that there was no output in the Tasmota console of the web UI after I executed the command CheapPower (or CheapPower FI in the version that I did not publish yet), until the eventual restart messages, where the first few messages had a timestamp close to 00:00 and the last ones with the current time. The hang or crash is reproducible just by replacing the http: with https: in the published code, installing the .tapp file, and executing CheapPower from the Tasmota console right after restart. With http: this code has been stable, running for several weeks.

I will try to narrow down this "test case" for reproducing the anomaly. That will take a few days, though. I think I should use an https URL that is expected to return a constant dataset, instead of something that depends on the time of the day.

[barbudor]

Print goes nowhere
Either use Arduinos's 'Serial.print' to send to serial
Or tasmota's 'AddLog' for the webconsole

[sfromis]

Of course, Serial.print or AddLog does not apply to coding in Berry, where you instead have print() or log(). As a crash-proof technique, you could also write it to a file opened for append, and flush() the buffer after writes.

If the failure is first time in a code path, you could also create a runtime error at a certain point in the code, if you get that, you know that it was safe that far.

If the failure is not first time in the code path, it might be a memory leak, which you can check by following free memory. If it keeps depleting, something is rotten.....

[dr-m]

I think that this anomaly will occur on the first time after restart or reboot. Which Berry code would you recommend for injecting a runtime error?

[sfromis]

The Berry language does allow such:
raise 'crashing'
Of course, also trivial runtime errors like division by zero
print(1/0)

[barbudor]

Of course, Serial.print or AddLog does not apply to coding in Berry, where you instead have print() or log(). As a crash-proof technique, you could also write it to a file opened for append, and flush() the buffer after writes.

Sorry
I'm too tired, too much work

[dr-m]

I debugged this a little. I removed the .tapp and input the code with the following modification to the Berry Console:

--- tasmota/berry/modules/cheap_power/cheap_power.be	2024-12-15 21:00:30.860176636 +0200
+++ tasmota/berry/modules/cheap_power/cp.be	2024-12-24 09:25:48.127720826 +0200
@@ -1,9 +1,6 @@
 import webserver
 import json
 
-var cheap_power = module("cheap_power")
-
-cheap_power.init = def (m)
 class CheapPower
   var prices # future prices for up to 48 hours
   var times  # start times of the prices
@@ -20,7 +17,7 @@
     "<td style='width:25%'><button onclick='la(\"&op=2\");'>🔄</button></td>"
     "<td style='width:25%'><button onclick='la(\"&op=3\");'>⏭</button></td>"
     "</tr></table>"
-  static var URL0 = 'http://sahkotin.fi/prices?start=', URL1 = '&end='
+  static var URL0 = 'https://sahkotin.fi/prices?start=', URL1 = '&end='
   static var URLTIME = '%Y-%m-%dT%H:00:00.000Z'
 
   def init()
@@ -29,15 +26,9 @@
   end
 
   def start(idx)
-    if idx == nil || idx < 1 || idx > tasmota.global.devices_present
-      tasmota.log(f"CheapPower{idx} is not a valid Power output")
-      tasmota.resp_cmnd_failed()
-    else
       self.channel = idx - 1
       tasmota.add_driver(self)
       self.update()
-      tasmota.resp_cmnd_done()
-    end
   end
 
   def power(on) tasmota.set_power(self.channel, on) end
@@ -45,9 +36,8 @@
   # fetch the prices for the next 24 to 48 hours
   def update()
     var wc = webclient()
-    var rtc = tasmota.rtc()
-    self.tz = rtc['timezone'] * 60
-    var now = rtc['utc']
+    self.tz = 120 * 60
+    var now = 1734822000
     var url = self.URL0 +
       tasmota.strftime(self.URLTIME, now) + self.URL1 +
       tasmota.strftime(self.URLTIME, now + 172800)
@@ -149,6 +139,6 @@
     tasmota.web_send_decimal(status)
   end
 end
-return CheapPower()
-end
-return cheap_power
+var cheap_power = CheapPower()
+cheap_power.start(1)
+print(cheap_power.chosen)

Right after rebooting the Shelly Pro 2 into Tasmota 14.4.0.1 (b3b9699-tasmota32), when I input the code with the above modification to the Berry Console, the print statement will display 2. (This should be deterministic, because I hard-coded the URL by hard-coding the var now to a past date.)

To my surprise, the code would run just fine, even though I’m now using https. I tried again without the penultimate (3rd) hunk, that is, letting it use the current time, and it still seems to work, with size(cheap_power.prices) being 16 and cheap_power.chosen being 15 at the moment. The source of the instability would seem to be the tasmota.add_cmd() or the tasmota.resp_cmnd_done().

[sfromis]

AFAICT, the "culprit" was the call hierarchy of

command handler
cheap_power.start
self.update
self.schedule_chosen
self.power
tasmota.set_power
Power command

The thing is that issuing commands can be quite problematic when being invoked from an existing Tasmota core-related callback (especially a command callback), and tasmota.set_power will internally be issuing a Power command (as visible in the console).

In many cases, you can use a workaround of breaking the command hierarchy, using something like tasmota.add_timer(0, self.update), deferring the function call to right after returning from the current call hierarchy.

[dr-m]

@sfromis Thank you for the advice; I will try that. I was thinking of tasmota.set_timer(); maybe you meant that by tasmota.add_timer().

For the record, f7fc732 is my current development, implementing an interface for Swedish prices. With http://sahkotin.fi it works, but with https it will cause the system to hang as soon as CheapPower1 FI is invoked. The code worked fine with https when I used a similar patch that I had posted above (not installing any Tasmota command hooks). I tried to make the test conditions as deterministic as possible, by restarting the system after updating the file using the file manager, and right after restart, invoke CheapPower1 FI from the console.

Edit: Yes, it was this simple. Great!

diff --git a/tasmota/berry/modules/cheap_power/cheap_power.be b/tasmota/berry/modules/cheap_power/cheap_power.be
index 238fa733a..42b4c531d 100644
--- a/tasmota/berry/modules/cheap_power/cheap_power.be
+++ b/tasmota/berry/modules/cheap_power/cheap_power.be
@@ -42,7 +42,7 @@ class CheapPower
     if !payload
       tasmota.log(f"CheapPower{idx}: a price zone name is expected")
     elif payload == 'FI'
-      self.p_url = 'http://sahkotin.fi/prices?start='
+      self.p_url = 'https://sahkotin.fi/prices?start='
       self.p_kWh = '¢'
     elif re.match('^SE[1-4]$', payload)
       self.p_url = 'https://mgrey.se/espot?format=json&domain=' + payload +
@@ -58,7 +58,7 @@ class CheapPower
     self.channel = idx - 1
     self.p_zone = payload
     tasmota.add_driver(self)
-    self.update()
+    tasmota.set_timer(0, /->self.update())
     tasmota.resp_cmnd_done()
   end