Feature: Added "Tibber" Power Meter to support Tibber Pulse (via Tibber Bridge)

[marvincarstensen]

I've extended the "HTTP(S) + JSON" Power Meter (#153) to support the Tibber Pulse. The current power drawn from the grid is fetched using the local web server from the Tibber Bridge. No external requests through the internet (Tibber API).

Instructions on how to enable the web server permanently are also included in German and English:

Thank you to gvzdus and the discussion #123!

Pre-compiled binaries to directly test the feature:
opendtu-onbattery-generic_esp32.bin.zip
opendtu-onbattery-generic_esp32.factory.bin.zip

[schlimmchen]

So, the Tibber Pulse is a fancy Hichi?

Thanks for this pull-request! The part which reads the Tibber data stream and pushes it through the SML handler is very valuable.

However, I do not approve the current implementation.

1. The info text to setup the connection/bridge is actually nice, but it needs to go into the docs, not into the web application.
2. This power meter works significantly different than the HTTPS+JSON power meter implementation. You must no hijack this implementation and add a boolean switch to castrate it in multiple spots. Especially the change in tryGetFloatValueForPhase() is not acceptable. This will be a nightmare to maintain.

ToDo:

1. This must become a separate PoweMeter implementation. A new C++ class, say TibberPowerMeterClass.
2. Common parts, e.g., the basic authentication stuff, needs to go into a common base class (let's say HttpPowerMeterBase) of the new TibberPowerMeterClass and the existing HttpPowerMeterClass to avoid code duplication.
3. This must become a new type of PowerMeter selectable in the UI as such. It shall not hijack the HTTPS+JSON power meter.

In my opinion, 3. is absolutely mandatory. If you manage to implement 3., I will merge this despite my strong objections. I can address those when I tackle the PowerMeter refactoring. Add a new POWERMETER_TIBBER_CONFIG_T in the Configuration, which has only the needed struct members. Introduce a new power meter type value (config.PowerMeter.Source), both in the firmware and UI. Copy'n'paste the needed UI elements, with adjusted IDs, and do not re-use the HTTPS+JSON inputs.

Edit: Remove all files in webapp_dist from the changeset (do not delete them from the repo, just make sure your commits do not include changes to those files).

[schlimmchen]

@marvincarstensen Share your struggle with compiling, maybe I can help you easily. Are you compiling for the generic or generic_esp32 environment? Please note #950.

[marvincarstensen]

@schlimmchen I'm actually able to compile the firmware. It looks like a boot-loop with the following log:

18:11:34.916 > rst:0x3 (SW_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
18:11:34.922 > configsip: 0, SPIWP:0xee
18:11:34.924 > clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
18:11:34.930 > mode:DIO, clock div:2
18:11:34.933 > load:0x3fff0030,len:1184
18:11:34.935 > load:0x40078000,len:13232
18:11:34.935 > load:0x40080400,len:3028
18:11:34.938 > entry 0x400805e4
18:11:35.674 > ets Jul 29 2019 12:21:46

I stripped down my code back to the development branch, but I'm still getting the boot-loop. As soon as I switch back to the "development"-branch on the same workspace and recompile, it works fine again.

[schlimmchen]

Yes, that's #950. Please work on an earlier version of the development branch, or wait a couple of hours when I hopefully have freed some flash such that non-S3 builds work again (building and booting).

[marvincarstensen]

Thank you for your feedback and hint while debugging @schlimmchen!

So, the Tibber Pulse is a fancy Hichi?

Yes, but "Tibber" is actually a power company which fetches the power consumption in realtime from the power meter and charges only the current market prices. I just tried to "repurpose" the Tibber pulse because it already blocks the power meter.

The info text to setup the connection/bridge is actually nice, but it needs to go into the docs, not into the web application.

The info text got moved to the wiki: https://github.com/helgeerbe/OpenDTU-OnBattery/wiki/Power-Meter

This power meter works significantly different than the HTTPS+JSON power meter implementation. You must no hijack this implementation and add a boolean switch to castrate it in multiple spots. Especially the change in tryGetFloatValueForPhase() is not acceptable. This will be a nightmare to maintain.

I'm totally fine with separating the Tibber into another PowerMeter type. I just didn't want to create redundant code, as the Tibber technically also is an "HTTP" Power Meter.

This must become a separate PoweMeter implementation. A new C++ class, say TibberPowerMeterClass.

Done.

Common parts, e.g., the basic authentication stuff, needs to go into a common base class (let's say HttpPowerMeterBase) of the new TibberPowerMeterClass and the existing HttpPowerMeterClass to avoid code duplication.

I duplicated the HttpPowerMeterClass and removed all unnecessary function, constants, etc. and added the modifications for Tibber. But I did not create a base class, but feel free to do that while refactoring the PowerMeter.

This must become a new type of PowerMeter selectable in the UI as such. It shall not hijack the HTTPS+JSON power meter.

Done.

In my opinion, 3. is absolutely mandatory. If you manage to implement 3., I will merge this despite my strong objections. I can address those when I tackle the PowerMeter refactoring. Add a new POWERMETER_TIBBER_CONFIG_T in the Configuration, which has only the needed struct members. Introduce a new power meter type value (config.PowerMeter.Source), both in the firmware and UI. Copy'n'paste the needed UI elements, with adjusted IDs, and do not re-use the HTTPS+JSON inputs.

Also done.

Edit: Remove all files in webapp_dist from the changeset (do not delete them from the repo, just make sure your commits do not include changes to those files).

I removed the files in webapp_dist from the changeset.

[schlimmchen]

Alright, this looks much better! Thanks for your efforts.

I will rebase this onto the current development branch, solve the conflicts and update parts of your code for compatibility with ArduinoJson7. I then need you to test the feature once more on short notice, so we can merge this. Merging this is important to me, because as of today, I started the power meter refactoring, and it's best if my work is based on yours. However, I want to be sure that the feature is working okay before it is merged. If we need to do a maintenance release for some reasons, the feature will be included and published, and it shall be working then.

Are you available to re-test on short notice?

[schlimmchen]

Does the username have to be configurable, or will it always be "admin" since it cannot be changed on the bridge?

The URL must be http://{IP}/data.json?node_id=1? Will it be sufficient for the user to specify the IP address of the bridge only? Does a user want to change anything about that URL?

Nevermind. I think we should keep it generic, as there are possibly other devices out there which work the same way (HTTP server handing out SML data).

[schlimmchen]

@marvincarstensen Can you please test the firmware available in the PR build run?

Could you also be so kind and download a copy of the binary obtained from your Tibber bridge at http://{IP}/data.json?node_id=1 for testing purposes? I think I crafted a dummy SML binary, but it would be much better if we had a real example file.

[noone2k]

did check it ... and it works as expected.
some more sml values would be great to be exported to mqtt ( like import/export etc. ).
// edit ^^ I've already seen a pull request...

would reduce the need of another esp(32) for only "pulse" reading ...

attaching a dump from my bridge ( serial cleared ... hopefully none crc-checks chime in :) )

pulse.zip

btw., the pulse only sends the raw sml to the bridge over 433mhz ( longer range then most other non-wifi version, except lora, if any exists )

[marvincarstensen]

Are you available to re-test on short notice?

Sure! No problem.

@marvincarstensen Can you please test the firmware available in the PR build run?

I've tested that build too and it works as excepted.

Could you also be so kind and download a copy of the binary obtained from your Tibber bridge at http://{IP}/data.json?node_id=1 for testing purposes? I think I crafted a dummy SML binary, but it would be much better if we had a real example file.

I hope the dump of @noone2k works for you (thanks @noone2k !).

[schlimmchen]

Thanks for your feedback, @marvincarstensen @noone2k.

The binary you provided works: I can have it served by a webserver and the power meter implementation will spit out 21W as result. I guess that's reasonable.

btw., the pulse only sends the raw sml to the bridge over 433mhz

Does that mean that the actual power meter, to which the pulse is attached, will determine what data is received? Our goal should be to define SML handlers for the most common data? Something like this?

(from https://tasmota-sml-parser.dicp.net/decode)

I realized, while continuing the power meter refactoring, that this feature is actually very generic: HTTP/GET a binary from a webserver and interpret it as SML. I can see how other devices might do the same. That's why I am going to rename the class and settings to represent "SML over HTTP(S)" functionality. The GUI will mention "Tibber Pulse" explicitly as an example. The Wiki/docs are ready to help users setup this power meter interface to communicate with the Tibber bridge. As this will break the existing config you guys have, I will postpone including this feature in a public release. Please use the firmware from the PR build run you already tested to have support for this power meter.

Let me know if this works for you.

I will make sure that you, @marvincarstensen, remain author of the commit that introduces this feature.

[noone2k]

Thanks for your feedback, @marvincarstensen @noone2k.

The binary you provided works: I can have it served by a webserver and the power meter implementation will spit out 21W as result. I guess that's reasonable.

should be right ... ( have to lookup, to be 100% certain ) ...
// edit ... yes, looks reasonable ... the values in my db this time around is in this range ( 19-24 ) ...

btw., the pulse only sends the raw sml to the bridge over 433mhz

Does that mean that the actual power meter, to which the pulse is attached, will determine what data is received?

yes ... the pulse send the raw data 1:1 and doesnt alter anything.
to be more generic: the pulse is a 433mhz sml-bridge, where we can grab the sml via http from the "power-plug" ...

this way: the same rules for hichi and co applies to this device.

so, the data received depends on the meter(-settings) ( short, long, advanced dataset )
the long dataset should contain at least: im-/export ( 1|2.8.0 ) / current / voltage / power ..
for the power information ( 16.7.0 ) the long dataset is needed by my used model,
but this is up to the user to setup the (smart-) meter right ...

^^should cover the next part ...

Our goal should be to define SML handlers for the most common data? Something like this?

(from https://tasmota-sml-parser.dicp.net/decode)

I realized, while continuing the power meter refactoring, that this feature is actually very generic: HTTP/GET a binary from a webserver and interpret it as SML. I can see how other devices might do the same. That's why I am going to rename the class and settings to represent "SML over HTTP(S)" functionality. The GUI will mention "Tibber Pulse" explicitly as an example. The Wiki/docs are ready to help users setup this power meter interface to communicate with the Tibber bridge. As this will break the existing config you guys have, I will postpone including this feature in a public release. Please use the firmware from the PR build run you already tested to have support for this power meter.

Let me know if this works for you.

i'm fine with it ... but this question is more for @marvincarstensen :)

I will make sure that you, @marvincarstensen, remain author of the commit that introduces this feature.

[marvincarstensen]

I realized, while continuing the power meter refactoring, that this feature is actually very generic: HTTP/GET a binary from a webserver and interpret it as SML. I can see how other devices might do the same. That's why I am going to rename the class and settings to represent "SML over HTTP(S)" functionality. The GUI will mention "Tibber Pulse" explicitly as an example. The Wiki/docs are ready to help users setup this power meter interface to communicate with the Tibber bridge. As this will break the existing config you guys have, I will postpone including this feature in a public release. Please use the firmware from the PR build run you already tested to have support for this power meter.

Sounds good to me! Update us as soon as you have refactored the power meters and implemented the "SML over HTTP(S)". I will here to test the implementation.

[arnhei]

I've extended the "HTTP(S) + JSON" Power Meter (#153) to support the Tibber Pulse. The current power drawn from the grid is fetched using the local web server from the Tibber Bridge. No external requests through the internet (Tibber API).

Instructions on how to enable the web server permanently are also included in German and English:

Thank you to gvzdus and the discussion #123!

Pre-compiled binaries to directly test the feature: opendtu-onbattery-generic_esp32.bin.zip opendtu-onbattery-generic_esp32.factory.bin.zip

Hello, I am also interested in that topic. I currently use a script in iobroker to get the power from the bridge and provide the value with mqtt. Your solution would be much better (no iobroker necessary).
I tested your binary but the inverter was not working. What did I made wrong.
Waiting for an official feature in opendtu...
Ragards

[schlimmchen]

@arnhei Did you install one of these: https://github.com/helgeerbe/OpenDTU-OnBattery/actions/runs/8974757062?pr=915 ?

Did you install the correct flavor? Maybe you installed "generic_esp32" but need "generic", as the latter has some default pins defined for the NRF24 module, but the former does not. If you are interested in cleaning that up, install an appropriate Device Profile.

[Megathomas-de]

Hello,
I use the firmware offered here - it works!
If I understand correctly as a newbie, this feature is waiting to be incorporated into the official firmware. Will that happen or is it not certain?
Then I read that oDTUoB should no longer be continued on ESPs with 4MB flash memory. This means that when it is finalized at some point, it won't run on the old ESPs? Or you can also use this version for longer if it is not recorded or no longer runs on the old ESPs.

(I used Google Translate in case some things sound strange)

Best regards

[schlimmchen]

This feature will certainly be merged, however, not via this pull request. I will let you know (add a comment here) when the feature is merged into the development branch. You will have to observe when the first release is made which includes the feature (see changelogs).

I highly anticipate that there will be another release which includes this feature, which will also run on ESP32 with 4MB of flash.

Or you can also use this version for longer if it is not recorded or no longer runs on the old ESPs.

This sentence got garbled up... At least I don't understand what you are asking.

German: Du möchtest wissen, ob du die oben verlinkte und im Kontext dieses pull request gebaute Firmware einfach weiterverwenden kannst? Klar. Zum einen basiert diese aber auf einem etwas älteren Stand des Projektes, und zum anderen wird dieser pull request wird nicht gepflegt, d.h. du wirst irgendwann abgehängt sein, wenn du neue Features nutzen möchtest, die Teil von Releases werden. Beantwortet das deine Fragen?

Ich habe dieses Feature noch einmal auf den aktuellen development Branch aufgesetzt. Die daraus resultierende neue Firmware findest du im neuen Build Run. Die ist dann wie 2024.06.03 mit Tibber.

[Megathomas-de]

Hallo Schlimmchen,
alle meine Fragen wurden komplett beantwortet - auch der Teil der in der Übersetzung verloren gegangen ist. Vielen Dank!
Ich benutze also den alten ESP32 solange es geht und im Normalfall wird das hier in die offizielle Version eingehen - früher oder später. Vielen Dank auch für diese aktuelle Version MIT Tibber Pulse Unterstützung!

Ich wollte nicht unhöflich sein und einfach auf Deutsch schreiben, vielen Dank für den Klartext!

LG Thomas

[bikerbiker12]

Hallo zusammen,

ich verstehe es mal wieder nicht. Wo finde ich denn das aktuelle Release für den Tibber. Mit der offiziellen von Helge läuft es ja nicht oder? Zumindest bei mir nicht.
Wer kann mir helfen

Grüße Marco

Update: man sollte sich einloggen und schon klappt der download. Allerdings komme ich trotzdem nicht an die Daten des Tibber. Die aktivierung des Webservers war erfolgreich. Die IP-Adresse stimmt auch, sowie auch die Zugangsdaten zum Tibber. Was kann man tun?

[Megathomas-de]

Hallo Marco,
nehme die Version aus dem letzten Beitrag von Schlimmchen "im neuen Build Run", das funktioniert genauso wie Ursprüngliche hier aus dem Tread. Das Offizielle von Helge unterstützt das noch nicht.

Zum Update: Ich habe da auch so meine Probleme, weiß aber nicht genau was schief läuft. Zuerst hatte ich eine tibber-host.fritz.box Adresse die hinter aber nicht mehr wollte. Also nun direkt eingetragen. Und das Passwort will openDTU onBattery als echte Eingabe nicht als Copy and Paste. Musste ich mehrfach ausprobieren bis ich es geschafft habe.

[bikerbiker12]

Hi, habe ich gemacht. War nicht eingeloggt, daher kein Download. Allerdings klappt es trotzdem nicht… Hab mal ne Screenshot angehängt Gruß marco Von: Megathomas-de ***@***.***> Gesendet: Freitag, 14. Juni 2024 14:37 An: helgeerbe/OpenDTU-OnBattery ***@***.***> Cc: Marco Matterne ***@***.***>; Comment ***@***.***> Betreff: Re: [helgeerbe/OpenDTU-OnBattery] Feature: Added "Tibber" Power Meter to support Tibber Pulse (via Tibber Bridge) (PR #915) Hallo Marco, nehme die Version aus dem letzten Beitrag von Schlimmchen "im neuen Build Run", das funktioniert genauso wie Ursprüngliche hier aus dem Tread. Das Offizielle von Helge unterstützt das noch nicht. — Reply to this email directly, view it on GitHub<#915 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A6GKDTH4XSL3TPQUC77EBCLZHLPYDAVCNFSM6AAAAABGVMZZQ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRXHE2DCOBWGY>. You are receiving this because you commented.Message ID: ***@***.***>

[Megathomas-de]

Leider sehe ich kein Bild

[bikerbiker12]

Hallo Marco, nehme die Version aus dem letzten Beitrag von Schlimmchen "im neuen Build Run", das funktioniert genauso wie Ursprüngliche hier aus dem Tread. Das Offizielle von Helge unterstützt das noch nicht.

Zum Update: Ich habe da auch so meine Probleme, weiß aber nicht genau was schief läuft. Zuerst hatte ich eine tibber-host.fritz.box Adresse die hinter aber nicht mehr wollte. Also nun direkt eingetragen. Und das Passwort will openDTU onBattery als echte Eingabe nicht als Copy and Paste. Musste ich mehrfach ausprobieren bis ich es geschafft habe.

Der Webserver hatte das "true" nicht übernommen. Allerdings bekomme ich jetzt bei Test folgende Meldung:

[HttpPowerMeter] Unable to parse server response as JSON

Muss den beim "JSON Pfad" noch etwas eingetragen werden?

[Megathomas-de]

Wenn du das Feld JSON-Pfad sehen kannst, hast du etwas falsch eingestellt. Bei Stromzählertyp muss "Tibber Pulse (via Tibber Bridge" ausgewählt sein.

[bikerbiker12]

Wenn du das Feld JSON-Pfad sehen kannst, hast du etwas falsch eingestellt. Bei Stromzählertyp muss "Tibber Pulse (via Tibber Bridge" ausgewählt sein.

Hmm, das kann ich nicht auswählen da nicht vorhanden.

SDK-Version | v4.4.6-dirty -- | -- 0.1.28 2024.06.03 SDK-Version v4.4.6-dirty Konfigurationsversion 0.1.28 Firmwareversion / git Hash [2024.06.03](https://github.com/helgeerbe/OpenDTU-OnBattery/releases/tag/2024.06.03)

[Megathomas-de]

Dann ist das nicht die Firmware hier aus dem Thread, nehme ich mal stark an. Meine heißt helgeerbe/OpenDTU-OnBattery/pr915-202406061134.

[bikerbiker12]

Dann ist das nicht die Firmware hier aus dem Thread, nehme ich mal stark an. Meine heißt heldeerbe/OpenDTU-OnBattery/pr915-202406061134.

Asche über mein Haupt. Manchmal sollte man solche Sachen nicht zwischen Tür und Angel machen. Ich hatte tatsächlich die falsche Firmware erwischt.
Vielen Dank für die Hilfe...läuft jetzt

LG Marco

[schlimmchen]

Superseded by #1077.

[JochenKlenk]

Ich bekomme die Firmware nicht installiert. Bekomme immer einen invalid header nach der Installation.

invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0�ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI

Habt ihr eine Idee?

[schlimmchen]

Nimm bitte Firmware aus #1077 oder aus dem letzten Build Run des development Branches. Du hast vermutlich die factory.bin über OTA ge-flash't, oder die non-factory.bin über esptool mit dem falschen Offset, oder du hast eine zu große Firmware in eine zu kleine Partition installiert, oder oder oder 😉 Lösche den Flash deines ESP32 und fang von Vorn an.

[JochenKlenk]

Hab ich alles durchprobiert, denke ich. Die normale Webinstallation läuft ja.
????

[schlimmchen]

Die normale Webinstallation läuft ja.

Dann hast du ja eine funktionierende OpenDTU-OnBattery, mit der du lediglich noch ein OTA Update machen musst. Beschreib bitte mal exact, was du tust. Welchen ESP hast du? Welche Datei verwendest du?

[JochenKlenk]

Nach langem hin und her hat endlich alles geklappt. Vielen Dank für die Hilfe.
Habe nun ein ESP32-S3 Dev Board in Verwendung.
Leider fliegt sporadisch immer wieder die Verbindung zum Hoymiles weg. Also er erreicht ihn nicht.
Davor mit ahoy hat es ohne Probleme funktioniert.
Gleiches NRFl Modul.
Jemand eine Idee?

[JochenKlenk]

okay warum auch immer. Es war das Netzteil. Sieht so aus als ob das Dev Board Probleme mit der Stromversorgung hat bei stärkeren Netzteilen oder das Netzteils Mist ist. Danke an alle.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion or issue for related concerns.