Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Power meter support: HTTP(S) + JSON (Shelly 3EM, Tasmota, Volkszähler etc.) #153

Merged
merged 5 commits into from
Apr 3, 2023

Conversation

berni2288
Copy link

@berni2288 berni2288 commented Apr 1, 2023

It's finally implemented :)

  • Support for any HTTP(S) JSON based Power meters like Shelly 3EM, Tasmota, Volkszähler etc.
  • HTTPS is also supported without certificate checks (MITM attacks are possible)
  • Requests are blocking
  • Very user friendly UI with HTTP request test button
  • Lots of customization options: Basic Auth; Custom request header; Timeout
  • You can specify the json field (Uses https://github.com/mobizt/FirebaseJson) - Example: testarray/[2]/power
  • You can choose whether you want individual HTTP requests per phase, or just one (First phase can be used for the total power of all 3 phases)
  • Fixed the _lastPowerMeterUpdate safety check not working anymore in the Power Limiter (the variable was always updated regardless of the success)

image

@berni2288 berni2288 force-pushed the feature-http-json-powermeter branch from cf40e78 to 0f88cce Compare April 1, 2023 16:02
@berni2288 berni2288 requested a review from helgeerbe April 1, 2023 16:25
@berni2288 berni2288 added the enhancement New feature or request label Apr 1, 2023
@madmartin
Copy link

Hello @berni2288

I am suprised and impressed.... this is exactly what I missed some minutes ago when I started playing with this OpenDTU-branch....

I will test.

@berni2288 berni2288 force-pushed the feature-http-json-powermeter branch from 0f88cce to 852ae06 Compare April 2, 2023 09:15
@berni2288
Copy link
Author

berni2288 commented Apr 2, 2023

I've just rebased the branch on the latest changes of development branch to fix a crash :)
Please also note that you don't have to configure all 3 phases, you can just use the first phase with the total power value, if that is available in your API.

You also have to run (cd webapp && yarn build) in order to compile the webapp before you build the firmware.

@madmartin
Copy link

You also have to run (cd webapp && yarn build) in order to compile the webapp before you build the firmware.

I got that, no problem.

Here are my testing results:

First: while playing around with the JSON path examples, I figured out that one of the json examples is not properly formatted:

diff --git a/webapp/src/views/PowerMeterAdminView.vue b/webapp/src/views/PowerMeterAdminView.vue
index ed6d0ec..785b090 100644
--- a/webapp/src/views/PowerMeterAdminView.vue
+++ b/webapp/src/views/PowerMeterAdminView.vue
@@ -170,7 +170,7 @@
 
                 <h2>JSON path examples:</h2>
                 <ul>
-                    <li>total_power - { othervalue: "blah", "total_power": 123.4 }</li>
+                    <li>total_power - { "othervalue": "blah", "total_power": 123.4 }</li>
                     <li>testarray/[2]/myvalue - { "testarray": [ {}, { "power": 123.4 } ] }</li>
                 </ul>
 

I have a tasmota-driven SML power meter adapter.
I do a read test with curl and jq. The tasmota command to send is "status 8" and must be url-encoded to status%208 - this should appear in the example section or in a documentation with usage examples.

~ $ curl -s "http://tasmota-62a4da-1242.home/cm?cmnd=status%208" | jq
{
  "StatusSNS": {
    "Time": "2023-04-02T13:38:36",
    "Strom": {
      "Total_in": 19902.6084,
      "L1": 219.12,
      "Meter_Number": "0a01445a470002fb2769"
    }
  }
}

The actual power consumption is the "L1" value, so the jq command

~ $ curl -s "http://tasmota-62a4da-1242.home/cm?cmnd=status%208" | jq '.StatusSNS.Strom.L1'
219.12

gives the correct result.

Translated to the new power meter input dialog I get this result:

grafik

The whole http get response data is 124 bytes - is that really too much?
I did not read the code yet, maybe I find the solution myself. But I wanted you to give a fast feedback.

@berni2288
Copy link
Author

Thanks, I'll change the examples.

The response buffer is 2000 bytes I think, are you sure that your URL does not require authentication?
I can check again if I find some bug...

@madmartin
Copy link

Thanks, I'll change the examples.

The response buffer is 2000 bytes I think, are you sure that your URL does not require authentication? I can check again if I find some bug...

Yes, the tasmota device does not need authentication. The curl without the -s option shows a statistic:

martin@bln9716cm ~ $ curl  "http://tasmota-62a4da-1242.home/cm?cmnd=status%208" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   126    0   126    0     0   2747      0 --:--:-- --:--:-- --:--:--  2800
{
  "StatusSNS": {
    "Time": "2023-04-02T14:08:03",
    "Strom": {
      "Total_in": 19902.6850,
      "L1": 145.94,
      "Meter_Number": "0a01445a470002fb2769"
    }
  }
}
martin@bln9716cm ~ $ 

where you can see the response is 126 byte.

@berni2288
Copy link
Author

berni2288 commented Apr 2, 2023

I have saved your json to my webserver and hit the Test button and I get "Success! Power: 145.94W". So it works for me.

Can you maybe try the following CURL command?

curl \
	--header "User-Agent: OpenDTU-OnBattery" \
	--header "Content-Type: application/json" \
	--header "Accept: application/json" \
	"http://tasmota-62a4da-1242.home/cm?cmnd=status%208"

Thanks!

@madmartin
Copy link

Exactly the same:

martin@bln9716cm ~ $ curl \
        --header "User-Agent: OpenDTU-OnBattery" \
        --header "Content-Type: application/json" \
        --header "Accept: application/json" \
        "http://tasmota-62a4da-1242.home/cm?cmnd=status%208"
{"StatusSNS":{"Time":"2023-04-02T14:37:55","Strom":{"Total_in":19902.7331,"L1":0.00,"Meter_Number":"0a01445a470002fb2769"}}}martin@bln9716cm ~ $

the same with jq:

martin@bln9716cm ~ $ curl       --header "User-Agent: OpenDTU-OnBattery"        --header "Content-Type: application/json"       --header "Accept: application/json"        "http://tasmota-62a4da-1242.home/cm?cmnd=status%208" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   125    0   125    0     0   3288      0 --:--:-- --:--:-- --:--:--  3378
{
  "StatusSNS": {
    "Time": "2023-04-02T14:39:24",
    "Strom": {
      "Total_in": 19902.7340,
      "L1": 77.93,
      "Meter_Number": "0a01445a470002fb2769"
    }
  }
}
martin@bln9716cm ~ $ 

@madmartin
Copy link

I looked at commit dfeafb2 - looks good!

I think I need some debug output on the console to see more. I have to fetch the DTU from above and connect USB...

@berni2288
Copy link
Author

berni2288 commented Apr 2, 2023

You can also find a Web Console in the menu "Info".
MessageOutput.printf prints to Web console and Serial console :)

You can replace this code in src/HttpPowerMeter.cpp:

snprintf_P(error, errorSize, "Response too large!");

with:

            snprintf_P(error, errorSize, "Response too large! Response length: %d Body: %s",
                httpClient.getSize(), httpClient.getString().c_str());

Then you get the beginning of the response body in the UI directly.

@madmartin
Copy link

Hmm.... tried with the last commit and I receive

Response too large! Response length: -1 Body: {"StatusSNS":{"Time":"2023-04-02T21:45:01","Strom":{"Total_in":19904.8576,"L1":513.89,"Meter_Number":"0a01445a470002fb2769"}}}

WTF could be wrong here?
I compiled 2 times, last time with an extra platformio run -e generic -t clean to avoid any dependency fuzz.

The Content-Length header is not always returned by a web server
@berni2288
Copy link
Author

WTF could be wrong here? I compiled 2 times, last time with an extra platformio run -e generic -t clean to avoid any dependency fuzz.

Thanks @madmartin . That helped me to identify the problem and I've fixed it now.
The Content-Length header is not returned by Tasmota it seems, or is 0. That' why the getSize() method returned -1.
And comparing a signed integer against an unsigned integer is not a good idea it seems, as -1 is indeed greater than 1999 in this case (integer overflow).

Can you test again now? thanks!

@madmartin
Copy link

@berni2288 Success, it works! Good job!

One small fix to make the console output more readable: add CR+LF

diff --git a/src/PowerMeter.cpp b/src/PowerMeter.cpp
index 6bfe7e0..da2fc2d 100644
--- a/src/PowerMeter.cpp
+++ b/src/PowerMeter.cpp
@@ -68,7 +68,7 @@ void PowerMeterClass::onMqttMessage(const espMqttClientTypes::MessageProperties&
         _powerMeter3Power = std::stof(std::string(reinterpret_cast<const char*>(payload), (unsigned int)len));
     }
 
-    MessageOutput.printf("PowerMeterClass: TotalPower: %5.2f\n", getPowerTotal());
+    MessageOutput.printf("PowerMeterClass: TotalPower: %5.2f\r\n", getPowerTotal());
 
     _lastPowerMeterUpdate = millis();
 }
@@ -143,7 +143,7 @@ void PowerMeterClass::loop()
         }
     }
 
-    MessageOutput.printf("PowerMeterClass: TotalPower: %5.2f\n", getPowerTotal());
+    MessageOutput.printf("PowerMeterClass: TotalPower: %5.2f\r\n", getPowerTotal());
 
     mqtt();
 

@berni2288
Copy link
Author

Thanks for testing!

@STE2k
Copy link

STE2k commented Apr 15, 2023

I can´t find the power meter settings page as posted above.. what am I doing wrong? Or where can i find these settings? Do I have to activate Dynamics Power Limit first? I checked everything, nowhere to find the menu
I used OpenDTU-OnBattery-feature-http-json-powermeter. Do I have to erase my ESP32 completely because of it is using the old web UI?

@madmartin
Copy link

madmartin commented Apr 15, 2023

@STE2k You have to reload the WebUI in the browser! Otherwise the browser continues to use the cached WebUI and code.

Internet says:
To hard refresh on Google Chrome, there are two ways you can do it:

Hold down Ctrl and click the Reload button.
Or Hold down Ctrl and press F5.

To hard refresh on Firefox, there are also two easy hotkey commands you can use:

Hold down Ctrl, Shift and the ‘R’ key.
Or Hold down Ctrl and press F5.

@STE2k
Copy link

STE2k commented Apr 15, 2023

Working, OMG.. I´m the noob. Thx

@madmartin
Copy link

No problem.

@berni2288 berni2288 deleted the feature-http-json-powermeter branch April 15, 2023 20:22
@berni2288
Copy link
Author

That just reminded me to delete old branches. Please use the development branch instead!

@kloppy1984
Copy link

I have the same problem, and can not find the new menu items

@berni2288
Copy link
Author

Please refer to the README.md on how to build the webapp and the firmware. All changes are merged into development. I'll lock this PR now.

@hoylabs hoylabs locked as off-topic and limited conversation to collaborators Apr 17, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants