New esphome component for pvvx firmware + "Custom" advertising

[airy10]

Available here :
https://github.com/airy10/pvvx_mithermometer

Mainly a copy of the esphome atc_mithermometer component with updated decoding code

[pvvx]

More relevant is support for Xiaomi Mijia encoding and fix drops when receiving ads in ESP32.
I have not found any examples with AES CCM decoding for ESP32.

[airy10]

Ads format from the original firmware is already supported by the bundled xiaomi_lywsd03mmc components.
Or you mean custom format + encryption ?

[pvvx]

ESPHome does not work on Android and Windows. Requires specific knowledge, additional costs for unnecessary components and assembly of the system from ordinary users.
In ESPHome It is impossible to solve the simple problem of receiving and parsing advertising packages for the entire BLE group, which has a couple of formats directly on ESP32.

[airy10]

Well, ESPHome is working great for me as a cheap Wifi bridge for my LYWSD03MMC devices and I just thought this component might be useful for other people too...

[pvvx]

The component can be useful when fixing ad-packet reception skipping in ESP32 drivers. Otherwise it is not a working solution.

[rjblake]

Well, ESPHome is working great for me as a cheap Wifi bridge for my LYWSD03MMC devices and I just thought this component might be useful for other people too...

I can see the benefit when using Home Assistant or other Home Automation. In my case, I need additional ways to listen for advertising packages (home automation server bluetooth does not reach all corners of house) and relay them back to main server; so in my setup (Xiaomi LYWSD03MMC->ESP32 Tasmota->Node Red->Home Automation Server). I have not had any joy with ESPHome and my ESP32s, so curious if this would work

[pvvx]

ESPHome, ESP32-wrover-b, 3 sensors are registered.
Half an hour charts. Each graph must contain 180 points (!):

Log Debug says that only 0.5 AD packets per second were received from all surrounding BLE devices.
The Sniffer says that the loss does not exceed 5% of the packets.

[pvvx]

I need additional ways to listen for advertising packages (home automation server bluetooth does not reach all corners of house) and relay them back to main server;

Reception of advertisements on JDY-10 (TLSR8266) or CC2540 USB Dongle.
https://github.com/pvvx/UBIA/tree/master/Sniffer/OpenWRT_SnifferAd
Reaches several hundred ad packages per second...
https://www.youtube.com/watch?v=MqFcY5Hovpw

[airy10]

Even if packets are lost, even receiving new values for temp and humidity every couple of minutes is good enough for me for that kind of sensors. I'm using it with Home Assistant for sensors that are too far from my Raspberry Pi. I can understand that it might not fit some people needs, but it's very much good enough for my needs, especially for the price and easy of use.

…

Le 12 avr. 2021 à 17:33, Victor ***@***.***> a écrit : I need additional ways to listen for advertising packages (home automation server bluetooth does not reach all corners of house) and relay them back to main server; Reception of advertisements on JDY-10 (TLSR8266) or CC2540 USB Dongle. https://github.com/pvvx/UBIA/tree/master/Sniffer/OpenWRT_SnifferAd Reaches several hundred ad packages per second... https://www.youtube.com/watch?v=MqFcY5Hovpw — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[pvvx]

Even if packets are lost, even receiving new values for temp and humidity every couple of minutes is good enough for me for that kind of sensors.

Smart Home users are not interested in sensor readings. They are interested in the execution of scripts, which requires a certain response of timely triggering according to data from the sensors. And also the reliability of the system, which is not provided in the HA.

According to the scheme of building a smart home system with ESPHome and BLE, the most unstable component today is ESP32.
Due to the instability of ESP32 reception and operation, difficulties with the installation for ordinary people of ESPHome, I cannot advise users to use ESPHome.
The situation is similar with "HA" - there is no support for standard devices for placing the "HA" system on the existing equipment of users. Typical equipment is old cheap smartphones, computers, routers and other household junk.

[rjblake]

Smart Home users are not interested in sensor readings. They are interested in the execution of scripts, which requires a certain response of timely triggering according to data from the sensors.

Indeed, in the case of my switches and automations. My heating is a different story (updates every minute).

According to the scheme of building a smart home system with ESPHome and BLE, the most unstable component today is ESP32.

Guess I've been lucky, as these have been stable without a reboot/restart for over 4 months now.

Typical equipment is old cheap smartphones, computers, routers and other household junk.

..or in my case a few thousand Euros of equipment; but understand many HA systems are hacked together bits and pieces that require never ending maintenance to keep running.

[pvvx]

Guess I've been lucky, as these have been stable without a reboot/restart for over 4 months now.

This is another advertisement without confirmation.

Xiaomi LYWSD03MMC with custom firmware is able to switch heating without assistance.

[mr-sneezy]

Don't know if anyone cares, but my results with ESPhome on ESP32 have been outstanding with my three reflashed MiThermo sensors. This one is even locked up inside my refrigerator, and I am very interested in the charts recorded by HA to see how my fridge performs....

PS. My HA system IS hacked together with bit & pieces, and once I ironed out a few bugs (like SD card failures) and run a SSD drive with the RPi (instead of a big SD card) the reliability has also been outstanding.

[rjblake]

Guess I've been lucky, as these have been stable without a reboot/restart for over 4 months now.

This is another advertisement without confirmation.

Xiaomi LYWSD03MMC with custom firmware is able to switch heating without assistance.

Here is the confirmation. Data at 5 minute frequency written to Domoticz. My heating system is not an On/Off, but a Modulating boiler that is adjusted based on heating requirements for my underfloor heating on the ground floor, wall mounted radiators on other floors and control valves for these, combined with the heat loss/retention properties of the different rooms. So, unfortunately a LYWSD03MMC doesn't meet my needs as the thermostat. I'm certainly not 'advertising ESP32s', as mentioned perhaps I've been lucky so far.

[pvvx]

Graphs in the thermometer itself are recorded in more detail. And we are talking about the speed of receiving information and without dropout. Actual for PID and more accurate adjustments.

Show on your schedule actually accepted glasses that the thermometer transmits every 10 seconds. Not a diagram that has interpolation to display.

ESPHome with ESP32 forces you to duplicate ad packets and transfer them as often as possible, which greatly affects the battery consumption of the sensor.
With an advertisement interval of 2.5 seconds with a sensor polling every 10 seconds, 12 packets are transmitted. Any other BLE receiver in noisy radio air, it loses no more than 8% of packets, then ESP32 in ESPHome loses more than 90%.
With an increase in the number of devices, this is already critical.

Comparison of the modified hcitool says that the ESP32 program needs to be fixed.

[kelchm]

I've also observed issues with using ESPHome for receiving BLE advertisements. I've spent a little bit of time trying to fine tune the scan_parameters, but the performance is still pretty abysmal.

Fundamentally I think the problem is that an ESP32 can't be listening for BLE advertisements while simultaneously sending/responding to network requests over WiFi. I suppose you could set one up to use ethernet instead, but it seems like a rather roundabout solution.

Here's a graph of the number of advertisements received over a 24 hours period of three of my BLE sensors grouped by periods of 5 minutes. While I certainly don't expect this to be a perfectly flat line, less than one advertisement being captures for a given sensor over a period of five minutes just doesn't work for the use cases I have in mind.

[mr-sneezy]

@

Fundamentally I think the problem is that an ESP32 can't be listening for BLE advertisements while simultaneously sending/responding to network requests over WiFi. I suppose you could set one up to use ethernet instead, but it seems like a rather roundabout solution.

I guess one way to prove where the issue is in the ESP32 system is combine an ESP32 with a ESP8266 (or second ESP32), and send the received data from the ESP32 listener (without any WiFi functions) to the ESP8266 via serial for the ESP8266 to send to HA (or wherever the target is). Could use MQTT instead of ESPhome service as well...
Or is there a better HW just as cheap as an ESP32 we could use ?

[pvvx]

I guess one way to prove where the issue is in the ESP32 system is combine an ESP32 with a ESP8266 (or second ESP32), and send the received data from the ESP32 listener (without any WiFi functions) to the ESP8266 via serial for the ESP8266 to send to HA (or wherever the target is). Could use MQTT instead of ESPhome service as well...

Questions about missing advertise packages for ESP32 are available on many Internet resources.
The selection of scan intervals does not help.

For comparison, I tried many options:

• BLE Sniffers / TLSR8266
• Modified hcitool
• RTL8722DM
• USB-BT adapters also differ in the number of AD packets received per unit of time: https://www.youtube.com/watch?v=3getuFLcMko

MQTT on weak platforms limits performance and does not have time to service a large network of sensors.
mqtt-performance-tests

[pvvx]

Or is there a better HW just as cheap as an ESP32 we could use ?

Xiaomi LYWSD03MMC. TLSR8251 has USB2.0 FS. USB pins are available on the board...
JDY-10 (TLSR8266) = $1
TB-03F, TB-04 (TLSR8253) BLE, ZigBee, ...
BW16 (RTL8720DN) BLE + WiFi 5GHz, USB2.0 HS
...

[airy10]

Note that while the ESP32 will never be the device to use if you need high availability for BT+Wifi, you can improve BLE adv reception by enlarging the scanning window, which is very small by default. I'm getting about 35% of them with these parameters in a quite noisy environment - which is enough for what I'm using it for - you'll decide if it is good enough for you :)

 esp32_ble_tracker: 
    scan_parameters: 
      active: false
      interval: 211ms
      window: 120ms

Default is 350ms/30ms
(using it my LYWSD03MMC using 10s advertising interval/ and x10 measure interval - which is much less that the default setting but I don't need more points)
I wouldn't use that for life-saving features, but it's definitively good enough for me for remote checking temperatures which aren't changing quickly).

[pvvx]

What prevents the RF receiver in ESP32 from working permanently? The CPU does not take part in the reception. Usually, a packet received by the radio path causes an interrupt and is processed by DMA.
But in reality, it turns out that the proprietary ESP32 drivers cannot cope with the 1 Mbit / s stream. :)
In the Espressif documentation, the reception level in dB is higher than that of other cheap ICs, but in fact the opposite is true. Perhaps Espressif has no specialists or there are errors in the microcircuit...

I compared the number of dropped ESP32 packets without Wi-Fi enabled with the output to the UART. Changes to scan parameters have very little effect on packet rejection rate. At the same time, other chips with weaker processors and characteristics of the radio path have a lower percentage of drops. Perhaps this is due to a malfunction of the automatic reception level control system in the ESP32. But it is more likely that these are algorithms applied in the RF driver itself.

It was noticed that the processing time of data from ad packages also affects the number of lost packages. It looks like the drivers are not buffered and not designed to work with two cores ...

[kelchm]

I appreciate the insights @pvvx -- honestly this isn't something I've had much time to research thoroughly.

Are there any "plug and play" solutions that could be either used with MQTT or easily adapted to use with Home Assistant? Not that I don't like tinkering with these things, but my time is at a premium right now. Part of the appeal of ESPHome was the fact that the time it took to set it up was trivial.

[elRadix]

@kelchm you can use the custom component below to integrate your devices with home assistant. It works great.

https://github.com/custom-components/ble_monitor

[zdebel]

This is from my custom setup, I use my own sketch on ESP32 for BLE -> MQTT deal, influxdb for database and grafana for presentation. I also log via uart each packet, I get one every few seconds. What bothers me more, is the fact that your firmware drains the battery faster (default settings) than the 'original' atc1441 as can be seen here:

I confirmed it using my multimeter.

[zdebel]

What prevents the RF receiver in ESP32 from working permanently? The CPU does not take part in the reception. Usually, a packet received by the radio path causes an interrupt and is processed by DMA.

From what I understand, WiFi and BLE are multiplexed, thus you can't be in 'BLE rx' mode all the time.

[stevoh6]

One important note, why is this important: ESPhome + temp sensor (as those ble ) is perfect solution for smart automation, especialy heating. ESPhome has native support of checking availabilty of HomeAssistant ( whole automations can be run in super "smart" way, HomeAssistant+NodeRed+whatever you want) but in case of power outage, server failure etc, ESPhome can automaticly "switch" to safe mode and act as regular thermostat (maybe little bit smarter with some simple automations written directly in esp). Im using this "safe" scenario even for wall switches.

[pvvx]

This is from my custom setup, I use my own sketch on ESP32 for BLE -> MQTT deal, influxdb for database and grafana for presentation. I also log via uart each packet, I get one every few seconds.

Your graph should have 1080 points in 3 hours. At default settings, the data changes every 10 seconds, but this is not on your graph. 80% of packages are missing.
3 hours x 60 minutes 60 seconds in 10 second increments creates 1080 points. And on the presented chart there are only about 160 points in 3 hours. :)

What bothers me more, is the fact that your firmware drains the battery faster (default settings) than the 'original' atc1441 as can be seen here:

I confirmed it using my multimeter.

Is the battery charging in the center of the graph? :)

[pvvx]

What prevents the RF receiver in ESP32 from working permanently? The CPU does not take part in the reception. Usually, a packet received by the radio path causes an interrupt and is processed by DMA.

From what I understand, WiFi and BLE are multiplexed, thus you can't be in 'BLE rx' mode all the time.

Testing was carried out with the pin in the UART. WiFi is off.

[pvvx]

MHO-C401 records from Flash Memo. Step 10 minutes.


The battery (СR2032) voltage always repeats the temperature schedule.

[zdebel]

This is from my custom setup, I use my own sketch on ESP32 for BLE -> MQTT deal, influxdb for database and grafana for presentation. I also log via uart each packet, I get one every few seconds.

Your graph should have 1080 points in 3 hours. At default settings, the data changes every 10 seconds, but this is not on your graph. 80% of packages are missing.
3 hours x 60 minutes 60 seconds in 10 second increments creates 1080 points. And on the presented chart there are only about 160 points in 3 hours. :)

What bothers me more, is the fact that your firmware drains the battery faster (default settings) than the 'original' atc1441 as can be seen here:

I confirmed it using my multimeter.

Is the battery charging in the center of the graph? :)

I did a query on my db, around 400 points every hour. Disabling logging to flash made all the difference in regard to power consumption. Thanks for your hard work man, all this makes the device very attractive.

[pvvx]

I did a query on my db, around 400 points every hour.

If your script does not take into account the change of the metering counter, then for an hour there should be 60 * 60 / 2.5 = 1440 received AD packets.
If we take into account that there are other RF sources operating at 2.4 GHz between the sensor and the receiver, then the number of beaten packets will be at the level of several percent...
In a noisy radio air, in the presence of a WiFi6 router, Xiaomi 3 gateway and a dozen other devices with WiFi, BLE, MESH, Zigbee between the sensor and a cheap sniffer (distance 10 m), the number of dropped packets does not exceed 7%.
These are private measurements and may differ depending on the channel settings of the Wifi routers, the number of BLE devices and advertising intervals.

Disabling logging to flash made all the difference in regard to power consumption.

Disabling the recording of measurements in Flash will give no more than 1% battery savings.
The main difference in the consumption of the current firmware from atc1441 is in the interval of advertising and more economical polling of the sensor. atc1441 has an interval of 1.85 seconds and the current firmware is 2.5 seconds. Сonsumption by this parameter alone is already 2.5 / 1.85 = 1.35 times lower...
The RF-TX power level of the original Xiaomi firmware is set below 0 dB. In the current version, the default settings are also set to 0 dB, which reduces power consumption. This is enough for the sensor to work in the same room (creates fewer signal reflections), the level increase can be applied in other cases ...

[camaz]

As someone who also uses HA (Homeassistant) withy ESPHome, I can say I also experienced degraded ADV RX performance on an ESP32 (M5Stack Atom Lite); somewhere above 80% packet loss.

I have switched to a raspberry pi zero w with BlueZ and (the unfortunately deprecated) HCITool. I now get just 10~20% packet loss, depending on connection distance.

Many thanks for the project PVVX!

[deepcoder]

@camaz I too struggled with bluez and python ble on rpi and ubuntu. I wrote a rather rough c app that runs on both rpi and intel ubuntu. I collect from 20 sensors with good success for going on 6 months now. I don't collect missed packet data as I collect from govee sensors as well as xiaomi. That said from sensors that update sub-minute I see those sensor reading consistently. Perhaps it will give you some ideas : https://github.com/deepcoder/bluetooth-temperature-sensors

[pvvx]

https://github.com/pvvx/ble_monitor
NanoPi R1, NanoPi Neo Core2 + USB BT, ...

[metronidazole]

Thanks - was wondering why I couldn't get any sensor data using this firmware. Any plans to get (attempt) this merged with upstream?

[jddonovan]

Just as a side note, the C3-versions of the ESP32 chips seem to have a much better BT stack and hardware than earlier editions of the ESP32 chips. There's still around 15% packet loss, but this already means that with 4x broadcast per value the drop rate is inconsequential. It might be possible to drop the package loss further by optimising the code on top of the stack.

[kelchm]

@jddonovan that's great news, thanks for sharing. I will definitely have to order some to give it a try.

I spent some time trying to get a scanner up and running on the Wio Terminal (which has the ATSAMD51P19 + dedicated RTL8720DN for bluetooth + wifi), but never got it fully working. I think there are some bugs in the libraries provided by Seed studio.

[jddonovan]

@kelchm just be warned that support for C3 is still limited. Esphome is practically unusable: using I2C oled display freezes the board after around 15 mins (memory leak?), bluetooth libraries and MQTT support is missing and PWM output silently fails.

My tests were with using Arduino, the readily available BT library and MQTT. I had no issues with this simple setup. Your mileage may vary though. :)

[pvvx]

USB-BT and other BT adapters on Windows do not accept BLE ads. But you can do it on a cheap JDY-10 module or similar - AdScanerTrg - BLE ADV repeater. When connected to AdScanerTrg, it transmits ad packets from other devices. The losses are minimal, less than that of the ESPxxxx.

---

[pvvx]

I spent some time trying to get a scanner up and running on the Wio Terminal (which has the ATSAMD51P19 + dedicated RTL8720DN for bluetooth + wifi), but never got it fully working. I think there are some bugs in the libraries provided by Seed studio.

Seed studio uses the interface on SPI and after fixing to UART (reverse recovery in AT-files from files from SDK Ameba) everything works successfully.