What determines when the device sends an update?

[penright]

I felt like I was highjacking #23 question. An @MichielioZ pointed out that I was getting confused between the device sending advertising and updates. I just assumed that the device would send data with every advertising packet. I can see a power-saving by not turning on the measuring sensor, but if you are turning on the radio to send advertising would sending whatever the last data use that much more power?
What is the maximum time between updates?

Let me give a little more context for my question.
I have just started with ATC device and ESPHome. Seeing the update was asynchronous and ESPHome would not know if a device quit reporting, I started trying to figure out how to know if a device has not sent an update in X time. That way HA could send an alert that temp sensor XYZ is down. I was going to use the state property "last_updated". Because of some HA and ESPHome internals, I was going to create a Heartbeat based on the "on_value" event in ESPHome. That is when I notice that there are long times between data updates. That is why I am asking for the maximum time between updates, so I know when to send an alert that a sensor may be down.

[pvvx]

Device send data with every advertising packet.
The period for the transmission of advertising packages is set as:

The time between data updates in advertising packages is equal to the sensor polling time:

In advertising packages, a counter for polling sensors is transmitted to detect data updates.

ESPHome has flaws in the ESP32 BLE drivers. The scan settings prevent ESP32 from receiving packets all the time and there are many unaccepted packets. Other equipment is free of these shortcomings.

[apaperclip]

@pvvx
Does this help with the scanning flaw you mention?

esp32_ble_tracker:
scan_parameters:
active: false

[pvvx]

Does this help with the scanning flaw you mention?

esp32_ble_tracker:
scan_parameters:
active: false

These are bugs in the ESP32 BLE driver SDK.
Other SoCs work well - no more than 8% dropouts in heavily clogged radio air.
#59
https://esp8266.ru/forum/threads/ochen-chasto-esp32-propuskaet-priem-reklamnyx-paketov.5533/post-79166

RF driver code, the operating system itself is closed for all ESPs - proprietary and does not belong to open-source.
Request a fix from Espressif.

[penright]

@pvvx I hope my questions did not come across as complaining. I was just trying to understand. As I mention in "setting context", I connected the dots that whatever the receiver is would not know when a device was offline other than X amount of time with no updates. So I started different tests seeing what should I set X time for. The first lesson learned is HA only updates when a value changes. So even when the device and ESP are communicating, the last_updated value was not changing. So I created a "heartbeat" sensor that the ESP creates and updates on each received from the device. That is when I asked about what are the expectations. From what I have read here and saw in practice, my X time, with default settings, is around 15 minutes. In fact, my ESP could not connect to HA, which a reboot of HA is what fixed it, help me see it in this graph. You can see the stretch in dark blue when it was the same value for a long time. Right now the time is from the ESP and I am not worried about it being right or better said "set right", just that it is changing.

[swiergot]

@pvvx Just found this old issue. I'm trying to request the last 50 readings from ESP32 and I'm getting only 15-20 notifications each time (there are gaps in the sequence numbers) so it looks like some are lost. Could it be because of ESP32 dropping packets? Are you aware of any workarounds? I thought indications should solve the problem but apparently they are not supported by this firmware.

[pvvx]

After the "Read memory measures" command, the thermometer changes the "connection interval" to a faster one.
After sending the requested number of measurements, the "Connection latency" is restored.
Otherwise, reading 19632 measurements takes a very long time.

Default "Connection interval" is 20ms. And "Connection latency" is designed for 2.5 seconds.
As a result, in normal connection mode, packets follow on average every 2.5 seconds, and during "memory measures" transmission - 20 ms.

Changes in "Connection intervals/latency" are always consistent in the receiver and transmitter. If any of them do not agree with the change request, then the changes are not accepted.

• The minimum "Connection Interval" in the BLE standard is 7.5ms. But many Apple devices do not support this interval. And the current firmware uses 20ms.

The BLE in the ESP32 can receive all the packets, but not have time to process them at the top level of the software?
Enabled debugging in the UART on the ESP32 will provide a bunch of skips.

PS: When receiving BLE advertisement packets, if there are many BLE devices, then the receiver receives BLE advertisements every 3 ms. UART speeds in 115200 Baud and even USB1.1 are not enough to transfer information in binary form in the HCI format. Requires BT adapter with USB2.0 HS.

[swiergot]

Ok, so I tried to build the firmware with bls_pm_setManualLatency(24) in line 485, which should be 500 ms. But the transfer of records takes the same time as before.

I changed cmd_parser.c and ran make, then took ATC_Thermometer.bin. What am I missing?

EDIT: Just saw your edit. Perhaps ESP32 does not accept 500 ms?

[pvvx]

"Connection latency" does not affect the transmission interval from the device. "Connection latency" is the amount of time a device may not respond or accept requests. But it can transmit with the "Connection interval" step. The receiver still needs to be prepared to accept every "connection interval".
See the BLE standards for a more precise description.
I do not know English well and it is difficult for me to describe through Google translator...

The code sets bls_pm_setManualLatency(0) so that the chip wakes up every ""Connection interval" and everything is unambiguous. To increase the step, you need to change the "connection interval".

[pvvx]

In the "connection intervals" you can change the ESP32 by sending a request after connecting.

Additionally https://novelbits.io/ble-connection-intervals
https://microchipdeveloper.com/faq:279

[swiergot]

I can't see any option to change it after connecting:

https://docs.micropython.org/en/latest/library/bluetooth.html

Maybe MicroPython does not expose this API.

[swiergot]

Based on the links you provided, it seems that even setting a longer connection interval does not guarantee that packets will be transmitted slower, as multiple packets can be transmitted in a single connection interval.

[pvvx]

No more than the set MTU can be transmitted in one interval.
The size of one package depends on the version of BT.
Default maximum MTU size of 23 bytes -> one packet to support BT4.0.
As a result, 1 packet is transmitted during the "Connection interval" time.

You can request an increase in MTU during a connection. But that's not useful for these thermometers...

---

Arduno ESP32 Test...

I inserted a piece of code

		uint8_t blk[3];
        blk[0] = 0x35;
        blk[1] = 50;
        blk[2] = 0;
        Serial.printf("*** Get Memo\r\n");
        pRemoteCharacteristic->writeValue(blk,3); // get Memo

into the sketch for ESP32 for testing.
And I get the following logs via UART:

Found service '181a' data len: 19, 12161a18faf80538c1a4fefea122dc0a400a0f
MAC: a4c13805f8fa
Temp: -2.58°, Humidity: 88.65%, Vbatt: 2780, Battery: 64%, flg: 0x0f, count: 10
*** Get Memo
Time since boot: 246. New scan.
 ------ delta: 5006 ms, lost: 38, rx total: 44
Found service '181a' data len: 19, 12161a18faf80538c1a4fefea122dc0a400a0f
MAC: a4c13805f8fa
Temp: -2.58°, Humidity: 88.65%, Vbatt: 2780, Battery: 64%, flg: 0x0f, count: 10
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3501004ae97d63bb09f00d140c
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3502003de67d63a1090b0e1f0c
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 35030031e37d637e092a0e870c
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3504003ee07d636a09630e890c
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3505005add7d6321099a0e290c
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3506006eda7d6361093f0e330a
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3507003bea7d63f008370f3909
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 35080052e77d6306099b0f3409
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 35090082e47d630009b80f3709
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 350a00b2e17d63fa08be0f3409
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 350b00e2de7d63f308bc0f3109
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 350c0012dc7d63eb08940f3009
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 350d0042d97d63ef086d0f3109
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 350e0072d67d63f0089d0f3109
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 350f00a2d37d63ea088d0f3009
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 351000d1d07d63e0085f0f2f09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 35110001ce7d63f008d20f3109
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 35120031cb7d63ec0807103009
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 35130061c87d63e30818102f09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 35140091c57d63d50825102e09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 351500c1c27d63c30816102c09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 351600f1bf7d63b908dc0f2a09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 35170021bd7d63bc08b70f2b09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 35180051ba7d63bc089b0f2c09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 35190080b77d63cb0819102d09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 351a00b0b47d63c2082c102c09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 351b00e0b17d63bc083a102b09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 351c0010af7d63af083f102a09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 351d0040ac7d63a2084c102909
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 351e0070a97d63930861102809
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 351f00a0a67d63890867102709
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 352000d0a37d6379086e102a09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 35210000a17d63670879101e09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3522002f9e7d63570875101809
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3523005f9b7d63400861101409
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3524008f987d632e082d101409
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 352500bf957d633d081c101609
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 352600ef927d63580865101909
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3527001f907d635108bc101709
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3528004f8d7d633908a9101409
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3529007f8a7d632c086a101409
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 352a00af877d632b08bb101209
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 352b00de847d631908bd101009
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 352c000e827d630008b6100e09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 352d003e7f7d63ed0792100c09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 352e006e7c7d63e40783100c09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 352f009e797d63ea078a100c09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 353000ce767d63ed078a100c09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 353100fe737d63ec075c100d09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 13
data: 3532002e717d63ee074d100e09
Notify callback for characteristic 00001f1f-0000-1000-8000-00805f9b34fb of data length 3
data: 350000

50 measurements are transferred to the ESP32 in 12 seconds

(power СR2032 mA/ms)

ESP32 does not have time to work out and the thermometer transmits duplicates ... Battery...

[swiergot]

Funny, yesterday's night I also tried Arduino :) And indeed, results for me were also much better. However, my sketch (which is based on this example) has pClient->setMTU(517) whilst yours does not. When you mentioned MTU, I immediately thought this might be related, but since your sketch does not set it...

One more thing I noticed yesterday in Arduino is that it was skipping the first 7 packets, i.e. I was getting 43 instead of 50. I had to add a small delay after subscribing to notifications and before sending a request. It doesn't seem you are doing such thing in your sketch so something else must be going on. I will try different combinations.

[pvvx]

ESP32 cannot successfully handle BLE in Arduino at 160MHz CLK. Even at 240 MHz, packet acknowledgments are delayed, which greatly affects the batteries of the sensors.

TelinkMiFlasher.html read 50 measurements (0.2 sec):

(power СR2032 mA/ms)

---

When connecting, ESP32 requests the entire list of UUIDs each time, and does not take from memory from the previous connection. But further works on an index.
This behavior greatly increases the battery consumption of any standalone BLE device.
Normal devices remember the UUID indexes and, when reconnecting, immediately access the index number, without downloading the entire UUID database from the device for indexing.
(If the UUIDs on the device change, then there is a UUID for that with an indication that this has happened...)

PS: All of the above, if using ESP32, will result in a short life of the CR2032 in the sensor - up to a month, if not less...

[swiergot]

Excellent point. I don't intend to query the sensor during a normal operation, but only when if ESP32 for some reason has gaps in readings from advertisements. Besides, I suppose the library can be easily changed to have fixed service indices.

Big thanks for all your help!