Update rate and RuntimeError: Event loop stopped before Future completed.

[pvvx]

I need to receive all packages with the maximum data update rate.
But that doesn't work in ble_monitor.
In the working environment of radio broadcasting, BT adapters receive more than several hundred advertising packets per second from dozens of BLE sensors.
Setting the period to "0" in the log, 'ble_monitor' generates many errors (RuntimeError: Event loop stopped before Future completed) per hour ...

Does "Ble_monitor" cause BLE internal buffer overflows due to Python scripting performance or incorrect algorithms?
If, before calling ble_parser (), you enter a packet limit (discard most of it), reducing the number of transmitted packets to a couple per second, then no errors appear.
Tried various hardware platforms. Python3.9.5+, all packages are the latest versions.
Samples (the "line" numbers in the script 'ble_monitor' may be different - edited to clarify the problem.):

2021-05-24 02:14:12 ERROR (Thread-10) [root] Uncaught thread exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/root/.homeassistant/custom_components/ble_monitor/__init__.py", line 521, in run
conn[hci], btctrl[hci] = self._event_loop.run_until_complete(fac[hci])
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 640, in run_until_complete
raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.
2021-05-24 02:14:15 ERROR (Thread-11) [root] Uncaught thread exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/root/.homeassistant/custom_components/ble_monitor/__init__.py", line 521, in run
conn[hci], btctrl[hci] = self._event_loop.run_until_complete(fac[hci])
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 640, in run_until_complete
raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.

All BLE MESH packets on the network are also passed to the 'ble_parser()' procedure. BT-BLE adapters with UART completely fill the 1.5 Megabit connection...

[Ernst79]

@pvvx Let me first try to explain how BLE monitor works.

• In __init__.py we use the pypi package aioblescan to make a HCIdump of BLE advertisements.
• Each received BLE advertisement is passed to the ble_parser procedure (in \ble_parser\__init__.py), where we three or 4 bytes (including the UUID) to see if it is supported, the rest is ignored.
• If the bytes are supported, one of the parsers is used to translate the bytes to a dictionary with {measurement: value}, including things like RSSI, device type, etc. In this parser, a filter is applied to only process "new" BLE advertisements, based on the "counter" that is in each message. If it is the same, the advertisement will be ignored.
• This result is passed on the sensor.py or binary_sensor.py, where the HA sensors are created. In sensor.py we actually have two type of sensors, the "measuring" sensors that collect data for the set period, do some averaging over this period and update the result at the end of the period. The other type is more of an "instant" sensor (e.g. the remote sensors). These sensors will update their value on each received advertisement (although it still does the filtering of the "counter").

Now back to the problem, the RuntimeError: Event loop stopped before Future completed.. This error is an error that is caused by the aioblescan pypi package. The aioblescan package is called with aiobs in BLE monitor.

            for hci in self._interfaces:
                try:
                    mysocket[hci] = aiobs.create_bt_socket(hci)
                except OSError as error:
                    _LOGGER.error("HCIdump thread: OS error (hci%i): %s", hci, error)
                else:
                    fac[hci] = getattr(self._event_loop, "_create_connection_transport")(
                        mysocket[hci], aiobs.BLEScanRequester, None, None
                    )
                    conn[hci], btctrl[hci] = self._event_loop.run_until_complete(fac[hci])
                    _LOGGER.debug("HCIdump thread: connected to hci%i", hci)
                    btctrl[hci].process = self.process_hci_events
                    try:
                        self._event_loop.run_until_complete(btctrl[hci].send_scan_request(self._active))
                    except RuntimeError as error:
                        _LOGGER.error(
                            "HCIdump thread: Runtime error while sending scan request on hci%i: %s", hci, error
                        )

This part of the code is stopped and started each period. This code will tell aioblescan to set up a connection to a hci port and (further in the code) make a HCIdump and pass on each BLE advertisement in RAW format. This is done instantly, not at the end of the period at once. But the connection is closed and started over at the end of each period.

I think what happens is that if you set the period to 0 seconds, it will start a new connection before it was even able to close the previous connection, causing the mentioned Error.

I'm not sure what you exactly try to aim, but I think an "instant" sensor is what you are looking for. For these sensor, the 'period' isn't waited for, messages are updated right away.

Are you trying to do this for ATC firmware sensor? Temperature and humidity are measuring sensors, so the won’t update till the end of the period. But we could make an option to update instantly.

Actually, the reason that it doesn’t update instantly is because sensor sometimes send weird results, part of these spikes are filtered out already, and averaging filters out the rest.

[pvvx]

Is this averaging for the Xiaomi sensor? :)

How do you turn off your data averaging option?

Special firmware will support instant sensors. In the current version, the custom firmware XiaomiLYWSD03MMC reports several bits of binary switches and a trigger output to the GPIO and programmed to thresholds with histresis in temperature and temperature.

Trigger bits directly control other BLE devices as a fallback scenario when server hardware is de-energized and other options.
The transfer of the state of the reed switch is supported. This is DIY - requires soldering into a reed switch thermometer.

---

Averaging for advertising in encrypted Mi format is done by the thermometer itself according to the user's settings in it.
Everything is very good and without averaging data from the sensor:

In the ad-beacon of these sensors with various options for software and hardware registration of failures and incorrect data were not found. Correct the package parsing sources.

As a result, I started rewriting your 'ble_monitor' to correctly parse structures in BLE advertising packages ...

PS: XiaomiLYWSD03MMC and over, Custom firmware Ver 3.2 - Added new encrypted beacon formats, reed switch maintenance.

[Ernst79]

Is this averaging for the Xiaomi sensor? :)

First picture, this is a "bug" in the original firmware. In sensors.py there is a check for this increasing humidity behavior. There is a check for the device firmware, that sets a self._jagged flag to fix this behavior. For more info about this bug, see this post #7 (comment) As the firmware attribute is "Xiaomi (Encrypted)" and not "Xiaomi (MiBeacon V4/V5 encrypted)". it won't correct for this behavior. Try to change the firmware to "Xiaomi (MiBeacon V4/V5 encrypted)"

        # LYWSD03MMC / MHO-C401 "jagged" humidity workaround
        if devtype in ('LYWSD03MMC', 'MHO-C401'):
            if self._device_firmware == "Xiaomi (MiBeacon)" or self._device_firmware == "Xiaomi (MiBeacon V4/V5 encrypted)":
                self._jagged = True

How do you turn off your data averaging option?

At the moment, you can't. I will have to change that in the code.

Special firmware will support instant sensors. In the current version, the custom firmware XiaomiLYWSD03MMC reports several bits of binary switches and a trigger output to the GPIO and programmed to thresholds with histresis in temperature and temperature.

Trigger bits directly control other BLE devices as a fallback scenario when server hardware is de-energized and other options.
The transfer of the state of the reed switch is supported. This is DIY - requires soldering into a reed switch thermometer.

Averaging for advertising in encrypted Mi format is done by the thermometer itself according to the user's settings in it.
Everything is very good and without averaging data from the sensor:

I understand, but this is only the case for your optimized ATC firmware. BLE monitor has to deal with a lot more sensors, which turn out to send wrong data once in a while. But I will look into an option to disable averaging in the future.

In the ad-beacon of these sensors with various options for software and hardware registration of failures and incorrect data were not found. Correct the package parsing sources.

As a result, I started rewriting your 'ble_monitor' to correctly parse structures in BLE advertising packages ...

Ok, but I would propose to implement the updates in BLE monitor here. I'm totally open for improvements 😄

PS: XiaomiLYWSD03MMC and over, Custom firmware Ver 3.2 - Added new encrypted beacon formats, reed switch maintenance.

Will look into that as well.

[Ernst79]

@pvvx I like the changes you made to the ble_parser, I will implement this as well. But I will have to keep the compatibility with the MiScale and Kegtron sensors of course. This will make it easier to get your changes implemented.

[pvvx]

I haven't figured it all out yet in your ble_monitor.
I only use HA for tests. When 7 sensors are connected to it, 1..2 GiB of records are created on SD or eMMC per day.
Opening charts containing 8 thousand points per day takes a lot of time ...
All this needs to be fixed, otherwise it is not a working option. But there is no time for everything ...

How is it meant to include BT-MESH sensors processing in ble_monitor?

---

Ok, but I would propose to implement the updates in BLE monitor here. I'm totally open for improvements 😄

I don't know English well and I don't want to learn it. As a result, it's easier for me to describe everything in code ...

[pvvx]

First picture, this is a "bug" in the original firmware. In sensors.py there is a check for this increasing humidity behavior. There is a check for the device firmware, that sets a self._jagged flag to fix this behavior. For more info about this bug, see this post #7 (comment) As the firmware attribute is "Xiaomi (Encrypted)" and not "Xiaomi (MiBeacon V4/V5 encrypted)". it won't correct for this behavior. Try to change the firmware to "Xiaomi (MiBeacon V4/V5 encrypted)"

Different platforms, same sensor, same time:

Xiaomi Gateway 3 does not give out a fractional part of the moisture value from the original Xiaomi LYWSD03MMC:

[Ernst79]

Yes, that is exactly what this "jagged" option is doing, it removes the decimal part. But this isn't done in your case, as the code in sensor.py checks for firmware = "Xiaomi (MiBeacon)" or "Xiaomi (MiBeacon V4/V5 encrypted)", and your firmware is now called "Xiaomi (Encrypted)". So either rename firmware, or add "Xiaomi (Encrypted)" to the code.

        # LYWSD03MMC / MHO-C401 "jagged" humidity workaround
        if devtype in ('LYWSD03MMC', 'MHO-C401'):
            if self._device_firmware == "Xiaomi (MiBeacon)" or self._device_firmware == "Xiaomi (MiBeacon V4/V5 encrypted)":
                self._jagged = True

Here you can see the behavior in more detail, it adds 0,1% every step, till 0.9 and than starts over at 0.0 again.

[Ernst79]

Support for Encrypted ATC advertisements has been added in 2.7.0-beta. It also includes automatically selecting the advertisements with the highest accuracy, when sensor is set to send in all adv. formats.

Binary sensors (opening and switch) are not added yet, but are already in the output of the ble_parser. If you want these two sensors, you only have to add them in const.py. But I don't exactly understand what these two sensors actually report.

[pvvx]

Custom Firmware Xiaomi LYWSD03MMC options allow you to:

1. Xiaomi LYWSD03MMC has a PCB soldering test point. Available without disassembling the case and marked "Reset". Use as a trigger output with adjustable temp and / or humidity hysteresis. Hysteresis and thresholds are set in TelinkMiFlasher.html. The output status is displayed in the ad package.
2. It is possible to solder a reed switch on the LYWSD03MMC board to the pins marked "P10" and GND. The state of the reed switch is transferred to the advertising package.

[paralichko]

Is this averaging for the Xiaomi sensor? :) How do you turn off your data averaging option?

Special firmware will support instant sensors. In the current version, the custom firmware XiaomiLYWSD03MMC reports several bits of binary switches and a trigger output to the GPIO and programmed to thresholds with histresis in temperature and temperature. Trigger bits directly control other BLE devices as a fallback scenario when server hardware is de-energized and other options. The transfer of the state of the reed switch is supported. This is DIY - requires soldering into a reed switch thermometer.

Averaging for advertising in encrypted Mi format is done by the thermometer itself according to the user's settings in it. Everything is very good and without averaging data from the sensor:

In the ad-beacon of these sensors with various options for software and hardware registration of failures and incorrect data were not found. Correct the package parsing sources.

As a result, I started rewriting your 'ble_monitor' to correctly parse structures in BLE advertising packages ...

PS: XiaomiLYWSD03MMC and over, Custom firmware Ver 3.2 - Added new encrypted beacon formats, reed switch maintenance.

Have You achieved the opening sensor to work ?