bluez: Linux kernel always de-duplicates advertisements

[dhalbert]

• bleak version: 0.7.0
• Python version: 3.8.x
• Operating System: Linux
• BlueZ version (bluetoothctl -v) in case of Linux: 5.53

One issue with scanning using bluez is that the current Linux kernel support always turns on de-duplication of advertisements. This makes it difficult or impossible to use multiple advertisements to transmit changing data in a connectionless way. See this thread in the linux-bluetooth mailing list. @tannewt first encountered this while working on Adafruit_CircuitPython_BLE_BroadcastNet.
Around line 875 (right now) in https://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next.git/tree/net/bluetooth/hci_request.c:

		ext_enable_cp.filter_dup = LE_SCAN_FILTER_DUP_ENABLE;

@dlech noted:

...it looks like BlueZ is getting better advertisement support: https://github.com/bluez/bluez/blob/master/doc/advertisement-monitor-api.txt - might be good to make sure it does what is needed while it is still experimental)

In Adafruit_Blinka_bleio, we have gotten around this for now by using hcitool or bluetoothctl to initiate scanning, and hcidump to look at the raw data. However, this requires adding privileges to those commands. See our README for more details. The code is here, in particular _use_hcitool() and _start_scan_hcitool().

(Originally mentioned in #206 (comment) and #230 (comment))

[bojanpotocnik]

@dhalbert thank you for this post. I was developing and testing my application on ThinkPad laptop and everything worked as expected, but when deployed on Raspberry Pi, not all packets were visible in bluetoothctl. But hcitool lescan --duplicates worked properly, as you described (and as in SO question Disable “Filter duplicates” setting for LE Set Scan Enable command - I also posted there).
I don't know why is it working as expected on my laptop, but after two days of frustration, I finally got on the right track and discovered this post.

This is my workaround: After scanning is started in bleak/bluetoothctl, I execute:

# First, disable ongoing scan enabled by bluetoothctl - if this is not executed
# then next command to enable scanning will result in Command Disallowed (0x0c)
# status. Fortunatelly, bluetoothctl seems not to be bothered by execution of
# this commands.
hcitool cmd 0x08 0x000C 0x00 0x00
# This results in
# < HCI Command: LE Set Scan Enable (0x08|0x000c) plen 2
#         Scanning: Disabled (0x00)
#         Filter duplicates: Disabled (0x00)
# > HCI Event: Command Complete (0x0e) plen 4
#       LE Set Scan Enable (0x08|0x000c) ncmd 1
#         Status: Success (0x00)
# Now, enable scanning with duplicate filtering disabled
hcitool cmd 0x08 0x000C 0x01 0x00
# This results in
# < HCI Command: LE Set Scan Enable (0x08|0x000c) plen 2
#         Scanning: Enabled (0x01)
#         Filter duplicates: Disabled (0x00)
# > HCI Event: Command Complete (0x0e) plen 4
#       LE Set Scan Enable (0x08|0x000c) ncmd 1
#         Status: Success (0x00)

However, one additional problem still persists: every 11 seconds, MGMT Command: Start Service Discovery will be executed, causing the following series of commands:

< LE Set Scan Enable (Scanning: Disabled (0x00))
< Inquiry (General Inquiry)
< LE Set Random Address (Non-Resolvable)
< LE Set Scan Parameters (Active (0x01), Interval: 11.250 msec)
< LE Set Scan Enable (Scanning: Enabled (0x01), Filter duplicates: Enabled (0x01))

which causes filters to be enabled again. I cannot get around those 10 seconds even by executing e.g. (sudo btmgmt):

[mgmt]# scan-params 60 60

[emece67]

• bleak version: 0.20.2-1
• Python version: 3.11.2
• Operating System: Linux (uname -r gave 6.1.21-v8+)
• BlueZ version (bluetoothctl -v) in case of Linux: 5.66

Hi,

Just a newcomer into Bleak/BlueZ, but I think I was hit by this same issue. I'm monitoring some Xiaomi BLE thermometers whose firmware was switched to that of pvvx Victor. With this new firmware such thermometers send the measured temperature/humidity/battery level on their advertisement data, and, in order to save battery, they are supposed to be scanned passively.

On Windows I'm able to monitor them using a callback registered when calling the bleak.BleakScanner constructor. The callback is invoked each time an advertisement from these thermometers is received, no deduplication.

But on Linux (on a Raspberry Zero 2W) it is mandatory to pass argument or_patters to BlueZ when passively scanning. I use or_patterns = [OrPattern(0, AdvertisementDataType.SERVICE_DATA_UUID16, b"\x1a\x18") to match the UUID of these thermometers, but then the callback is only called once for each thermometer, there is deduplication.

Passing a filters argument to Bluez setting or resetting DuplicateData does not solve the issue. This was expected as, in some place, docs say this parameter is ignored when passively scanning.

But, being the scan started, doing: sudo hcitool cmd 0x08 0x000C 0x00 0x00; sudo hcitool cmd 0x08 0x000C 0x01 0x00 (as stated above) immediately caused no deduplication, all received advertisements invoke the callback, and my monitoring script works OK forever (not any issue every 11 seconds).

Also, actively scanning in Linux didn't show the issue, so it seems restricted to passive scanning in Linux.

Hope this helps, thanks and regards.

[emece67]

Hi again,

I have just noticed that this de-duplication (on Linux —Debian Bookworm— on a Raspberry Pi Zero 2 W) when passively scanning does not happen using an external Bluetooth dongle (an rtl8761 based one). Curious…

Regards.