basic spam control / throttle

[ivanfmartinez]

There are some reports of devices that are spamming on network.

Some examples :

• Debounce not working for exact duplicate messages in the same second. #17984
• [New device support]: TS0601 _TZE200_lvkk0hdg (EPTTECH Tank Level Monitor Zigbee) #21015

In my case the spamming devices are ambient sensors and water level sensors, that are sending multiple messages per minute, some send one message per second, and the values does not change at this period and can be just ignored.

We cannot do nothing about the zigbee network, but at least we can reduce the overhead on MQTT and storage on homeassistant

This patch just ignore messages that came from device faster than a min_elapsed value.

This case is different from the debounce that combine messages and send after a specified no messages time.

@Koenkk I dont think that is the best solution, but was the easiest that I found with little changes, I used the existing elapsed variable to control the time.

All tests are passing

[Koenkk]

This case is different from the debounce that combine messages and send after a specified no messages time.

I want to understand better why debounce does not work for this use case, could you elaborate on this?

[ivanfmartinez]

This case is different from the debounce that combine messages and send after a specified no messages time.

I want to understand better why debounce does not work for this use case, could you elaborate on this?

@Koenkk as I have explained before at : #17984 (comment)

debounce expects that will be a "SILENT" time where the sensor does not send messages

https://www.zigbee2mqtt.io/guide/configuration/devices-groups.html#common-device-options

Debounces messages of this device. When setting e.g. debounce: 1 and a message from a device is received, Zigbee2MQTT will not immediately publish this message. But it will combine it with other messages that are received max 1 seconds apart from each other. So there needs to be "N second of silence" from the device before the combined message is sent out.

For example debounce: 5 means that there needs to be "5 seconds of silence" from the device, before combined messages is sent out.

This is handy for e.g. the WSDCGQ11LM which publishes humidity, temperature and pressure at the same time but as 3 different messages.

Don't configure debounce to be higher that sensors update interval as that would cause all the messages to be debouncer and messages won't be sent out at all.

The condition here is that the device KEEPs sending sensor data (water level, ambient temperature, humidity) each second (sometimes more than one message per second). No silence from the sensor and keep sending messages to MQTT. This change will ignore messages for some time after one message is processed/sent to mqtt.

These sensors (water level, temperature, humidity) does not change so fast to need the values to be sent each second. Then I get first message, discard received messages for specified time, accept again and ignore more messages again.

This reduces MQTT load, zigbee2mqtt web ui responses better, and homeassistant database become smaller.

Before doing this my homeassistant db was over 40GB with less than 10 days storage. Now I have 20 days of storage and my db have less than 7GB.

The change allow user to activate this "ignoring" behaviour per device using a min_elapsed configuration option, or just putting "SPAMMER" in the device description to use a default 30 second min_elapsed.

Of course this change will not work if a sensor that have this behaviour also have something like actions that must be sent. But as I have seen in many issues the people that have my same problem have metering sensors.

In some cases people can just change sensors that will be better for the health of zigbee network. But there are cases that we dont have much options. The only zigbee ultrasonic water level sensor that I can purchase is one of the spamming devices.

I don't have much knowledge of how zigbee2mqtt works, neither JS programming and maybe you can have a better generic solution. But many people are having problems with this case that are not considered yet in z2m, the issues that I have listed are just some issues that I found that appear to be related to this case.

[Koenkk]

I think we should introduce a throttle option, this library can be used for that.

[ivanfmartinez]

I think we should introduce a throttle option, this library can be used for that.

@Koenkk this appear to be correct way to use this library ?

The result will be a little different from my old code, now it will send the last message if the device stops, appear to be a better behaviour than with my code.

[Koenkk]

Thanks!

[mariusvr]

Just wanted to say thank you. I have motion sensors that can spam mqqt 5x per second. Is there a way I can test this or when will it be included in the main release? Cheers, Marius

[ivanfmartinez]

Just wanted to say thank you. I have motion sensors that can spam mqqt 5x per second. Is there a way I can test this or when will it be included in the main release? Cheers, Marius

@mariusvr https://www.zigbee2mqtt.io/advanced/more/switch-to-dev-branch.html

[mariusvr]

Thanks have the latest dev version now. How do I configure the throttle parameter? Can't see it in the front-end and I get an error if I try adding it to the YAML config file under my sensor config. Cheers, Marius

[happyjaxx]

Hi,

This could be a life saver !

I'm trying to diagnose why HA+Homebridge was draining my iPhone and AppleWatch when I remembered my tuya based power meters TS0601 _TZE204_ac0fhfiq (got one at grid entry and on the solar farm) tend to spam the network alot (and HA might be pushing updates too much, homebridge was nicer but disfunctionnal 🤷🏻‍♂️)
That spamming also makes clicking in the UI pretty tough as the page is continuously redrawn over and over again.

So, to minimize the noise, went ahead, updated Z2M to 1.41.0 (using the update.sh script), edited the configuration.yaml:
'0xa4c13873b945ccd4':
friendly_name: power_meter_solar
homeassistant: {}
optimistic: true
throttle: 5

restarted Z2M...
But they still keep pouring in (receive.ts is up to date, the main branch is looking good)

Did I miss anything?

Thanks,
JaXX./.

[ivanfmartinez]

@happyjaxx your config indicates power_meter_solar and your screenshot indicated power_meter_main

[happyjaxx]

Yup, I have two exact same Tuya meters, one at power grid arrival, the second on the solar panels lead, they both have the exact same symptom, and I set the throttle on both of course

  '0xa4c138c1700ba5ee':
    friendly_name: power_meter_main
    throttle: 5
    homeassistant: {}
    optimistic: true
  '0xa4c13873b945ccd4':
    friendly_name: power_meter_solar
    homeassistant: {}
    optimistic: true
    throttle: 5