tubeszb-cc2652-poe-2023 connection closed

[vobelic]

Hello,

I have a single HA/Z2M instance that is managing two physical locations. The HA/Z2M is at location1, while the coordinator is at location2.
Between the locations i've a low latency tunnel that is rock solid (500/500Mbps link) and is working for years without issues. Only caveat is once in 24hrs the tunnel goes down for 30-40seconds (ISP enforced session teardown).

When the tunnel is back, every 3-4th time Z2M loses the connection to the coordinator, Z2M goes down and can't restart.
What actually happens is that connections to coordinator on port TCP-6638 are reset.
I can immediately see that the issue is present if i try

% telnet x.x.x.x 6638
Trying x.x.x.x...
Connected to x.x.x.x
Escape character is '^]'.
Connection closed by foreign host.

Normally ofcourse telnet would remain connected and would spew gibberish during normal operation.
This happens only after the tunnel goes down/up, and never during the rest of the day, hence the connection and tunnel itself aren't an issue.

My assumption is that the coordinator ends up in some kind of loop when TCP connection isn't terminated gracefully and ever so often the process ends up in some kind of locked up state.
The only solution is to fully reboot the coordinator.

Haven't had the opportunity to take some packet traces as it's not really 100% reproducible, but i could if needed.

Is there any way to optimize this behavior on the coordinator or lead me through how to extract some debugging info, please?

Some stats (I've got Tuya based thermostat with debounce set to 1, and some window blinds modules):

Zigbee2MQTT version
    [1.34.0]
Coordinator type
    zStack3x0
Coordinator revision
    20221226
Frontend version
    0.6.147
Zigbee-herdsman-converters version
    15.130.1
Zigbee-herdsman version
    0.25.0
Stats
        Total 4
        By device type
        Router: 4
        By power source
        Mains (single phase): 4
        By vendor
        _TZ3000_1dd0d5yi: 3
        _TZE200_aoclfnxz: 1
        By model
        TS130F: 3
        TS0601: 1

Thanks in advance for any suggestions (aside the obvious to have a HA instance on both locations haha)!

[LordNex]

Hello,

I have a single HA/Z2M instance that is managing two physical locations. The HA/Z2M is at location1, while the coordinator is at location2.

Between the locations i've a low latency tunnel that is rock solid (500/500Mbps link) and is working for years without issues. Only caveat is once in 24hrs the tunnel goes down for 30-40seconds (ISP enforced session teardown).

When the tunnel is back, every 3-4th time Z2M loses the connection to the coordinator, Z2M goes down and can't restart.

What actually happens is that connections to coordinator on port TCP-6638 are reset.

I can immediately see that the issue is present if i try

% telnet x.x.x.x 6638

Trying x.x.x.x...

Connected to x.x.x.x

Escape character is '^]'.

Connection closed by foreign host.

Normally ofcourse telnet would remain connected and would spew gibberish during normal operation.

This happens only after the tunnel goes down/up, and never during the rest of the day, hence the connection and tunnel itself aren't an issue.

My assumption is that the coordinator ends up in some kind of loop when TCP connection isn't terminated gracefully and ever so often the process ends up in some kind of locked up state.

The only solution is to fully reboot the coordinator.

Haven't had the opportunity to take some packet traces as it's not really 100% reproducible, but i could if needed.

Is there any way to optimize this behavior on the coordinator or lead me through how to extract some debugging info, please?

Some stats (I've got Tuya based thermostat with debounce set to 1, and some window blinds modules):

Zigbee2MQTT version

    [1.34.0]

Coordinator type

    zStack3x0

Coordinator revision

    20221226

Frontend version

    0.6.147

Zigbee-herdsman-converters version

    15.130.1

Zigbee-herdsman version

    0.25.0

Stats

        Total 4

        By device type

        Router: 4

        By power source

        Mains (single phase): 4

        By vendor

        _TZ3000_1dd0d5yi: 3

        _TZE200_aoclfnxz: 1

        By model

        TS130F: 3

        TS0601: 1

Thanks in advance for any suggestions (aside the obvious to have a HA instance on both locations haha)!

My suggestion, and others may have a better solution is to A. Install a good quality mains powered Zigbee router at the location that does not have a coordinator. That should reinitialize the primary connection once the backhaul is reconnected. Or B. Have Home Assistant restart the ZHA/Z2M integration which should again force a resync of the network. Just my 2cp.

[vobelic]

Not sure we synced here...
The problem is that the coordinator at location2 goes berserk after the connection is broken (i.e. only remedy is to reboot it).
Adding a coordinator at location1 (2km away...) won't help restore the coordinator back to operational state at location2.

[tube0013]

Tunneling the connection is not recommended. Zigbee2mqtt and zha both require a constant serial connection.

My suggested solution is to use a Pi - even a 2 or 3 and set up a local zigbee2mqtt instance at the remote location and set up a tunnel for the mqtt connection which will be much more forgiving with disconnects.

[LordNex]

I completely agree. But technically a backhauled connection, as long as it's steady, should be just as good as having the coordinator attached to a separate switch. Obviously the wireless mesh is. It going to function and will not work as intended so I think your answer is by far a better solution. But the one thing I know is networking and Microsoft AD. And if he has the right equipment and the right routes in place. It should be doable, although not the best solution

The Aqara Zigbee Repeater does pretty well as it uses the same EFR32 chip that IMO your best device is made from. But at $30-35 he might as well just split the networks and send the commands over MQTT as you suggested. I just wish RPis were still that price.

[vobelic]

Tunneling the connection is not recommended. Zigbee2mqtt and zha both require a constant serial connection.

My suggested solution is to use a Pi - even a 2 or 3 and set up a local zigbee2mqtt instance at the remote location and set up a tunnel for the mqtt connection which will be much more forgiving with disconnects.

I agree with you there, it's not an ideal setup as you say. And it's probably the easiest to just install a Z2M instance somehow at the other location, and I'll most probably pursue that road.

But still, I would strongly argue that dropped serial connection for any reason should NOT kill the coordinator itself - only the HA/Z2M should lose the connection and Z2M should be impacted, but that's it.
I would like to look at this issue without being biased by the tunnel topic - the car engine shouldn't just fall off because the car hit a bump on the road, now should it?

[LordNex]

Again not the best solution, if you could setup a local ping integration to monitor the backhaul. Probably best by querying the coordinator from the remote side depending on what hardware is available there.

If it drops x amount of packets over x amount of time. Have an automation that restarts just z2m. It would still take a few min to reestablish and communicate again, but minus doing a local install, would be the easiest and cheapest resolution.

But they are both correct. It's not the best way and could cause issues so YMMV

[vobelic]

I just realized, if i were to have a mqtt broker at location2, then I would end up in the same problem when I install a coordinator on location1.
AFAIK on HA you can have only one MQTT integration.
Apparently running two full HA instances, i.e. separate two instances for location1 and location2 seems the way to go.
My issue is that I need some HW on location2 which I wanted to avoid, oh well.

[LordNex]

I just realized, if i were to have a mqtt broker at location2, then I would end up in the same problem when I install a coordinator on location1.

AFAIK on HA you can have only one MQTT integration.

Apparently running two full HA instances, i.e. separate two instances for location1 and location2 seems the way to go.

My issue is that I need some HW on location2 which I wanted to avoid, oh well.

Depending on your network hardware you might be able to create a VPN between the two sites. Huge fan of Firewalla.com for making things just so easy. And technically you could run another docker container on the Firewalla with an MQTT broker or something there. Might even be able to run a minimal install of ZHA with the proper dongle and drivers. I've seen people do some pretty crazy things with them. Me I just want a really fast rock solid Firewall/IDS/IPS. You wouldn't believe how much internet cesspool hits my WAN. Just can't trust any old router anymore.