Gateway Cascade

[ebaauw]

I've got two Pi-s, one production and one test, both with a RaspBee running deCONZ and homebridge-hue on the same (wired) subnet. This afternoon I upgraded both to v2.04.91. Since then, I'm seeing some disturbing log messages that I haven't seen before. On my production Pi:

15:00:00:001 API error 1, api/2c34bcf4-c/groups, unauthorized user
15:00:05:002 API error 1, api/2c34bcf4-c/config, unauthorized user
15:00:10:002 API error 1, api/2c34bcf4-c/groups, unauthorized user
15:00:15:001 API error 1, api/2c34bcf4-c/config, unauthorized user
15:00:20:002 API error 1, api/2c34bcf4-c/groups, unauthorized user
15:00:25:002 API error 1, api/2c34bcf4-c/config, unauthorized user
15:00:30:002 API error 1, api/2c34bcf4-c/groups, unauthorized user
15:00:35:001 API error 1, api/2c34bcf4-c/config, unauthorized user
15:00:40:002 API error 1, api/2c34bcf4-c/groups, unauthorized user
15:00:45:002 API error 1, api/2c34bcf4-c/config, unauthorized user
15:00:50:001 API error 1, api/2c34bcf4-c/groups, unauthorized user

and

15:00:00:605 unhandled http status code in connected state 403 switch to offline state
15:00:10:606 unhandled http status code in connected state 403 switch to offline state
15:00:20:604 unhandled http status code in connected state 403 switch to offline state
15:00:30:604 unhandled http status code in connected state 403 switch to offline state
15:00:40:605 unhandled http status code in connected state 403 switch to offline state
15:00:50:603 unhandled http status code in connected state 403 switch to offline state

These message have been appearing every 10 seconds.

Same on my test pi:

15:00:00:607 API error 1, api/5728cec5-0/groups, unauthorized user
15:00:05:607 API error 1, api/5728cec5-0/config, unauthorized user
15:00:10:607 API error 1, api/5728cec5-0/groups, unauthorized user
15:00:15:609 API error 1, api/5728cec5-0/config, unauthorized user
15:00:20:607 API error 1, api/5728cec5-0/groups, unauthorized user
15:00:25:607 API error 1, api/5728cec5-0/config, unauthorized user
15:00:30:607 API error 1, api/5728cec5-0/groups, unauthorized user
15:00:35:609 API error 1, api/5728cec5-0/config, unauthorized user
15:00:40:606 API error 1, api/5728cec5-0/groups, unauthorized user
15:00:45:609 API error 1, api/5728cec5-0/config, unauthorized user
15:00:50:607 API error 1, api/5728cec5-0/groups, unauthorized user
15:00:55:609 API error 1, api/5728cec5-0/config, unauthorized user

I don't have these usernames configured anywhere. I double-checked that no rules are owned by these users, and that no schedule command contains these.

Apart from the deCONZ webapp and homebridge-hue I don't use any deCONZ clients. I wanted to double-checked using netstat that there's no weird connections to the port (80) where the webapp runs, but it would seem that the two deCONZ instances keep a connection to each other. Indeed, the messages on the production Pi stop, when quitting deCONZ on the test Pi, and resume when starting deCONZ on the test Pi.

What is happening here?!

[manup]

yes deconz tries to discover other gateway in the same network. The reason behind this is a feature called gateway cascades which allows for example forward a switch event to another gatway.

I can can add a commandline switch to disable it. Or at least prevent the debug output 🙂

[ebaauw]

Or at least prevent the debug output

LOL. I'd rather they take the hint that the usernames aren't configured.

I can can add a commandline switch to disable it.

Or... (if I may suggest a crazy idea ;-) you could document the feature? Seriously, this could be very useful to break a larger ZigBee network into two to reduce the polling time. I assume the cascading gateways are supposed to run at different ZigBee network channels and PAN IDs? Would they forward CLIP events as well?

[manup]

LOL. I'd rather they take the hint that the usernames aren't configured.

The implementation hasn't been touched in a while and is fairly basic to forward only group/scene commands. If I recall correctly the periodic polling is to check if the other gateway(s) are reachable, regardless of a configured username.

There is a hidden page in the webapp: http://<gateway-ip>/edit_gateways.html were the connection can be startet.

Or... (if I may suggest a crazy idea ;-) you could document the feature?

Totally agree, documentation in general is currently a real pain point and one of the next bigger tasks in early 2018.

Seriously, this could be very useful to break a larger ZigBee network into two to reduce the polling time. I assume the cascading gateways are supposed to run at different ZigBee network channels and PAN IDs?

That's exactly what it is used for, we have larger projects with > 500 nodes where multiple gateways and their devices are connected by this feature. Every gateway has its own ZigBee network settings they are sometimes not even in radio signal range but only connected by IP/LAN.

Would they forward CLIP events as well?
I think it can and should be extended so that every API request can be forwarded: which is basically providing webhooks in the API.

You can see current state in the here:
https://github.com/dresden-elektronik/deconz-rest-plugin/blob/master/rest_gateways.cpp

[ebaauw]

OK, after some experimentation and code reading, I managed to setup:

$ ph_get /gateways
{
  "1": {
    "cascadegroups": [
      {
        "local": "19250",
        "remote": "2"
      }
    ],
    "groups": {
      "2": "Test"
    },
    "ip": "192.168.xxx.xxx",
    "name": "pi",
    "pairing": true,
    "port": 80,
    "state": "connected",
    "uuid": "00212EFFFFxxxxxx"
  }
}

I had to create the username on the remote gateway manually. After that, it retrieved the groups from the remote gateway. I created the cascadegroup through the REST API, the local group is linked to one of my IKEA Trådfri remotes. When I press the On/Off button, I see in the local log:

Dec 20 22:31:28 pi2 deCONZ[1099]: 22:31:27:551 GW pi forward command 0x02 on cluster 0x0006 on group 0x4B32 to remote group 0x0002

That looks promising, however, in the log of the remote gateway, I see

Dec 20 22:31:29 pi deCONZ[845]: 22:31:27:608 Content not completely loaded (got 0 of 0), wait 20ms
Dec 20 22:31:29 pi deCONZ[845]: 22:31:27:628 Content not completely loaded (got 0 of 0), fetch rest [2]
Dec 20 22:31:29 pi deCONZ[845]: 22:31:27:651 API error 2, /groups/2/action, body contains invalid JSON

Now that's interesting: outgoing, deCONZ forwards ZigBee group commands, but incoming, it expects an API call with JSON? That's not going to work - there's no API equivalent for the Toggle command (0x02) of the OnOff cluster (0x0006).

When issuing ph_put /groups/19250/action '{"on":true}', it looks better. The local gateway reports:

Dec 20 22:42:31 pi2 deCONZ[1099]: 22:42:31:030 GW pi forward command 0x01 on cluster 0x0006 on group 0x4B32 to remote group 0x0002

and the remote gateway:

Dec 20 22:42:31 pi deCONZ[845]: 22:42:31:082 Content not completely loaded (got 0 of 0), wait 20ms
Dec 20 22:42:31 pi deCONZ[845]: 22:42:31:102 Content not completely loaded (got 0 of 0), fetch rest [2]
Dec 20 22:42:31 pi deCONZ[845]: 22:42:31:131 add task 21 type 14 to group 0x0002 cluster 0x0006 req.id 153
Dec 20 22:42:31 pi deCONZ[845]: 22:42:31:131 APS-DATA.request id: 153, addrmode: 0x01, addr: 0x0002, profile: 0x0104, cluster: 0x0006, ep: 0xFF queue: 0 len: 3
Dec 20 22:42:31 pi deCONZ[845]: 22:42:31:231 Erase task req-id: 153, type: 14 zcl seqno: 1 send time 0, profileId: 0x0104, clusterId: 0x0006
Dec 20 22:42:31 pi deCONZ[845]: 22:42:31:231 APS-DATA.confirm id: 153, status: 0x00 SUCCESS

Bingo!

[ebaauw]

Plugged in my Hue bridge. It's discovered as a gateway by deCONZ. Hacked the deCONZ database to provide a valid Hue bridge username (the Hue API no longer allows to specify a username when creating a user) and it seems to work - at least the groups from the Hue bridge are retrieved and group commands are forwarded:

$ ph_get /gateways/2
{
  "cascadegroups": [
    {
      "local": "19250",
      "remote": "1"
    }
  ],
  "groups": {
    "1": "Living room"
  },
  "ip": "192.168.xxx.xxx",
  "name": "philipshue",
  "pairing": true,
  "port": 80,
  "state": "connected",
  "uuid": "001788FFFExxxxxx"
}

Dec 21 08:41:52 pi2 deCONZ[6856]: 08:41:49:149 GW pi forward command 0x02 on cluster 0x0006 on group 0x4B32 to remote group 0x0002
Dec 21 08:41:52 pi2 deCONZ[6856]: 08:41:49:149 GW philipshue forward command 0x02 on cluster 0x0006 on group 0x4B32 to remote group 0x0001

Of course this will break on API incompatibilities for scenes (see #29), but otherwise I see a lot of possibilities.

• I could offload all my Hue lights to the Hue bridge, reducing the polling time needed by deCONZ. The OSRAM, IKEA, and ubisys lights support attribute reporting, so deCONZ only needs to poll my 5 innr lights. I'd keep all my sensors, switches, rules, and schedules on deCONZ and cascade to the Hue bridge the group commands to interact with the Hue lights. I need to test how stable my network would be, in particular if the remaining non-Hue lights provide enough network coverage for all my sensors and switches to connect.
• By exposing both the deCONZ gateway and the Hue bridge to HomeKit (either natively or through homebridge-hue), I'd still have a complete view in HomeKit. I'd probably use the native v2 Hue bridge HomeKit feature, since that pushes changes to HomeKit, rather than having homebridge-hue poll the bridge;
• I might even dust off my old v1 Hue bridges and distribute the Hue lights amongst these and the v2 bridge, to reduce further the polling time on the Hue bridge. Of course, I'd need homebridge-hue to expose the lights connected to the v1 bridges.
• With the reduced polling time, state.reachable would reflect more timely the actual reachability of the lights, enabling workarounds (in homebridge-hue) for the unholy power-on defaults. For workarounds in the gateway (or Hue bridge) rules, we'd need rule support for state.reachable (see Enable the use of the reachable attribute in rules #170);
• Since the Hue lights are on the Hue bridge, they'd receive their firmware updates automatically;
• Since the Hue lights are on the Hue bridge, they could be used for Hue Entertainment (issue Philips Hue Entertainment #167);

[ebaauw]

OK, I moved two of my Living Room lights to the Hue bridge, and setup the cascade from the Hue dimmer switch group on the deCONZ gateway to the Hue bridge. Had to patch gateway.cpp, since the Hue dimmer sends Off with effect (0x40) instead of plain Off (0x00). While doing that, fixed the invalid cascade of Toggle. See PR #338.

When switching the lights on with the Hue dimmer switch or through the API, the delay as hardly noticeable, comparable to switching the lights on through a rule on the gateway. When switching the lights off, the delay is somewhat more noticeable.

I grouped four lights connected by deCONZ and exposed through homebridge-hue and two lights exposed by the Hue bridge in one HomeKit group. When toggling the HomeKit group, all six lights react simultaneously.

[ebaauw]

My gateway rules control the lights through scene recall and off commands. Both are now cascaded to the Hue bridge in full. I map the deCONZ scene /api/username/groups/12/scene/1 to /scenes/g12s1 on the Hue bridge. While they're deprecated, you can still create v1 scenes, where you can specify the scene ID, e.g.:

ph_put /scenes/g1s1 '{"name": "s1g1", "lights":["1","2"]}'

The Hue dimmer switch in direct mode is now fully functional as well, controlling four lights connected to the deCONZ gateway and two connected to the Hue bridge. On, Off, DimUp, DimDown are all cascaded to the Hue bridge and work with very acceptable response times. Note that you need bri_inc for DimUp and DimDown, so you cannot yet cascade these to another deCONZ gateway (see #84). Note that, while lights connected to the local network can still be controlled when the gateway is down, the cascade only works when both the gateway and Hue bridge are up.

Support for the IKEA remote in direct mode is harder. There's no equivalent for Toggle in the REST API. Either we need to add that (but that will work only for deCONZ), or check the value of state.all_on to decide whether to turn the remote lights on or off. Not sure about the timing, though. I think the gateway listens in to the commands sent to the ZigBee network, so state.all_on might or might not already be updated. The DimUp and DimDown buttons send different Move and Step commands on press vs hold vs release, with of without On/Off. Tedious but doable. Mapping the manufacturer-specific scene commands for Previous and Next to ct_inc seems tedious but doable as well.