Future Development of Homebridge Hue

[ebaauw]

UPDATE DEC 2023
UPDATE AUG 2023

Homebridge Hue is my oldest plugin. I started working on it in February 2016, and published the first version on GitHub in October 2016. I added support for deCONZ in the summer of 2017, including web socket notifications. In February 2020, I added HTTPS support for the gen-2 Hue bridge, followed by API v2 push notifications in December 2021.
Over that period, the number of different devices supported by Homebridge Hue has increased dramatically, fuelled by the consumerisation of Zigbee devices. Starting with (then) Philips Hue, OSRAM Lightify, and IKEA Trådfri, followed by Xiaomi, and many OEM devices, and even devices from Lidl and Aldi.
The code base has become increasingly complex over the years, and is due for a refactor. Homebridge Hue still uses Homebridge's traditional static platform accessory model (see #4), where most recent plugins use the dynamic platform accessory model, which offers many benefits. The Hue and deCONZ APIs have been diverging, and their v2 APIs are completely different, making it increasingly hard to combine Hue bridge and deCONZ gateway support in a single plugin.

Homebridge Hue is also my most used plugin. I don't collect any statistics, but the NPM registry mentions up to 8,000 weekly downloads, and GitHub Insights lists over 3,400 unique visitors.
Homebridge Hue is also, by far, my largest plugin; I'm exposing over 140 accessories from my deCONZ gateway. I would take me half a day to reconfigure HomeKit if those accessories would be removed from HomeKit and re-exposed.
Because of this, I've been hesitant to refactor Homebridge Hue. While I can technically manage to expose my accessories using a dynamic platform plugin, without HomeKit losing them, this is tricky and requires a deep understanding of the internal working of Homebridge and HAP-NodeJS. For me, a big-bang update would be out of the question, as I think for most users.

Going forward, I will create a new plugin for deCONZ, homebridge-deconz, and a new plugin for Hue APIv2, homebridge-hue2, while keeping the current Homebridge Hue plugin. These three plugins can be run in parallel, using different child bridges, connecting to the same Hue bridge(s) and/or deCONZ gateway(s). This allows you to setup a new HomeKit home for testing. By connecting the child bridges for all plugins to the same Home, white- and blacklisting can be used to move accessories over to the new plugins one-by-one. This will also allow me to release work-in-progress versions of the new plugins, not yet supporting all device types.
Eventually the current Homebridge Hue plugin will drop support for the gen-2 Hue bridge and the deCONZ gateway. After that, I might release a new version with the dynamic platform accessory model. I don't think there are enough users left using the gen-1 Hue bridge or HA-Bridge or Tasmote to justify the effort needed to provide a gradual migration path.

[HARDIK-TSH1392]

Hey, I am new to open source and I am finding it really difficult as to how to start doing anything. I just came across your project, could you guide/explain as to what should I do because I am interested in contributing to open source, so taking your project as a start can you help me contribute?

[ebaauw]

@HARDIK-TSH1392 better start with something far less technically complex.

[sschuste]

Wow. That's a lot of work you have to do. So much code and I'm only using it for two devices. And I don't have enough programming skills to contribute anything essential. If you're ever in Frankfurt am Main, your dinner's on me. But maybe you have some better ideas how we can support you.

[Olepir]

@ebaauw thank you for your job

my whole smart home is based on you plugin

can’t wait for your next project

[vvdheuvel]

Hi Erik, wise decision! 🙏🏻 I think this will take out some complexity and better focus on both platforms. When do you suspect it will stable for usage? Will there be a possibility to migrate?

[ebaauw]

Homebridge deCONZ should be pretty stable by now.

I cannot opaquely move accessories from Homebridge Hue to Homebridge deCONZ or Homebridge Hue2, without HomeKit noticing. In theory this would only be possible if all plugins use the same Homebridge installation (without child bridges). As Homebridge deCONZ and Homebridge Hue2 generate the same accessory UUID as Homebridge Hue, HomeKit wouldn't care which plugin handles the accessory (and doesn't even know), as long as it's exposed by the same HomeKit bridge (HAP server). However, only one plugin can expose that accessory, or you would see the dreaded "cannot add another accessory with the same UUID" message.

Homebridge Hue still uses the (ancient) static platform accessory model, where all accessories need to be declared to Homebridge, before it starts the HAP server. Homebridge deCONZ and Homebridge Hue2 use the (newer) dynamic platform accessory model, where a plugin can add (and remove) accessories at runtime, after Homebridge has started the HAP server. So I would need to restart Homebridge for Homebridge Hue to no longer expose an accessory, but Homebridge deCONZ and Homebridge Hue2, can only re-create that accessory, after the HAP server has started. This would leave a small window where the accessory isn't exposed, causing HomeKit to remove the accessory, and to treat the re-created accessory as a new one. I did have a hack in homebridge-lib, to delay the startup of Homebridge from a dynamic platform plugin, but that's too complex to share and no longer works with child bridges. I discussed with the team to include this feature in Homebridge, but it was deemed too dangerous, as too many users wouldn't understand why Homebridge wouldn't be visible in HomeKit, even though it is running.

To make a long story short: migration consists of removing the accessories by Homebridge Hue from HomeKit, and re-creating them by Homebridge deCONZ or Homebridge Hue2, losing any associations to HomeKit rooms, groups, scenes, automations. This sucks, but then, so does life. To mitigate the impact, I support manual migration per accessory, so you don't have to handle all the re-configuration at once. And it also allows me to release Homebridge deCONZ and Hue2 without full support for all devices.

Make sure to run Homebridge deCONZ and/or Homebridge Hue2 on a separate HAP server than Homebridge Hue. I strongly recommend to run the new plugins in their own child bridge. I would pair it to a new test home at first, to familiarise yourself with it, and to check which accessories you feel are production ready. Then un-expose all accessories, remove the child bridge from the test home, and pair it to your production home.

In the production home, select an accessory to expose. In HomeKit, you'll now see two versions of the same accessory, allowing you to add the new accessory to the same room, groups, scenes, and automations, while still being able to see how the old accessory is configured. Homebridge deCONZ now creates a migration resource link, containing all the resources it exposes. Homebridge Hue2 will do the same. Homebridge Hue can be configured to take into account these resource links (through the new homebridgeDeconz and homebridgeHue2 config.json keys). When you're satisfied with the config for the new accessory, restart Homebridge Hue, and the old accessory is removed.

Note that Eve stores the history under the accessory UUID, so the old and new accessories share the history, causing double entries. For accessories with Eve history, you want to make sure to load the history from the old accessory, before exposing the new one, and to remove the old accessory shortly after exposing the new accessory.

[jan666]

Oh, that sounds really horrible.

I guess its safe to just use the old homebridge-hue plugin forever? Re-doing everyhing in the Home App will be a real pain

[ebaauw]

Yes, that is horrible. That's why I had been delaying the refactor of Homebridge Hue for so long, up to the point where it's no longer sustainable to maintain it.

I cannot prevent that pain, unfortunately, but I won't force you when to take it, nor to take all of it at once. I will stop adding new devices and features in Homebridge Hue pretty soon, after both Homebridge deCONZ and Homebridge Hue2 are ready for mainstream production. I will eventually release a version 1.0 of Homebridge Hue that no longer supports the deCONZ gateway nor the gen-2 Hue bridge, but not within the next six months. Of course, you can continue to run (and even re-install) the 0.x version of Homebridge Hue.

[Ar-eh]

Does using an app like "Controller for HomeKit", with the ability to backup a HomeKit configuration and restore from that backup, address the migration difficulties? I've found that when migrating between Homebridge Hue and the HomeKit integration in HomeAssistant, the backup does not work perfectly due to differences in how accessories are exposed. But perhaps it would be smoother in migrating from Homebridge Hue to HomeBridge deCONZ?

[ebaauw]

I really don’t know.

[jan666]

@ebaauw can you provide a "migration guide"?

And do you think https://controllerforhomekit.com/features/backup-restore/ can help doing the migration? https://controllerforhomekit.com/2021/06/22/how-to-replace-an-accessory/

[ebaauw]

As I said before: I really don’t know. I have no experience with Controller for HomeKit.

I’m currently running Homebridge deCONZ in a test home. When I think it’ll be ready to replace Homebridge Hue in my real home, I’ll release v1.0 including a WiKi on migration.

[mhin76]

So much appreciate your work and commitment!! It is the most important plugin on my homebridge for about 30 devices. Good luck for the refactoring and thanks for your dedication!

[jaydisc]

Controller4HomeKit would handle this transition perfectly. I moved all my accessories from a Hue Bridge to deCONZ a year or so ago... after doing so, I matched things up in Controller, and it rebuilt all room assignments, scenes and automations in seconds. It's quite amazing, and imo, removes any worry or concern about migrating between homebridge-hue and homebridge-deconz, etc.

[quetsal]

I just installed homebridge-deconz, but there is no configuration at all. I can type something directly in the config editor, but it’s not supposed to be like this I suppose. I disabled the hue plug-in for testing.
Anyway, it does not find anything.

[krysly]

I'm just getting into the topic of homebridge and Hue and would like to use the new v2 version directly for this.
The v2 must currently be configured in the config file. However, there is currently no wiki or example for this as it exists in the first version in github.
Would it be possible to provide a basic configuration for v2 as an example ?
I tried a v1 config but that doesn't work.
Many thanks
Christian

[ebaauw]

Homebridge Hue2 doesn't yet do anything.

[ghdbwise]

Hi Eric,
does homebridge-deconz do anything? I see a stateless switch under devices and in homekit. When restarting homebridge there is no pairing request. The deconz is in pairing mode, but nothing happens.
Many thanks so far for homebridge-hue.
Regards Peter.

[ebaauw]

Yes, it’s still under development, but functional. You need to select what to expose in Eve or another HomeKit app, see the README.

[ghost]

Well I'm using the hue emulation on a Tasmota device. :-(

[ebaauw]

Familiarise yourself with Homebridge deCONZ

1. Install Homebridge deCONZ in a child bridge. See Homebridge deCONZ README for details, incl. link how to setup a child bridge;
2. Pair Homebridge deCONZ to your deCONZ gateway; by default, it will expose all zigbee devices connected to the deCONZ gateway.
3. Temporarily create a new HomeKit Home, and pair the Homebridge deCONZ child bridge to that Home;
4. Check that your devices are exposed correctly. Log an issue for devices that aren't exposed correctly or for which you need additional feature requests.
5. Play around with the dynamic configuration through the ui command-line tool, un-exposing and exposing devices, changing log level, how to expose devices, etc.

Move Homebridge deCONZ to production

7. Un-expose all devices in Homebridge deCONZ;
8. Remove Homebridge deCONZ from your test HomeKit home, and pair it to your real production HomeKit home. Remove the test HomeKit home;
9. Expose a device (that passes the checks in step 4) in Homebridge deCONZ. Note the message of the migration resource link in the Homebridge log.
   You'll now see the accessory twice in HomeKit, once exposed by Homebridge Hue, and once by Homebridge deCONZ.
10. Configure the Homebridge deCONZ accessory in HomeKit: assigning it to the HomeKit rooms, groups, scenes, automations like the Homebridge Hue exposed accessory.
11. Update the Homebridge Hue config.json setting homebridgeDeconz to the ID of the migration resourcelink by Homebridge deCONZ from step 8.
12. Restart Homebridge Hue to apply the migration resourcelink, blacklisting the devices exposed by Homebridge deCONZ.
13. At your convenience, repeat steps 8, 9, 11 for each device connected to the deCONZ gateway.

Cleanup Homebridge Hue

14. After all devices have been migrated, remove the deCONZ gateway from the Homebridge Hue configuration. When listing bridges and gateways using hosts, remove the IP address and port of the deCONZ gateway from the array. When using automatic discovery, set noDeconz.
15. Uninstall Homebridge Hue once all gateways are migrated (unless you still use it for any Hue bridge).

[kolorfilm]

Thank you @ebaauw for the very detailed migration guide! 🙏🏻

I‘m a little bit confused about step 7
Un-expose all devices in Homebridge deCONZ

How can I do this? Also when I add the gateway it automatically expose all devices right? How can I prevent this to only expose one by one as you described starting from step 9?

What does this „migration resource link“ thing? I do not completely understand it. I mean I could also just add the new gateway, remove the old one from HomeKit and resetup everything?

How does the „migration resource link“ help here? And also when does it show up and where?

Thank you already for your help!

[ebaauw]

I‘m a little bit confused about step 7

Un-expose all devices in Homebridge deCONZ
How can I do this?

Issue ui put /accessories/id '{"expose": false}' for each accessory. Hence: familiarise yourself with Homebridge deCONZ, notably step 5.

What does this „migration resource link“ thing?

It lists the resources exposed by Homebridge deCONZ, so Homebridge Hue can stop exposing these.

I mean I could also just add the new gateway, remove the old one from HomeKit and resetup everything?

Yes, you could, if you want to do a big-bang migration. Depending on the size of your HomeKit home, this might take you a full day or longer.

How does the „migration resource link“ help here? And also when does it show up and where?

It's a /resourcelinks resource on the deCONZ gateway, maintained by Homebridge deCONZ, and read by Homebridge Hue. Homebridge deCONZ writes a log messsage, whenever the exposed devices change, or when issuing Identify on the gateway accessory.

[8/12/2023, 11:30:07 AM] [deCONZ] pi5: migration: /resourcelinks/3: 35 resources

You would enter the ID, 3, in the Homebridge Hue configuration.

[kolorfilm]

Works like a charm! Thanks. I made a fresh Installation btw. The HomeKit wizard is really simple. It gave me for all devices a popup to already move the new fetched devices to existing groups.

I was wondering if there is a plan to bring back the adaptive live adjustments to the new version. For Homebridge hue there is a option to make the lighting not too red.

Also there is an option to set up what type of devices should be exposed. As far as I understood with the new version it only allows me to not expose devices that I set to expose false one by one and not like „all switches“?

I also recognised that when I turned on adaptive light for a group in a room like (lamp 1-3 grouped into a single lamp group) the brightness doesn’t set correctly (when I set the group to brightness 50 is lacks means doesn’t set up to 50 directly. just after a few tries). For single lamps this works for adaptive light. When adaptive light is off it works also for groups. I was wondering if this is more a Deconz problem itself. I mainly use ikea lamps btw.

[ebaauw]

Please stay on topic and don't pollute this issue with other questions.

[Knot3n]

What will happen with the automations from the exposed devices?
Do we need to "link" them again to the correct device/automation?

[ebaauw]

HomeKit sees the devices exposed by Homebridge deCONZ as different accessories as when exposed by Homebridge Hue. When you no longer expose the accessories by Homebridge Hue, any associations with HomeKit rooms, groups, scenes, and automation are lost. You need to change these to include the Homebridge deCONZ accessories instead.

[Spy-1]

I‘m a little bit confused about step 14

i somehow cannot delete the "old" deconz Gateway. It is always required by the Hue plugin to press the link button.
The ResourceLink is inserted correctly.

How do I remove the Deconz from the Hue Plugin configuration?

[ebaauw]

That’s a good point: as long as you use auto discovery, Homebridge Hue will discover the deCONZ gateway and block Homebridge startup, until it has connected to it. You can disable auto discovery by listing your Hue bridge(s) in config.json, under hosts, but that only works when the Hue bridges have a fixed IP address.

I probably need to cut a Homebridge Hue release that (optionally?) only auto discovers Hue bridges.

[Marfre888]

UPDATE AUG 2023 I've bumped Homebridge deCONZ to v1, as I think it's production ready. I concluded my migration to Homebridge deCONZ in May, and have been running Homebridge deCONZ exclusively for three months now, without any issues.

Consequently, Homebridge Hue support for deCONZ gateways has been deprecated, meaning you should start planning a migration to Homebridge deCONZ. For now, Homebridge Hue continues to expose devices connected to a deCONZ gateway, but I won't be adding support for new devices nor implement any feature requests not supported by the Hue bridge.

Note that Homebridge Hue2 is still just a placeholder, so continue to use Homebridge Hue to expose devices connected to a Hue bridge.

@ebaauw I noticed Homebridge-Hue has a migrate feature but I can't find any tutorials or wikis on how to do it. Is there any way to easily jump to Homebridge-Deconz without having to remove all the lights/switches from Homekit, I have so many that it will be a monumental task to set up all the scenes and automations again.

[ebaauw]

Please don't hijack this issue.

[ebaauw]

v0.13.67 adds the noDeconz option, not to discover any deCONZ gateways.

[romanbalsiger]

Familiarise yourself with Homebridge deCONZ

1. Install Homebridge deCONZ in a child bridge. See Homebridge deCONZ README for details, incl. link how to setup a child bridge;
2. Pair Homebridge deCONZ to your deCONZ gateway; by default, it will expose all zigbee devices connected to the deCONZ gateway.
3. Temporarily create a new HomeKit Home, and pair the Homebridge deCONZ child bridge to that Home;
4. Check that your devices are exposed correctly. Log an issue for devices that aren't exposed correctly or for which you need additional feature requests.
5. Play around with the dynamic configuration through the ui command-line tool, un-exposing and exposing devices, changing log level, how to expose devices, etc.

Move Homebridge deCONZ to production

7. Un-expose all devices in Homebridge deCONZ;
8. Remove Homebridge deCONZ from your test HomeKit home, and pair it to your real production HomeKit home. Remove the test HomeKit home;
9. Expose a device (that passes the checks in step 4) in Homebridge deCONZ. Note the message of the migration resource link in the Homebridge log.
   You'll now see the accessory twice in HomeKit, once exposed by Homebridge Hue, and once by Homebridge deCONZ.
10. Configure the Homebridge deCONZ accessory in HomeKit: assigning it to the HomeKit rooms, groups, scenes, automations like the Homebridge Hue exposed accessory.
11. Update the Homebridge Hue config.json setting homebridgeDeconz to the ID of the migration resourcelink by Homebridge deCONZ from step 8.
12. Restart Homebridge Hue to apply the migration resourcelink, blacklisting the devices exposed by Homebridge deCONZ.
13. At your convenience, repeat steps 8, 9, 11 for each device connected to the deCONZ gateway.

Cleanup Homebridge Hue

14. After all devices have been migrated, remove the deCONZ gateway from the Homebridge Hue configuration. When listing bridges and gateways using hosts, remove the IP address and port of the deCONZ gateway from the array. When using automatic discovery, set noDeconz.
15. Uninstall Homebridge Hue once all gateways are migrated (unless you still use it for any Hue bridge).

Thank you very much for those instructions! BTW the BackUp function of the "Controller" App helped a lot

[ebaauw]

UPDATE DEC 2023
I've removed mention of the deCONZ gateway from the Homebridge Hue documentation and Wiki. Homebridge Hue v0.13.70 is the last version that still supports the deCONZ gateway. In januari, I will publish a v0.14 of Homebridge Hue, that will no longer support the deCONZ gateway.

As of v0.13.70, ph no longer supports the deCONZ gateway. Use deconz to interact with the deCONZ API from the command line. This utility is bundled with Homebridge deCONZ, or can be installed standalone from the hb-deconz-tools package.

In other words: plan your migration to Homebridge deCONZ now. See #1070 (comment) for details.

Note that Homebridge Hue2 is still a placeholder, so please continue to use Homebridge Hue for the gen-2 Hue bridge.

[Knot3n]

Familiarise yourself with Homebridge deCONZ

1. Install Homebridge deCONZ in a child bridge.  See Homebridge deCONZ README for details, incl. link how to setup a child bridge;

2. Pair Homebridge deCONZ to your deCONZ gateway; by default, it will expose all zigbee devices connected to the deCONZ gateway.

3. Temporarily create a new HomeKit Home, and pair the Homebridge deCONZ child bridge to that Home;

4. Check that your devices are exposed correctly.  Log an issue for devices that aren't exposed correctly or for which you need additional feature requests.

5. Play around with the dynamic configuration through the `ui` command-line tool, un-exposing and exposing devices, changing log level, how to expose devices, etc.

Move Homebridge deCONZ to production

7. Un-expose all devices in Homebridge deCONZ;

8. Remove Homebridge deCONZ from your test HomeKit home, and pair it to your real production HomeKit home.  Remove the test HomeKit home;

9. Expose a device (that passes the checks in step 4) in Homebridge deCONZ.  Note the message of the migration resource link in the Homebridge log.
   You'll now see the accessory twice in HomeKit, once exposed by Homebridge Hue, and once by Homebridge deCONZ.

10. Configure the Homebridge deCONZ accessory in HomeKit: assigning it to the HomeKit rooms, groups, scenes, automations like the Homebridge Hue exposed accessory.

11. Update the Homebridge Hue config.json setting `homebridgeDeconz` to the ID of the migration resourcelink by Homebridge deCONZ from step 8.

12. Restart Homebridge Hue to apply the migration resourcelink, blacklisting the devices exposed by Homebridge deCONZ.

13. At your convenience, repeat steps 8, 9, 11 for each device connected to the deCONZ gateway.

Cleanup Homebridge Hue

14. After all devices have been migrated, remove the deCONZ gateway from the Homebridge Hue configuration.  When listing bridges and gateways using `hosts`, remove the IP address and port of the deCONZ gateway from the array.  When using automatic discovery, set `noDeconz`.

15. Uninstall Homebridge Hue once all gateways are migrated (unless you still use it for any Hue bridge).

What will happen with automatins within Homekit?
Will they break when i'll migrate all the devices?

[ebaauw]

What will happen with automatins within Homekit?
Will they break when i'll migrate all the devices?

That's already been answered above.

[Knot3n]

What will happen with automatins within Homekit?
Will they break when i'll migrate all the devices?

That's already been answered above.

Sorry ... totally forgot that i already asked.. my bad. Christmas stress :=)

[spewu]

Sorry if this is considered spam, but I thought maybe my script could help someone else 🙂

I have a lot of devices, to calling the ui tool to un-expose each device is very tedious, so I created a small script to do it for me.

First get a json file with all of your devices:

ui get /devices > devices.json

Then create a simple script like this (just put this into a file called un-expose-all.sh or whatever you wanna call it):

#!/bin/bash

# Path to the JSON file
FILE="devices.json"

# Check if the JSON file exists
if [ ! -f "$FILE" ]; then
    echo "Error: File $FILE does not exist."
    exit 1
fi

# Read and process each device in the JSON file
jq -c 'to_entries[] | {id: .key, expose: false}' $FILE | while IFS= read -r line; do
    id=$(echo $line | jq -r '.id')
    # Call the 'ui' app with the updated device information
    ui put "/devices/$id" "{\"expose\": false}"
done

echo "Devices updated successfully."

Make the file executable:

chmod +x un-expose-all.sh

Then execute it:

./un-expose-all.sh

Once it has completed, all of your devices will be un-exposed.

[HDeKnop]

I am attempting to set up a semi-DIY "smarthome" built around the Apple eco-system.
Most of the lights are ZigBee. Different brands but controlled from the Hue v2 bridge for two reasons:

1. I've been working with Philips for 15 years now, including with the Hue team in its earliest days. One might say I bleed "Philips" blue.
2. Wife knows the Hue app.
3. Relatively simple API ... that I can manage quite easily linking to Shelly switches.

Hue already exposes all the lights to Homekit. However through the Homebridge plugin they respond significantly better / more consistently. However still not perfect.

I see now that you are building 2 separate plugins - and that DeConz seems to support more features.

Q: is there a benefit from transitioning to DeCONZ and the DECONZ plugin now? or vice-versa a benefit to Hue? Alternatively: assuming you had already refactored the plugins and ì did not know Hue yet ... which direction would you advise to go - Hue2 or DeCONZ?

Considering support on Homebridge, but also more generally beyond with Node-Red etc.

[ebaauw]

See for https://github.com/ebaauw/homebridge-hue/wiki/Features comparison of HomeKit support of Hue bridge's native HomeKit feature vs Hue bridge's native Matter feature vs Hue bridge with Homebridge Hue vs deCONZ with Homebridge deCONZ. It largely depends on the devices you have. Next to HomeKit you might have other considerations like the Hue app, dependence on cloud services, or features like Hue Entertainment.