[Feature request]: Add support for scenes to Home Assistant extension

[Kars-de-Jong]

Is your feature request related to a problem? Please describe

Scenes created with Zigbee2MQTT are not available in Home Assistant.
You have to create them manually, they are not linked to devices.

Describe the solution you'd like

I'd like scenes that are created in Zigbee2MQTT to be automatically added to Home Assistant through MQTT discovery.
Home Assistant has support for this (through the MQTT Scene integration).

Describe alternatives you've considered

The alternative is manually adding MQTT scenes, which is not very user friendly.

Additional context

I think this is a must-have feature ;-)

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[Kars-de-Jong]

It may be stale, but I still think this would be a good feature.

[nakedible]

I think this is an important feature - in my set up, it doesn't make sense to ever configure any lights without setting a scene. All my lights are color temperature lights, so simply turning a light on is pointless (brightness is likely to be 1%) and also adjusting only brightness causes nasty results (really bright yellow lights or really dim blue lights) and also single light control is pointless as most rooms have more than one light source and just controlling one just creates a weird scene with one side of the room lit. And for me, evening dim lighting might mean ceiling spots fully off, and only the indirect light strips on, so not every light in a group is going to have the same settings for a scene.

So the only thing I want to call from the Home Assistant side is activating a scene for a certain group.

As I understand it, implementing this would mean extending https://github.com/Koenkk/zigbee2mqtt/blob/master/lib/extension/homeassistant.ts to somehow also represent active scenes for a group in the discovery topics. I don't know enough of Home Assistant to know what's the best way to do that, since Home Assistant scene support is also kind of non-existent - but at least some workarounds are definitely possible, like exposing each scene for a group as a separate "light" with just the "turn_on" action.

No promises I have time to actually conclude this, but @Koenkk do you think I should take a stab at implementing this? Would it likely get merged?

[Koenkk]

@nakedible I'm fine with that. Some tips:

• For the HA part https://www.home-assistant.io/integrations/scene.mqtt/ seems to fit well.
• You can access all scenes of a device in homeassistant.ts via endpoint.getDevice().meta.scenes

[nakedible]

For @Kars-de-Jong and others who stumble upon this... indeed manually configuring MQTT scenes does make them behave somewhat decently:

mqtt:
  scene:
    - name: Kaikki Päivä
      command_topic: zigbee2mqtt/Kaikki/set
      payload_on: '{ "scene_recall": 1 }'
      unique_id: 666_1_scene_zigbee2mqtt
      availability:
        - topic: zigbee2mqtt/bridge/state
    - name: Kaikki Ilta
      command_topic: zigbee2mqtt/Kaikki/set
      payload_on: '{ "scene_recall": 0 }'
      unique_id: 666_0_scene_zigbee2mqtt
      availability:
        - topic: zigbee2mqtt/bridge/state
    - name: Kaikki Yö
      command_topic: zigbee2mqtt/Kaikki/set
      payload_on: '{ "scene_recall": 5 }'
      unique_id: 666_5_scene_zigbee2mqtt
      availability:
        - topic: zigbee2mqtt/bridge/state
    - name: Kaikki Pois
      command_topic: zigbee2mqtt/Kaikki/set
      payload_on: '{ "scene_recall": 4 }'
      unique_id: 666_4_scene_zigbee2mqtt
      availability:
        - topic: zigbee2mqtt/bridge/state

I'll play around a bit with these, or possibly templated lights as well, to figure out what's the best way for these to be represented on the HA side.

[Kars-de-Jong]

Thanks for looking at this feature request, much appreciated!

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[Kars-de-Jong]

Any progress on this?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[Kars-de-Jong]

Would still love to see this implemented!

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[Kars-de-Jong]

@nakedible Are you still planning on adding this?

[gdgib]

Same question: is anyone working on this?

I'll need some kind of automated solution. Adding it to the Z2M HA integration isn't necessarily the fastest/cheapest route, but I may take a stab at it just to avoid maintaining some custom built nonsense that no one else cares about. No promises.

@nakedible if you're working on this, please let me know so I don't duplicate your effort.

[gdgib]

Okay, 3 weeks no reply. Looks like I may have to dig in, because I want to migrate a lot of scenes to z2m and I really don't want to configure HA manually, so....

Looks like a matter of getting the scenes exposed by the device as suggested above, and creating the relevant discovery entry. This should be done for all device types & groups, though I'll only really be testing lights and groups of lights for myself for now. It appears augmenting getConfigs(...) is probably the right way to go, maybe just pushing more discovery entries at the end of the method before the return.

My open questions (mostly for myself I suppose):

1. What should the relevant discovery entries be? I think the goal will be to replicate https://www.home-assistant.io/integrations/scene.mqtt/ for obvious reasons. That probably means creating discovery entries matching the schema in the docs. For my own sanity code is here - https://github.com/home-assistant/core/blob/dev/homeassistant/components/mqtt/scene.py
2. device.endpoint.getDevice().meta.scenes seems to apply for devices, but how would I get the scenes for a group? Is that even a meaningful question, or have I misunderstood the z2m data model?
3. What's the type of the scenes property? I should be able to track this down through the endpoint starting in Device.ts
4. How will I unit test any of this? I guess through https://github.com/Koenkk/zigbee2mqtt/blob/master/test/homeassistant.test.js by following https://github.com/Koenkk/zigbee2mqtt#developing

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[gdgib]

Despite the bot being a antsy, I'm personally unlikely to stop thinking about this for a while, since I've got dozens of scenes and no interest in doing it by hand. Anyone who finds this is welcome to @ me.

[mundschenk-at]

No idea how to best implement this on the TS side, but I can help creating discovery entries for this, @gdgib

[gdgib]

That'd be fantastic. I speak pretty much every programming language, so if you even just have some examples I could steal from or something I can translate to TS pretty easily, I suspect. Honestly, anything to get me a better understanding of how discovery entries work at all.

[mundschenk-at]

Great! I'll configure a few scenes statically using YAML and then I'll translate that to discovery shorthand and post the examples here. Have you had time to look at the Z2M data model to extract e.g. the device identifier so that we can properly link scenes to groups and devices? (Manually the ID is of course available via HA's debug information.)

[gdgib]

I'll take a swing at that this weekend, but I gotta get my debug/test env for z2m set up again too.

This has been lower priority for me with so many unknowns, your help changes that.

[mundschenk-at]

So auto-discovery messages are special MQTT topics following this format: <discovery_prefix>/<component>/[<node_id>/]<object_id>/config

The discovery_prefix is usually just homeassistant, but can be configured. For us, component is scene. The values for node_id and object_id depend on whether we have a unique ID for our scenes. I am assuming we don't, so I'd set node_id to the unique ID group or device in question.

So as an example, we'd get something like this string for the topic: homeassistant/scene/0x0123456789abcdef/scene_3/config.

The value is a JSON object containing either long- or short-form keys. An example could look like this (using long-form, as the existing Z2M discovery topics):

{
      "unique_id": "0x0123456789abcdef_scene_3_zigbee2mqtt",
      "name": "My Scene",
      "command_topic": "zigbee2mqtt/my_device/set",
      "availability": {
          "topic": "zigbee2mqtt/bridge/state",
          "value_template": "{{ value_json.state }}"
      },
      "payload_on": "{ \"scene_recall\": 3 }",
      "device": {
        "name": "My Device",
        "identifiers": [ "zigbee2mqtt_0x0123456789abcdef" ]
    },
    "origin": {
        "name": "Zigbee2MQTT",
        "sw": "1.33.0",
        "url": "https://www.zigbee2mqtt.io"
    }
}

Now that could be refined with a second availability topic (parse zigbee2mqtt/bridge/devices to check if the scene is listed for the device/group), but that's maybe for a second iteration. Also, the discovery entries of Z2M could be improved a bit, setting up the bridge as "hub device" in HA's device registry and using "services" instead of "devices" for groups (not a functional difference, but that's how the native Hue integration does it).

[gdgib]

Thank you. I was a little surprised I couldn't find a document somewhere explaining discovery messages. Did I miss it? If not, maybe one of us could open a PR explaining what you just did.

Anyway, that comment makes a huge difference. Thanks for trolling closed issues to help!

[mundschenk-at]

Yeah, it's a bit hidden away in https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery and the examples are not too clear. Basically, it's the same syntax as manual MQTT entity configuration in HA, but encoded as JSON instead of YAML (and with some optional shorthands).

[mundschenk-at]

@gdgib Did you get around to experimenting with this?

[gdgib]

No, with apologies. It'll be another few days, maybe another week. Something critical came up. Feel free to proceed without me if you prefer.

[mundschenk-at]

No problem. I'll have to wait for you, my JS/TS skills are mostly read-only, I fear.

[radclif]

Can I help? I'm a coder since many years. In PHP, Javascript though. Maybe some Python. I would love to get this fixed. I have 200+ zigbee devices though Z2M and it's a pain 👎

[mundschenk-at]

Sure, @radclif, AFAIK everything is written in TypeScript. I've got no knowledge of the z2m codebase, so I can't really tell you where to look, but what needs to be done is create discovery messages for all scenes as described in #13937 (comment).

[mundschenk-at]

Hi @gdgib & @radclif, did you find time to look at the z2m code yet?

[gdgib]

No, I expect to work on this probably the last week of 2023, given my current schedule/plans.

[mundschenk-at]

@Koenkk Is there any kind of whitepaper or high-level documentation how the modules in Z2M interact with each other?

From my experimenting I gather that there is currently no way to get a collection of scenes in homeassistant.ts except reimplementing the getScenes logic from bridge.ts, or am I missing something? While there is an onScenesChanged event published, it does not include any data for a callback, so that does not help either. The scenes for each device/group would be available as part of the bridge topic, but parsing from MQTT data seems ill-advised.

[Koenkk]

@mundschenk-at the getScenes can be moved to utils.ts, then it is accessible for both bridge and homeassistant extension.

[mundschenk-at]

Thanks, I managed to do that despite my rudimentary TS skills and I'm working on PR (only for group scenes right now, not sure if scenes for individual endpoints really are a thing?).

[mundschenk-at]

I have added a draft PR. It currently lacks support for individual endpoint scenes and (more importantly) test coverage.

[mundschenk-at]

@radclif, @gdgib If you'd like to test-drive my implementation, you can switch to https://github.com/mundschenk-at/zigbee2mqtt/tree/homeassistant_discovery. It should be up to date with dev (at least as of yesterday).