ZiGate users help wanted to test ZHA with fork of zigpy-zigate with new patches

[Hedda]

ZiGate users help wanted to test fork of zigpy-zigate with ZHA before developers submit pull requests:

Please see discussion and plea for help -> zigpy/zigpy-zigate#104

Help from testers with ZiGate adapters is wanted to "regression test" other Python packages of check if patches applied to a fork of zigpy-zigate look to have introduced any new bugs seen in ZHA integration.

The backstory:

The lead developer of zigpy-zigate (zigpy radio library for ZiGate based Zigbee Coordinator adapters used by the ZHA integration) does not have time to maintain or improve the zigpy-zigate library.

The good news is that some developers of a Zigbee implementation for Domoticz which will also use the same zigpy-zigate library have offered to submit bug-fixes and new features/functions.

The problem with that is that since those developers do not use ZHA at all they can not perform regression testing on ZHA to test if they introduce any other bugs or issues with their patches that can be seen in ZHA but not in their (non-ZHA) Zigbee implementation for Domoticz.

Since those developers of a Zigbee implementation for Domoticz do not have write access to the upstream zigpy-zigate repository on GitHub they have forked (copied) it to another repository and applied the patches there for now. That is why it is the library in that repository that will now need tested.

https://github.com/zigbeefordomoticz/zigpy-zigate

Volunteering testers that actually use a ZiGate adapter with ZHA is needed to help with testing the fork.

Once patches in the fork have been tested then developers can see to submitting patches upstream.

PS: Also now posted a copy of this general plea for help with testing in Home Assistant community forum here:

https://community.home-assistant.io/t/zigate-users-help-wanted-to-test-zha-with-fork-of-zigpy-zigate-with-new-patches/378362

[pipiche38]

Honestly it will be really hard to push our work to upstream if PRs are going to be handle like that.
zigpy/zigpy-zigate#114

I never hide the fact that our aim is to use zigpy-znp and other zigpy librairie but not the zigate one as our code will use zigate in a native way.
However the work we have done on our fork as been to put the zigpy-zigate library in a shape that it is usable and provides the expected features like broadcast, group, and communication flow .

Now if we have to proof by a+b any single PR that we are going to push , this is for sure something we are not ready to do.

So we tried with this simple one, but it looks like it doesn't fly, that is a shame

[Hedda]

One idea would be to campaign to find additional maintainers of the upstream zigpy-zigate repository that can also tested ZHA as well, see:

zigpy/zigpy-zigate#116

I have no idea how to proceed other than try to get new ZHA zigate testers and/or developers to help with the maintenance of zigpy-zigate.

Again, @doudz expressed that he, unfortunately, don't have enough spare time to spare on actively developing or maintaining zigpy-zigate.

[pipiche38]

Just for you guys to understand what we have done on our fork, is to make sure that zigpy-zigate behave as zigpy-znp. From our domoticz plugin we use only zigpy api() and except for some radio specific actions like Led , we are using some the standard one.

So it is hard to see why it our modifs should impact ZHA.

Last but not least one thing which is important to know and that could have indeed an impact to ZHA, is that we are not relying on the zigpy database (as we have our own one).

[Adminiuga]

@pipiche38 if you volunteer to maintain zigpy-zigate, then just let us know. Can give you permission over the repository

[pipiche38]

thanks but as said I'm focussing on the domoticz side and we still have a long way to go, our aims was to contribute into the enhancement of the zigpy-zigate library

Last I strongly recommend that you get someone from the ZHA community as we are using only a small piece of the zigpy layers ( not ZDO, no ZCl, no Databases ).

[Hedda]

@pipiche38 while slightly off-topic; can I suggest that one thing that you as developers of https://github.com/zigbeefordomoticz/Domoticz-Zigbee/tree/beta6 and users of ZiGate Zigbee Coordinators might be interested in developing for zigpy-zigate and zigpy-cli is support for backup and restore to file using the "Open ZigBee Coordinator Backup Format", and for that it will need to implement new radio API for zigpy:

Checkout zigpy/zigpy-cli#2 where @puddly added introduces a network backup/restore utility for any adapter implementing the new radio API, where the only zigpy library remaining is zigpy-zigate.

If zigpy-zigate supported this then your implementation too would be able to take advantage on this new universal backup format standard.

PS: This same "Open ZigBee Coordinator Backup Format" is not only used by ZHA and Jeedom via zigpy, but support for that format has also been implemented by zigbee-herdsman which is used by Zigbee2MQTT (Z2M) and IoBroker. This enables cross-implementation migrations.

[Hedda]

@fairecasoimeme if "ZiGate v1" firmware is no longer being developed (if Zigate firmware for JN5168 is now in maintenance mode only) could this feature + serial command be added "ZiGate v2" firmware to allow backup + restore solutions can be implemented?

fairecasoimeme/ZiGate#362

https://github.com/fairecasoimeme/ZiGatev2

PS: Again the goal would be that zigpy-zigate fully supported all features in new zigpy radio API, which include backup/restore:

zigpy/zigpy-cli#2

[pipiche38]

@Hedda I did play with @schrodingersket fork, and it is quiet stable. From my point of view it is backward compatible with the initial firmware, so it would even be interesting that zigpy-zigate aligned and use the features from @schrodingersket .

The main issue on zigpy-zigate is that it doesn't implement the firmware flow control which control the APS allocation

[schrodingersket]

The main issue on zigpy-zigate is that it doesn't implement the firmware flow control which control the APS allocation

If there's interest, I think I'd be willing to implement this for zigpy-zigate, at least for the ZiGate v1 firmware. You're referring to the control flow for loading PDM contents externally, correct?

I'd consider verifying it against ZiGate v2, but I unfortunately don't have the hardware necessary to test that. Unless something significant has changed in the codebase between v1 and v2, however, I imagine that the PDM management code sits low enough on the stack that support for this feature would be unchanged between v1 and v2.

[puddly]

The ZiGate platform supports very few features that zigpy requires (request flow control being one, network backup/restore, energy scanning, etc.). Backup would be useful to allow people to migrate to better-supported radios.

Is it possible to implement in firmware the commands to merely read out PDM contents? I'm sure the format isn't difficult to reverse engineer. This would allow for backup and restore to at least function, since right now even the PAN ID cannot be specified when forming a network.

[pipiche38]

I was more thinking on the send message protocol to prevent exhausting the PDUs.

send message with Ack
receive 0x8000
receive 0x8012 (APS data confirm, message routed)
receive 0x8011 (Ack) / 0x8702 (APS confirm failed)

send message without Ack
receive 0x8000
receive 0x8012 (APS data confirm, message routed)