Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement new radio API #2

Merged
merged 17 commits into from
Jun 28, 2022
Merged

Implement new radio API #2

merged 17 commits into from
Jun 28, 2022

Conversation

puddly
Copy link
Contributor

@puddly puddly commented Nov 14, 2021

This change introduces a network backup/restore utility for any adapter implementing the new radio API:

The following adapters also implement the new radio API but their firmware does not allow much network information to be read, making this form of migration impossible:


Instructions

You'll need access to a Python environment. zigpy-znp has documentation that describes how to do this for every common platform.

# Make a new venv, these packages are heavily modified
virtualenv -p 3 venv  # Python 3.7 or above is required
# Or python3 -m venv venv (if `virtualenv` isn't installed)
source venv/bin/activate

# Install HA OS packages (only if using HA OS)
apk add gcc musl-dev libffi-dev

# Install zigpy-cli
pip install git+https://github.com/zigpy/zigpy-cli.git

# Back up the old radio.      Note: this backup format will change in the future.

# Conbee/Raspbee
zigpy -vv radio deconz /dev/serial/by-id/... backup deconz.json

# CC2531, CC2652, etc.
# zigpy -vv radio znp /dev/serial/by-id/... backup znp.json

# HUSBZB-1, EFR32, etc.
# zigpy -vv radio ezsp /dev/serial/by-id/... backup ezsp.json

# Restore a backup to the new one
zigpy -vv radio znp /dev/serial/by-id/... restore deconz.json
# zigpy -vv radio ezsp /dev/serial/by-id/... restore deconz.json
# zigpy -vv radio deconz /dev/serial/by-id/... restore deconz.json


# Leave the virtualenv
deactivate

Once that's done, edit /config/.storage/core.config_entries (take a backup of the file first!) and change the coordinator's path to the new radio's /dev/serial/by-id/... path. If you're switching to a CC2652, make sure to delete the flow_control key and set the baudrate to 115200.

@puddly puddly added the enhancement New feature or request label Dec 12, 2021
@connoleg
Copy link

Hi,
I managed to backup and restore from/to a ConBee II using zigpy. But when I try to restore to a Sonoff 3 plus I get the following exception:

zigpy -vv radio znp /dev/ttyUSB0 restore deconz.json
2022-01-12 17:51:04.390 parallels-Parallels-Virtual-Platform asyncio DEBUG Using selector: EpollSelector
Traceback (most recent call last):
  File "/home/parallels/.local/bin/zigpy", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1134, in invoke
    Command.invoke(self, ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/parallels/.local/lib/python3.8/site-packages/zigpy_cli/cli.py", line 19, in inner
    return loop.run_until_complete(cmd(*args, **kwargs))
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/parallels/.local/lib/python3.8/site-packages/zigpy_cli/radio.py", line 51, in radio
    app = app_cls(config)
TypeError: Can't instantiate abstract class ControllerApplication with abstract methods connect, disconnect, start_network

Any idea whats going wrong?
Thanks

@puddly
Copy link
Contributor Author

puddly commented Jan 12, 2022

Make sure you're using a virtualenv. I believe pip install won't overwrite existing packages when you install from Git.

Or pip uninstall zigpy-znp, re-install it from git, then uninstall it again and install the normal version from PyPI.

@connoleg
Copy link

Thanks puddly, bang on the money! virtualenv did the trick. I hadnt figured I needed to run that way as I was not on my HomeAssistant box but a standalone ubuntu

Cheers

@toot
Copy link

toot commented Jan 12, 2022

I would also like to switch from a ConBee II to a Sonoff 3 plus with your tool, but the setup fails ...

I get the following errors after (I tried to) installed wheel

Failed building wheel for zigpy
Failed building wheel for zigpy-cli
Failed building wheel for zigpy-deconz
Failed building wheel for bellows

But at the end:
Successfully installed bellows-0.30.0.dev0 zigpy-0.41.0.dev0 zigpy-cli-0.0.1 zigpy-deconz-0.14.0.dev0 zigpy-znp-0.6.4

Did the installation work anyway?

Also I'm still struggeling with the next step:

Using "lsusb" I get this info about my conbee II:
Bus 001 Device 003: ID 1cf1:0030 Dresden Elektronik

Where do I get the remaining information for the backup command?
zigpy -vv radio deconz /dev/serial/by-id/... backup deconz.json

Sorry, I'm pretty new to this and I'm still learning...
Thanks in advance :)

@puddly
Copy link
Contributor Author

puddly commented Jan 12, 2022

You can ignore the wheel errors, those are just saying that Python didn't cache .whl files so it'll have to rebuild the modules from scratch if you reinstall them.

Run ls /dev/serial/by-id/ to list the serial ports. You'll see which one corresponds to which device (you don't have to have them both plugged in at once).

@toot
Copy link

toot commented Jan 12, 2022

Thanks! :)

Now the backup command results in different errors, stating that some distributions are missing:

bellows 0.30.0.dev0 requires click-log>=0.2.1, which is not installed.
bellows 0.30.0.dev0 requires pure_pcapy3==1.0.1, which is not installed.
bellows 0.30.0.dev0 requires pyserial, which is not installed.
bellows 0.30.0.dev0 requires pyserial-asyncio, which is not installed.
bellows 0.30.0.dev0 requires voluptuous, which is not installed.

I already manually installed: scapy, click and coloredlogs, the current error is:
The 'voluptuous' distribution was not found and is required by zigpy

But I tried installing python-voluptuous with no effect.

I'm afraid I missed something at the beginning.

@puddly
Copy link
Contributor Author

puddly commented Jan 12, 2022

This may be my mistake, I just updated the pip install command in the example. Instead of running:

$ pip install --upgrade --no-deps --force-reinstall ...

Try just running:

$ pip install ...

The dependencies should then be correctly installed.

@toot
Copy link

toot commented Jan 13, 2022

Thanks Putty, now everything works as intended :)

I had a friend of mine help me undo the changes I made yesterday and after that your guide worked very well.

Some feedback from him (to make it more fool proof):

  1. Adjusting the variable of the device from "..." and adding some additional input:

# Use TAB-autocomplete to get your device
YOUR_DEVICE=/dev/by-id/zigbeedevice
zigpy -vv radio deconz /dev/serial/by-id/$YOUR_DEVICE backup deconz.json

  1. Using another name (less generic) for the venv, to prevent hiccups if there is already another "venv"

But thanks alot again for this really useful tool :)

@puddly
Copy link
Contributor Author

puddly commented Jan 28, 2022

@toot @connoleg would either of you mind testing out something with your Conbee? They need to be relatively running recent firmware (mid 2020 or newer) for this to work:

CONBEE=/dev/serial/by-id/...

zigpy -vv radio deconz $CONBEE backup conbee1.json

# Edit `conbee1.json` and change `network_key.frame_counter` to something bigger by adding a `0` to the end

zigpy -vv radio deconz $CONBEE restore conbee1.json

# Unplug the Conbee and plug it back in (very important!)

zigpy -vv radio deconz $CONBEE backup conbee2.json

# Compare the two files
diff conbee1.json conbee2.json

The contents of conbee1.json and conbee2.json should be nearly identical and the frame counter value should have changed.

@stiang
Copy link

stiang commented Jan 29, 2022

I’m trying to migrate from a Raspberry Pi with RaspBee II to a proper server with the ConBee II USB dongle instead. I believe I have followed all the proper steps from the top of this issue, but getting this error when trying to backup the RaspBee:

(venv) bash-5.1# zigpy -vv radio deconz /dev/serial/by-id/usb-0658_0200-if00 backup deconz.json
2022-01-29 12:04:58.068 homeassistant asyncio DEBUG Using selector: EpollSelector
2022-01-29 12:04:58.242 homeassistant zigpy.appdb DEBUG SQLite version for <module 'sqlite3' from '/usr/local/lib/python3.9/sqlite3/__init__.py'>: 3.34.1
2022-01-29 12:04:58.273 homeassistant zigpy_deconz.uart DEBUG Connection made
2022-01-29 12:04:58.274 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.protocol_version: 34>, b'')
2022-01-29 12:04:58.275 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a02000800010022
2022-01-29 12:05:00.079 homeassistant zigpy_deconz.api WARNING No response to 'Command.read_parameter' command with seq id '0x02'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/venv/bin/zigpy", line 8, in <module>
    sys.exit(cli())
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/config/venv/lib/python3.9/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/config/venv/lib/python3.9/site-packages/zigpy_cli/cli.py", line 19, in inner
    return loop.run_until_complete(cmd(*args, **kwargs))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/config/venv/lib/python3.9/site-packages/zigpy_cli/radio.py", line 90, in backup
    await app.connect()
  File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/zigbee/application.py", line 71, in connect
    self.version = await api.version()
  File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 470, in version
    (self._proto_ver,) = await self[NetworkParameter.protocol_version]
  File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 435, in read_parameter
    r = await self._command(Command.read_parameter, 1 + len(data), param, data)
  File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 320, in _command
    return await asyncio.wait_for(fut, timeout=COMMAND_TIMEOUT)
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

Am I doing something wrong? I’m not 100% sure that the by-id ID is the correct one, but it’s the only entry in that folder and I get the same result when I use /dev/ttyACM0, which I know is the RaspBee.

@puddly
Copy link
Contributor Author

puddly commented Jan 29, 2022

0658:0200 looks like it's for an Aeotec Z-Wave stick. Try using /dev/ttyAMA0.

@stiang
Copy link

stiang commented Jan 29, 2022

Oof, so much for "which I know is the RaspBee"... Thanks, you are right, that looks like the correct device. And I get what appears to be promising output with no errors:

(venv) bash-5.1# zigpy -vv radio deconz /dev/ttyAMA0 backup deconz.json
2022-01-29 16:31:57.184 homeassistant asyncio DEBUG Using selector: EpollSelector
2022-01-29 16:31:57.360 homeassistant zigpy.appdb DEBUG SQLite version for <module 'sqlite3' from '/usr/local/lib/python3.9/sqlite3/__init__.py'>: 3.34.1
2022-01-29 16:31:57.387 homeassistant zigpy_deconz.uart DEBUG Connection made
2022-01-29 16:31:57.388 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.protocol_version: 34>, b'')
2022-01-29 16:31:57.389 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a02000800010022
2022-01-29 16:31:57.399 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a02000a000300220e01
2022-01-29 16:31:57.400 homeassistant zigpy_deconz.api DEBUG Read parameter protocol_version response: [270]
2022-01-29 16:31:57.400 homeassistant zigpy_deconz.api DEBUG Command Command.version (0,)
2022-01-29 16:31:57.401 homeassistant zigpy_deconz.uart DEBUG Send: 0x0d0300090000000000
2022-01-29 16:31:57.411 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0d0300090000076926
2022-01-29 16:31:57.411 homeassistant zigpy_deconz.api DEBUG Version response: 26690700
2022-01-29 16:31:57.414 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.mac_address: 1>, b'')
2022-01-29 16:31:57.414 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a04000800010001
2022-01-29 16:31:57.426 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a04001000090001fefd08ffff2e2100
2022-01-29 16:31:57.427 homeassistant zigpy_deconz.api DEBUG Read parameter mac_address response: [00:21:2e:ff:ff:08:fd:fe]
2022-01-29 16:31:57.428 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.aps_designed_coordinator: 9>, b'')
2022-01-29 16:31:57.429 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a05000800010009
2022-01-29 16:31:57.438 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0500090002000901
2022-01-29 16:31:57.439 homeassistant zigpy_deconz.api DEBUG Read parameter aps_designed_coordinator response: [1]
2022-01-29 16:31:57.440 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_address: 7>, b'')
2022-01-29 16:31:57.441 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a06000800010007
2022-01-29 16:31:57.451 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a06000a000300070000
2022-01-29 16:31:57.452 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_address response: [0x0000]
2022-01-29 16:31:57.453 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_panid: 5>, b'')
2022-01-29 16:31:57.453 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a07000800010005
2022-01-29 16:31:57.463 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a07000a000300054b32
2022-01-29 16:31:57.464 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_panid response: [0x324b]
2022-01-29 16:31:57.465 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.aps_extended_panid: 11>, b'')
2022-01-29 16:31:57.466 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0800080001000b
2022-01-29 16:31:57.477 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0800100009000b0000000000000000
2022-01-29 16:31:57.478 homeassistant zigpy_deconz.api DEBUG Read parameter aps_extended_panid response: [00:00:00:00:00:00:00:00]
2022-01-29 16:31:57.479 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_extended_panid: 8>, b'')
2022-01-29 16:31:57.480 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a09000800010008
2022-01-29 16:31:57.491 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a09001000090008fefd08ffff2e2100
2022-01-29 16:31:57.493 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_extended_panid response: [00:21:2e:ff:ff:08:fd:fe]
2022-01-29 16:31:57.493 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.current_channel: 28>, b'')
2022-01-29 16:31:57.494 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0a00080001001c
2022-01-29 16:31:57.504 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0a00090002001c0f
2022-01-29 16:31:57.505 homeassistant zigpy_deconz.api DEBUG Read parameter current_channel response: [15]
2022-01-29 16:31:57.505 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.channel_mask: 10>, b'')
2022-01-29 16:31:57.506 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0b00080001000a
2022-01-29 16:31:57.515 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0b000c0005000a00800000
2022-01-29 16:31:57.516 homeassistant zigpy_deconz.api DEBUG Read parameter channel_mask response: [<Channels.CHANNEL_15: 32768>]
2022-01-29 16:31:57.517 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_update_id: 36>, b'')
2022-01-29 16:31:57.518 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0c000800010024
2022-01-29 16:31:57.527 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0c00090002002400
2022-01-29 16:31:57.528 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_update_id response: [0]
2022-01-29 16:31:57.529 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (2, <NetworkParameter.network_key: 24>, b'\x00')
2022-01-29 16:31:57.530 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0d00090002001800
2022-01-29 16:31:57.544 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0d00190012001800af912adb53c0fe5b478542134b7796c9
2022-01-29 16:31:57.546 homeassistant zigpy_deconz.api DEBUG Read parameter network_key response: [0, [175, 145, 42, 219, 83, 192, 254, 91, 71, 133, 66, 19, 75, 119, 150, 201]]
2022-01-29 16:31:57.546 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_frame_counter: 39>, b'')
2022-01-29 16:31:57.547 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0e000800010027
2022-01-29 16:31:57.557 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0e000c00050027d6a60200
2022-01-29 16:31:57.558 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_frame_counter response: [173782]
2022-01-29 16:31:57.559 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.trust_center_address: 14>, b'')
2022-01-29 16:31:57.560 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0f00080001000e
2022-01-29 16:31:57.571 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0f00100009000efefd08ffff2e2100
2022-01-29 16:31:57.572 homeassistant zigpy_deconz.api DEBUG Read parameter trust_center_address response: [00:21:2e:ff:ff:08:fd:fe]
2022-01-29 16:31:57.573 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (9, <NetworkParameter.link_key: 25>, b'\xfe\xfd\x08\xff\xff.!\x00')
2022-01-29 16:31:57.574 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a10001000090019fefd08ffff2e2100
2022-01-29 16:31:57.591 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a10002000190019fefd08ffff2e21005a6967426565416c6c69616e63653039
2022-01-29 16:31:57.593 homeassistant zigpy_deconz.api DEBUG Read parameter link_key response: [00:21:2e:ff:ff:08:fd:fe, [90, 105, 103, 66, 101, 101, 65, 108, 108, 105, 97, 110, 99, 101, 48, 57]]
2022-01-29 16:31:57.594 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.security_mode: 16>, b'')
2022-01-29 16:31:57.595 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a11000800010010
2022-01-29 16:31:57.604 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a1100090002001003
2022-01-29 16:31:57.605 homeassistant zigpy_deconz.api DEBUG Read parameter security_mode response: [3]
2022-01-29 16:31:57.615 homeassistant zigpy_deconz.api DEBUG Command Command.write_parameter (5, <NetworkParameter.watchdog_ttl: 38>, b'\x00\x00\x00\x00')
2022-01-29 16:31:57.616 homeassistant zigpy_deconz.uart DEBUG Send: 0x0b12000c0005002600000000
2022-01-29 16:31:57.627 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0b12000800010026
2022-01-29 16:31:57.627 homeassistant zigpy_deconz.api DEBUG Write parameter watchdog_ttl: SUCCESS
2022-01-29 16:31:57.630 homeassistant zigpy_deconz.uart DEBUG Closed serial connection

However, the backup file is very short and shows no devices ("devices": []):

{
    "metadata": {
        "version": 1,
        "format": "zigpy/open-coordinator-backup",
        "source": "zigpy-cli@0.0.1",
        "internal": {
            "node": {
                "ieee": "00212effff08fdfe",
                "nwk": "0000",
                "type": "coordinator"
            },
            "network": {
                "tc_link_key": {
                    "key": "5a69674265________69616e63653039",
                    "frame_counter": 0
                },
                "tc_address": "00212effff08fdfe",
                "nwk_manager": "0000"
            },
            "link_key_seqs": {}
        }
    },
    "stack_specific": {},
    "coordinator_ieee": "00212effff08fdfe",
    "pan_id": "324b",
    "extended_pan_id": "00212effff08fdfe",
    "nwk_update_id": 0,
    "security_level": 5,
    "channel": 15,
    "channel_mask": [
        15
    ],
    "network_key": {
        "key": "af912adb53c0fe________134b7796c9",
        "sequence_number": 0,
        "frame_counter": 173782
    },
    "devices": []
}

In HA I have 14 devices and 30 entities, which I successfully control via the RaspBee, so something’s still off:

image

Any ideas what might cause the device list to be empty? I have disabled zha in HA while doing this. If I try to enable it I get different errors when trying to create the backup (about possible simultaneous access etc., which makes sense).

@puddly
Copy link
Contributor Author

puddly commented Jan 29, 2022

The coordinator normally doesn't keep track of any devices that aren't its direct children, but even that information isn't critical. The Conbee however expects the program controlling it to load in the child table at runtime since it'll be stored in the program's database. This is normal.

After you restore to the Conbee, can you unplug it, plug it back in, and take a second backup of it and post it here?

@stiang
Copy link

stiang commented Jan 29, 2022

@puddly Sorry if I’m being dense, but are you saying that the devices don’t need to be listed in the backup file, and are instead "mapped to" the RaspBee/ConBee through other means that are saved in the HA config/storage files? In other words, the zigbee backup file I have now should actually get me the devices I need if paired with a full HA backup?

I’m also unclear on what’s the best procedure to do the restore. Should I plug the ConBee into the Pi and do the restore there, then move the already restored ConBee to my Ubuntu server? Or plug the ConBee into the Ubuntu server, copy the backup file and then do the restore from there? Or doesn’t it matter?

I’ll post the info you want once I have successfully performed the restore. Thanks for helping out, really appreciate it!

@puddly
Copy link
Contributor Author

puddly commented Jan 29, 2022

Correct, there will be no devices if you're using a Raspbee or a Conbee.

It doesn't matter where you do the restore. Since you already have this virtual environment up and running, it's easiest to do it wherever you did the Raspbee backup.

@stiang
Copy link

stiang commented Jan 29, 2022

@puddly Sweet, looks like I was able to restore to the ConBee now. Will try to get it up and running on the new server shortly. Thanks again!

I followed the instructions you gave a few comments back about increasing the frame_counter value before restoring, then creating another backup. As far as I can see the value was reset, but that was all:

(venv) bash-5.1# diff conbee1.json conbee2.json
--- conbee1.json
+++ conbee2.json
@@ -33,7 +33,7 @@
     "network_key": {
         "key": "af912adb53c0fe5b478542134b7796c9",
         "sequence_number": 0,
-        "frame_counter": 1
+        "frame_counter": 0
     },
     "devices": []
 }
\ No newline at end of file
(venv) bash-5.1#

Let me know if your need anything else!

@puddly
Copy link
Contributor Author

puddly commented Jan 29, 2022

Looks like your Conbee is running an old firmware and doesn't support setting the frame counter. The one in your Raspbee backup was 173782 but the value read from the Conbee was 0.

Can you upgrade it to the latest firmware and re-run the migration from the Raspbee again? Some devices are strict about the frame counter rolling back and may not respond to commands sent from the Conbee.

@toot
Copy link

toot commented Jan 31, 2022

@toot @connoleg would either of you mind testing out something with your Conbee? They need to be relatively running recent firmware (mid 2020 or newer) for this to work:

The contents of conbee1.json and conbee2.json should be nearly identical and the frame counter value should have changed.

Unfortunately I've got my Conbee not around at the moment, I'm trying to convince a friend from HA and borrowed him mine :D

@robtpaton
Copy link

Done and fixed thank you! Can confirm I was able to move from a Conbee II to Sonoff Zigbee Dongle 👍

@mattague
Copy link

mattague commented Apr 22, 2022

I'm not entirely sure what might have gone wrong, but I can't seem to get this owrking. I am running in a virtual environment, but I can not get the commands to work. After running pip install... I get the following error at the end:

Successfully built zigpy zigpy-cli zigpy-znp zigpy-deconz bellows pure_pcapy3 scapy pyrsistent
Failed to build aiohttp pycryptodome frozenlist multidict yarl
ERROR: Could not build wheels for aiohttp, frozenlist, multidict, yarl, which is required to install pyproject.toml-based projects

And then trying to run zigpy -vv radio znp /dev/ttyACM0 backup znp.json gets me bash: zigpy: command not found.

Edit:

I did not get this working on my raspberry pi running Home assistant OS but I tried from another linux machine and it worked flawlessly, Thank you very much for the instructions.

@p71-yes
Copy link

p71-yes commented May 3, 2022

great work @puddly !!! huge thanks..

Once installed, it was a smooth move from deconz to a sonoff zigbee stick...

@hcrohland
Copy link

Just migrated from a raspbee to a conbee without problems

@francodutch
Copy link

great work @puddly !!! huge thanks..

Once installed, it was a smooth move from deconz to a sonoff zigbee stick...

I need to do the same/similar - raspbi on ZHA to sonoff dongle on ZHA. I've read through the chain but if you're interested we could do a clean procedure and test it for others to use?

@Hedda
Copy link

Hedda commented Jun 20, 2022

Originally posted by @pipiche38 in zigpy/zigpy-zigate#117 (comment)

@puddly , I found that https://github.com/fairecasoimeme/ZiGate/blob/5a64452e3cc4f32afe217a332d0c9b1d97e17d84/ModuleRadio/Firmware/src/ZiGate/Source/ZigbeeNodeControlBridge/app_start.c#L1191 which can be interested for you as regards to a Backup/Restore solution.
If we take in consideration this procedure is used when the PDM size is changed during a firmware upgrade, we might think that can be used for Backup and then Restore.

@doudz Do you have any input in regards to achieving ZiGate Zigbee network backup via this network backup/restore utility?

@puddly puddly marked this pull request as ready for review June 22, 2022 18:35
@puddly puddly merged commit f27b67b into zigpy:dev Jun 28, 2022
@thefjordkeeper
Copy link

@puddly - I'm fairly certain this worked okay for me on Ubuntu 22.04 but I am unable to locate the ezsp.json file. Where is that saved?
I'm not moving from one Coordinator to another, I just flashed a spare Sonoff EFR32MG21 dongle with the latest firmware in an attempt to iron out some Zigbee network behavior. I want to verify that the backup actually exists, it's a sickness that comes from working in IS.

Thank you!

@Hedda
Copy link

Hedda commented Aug 4, 2022

@puddly - I'm fairly certain this worked okay for me on Ubuntu 22.04 but I am unable to locate the ezsp.json file. Where is that saved?
I'm not moving from one Coordinator to another, I just flashed a spare Sonoff EFR32MG21 dongle with the latest firmware in an attempt to iron out some Zigbee network behavior. I want to verify that the backup actually exists, it's a sickness that comes from working in IS.

Suggest that now that this is merged (i.e. closed as merged) maybe instead post any additional questions as new issues https://github.com/zigpy/zigpy-cli/issues or discussions https://github.com/zigpy/zigpy-cli/discussions or ?

@balucanb
Copy link

@puddly If this is obvious sorry, I do not really understand yet how to use this tool but my question is this- I want to move my Zigbee2Mqtt network currently on Rpi4 w/ Sonoff 3.0 dongle to a a Home assistant Yellow and keep it on Z2M is that possible with this tool ( or any other way) Thanks

@asriva13
Copy link

Hi, I am trying to move from conbee2 to sonoff. While executing the backup command I am getting a timeout error (attached in the png file). Any help or directions would be appreciated.
error

@Hedda
Copy link

Hedda commented May 15, 2023

@asriva13 Have you first upgraded to the latest deconz firmware on your ConBee/RaspBee adapter?

For procedure step reference see example(s):

1. In the deCONZ app, under Settings > Gateway, check the firmware version. It should be version 26720700 or newer.

2. If the version is not up-to-date, select Update to 26720700.

Also see:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.