Configurations lost after upgrade

[xxxserxxx]

My problem is that an upgrade wiped out my configuration. The question is: what's the least amount of work needed to correct my situation?

I update the containers rarely, so it's possible that I'm hitting #5024. I'm not on Windows, and I'm not seeing a similar error. I have power cycled the server, restarted the zwavejs process, and unplugged the controller. I have performed a soft reset.

I'm running zwavejs in a container (via podman) on Linux. I have been successfully running it for over a year, mainly controlling devices through Home Assistant. I have added devices with S2 security, and some without, and have a network of 30 devices. For the container, I'm exposing the device by-id and linking in the zwave config directory with -v for external persistence.

I recently upgraded the container, and most of the configuration disappeared. All of the devices were there, but mqtt got turned on and the WS server got turned off; I'm slowly discovering what all got lost, but it appears the security keys also went away.

My issues are that:

1. No devices have security anymore
2. Several devices show "unknown manufacturer 0xXXX", "0xXXXX" for the product, and "Unknown product 0xXXXX" for the product code.
3. All devices lost their names and locations
4. One device, which I haven't tracked down, is stuck in a spinning red "ProtocolInfo" for the Interview. It's also an "Unknown product" victim, so I don't know which device it is.
5. Some devices will not respond to control. For these, although they are "unknown" manufacturer/products, they will have some detail, e.g. a "Multilevel switch v0" in the details with UI elements. However, changing values does nothing, and refreshing values still shows "undefined" for current value

There's a lot of borked stuff here, obviously. The various json files in the store directory look OK to me - they're valid json or jsonl files, although the node entries are all wiped of data -- the names/locations, for example, are ominously missing.

While I don't know what happened, what it looks like is a failed data migration. Some data came across... or maybe the only reason there's any data at all is because zwavejs pulled it from the controller device. Maybe it wiped everything.

In any case, what I really don't want to do is exclude/include all 30 devices in the house again. Some of them are in-wall, so I'd have to pull them out to get at the QR code to re-include with security. Is this going to be unavoidable, or is there some way to recover this?

Postfix: Re all of the data getting wiped: the config dir was, of course, not included in the nightly backups :-/. I'm assuming this was just an unfortunate sequence of events and that upgrading doesn't often wipe out the configuration; If I could reproduce it, I'd file a bug report. As it is, I have no evidence that this wasn't user error.

Controller: Silicon Labs HubZ ZWave + ZigBee USB
OS: Linux 4.13 x86_64
zwavejs: I can't say what versions were involved, but I'd guess it was 8.8.2 -> 8.8.3

[AlCalzone]

Please enable driver logs, loglevel debug and after startup attach it here as a file (drag & drop into the text field).

[AlCalzone]

Although I'm inclined to believe that your configuration is actually gone. In that case I hope you have your security keys saved somewhere else, or you really have to re-include all secure devices.

[xxxserxxx]

I'm inclined to believe that your configuration is actually gone

Yeah, that's what I was worried about. It was worth a shot (for me) to ask. Thanks for looking, though.

I'm attaching the two logs (z-ui_... and [logs.zip](https://github.com/zwave-js/node-zwave-js/files/10664754/logs.zip)) since they seem to contain different information.

Edit: the link above is incorrectly wrapped in Markdown comment quotes. It should be: logs.zip

[AlCalzone]

Something went wrong with uploading those it seems.

[xxxserxxx]

Hmm. No, they're uploaded; it's just that I somehow wrapped the link in code markup. Try this:

logs.zip

[AlCalzone]

That looks pretty normal, so its not just a hanging stick (which sometimes looks like all nodes are not ready)

[xxxserxxx]

Most everything was working; somehow, when zwavejsmqtt came up, it saw all of the devices, but the ones that had been set up with security couldn't be interviewed properly or controlled. As I mentioned, all name and location information was gone. Not knowing the how everything works, I assumed:

• The configuration was lost.
• ZWaveJS got a list of paired devices from the controller, but
• it had lost the security keys and so couldn't control those devices.

Some devices initially had "unknown" and the states were wonky -- half the motion detectors insisted they saw motion where there was none -- but after a day (or so) the non-security devices settled down and the interview information updated. No devices were "missing," and zwavejs refused to let me remove the security-paired devices as missing because they could still be pinged.

In the end, I only had to exclude and re-pair a couple of devices, so it wasn't horrible; one was a light switch and probably did not need security, but I re-added it anyway because somewhere along the line I've developed the impression that devices paired with security had better metadata and work more reliably. The other was a door lock and I wanted security on that -- I know from experience that this lock doesn't work as well without S2, but I'm assuming that's just an issue with the lock code, which is known to be not particularly good quality. In both cases, the most work was in Home Assistant, as the devices showed up as new, and I had to adjust several scripts and automations.

Anyway, it's all back up and stable now, and I've included the the store directory in nightly backups so I hope to not have this problem any more.

Advice for anyone finding this conversation

If you don't have a back-up of your store, you just have to suck it up and exclude/pair any S2 devices. I recommend figuring out exactly how you named and located the devices before the event. You may be able to get those names from Home Assistant. It's only a suspicion, but I think I may only have had trouble with the automations because I named the two S2 devices slightly differently when I re-included them. It's worth a shot, anyway.

As I still can't find anyone else complaining about an upgrade corrupting their configuration, I'm assuming that this was a PEBKAC issue.

@AlCalzone thanks for talking this through with me, and for all your work on ZWaveJS.

[AlCalzone]

zwavejs refused to let me remove the security-paired devices as missing because they could still be pinged.

This sounds like you were trying to use the "remove failed device" function. This is only for devices that are actually unreachable or no longer exist.
The normal way to exclude requires entering exclusion mode and pressing a button on the device.