Startup error: No enum constant tech.pegasys.teku.storage.server.StateStorageMode

[lucassaldanha]

Description

From Discord:

I just had a power failure, and my there is some problem getting up and running again. I get this message "main | FATAL | BeaconNodeCommand | No enum constant tech.pegasys.teku.storage.server.StateStorageMode."

This isn't the first time I have seen this issue happening. For some reason, the file <data_folder>/beacon/data-storage-mode.txt gets corrupted or something and causes this issue.

Need to investigate under what circumstances this happens and how we can better handle it.

[Detailed description of the problem and the impact it has]

Steps to Reproduce (Bug)

Modify an existing data-storage-mode.txt file in the beacon data directory with an invalid value, or empty. Then start teku.

Expected behavior:
Service should be able to start with no issues. Or should warn the user when something is wrong (and how to fix it).

1. If user set a storage option, we ALWAYS stick to that option. That option must override whatever is on data-storage-mode.txt. We also log a message to let the user know.
2. If user does not set a storage option, and we have a valid data-storage-mode.txt file, we use the value on the file to configure the database;
3. If user does not set the storage option, and we have a invalid data-storage-mode.txt file, we should warn the user and halt execution. In this case, we don't have a reliable way of deciding what mode makes sense. So we are going to stop and print an error message letting the user know that something is wrong, and that they should restart the node using the option --data-storage-mode to override the existing cached config. In the end, data-storage-mode.txt will be updated with the mode selected by the user using the option --data-storage-mode.

Actual behavior:
Node does not start and gives the error: main | FATAL | BeaconNodeCommand | No enum constant tech.pegasys.teku.storage.server.StateStorageMode.

Frequency:
Once the conditions to trigger this problem appear, they happen consistently, until some manual intervention happens (e.g. deleting the data-storage-mode.txt file so it gets recreated.

Versions (Add all that apply)

TODO

• Software version: [teku --version or look for log lines starting with Teku version:]
• Java version: [java -version]
• OS Name & Version: [cat /etc/*release]
• Docker Version: [docker version]
• Cloud VM, type, size: [Amazon Web Services I3-large]

[haavardf]

... 14 more
2024-06-06 09:55:16.845+10:00 | main | INFO | ValidatorConfig | '--validators-builder-registration-default-enabled' requires '--validators-proposer-blinded-blocks-enabled', enabling it
2024-06-06 09:55:16.864+10:00 | main | FATAL | BeaconNodeCommand | No enum constant tech.pegasys.teku.storage.server.StateStorageMode.
tech.pegasys.teku.infrastructure.exceptions.InvalidConfigurationException: No enum constant tech.pegasys.teku.storage.server.StateStorageMode.
at tech.pegasys.teku.cli.BeaconNodeCommand.tekuConfiguration(BeaconNodeCommand.java:420) [teku-24.4.0.jar:24.4.0]
at tech.pegasys.teku.cli.BeaconNodeCommand.call(BeaconNodeCommand.java:338) [teku-24.4.0.jar:24.4.0]
at tech.pegasys.teku.cli.BeaconNodeCommand.call(BeaconNodeCommand.java:77) [teku-24.4.0.jar:24.4.0]
at picocli.CommandLine.executeUserObject(CommandLine.java:2041) [picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine.access$1500(CommandLine.java:148) [picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) [picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) [picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) [picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) [picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) [picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine.execute(CommandLine.java:2170) [picocli-4.7.5.jar:4.7.5]
at tech.pegasys.teku.cli.BeaconNodeCommand.parse(BeaconNodeCommand.java:271) [teku-24.4.0.jar:24.4.0]
at tech.pegasys.teku.Teku.start(Teku.java:63) [teku-24.4.0.jar:24.4.0]
at tech.pegasys.teku.Teku.startFromCLIArgs(Teku.java:86) [teku-24.4.0.jar:24.4.0]
at tech.pegasys.teku.Teku.main(Teku.java:40) [teku-24.4.0.jar:24.4.0]
Caused by: java.lang.IllegalArgumentException: No enum constant tech.pegasys.teku.storage.server.StateStorageMode.
at java.base/java.lang.Enum.valueOf(Enum.java:274) ~[?:?]
at tech.pegasys.teku.storage.server.StateStorageMode.valueOf(StateStorageMode.java:16) ~[teku-storage-24.4.0.jar:24.4.0]
at tech.pegasys.teku.storage.server.StorageConfiguration$Builder.getStorageModeFromPersistedDatabase(StorageConfiguration.java:291) ~[teku-storage-24.4.0.jar:24.4.0]
at tech.pegasys.teku.storage.server.StorageConfiguration$Builder.determineDataStorageMode(StorageConfiguration.java:273) ~[teku-storage-24.4.0.jar:24.4.0]
at tech.pegasys.teku.storage.server.StorageConfiguration$Builder.build(StorageConfiguration.java:251) ~[teku-storage-24.4.0.jar:24.4.0]
at tech.pegasys.teku.config.TekuConfiguration$Builder.build(TekuConfiguration.java:236) ~[teku-24.4.0.jar:24.4.0]
at tech.pegasys.teku.cli.BeaconNodeCommand.tekuConfiguration(BeaconNodeCommand.java:418) [teku-24.4.0.jar:24.4.0]
... 14 more
2024-06-06 09:55:30.052+10:00 | main | INFO | ValidatorConfig | '--validators-builder-registration-default-enabled' requires '--validators-proposer-blinded-blocks-enabled', enabling it
2024-06-06 09:55:30.070+10:00 | main | FATAL | BeaconNodeCommand | No enum constant tech.pegasys.teku.storage.server.StateStorageMode.

[haavardf]

teku-24.3.1
Then upgraded to
teku-24.4.0
Same behaviour

oracle-java-20

Ubuntu 22.04.4 LTS

Configuration | Network: mainnet, Storage Mode: MINIMAL
Host Configuration | Maximum Heap Size: 10.00 GB, Total Memory: 62.82 GB, CPU Cores: 24