Solana 1.16.x and keep restarting

[mohsen-vybenetwork]

Hey Guys,
We've been using your fork in our production environment with Solana 1.14.x without any issues. However, since introducing version 1.16.x with the same configuration and a new build based on the latest version from the master branch, our RPC keeps restarting immediately after starting the Geyser plugin.

Just for context, we have different configurations with varying filters. It worked fine on V1.14.X, but we've encountered issues running a node with 1.16.x.

[2023-10-03T06:21:20.823418788Z WARN solana_core::validator] vote account: 35WnonKXQEbYSUZT3kwWEgMqinNw4djMptkzpapySrQ1 [2023-10-03T06:21:20.823429578Z INFO solana_metrics::metrics] datapoint: bootstrap-snapshot-download total_time_secs=72i get_rpc_nodes_time_secs=71i snapshot_download_time_secs=0i download_abort_count=0i blacklisted_nodes_count=12i [2023-10-03T06:21:20.823613263Z INFO solana_geyser_plugin_manager::geyser_plugin_service] Starting GeyserPluginService from config files: ["/solana/target/release/config.json", "/solana/target/release/config-m2m_zeta.json"] [2023-10-03T06:21:20.824975228Z INFO solana_accountsdb_plugin_kafka::plugin] Loading plugin "KafkaPlugin" from config_file "/solana/target/release/config.json" [2023-10-03T06:21:20.825231145Z INFO solana_accountsdb_plugin_kafka::plugin] rd_kafka_version: 0x10902ff, 1.9.2 [2023-10-03T06:21:20.830809551Z INFO solana_accountsdb_plugin_kafka::plugin] Created rdkafka::FutureProducer [2023-10-03T06:21:20.830824731Z INFO solana_accountsdb_plugin_kafka::plugin] Spawned producer [2023-10-03T06:21:22.628328201Z INFO solana_validator] solana-validator 1.16.14 (src:c932953e; feat:4033350765, client:SolanaLabs) [2023-10-03T06:21:22.628347112Z INFO solana_validator] Starting validator with: ArgsOs { inner: [ "/home/solana/.local/share/solana/install/active_release/bin/solana-validator", "--identity", "/home/solana/identity.json", "--entrypoint", "entrypoint.mainnet-beta.solana.com:8001", "--entrypoint", "entrypoint2.mainnet-beta.solana.com:8001",

[fanatid]

Hi @mohsen-vybenetwork, is it all logs from geyser? what is issue? node restart itself?
you also can check our grpc plugin and grpc -> kafka tool
https://github.com/rpcpool/yellowstone-grpc/tree/master/yellowstone-grpc-kafka

[mohsen-vybenetwork]

Hi @fanatid. Is there any way to exclude Geyser logs or increase the log level to know what is the issue?(RPC log severity is info) I've been checking the RPC logs, and the RPC itself keeps restarting just after starting the Geyser plugin. Everything is okay and works fine with Version 1.14.x. The main repository from Blockdaemon also works fine with Version 1.16.x, but it doesn't support filtering for accounts and transactions. However, we haven't been able to run your fork with Version 1.16.x.

[fanatid]

What is the node logs right before the restart?

[mohsen-vybenetwork]

Nothing special, just restarting after starting the geyser plugin and spawning the producer

[2023-10-03T06:21:20.822627517Z INFO solana_validator::bootstrap] RPC node root slot: 221258529
[2023-10-03T06:21:20.823405567Z WARN solana_core::validator] identity:
2Btqz4LWamivaC1fYkK3m2Udn55QGLksazqYR46DQw9q
[2023-10-03T06:21:20.823418788Z WARN solana_core::validator] vote account:
35WnonKXQEbYSUZT3kwWEgMqinNw4djMptkzpapySrQ1
[2023-10-03T06:21:20.823429578Z INFO solana_metrics::metrics] datapoint: bootstrap-snapshot-download total_time_secs=72i
get_rpc_nodes_time_secs=71i snapshot_download_time_secs=0i download_abort_count=0i blacklisted_nodes_count=12i
[2023-10-03T06:21:20.823613263Z INFO solana_geyser_plugin_manager::geyser_plugin_service] Starting GeyserPluginService from
config files: ["/solana/target/release/config.json", "/solana/target/release/config-m2m_zeta.json"]
[2023-10-03T06:21:20.824975228Z INFO solana_accountsdb_plugin_kafka::plugin] Loading plugin "KafkaPlugin" from config_file "/solana/target/release/config.json"
[2023-10-03T06:21:20.825231145Z INFO solana_accountsdb_plugin_kafka::plugin] rd_kafka_version: 0x10902ff, 1.9.2
[2023-10-03T06:21:20.830809551Z INFO solana_accountsdb_plugin_kafka::plugin] Created rdkafka::FutureProducer
[2023-10-03T06:21:20.830824731Z INFO solana_accountsdb_plugin_kafka::plugin] Spawned producer
[2023-10-03T06:21:22.628328201Z INFO solana_validator] solana-validator 1.16.14 (src:c932953e; feat:4033350765, client:SolanaLabs)
[2023-10-03T06:21:22.628347112Z INFO solana_validator] Starting validator with: ArgsOs {
inner: [
"/home/solana/.local/share/solana/install/active_release/bin/solana-validator",
"--identity",
"/home/solana/identity.json",
"--entrypoint",
"entrypoint.mainnet-beta.solana.com:8001",
"--entrypoint",
"entrypoint2.mainnet-beta.solana.com:8001",
"--entrypoint",
"entrypoint3.mainnet-beta.solana.com:8001",

[mohsen-vybenetwork]

@fanatid, I believe I've identified the issue. As I mentioned earlier, we have different configurations for various purposes and topics. When we use only one configuration, it works fine, but when we add more configurations, the RPC gets restarted. I suspect this may be related to supporting geyser hot reload in version 1.16.x. Supporting multiple configs works fine with V1.14.x.

--geyser-plugin-config /solana/target/release/config.json
--geyser-plugin-config /solana/target/release/config2.json \

[mohsen-vybenetwork]

@fanatid, is there any chance you could investigate and address the issue to support multiple configs with V1.16.x?

[fanatid]

1.16 support multiple configs, we run some nodes with few plugins

[mohsen-vybenetwork]

So interesting. Do you have any idea why my RPC is getting restarted when using multiple configurations? How should I define multiple plugin names? When I use a single library like this

'libpath': '/solana/target/release/libsolana_accountsdb_plugin_kafka.so',

it throws the following error:

'Failed to start validator: "Failed to load the Geyser plugin: FailedToLoadPlugin(Error { code: InvalidRequest, message: 'There already exists a plugin named KafkaPlugin loaded. Did not load requested plugin', data: None })'
I tried making a copy of the main library with a different name and defined it in the config like
'libpath': '/solana/target/release/libsolana_accountsdb_plugin_kafka2.so,'
but it still fails and got restarted just right after starting geyser
.

[fanatid]

to change the name you need recompile with changes string here

solana-accountsdb-plugin-kafka/src/plugin.rs

Line 51 in 648fc34

"KafkaPlugin"

[mohsen-vybenetwork]

Perfect. Solved my issue. Thank you @fanatid

[fanatid]

I still highly recommend you to check gRPC and gRPC -> kafka producer -- this would be much easier for you! If you have any questions you can open an issue in gRPC repo 🙂

[mohsen-vybenetwork]

Awesome. Thank you @fanatid for all your helps. Will check it.

[fanatid]

@mohsen-vybenetwork I created PR in Solana repo which should help with using the same plugins solana-labs/solana#33550