-
Notifications
You must be signed in to change notification settings - Fork 854
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Connection lost after changing General Configuration #1331
Comments
Hi @samy4sam. IOTGW-159 |
Hi @samy4sam,
|
Hi @samson0v tb_gateway.json: {
"thingsboard": {
"host": "mqtt.thingsboard.cloud",
"port": 1883,
"remoteShell": true,
"remoteConfiguration": true,
"statistics": {
"enable": true,
"statsSendPeriodInSeconds": 3600
},
"deviceFiltering": {
"enable": false,
"filterFile": "list.json"
},
"maxPayloadSizeBytes": 1024,
"minPackSendDelayMS": 200,
"minPackSizeToSend": 500,
"checkConnectorsConfigurationInSeconds": 60,
"handleDeviceRenaming": true,
"security": {
"type": "accessToken",
"accessToken": "<MY_TOKEN>"
},
"qos": 1,
"checkingDeviceActivity": {
"checkDeviceInactivity": false,
"inactivityTimeoutSeconds": 200,
"inactivityCheckPeriodSeconds": 500
},
"ts": 1710258944803
},
"storage": {
"type": "memory",
"read_records_count": 100,
"max_records_count": 100000,
"data_folder_path": "./data/",
"max_file_count": 10,
"max_read_records_count": 10,
"max_records_per_file": 10000,
"data_file_path": "./data/data.db",
"messages_ttl_check_in_hours": 1,
"messages_ttl_in_days": 7,
"ts": 1710258944803
},
"grpc": {
"enabled": false,
"serverPort": 9595,
"keepaliveTimeMs": 10000,
"keepaliveTimeoutMs": 5000,
"keepalivePermitWithoutCalls": true,
"maxPingsWithoutData": 0,
"minTimeBetweenPingsMs": 10000,
"minPingIntervalWithoutDataMs": 5000,
"keepAliveTimeMs": 10000,
"keepAliveTimeoutMs": 5000
},
"connectors": []
} When I try to enable the remote logs I have another bug. When I press save I see this on the linux server: tb-gateway | 2024-03-12 16:04:13 - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __process_remote_logging_update - 612 - Remote logging has being updated. Current logging level is: DEBUG
tb-gateway | 2024-03-12 16:04:13 - |INFO| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - process_config_request - 242 - Configuration update request received.
tb-gateway | 2024-03-12 16:04:14 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler databaseHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:04:14 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler serviceHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:04:14 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler connectorHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:04:14 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler converterHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:04:14 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler tb_connectionHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:04:14 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler storageHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:04:14 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler extensionHandler not found. Removing from configuration... When I then go back to the overview and reopen the general configuration view and go to the tab Logs, the remote logging is disabled again. So I am not able to enable remote logs. This is my logs.json file after my attempts to enable the logs: "version": 1,
"disable_existing_loggers": false,
"formatters": {
"LogFormatter": {
"class": "logging.Formatter",
"format": "%(asctime)s - |%(levelname)s| - [%(filename)s] - %(module)s - %(funcName)s - %(lineno)d - %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S"
}
},
"handlers": {
"consoleHandler": {
"class": "logging.StreamHandler",
"formatter": "LogFormatter",
"level": "DEBUG",
"stream": "ext://sys.stdout"
}
},
"loggers": {
"database": {
"handlers": [
"consoleHandler"
],
"level": "DEBUG",
"propagate": false
},
"service": {
"handlers": [
"consoleHandler"
],
"level": "INFO",
"propagate": false
},
"connector": {
"handlers": [
"consoleHandler"
],
"level": "INFO",
"propagate": false
},
"converter": {
"handlers": [
"consoleHandler"
],
"level": "INFO",
"propagate": false
},
"tb_connection": {
"handlers": [
"consoleHandler"
],
"level": "INFO",
"propagate": false
},
"storage": {
"handlers": [
"consoleHandler"
],
"level": "INFO",
"propagate": false
},
"extension": {
"handlers": [
"consoleHandler"
],
"level": "INFO",
"propagate": false
}
},
"root": {
"level": "ERROR",
"handlers": [
"consoleHandler"
]
},
"ts": 1710259453434
} |
I edited the logs.json manually on the server to receive debug logs on the server. This are the logs after the change of the remote shell settings and before the connection to the cloud is not working anymore: tb-gateway | 2024-03-12 16:23:39 - |INFO| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - process_config_request - 242 - Configuration update request received.
tb-gateway | 2024-03-12 16:23:39 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - process_config_request - 243 - Got config update request: {'RemoteLoggingLevel': 'NONE', 'logs_configuration': {'version': 1, 'disable_existing_loggers': False, 'formatters': {'LogFormatter': {'class': 'logging.Formatter', 'format': '%(asctime)s - |%(levelname)s| - [%(filename)s] - %(module)s - %(funcName)s - %(lineno)d - %(message)s', 'datefmt': '%Y-%m-%d %H:%M:%S'}}, 'handlers': {'consoleHandler': {'class': 'logging.StreamHandler', 'formatter': 'LogFormatter', 'level': 'DEBUG', 'stream': 'ext://sys.stdout'}, 'databaseHandler': {'class': 'thingsboard_gateway.tb_utility.tb_handler.TimedRotatingFileHandler', 'formatter': 'LogFormatter', 'filename': './logs/database.log', 'backupCount': 1, 'encoding': 'utf-8'}, 'serviceHandler': {'class': 'thingsboard_gateway.tb_utility.tb_handler.TimedRotatingFileHandler', 'formatter': 'LogFormatter', 'filename': './logs/service.log', 'backupCount': 7, 'interval': 3, 'when': 'D', 'encoding': 'utf-8'}, 'connectorHandler': {'class': 'thingsboard_gateway.tb_utility.tb_handler.TimedRotatingFileHandler', 'formatter': 'LogFormatter', 'filename': './logs/connector.log', 'backupCount': 7, 'interval': 3, 'when': 'D', 'encoding': 'utf-8'}, 'converterHandler': {'class': 'thingsboard_gateway.tb_utility.tb_handler.TimedRotatingFileHandler', 'formatter': 'LogFormatter', 'filename': './logs/converter.log', 'backupCount': 7, 'interval': 3, 'when': 'D', 'encoding': 'utf-8'}, 'tb_connectionHandler': {'class': 'thingsboard_gateway.tb_utility.tb_handler.TimedRotatingFileHandler', 'formatter': 'LogFormatter', 'filename': './logs/tb_connection.log', 'backupCount': 7, 'interval': 3, 'when': 'D', 'encoding': 'utf-8'}, 'storageHandler': {'class': 'thingsboard_gateway.tb_utility.tb_handler.TimedRotatingFileHandler', 'formatter': 'LogFormatter', 'filename': './logs/storage.log', 'backupCount': 7, 'interval': 3, 'when': 'D', 'encoding': 'utf-8'}, 'extensionHandler': {'class': 'thingsboard_gateway.tb_utility.tb_handler.TimedRotatingFileHandler', 'formatter': 'LogFormatter', 'filename': './logs/extension.log', 'backupCount': 7, 'interval': 3, 'when': 'D', 'encoding': 'utf-8'}}, 'loggers': {'database': {'handlers': ['databaseHandler', 'consoleHandler'], 'level': 'DEBUG', 'propagate': False}, 'service': {'handlers': ['serviceHandler', 'consoleHandler'], 'level': 'DEBUG', 'propagate': False}, 'connector': {'handlers': ['connectorHandler', 'consoleHandler'], 'level': 'INFO', 'propagate': False}, 'converter': {'handlers': ['converterHandler', 'consoleHandler'], 'level': 'INFO', 'propagate': False}, 'tb_connection': {'handlers': ['tb_connectionHandler', 'consoleHandler'], 'level': 'DEBUG', 'propagate': False}, 'storage': {'handlers': ['storageHandler', 'consoleHandler'], 'level': 'INFO', 'propagate': False}, 'extension': {'handlers': ['extensionHandler', 'consoleHandler'], 'level': 'INFO', 'propagate': False}}, 'root': {'level': 'ERROR', 'handlers': ['consoleHandler']}, 'ts': 1710260618743}, 'grpc_configuration': {'enabled': False, 'ts': 1710260618743}, 'storage_configuration': {'type': 'memory', 'read_records_count': 100, 'max_records_count': 100000, 'data_folder_path': './data/', 'max_file_count': 10, 'max_read_records_count': 10, 'max_records_per_file': 10000, 'data_file_path': './data/data.db', 'messages_ttl_check_in_hours': 1, 'messages_ttl_in_days': 7, 'ts': 1710260618743}, 'general_configuration': {'host': 'mqtt.thingsboard.cloud', 'port': 1883, 'remoteShell': True, 'remoteConfiguration': True, 'checkConnectorsConfigurationInSeconds': 60, 'statistics': {'enable': True, 'statsSendPeriodInSeconds': 3600, 'commands': []}, 'maxPayloadSizeBytes': 1024, 'minPackSendDelayMS': 200, 'minPackSizeToSend': 500, 'handleDeviceRenaming': True, 'checkingDeviceActivity': {'checkDeviceInactivity': False, 'inactivityTimeoutSeconds': 200, 'inactivityCheckPeriodSeconds': 500}, 'security': {'type': 'accessToken', 'accessToken': '<MY_TOKEN>'}, 'qos': 1, 'ts': 1710260618743}}
tb-gateway | 2024-03-12 16:23:39 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 386 - Processing logs configuration update...
tb-gateway | 2024-03-12 16:23:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler databaseHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:23:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler serviceHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:23:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler connectorHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:23:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler converterHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:23:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler tb_connectionHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:23:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler storageHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:23:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 396 - Handler extensionHandler not found. Removing from configuration...
tb-gateway | 2024-03-12 16:23:39 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 411 - Logs configuration has been updated.
tb-gateway | 2024-03-12 16:23:39 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_grpc_configuration_update - 360 - Processing GRPC configuration update...
tb-gateway | 2024-03-12 16:23:39 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_storage_configuration_update - 341 - Processing storage configuration update...
tb-gateway | 2024-03-12 16:23:39 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_storage_configuration_update - 357 - Processed storage configuration update successfully
tb-gateway | 2024-03-12 16:23:39 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_general_configuration_update - 286 - Processing general configuration update
tb-gateway | 2024-03-12 16:23:39 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_general_configuration_update - 288 - --- Checking connection configuration changes...
tb-gateway | 2024-03-12 16:23:39 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_general_configuration_update - 294 - ---- Connection configuration changed. Processing...
tb-gateway | 2024-03-12 16:23:39 - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_unsubscribe - 189 - Unsubscribed from *|*, subscription id '*'
tb-gateway | 2024-03-12 16:23:39 - |DEBUG| - [tb_client.py] - tb_client - run - 264 - connecting to ThingsBoard
tb-gateway | 2024-03-12 16:23:39 - |DEBUG| - [tb_client.py] - tb_client - _on_connect - 217 - TB client <paho.mqtt.client.Client object at 0x7fe38c0d0390> connected to ThingsBoard
tb-gateway | 2024-03-12 16:23:40 - |DEBUG| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - _on_subscribe - 70 - Service subscription to topic v1/gateway/attributes - successfully completed.
tb-gateway | 2024-03-12 16:23:40 - |DEBUG| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - _on_subscribe - 70 - Service subscription to topic v1/gateway/attributes/response - successfully completed.
tb-gateway | 2024-03-12 16:23:40 - |DEBUG| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - _on_subscribe - 70 - Service subscription to topic v1/gateway/rpc - successfully completed.
tb-gateway | 2024-03-12 16:23:40 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_general_configuration_update - 301 - --- Checking statistics configuration changes...
tb-gateway | 2024-03-12 16:23:40 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_general_configuration_update - 309 - --- Statistics configuration not changed.
tb-gateway | 2024-03-12 16:23:40 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_general_configuration_update - 311 - --- Checking device filtering configuration changes...
tb-gateway | 2024-03-12 16:23:40 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_general_configuration_update - 313 - ---- Device filtering configuration changed. Processing...
tb-gateway | 2024-03-12 16:23:40 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_general_configuration_update - 320 - --- Checking Remote Shell configuration changes...
tb-gateway | 2024-03-12 16:23:40 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_general_configuration_update - 322 - ---- Remote Shell configuration changed. Processing...
tb-gateway | 2024-03-12 16:23:40 - |WARNING| - [tb_gateway_service.py] - tb_gateway_service - init_remote_shell - 419 - Remote shell is enabled. Please be carefully with this feature.
tb-gateway | 2024-03-12 16:23:40 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_general_configuration_update - 329 - --- Checking other configuration parameters changes...
tb-gateway | 2024-03-12 16:23:40 - |DEBUG| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_general_configuration_update - 332 - --- Saving new general configuration... |
Hello @samson0v I am having the same issue as @samy4sam with regards to enabling remote logging on the gateway. Is this in the pipeline to be fixed and if not is there a way I can manually edit the logs.json like he did to enable remote logging if I'm using thingsboard cloud. Thank you |
Hi, it is an issue with re-connection, we will fix it asap. |
@imbeacon After reopening, the remote shell is off again: I also get now this error in the logs: tb-gateway | 2024-03-18 12:42:39 - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __process_remote_logging_update - 607 - Remote logging has being deactivated.
tb-gateway | 2024-03-18 12:42:39 - |INFO| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - process_config_request - 244 - Configuration update request received.
tb-gateway | 2024-03-18 12:42:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 400 - Handler databaseHandler not found. Removing from configuration...
tb-gateway | 2024-03-18 12:42:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 400 - Handler serviceHandler not found. Removing from configuration...
tb-gateway | 2024-03-18 12:42:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 400 - Handler connectorHandler not found. Removing from configuration...
tb-gateway | 2024-03-18 12:42:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 400 - Handler converterHandler not found. Removing from configuration...
tb-gateway | 2024-03-18 12:42:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 400 - Handler tb_connectionHandler not found. Removing from configuration...
tb-gateway | 2024-03-18 12:42:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 400 - Handler storageHandler not found. Removing from configuration...
tb-gateway | 2024-03-18 12:42:39 - |WARNING| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _handle_logs_configuration_update - 400 - Handler extensionHandler not found. Removing from configuration...
tb-gateway | 2024-03-18 12:42:40 - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_unsubscribe - 189 - Unsubscribed from *|*, subscription id '*'
tb-gateway | 2024-03-18 12:42:41 - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 181 - Subscribed to *|* with id 1 for device *
tb-gateway | 2024-03-18 12:42:41 - |ERROR| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - process_config_request - 263 - Unknown attribute update name (Available: general_configuration, storage_configuration, grpc_configuration, logs_configuration, active_connectors, RemoteLoggingLevel, (?=\D*\d?).*) |
Describe the bug
I encountered a critical issue with the gateway configuration that leads to loss of connectivity and necessitates a container reset. This problem arises when I modify the General Configuration settings of a newly set up gateway, specifically enabling the Remote Shell option to "true." Following this adjustment, the gateway becomes inaccessible. Attempts to regain connection, including restarting the gateway container, prove futile. The only remedy is to delete and recreate the docker container. Upon setting up a new container, the connection is restored, and the remote shell functionality is operational.
Moreover, since the introduction of cloud-based gateway configuration updates (replacing the previous YAML file setup), I have faced challenges in synchronizing the cloud settings with the gateway's local settings. This has raised concerns about the effectiveness of the new configuration process.
Questions:
Has anyone else experienced similar issues with connection loss after enabling the remote shell setting?
Are there known problems related to synchronizing cloud settings with the gateway since the shift from YAML file configuration to cloud-based configuration?
Steps to Reproduce:
Expected Behavior:
The gateway should remain accessible, and the remote shell functionality should be enabled without losing the connection or requiring a container reset.
Actual Behavior:
The connection to the gateway is lost upon enabling the remote shell setting. The connection cannot be re-established without deleting and creating a new docker container.
Workaround:
Deleting the existing gateway container and creating a new one restores the connection and enables the remote shell functionality.
Error traceback (If available):
Versions (please complete the following information):
The text was updated successfully, but these errors were encountered: