MQTT Connection interrupted with error: `libaws-c-mqtt: AWS_ERROR_MQTT_UNEXPECTED_HANGUP, The connection was closed unexpectedly

[ramg44]

After providing the endpoint in the .conf file also it is throwing error like below ERROR] {Config.cpp}: *** AWS IOT DEVICE CLIENT FATAL ERROR: Endpoint is missing

Loaded: loaded (/etc/systemd/system/aws-iot-device-client.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2024-07-12 12:57:18 EDT; 1min 15s ago
Process: 287010 ExecStart=/sbin/aws-iot-device-client --config-file $CONF_PATH (code=exited, status=1/FAILURE)
Main PID: 287010 (code=exited, status=1/FAILURE)

Jul 12 12:57:18 ubuntu aws-iot-device-client[287012]: 2024-07-12T16:57:18.763Z [INFO] {Config.cpp}: Succesfully create default file: /root/.aws-iot-device-client/sample-shadow/default-sample-shadow-doc>

Jul 12 12:57:18 ubuntu aws-iot-device-client[287012]: 2024-07-12T16:57:18.768Z [DEBUG] {Config.cpp}: Did not find a runtime configuration file, assuming Fleet Provisioning has not run for this device
Jul 12 12:57:18 ubuntu aws-iot-device-client[287012]: 2024-07-12T16:57:18.769Z [DEBUG] {Config.cpp}: Did not find a http proxy config file /root/.aws-iot-device-client/http-proxy.conf, assuming HTTP pro>
Jul 12 12:57:18 ubuntu aws-iot-device-client[287012]: 2024-07-12T16:57:18.770Z [ERROR] {Config.cpp}: *** AWS IOT DEVICE CLIENT FATAL ERROR: Endpoint is missing ***
Jul 12 12:57:18 ubuntu aws-iot-device-client[287012]: 2024-07-12T16:57:18.771Z [ERROR] {Main.cpp}: *** AWS IOT DEVICE CLIENT FATAL ERROR: AWS IoT Device Client must abort execution, reason: Invalid conf>
Jul 12 12:57:18 ubuntu aws-iot-device-client[287012]: 2024-07-12T16:57:18.772Z [DEBUG] {SharedCrtResourceManager.cpp}: Attempting to disconnect MQTT connection
Jul 12 12:57:18 ubuntu aws-iot-device-client[287012]: AWS IoT Device Client must abort execution, reason: Invalid configuration
Jul 12 12:57:18 ubuntu aws-iot-device-client[287012]: Please check the AWS IoT Device Client logs for more information
Jul 12 12:57:18 ubuntu systemd[1]: aws-iot-device-client.service: Main process exited, code=exited, status=1/FAILURE
Jul 12 12:57:18 ubuntu systemd[1]: aws-iot-device-client.service: Failed with result 'exit-code'.

Note :

But when i execute the bin file in /opt/aws-iot-device-client/build by passing the conf file getting below error
root@ubuntu:/opt/aws-iot-device-client/build# ./aws-iot-device-client --config-file /etc/.aws-iot-device-client/aws-iot-device-client.conf
AWS IoT Device Client must abort execution, reason: Failed to establish MQTT connection due to credential/configuration error
Please check the AWS IoT Device Client logs for more information
root@ubuntu:/opt/aws-iot-device-client/build#

below is the error i see in sdk.log

[ERROR] [2024-07-12T18:33:27Z] [0000ffff92ffbd50] [tls-handler] - ctx: Failed to add certificate and private key: No certificate in PEM (Error encountered in /opt/aws-iot-device-client/build/aws-iot-device-sdk-cpp-v2-src/crt/aws-crt-cpp/crt/s2n/crypto/s2n_certificate.c:55)

[HarshGandhi-AWS]

Hello @ramg44 , from the description of this issue it looks like you are not setting up your device correctly. I believe you might be making some mistake while starting the client. Can you share the steps to reproduce this issue?

[ramg44]

yeah, when starting it as a service seems like i have not set it up correctly. So what i have done is passed the config file to the executable file. which then picks up the config file but now i see a different error in the logs. could you please help me on this ?

Below is thee command i have executed in the build folder to start the client
root@ubuntu:/opt/aws-iot-device-client/build# ./aws-iot-device-client --config-file /etc/.aws-iot-device-client/aws-iot-device-client.conf

Below is the config file:

{
  "endpoint":	"a1***j6e99.iot.us-east-2.amazonaws.com",
  "cert":	"/opt/aws-iot-device-client/identity/certificate.pem.crt",
  "key":	"/opt/aws-iot-device-client/identity/private.pem.key",
  "root-ca":	"/opt/aws-iot-device-client/identity/ca.pem",
  "thing-name":	"greengrass-*-jetson",
  "logging":	{
    "level":	"DEBUG",
    "type":	"FILE",
    "file": "/var/log/aws-iot-device-client/aws-iot-device-client.log",
    "enable-sdk-logging":	true,
    "sdk-log-level":	"TRACE",
    "sdk-log-file": "/var/log/aws-iot-device-client/sdk.log"
  },
  "jobs":	{
    "enabled":	false,
    "handler-directory": "/etc/.aws-iot-device-client/jobs"
  },
  "tunneling":	{
    "enabled":	true
  },
  "device-defender":	{
    "enabled":	false,
    "interval": 300
  },
  "fleet-provisioning":	{
    "enabled":	false,
    "template-name": "",
    "template-parameters": "",
    "csr-file": "",
    "device-key": ""
  },
  "samples": {
    "pub-sub": {
      "enabled": false,
      "publish-topic": "",
      "publish-file": "/etc/.aws-iot-device-client/pubsub/publish-file.txt",
      "subscribe-topic": "",
      "subscribe-file": "/etc/.aws-iot-device-client/pubsub/subscribe-file.txt"
    }
  },
  "config-shadow":	{
    "enabled":	false
  },
  "sample-shadow": {
    "enabled": false,
    "shadow-name": "",
    "shadow-input-file": "",
    "shadow-output-file": ""
  }
}

Here i the error i see in the logs

2024-07-13T01:18:08.950Z [DEBUG] {Retry.cpp}: Retryable function returned unsuccessfully, sleeping for 900000 milliseconds
2024-07-13T01:33:08.954Z [INFO] {SharedCrtResourceManager.cpp}: Establishing MQTT connection with client id greengrass-poc-jetson...
2024-07-13T01:33:23.611Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection failed with error: aws-c-io: AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE, TLS (SSL) negotiation failed
2024-07-13T01:33:23.611Z [ERROR] {SharedCrtResourceManager.cpp}: Failed to establish shared MQTT connection, but will attempt retry...
2024-07-13T01:33:23.611Z [DEBUG] {Retry.cpp}: Retryable function returned unsuccessfully, sleeping for 900000 milliseconds
2024-07-13T01:48:23.615Z [INFO] {SharedCrtResourceManager.cpp}: Establishing MQTT connection with client id greengrass-poc-jetson...
2024-07-13T01:48:50.638Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection failed with error: aws-c-io: AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE, TLS (SSL) negotiation failed
2024-07-13T01:48:50.638Z [ERROR] {SharedCrtResourceManager.cpp}: Failed to establish shared MQTT connection, but will attempt retry...
2024-07-13T01:48:50.638Z [DEBUG] {Retry.cpp}: Retryable function returned unsuccessfully, sleeping for 900000 milliseconds
2024-07-13T02:03:50.642Z [INFO] {SharedCrtResourceManager.cpp}: Establishing MQTT connection with client id greengrass-poc-jetson...
2024-07-13T02:04:05.387Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection failed with error: aws-c-io: AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE, TLS (SSL) negotiation failed
2024-07-13T02:04:05.387Z [ERROR] {SharedCrtResourceManager.cpp}: Failed to establish shared MQTT connection, but will attempt retry...
2024-07-13T02:04:05.387Z [DEBUG] {Retry.cpp}: Retryable function returned unsuccessfully, sleeping for 900000 milliseconds
2024-07-13T02:19:05.392Z [INFO] {SharedCrtResourceManager.cpp}: Establishing MQTT connection with client id greengrass-poc-jetson...
2024-07-13T02:19:16.745Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection failed with error: aws-c-io: AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE, TLS (SSL) negotiation failed
2024-07-13T02:19:16.745Z [ERROR] {SharedCrtResourceManager.cpp}: Failed to establish shared MQTT connection, but will attempt retry...
2024-07-13T02:19:16.745Z [DEBUG] {Retry.cpp}: Retryable function returned unsuccessfully, sleeping for 900000 milliseconds
2024-07-13T02:34:16.749Z [INFO] {SharedCrtResourceManager.cpp}: Establishing MQTT connection with client id greengrass-poc-jetson...
2024-07-13T02:34:33.709Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection failed with error: aws-c-io: AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE, TLS (SSL) negotiation failed
2024-07-13T02:34:33.709Z [ERROR] {SharedCrtResourceManager.cpp}: Failed to establish shared MQTT connection, but will attempt retry...
2024-07-13T02:34:33.709Z [DEBUG] {Retry.cpp}: Retryable function returned unsuccessfully, sleeping for 900000 milliseconds
2024-07-13T02:49:33.713Z [INFO] {SharedCrtResourceManager.cpp}: Establishing MQTT connection with client id greengrass-poc-jetson...
2024-07-13T02:49:49.890Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection failed with error: aws-c-io: AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE, TLS (SSL) negotiation failed
2024-07-13T02:49:49.890Z [ERROR] {SharedCrtResourceManager.cpp}: Failed to establish shared MQTT connection, but will attempt retry...
2024-07-13T02:49:49.890Z [DEBUG] {Retry.cpp}: Retryable function returned unsuccessfully, sleeping for 900000 milliseconds

[ramg44]

@HarshGandhi-AWS now seeing the below error after using the ats endpoint

2024-07-15T13:46:11.859Z [INFO] {Main.cpp}: Sensor Publish is disabled
2024-07-15T13:46:11.859Z [INFO] {SharedCrtResourceManager.cpp}: Starting Device Client features.
2024-07-15T13:46:11.859Z [DEBUG] {FeatureRegistry.cpp}: Attempting to start Secure Tunneling
2024-07-15T13:46:11.859Z [INFO] {SecureTunnelingFeature.cpp}: Running Secure Tunneling!
2024-07-15T13:46:11.859Z [INFO] {Main.cpp}: Client base has been notified that Secure Tunneling has started
2024-07-15T13:46:11.930Z [DEBUG] {SecureTunnelingFeature.cpp}: Subscribed to tunnel notification topic
2024-07-15T13:46:12.883Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection interrupted with error: libaws-c-mqtt: AWS_ERROR_MQTT_UNEXPECTED_HANGUP, The connection was closed unexpectedly.. Device Client will retry connection until it is successfully connected to the core.
2024-07-15T13:46:28.177Z [INFO] {SharedCrtResourceManager.cpp}: MQTT connection resumed with return code: 0
2024-07-15T13:46:30.070Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection interrupted with error: libaws-c-mqtt: AWS_ERROR_MQTT_UNEXPECTED_HANGUP, The connection was closed unexpectedly.. Device Client will retry connection until it is successfully connected to the core.
2024-07-15T13:47:00.391Z [INFO] {SharedCrtResourceManager.cpp}: MQTT connection resumed with return code: 0
2024-07-15T13:47:01.492Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection interrupted with error: libaws-c-mqtt: AWS_ERROR_MQTT_UNEXPECTED_HANGUP, The connection was closed unexpectedly.. Device Client will retry connection until it is successfully connected to the core.
2024-07-15T13:48:01.829Z [INFO] {SharedCrtResourceManager.cpp}: MQTT connection resumed with return code: 0
2024-07-15T13:48:02.832Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection interrupted with error: libaws-c-mqtt: AWS_ERROR_MQTT_UNEXPECTED_HANGUP, The connection was closed unexpectedly.. Device Client will retry connection until it is successfully connected to the core.
2024-07-15T13:50:03.273Z [INFO] {SharedCrtResourceManager.cpp}: MQTT connection resumed with return code: 0
2024-07-15T13:50:03.911Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection interrupted with error: libaws-c-mqtt: AWS_ERROR_MQTT_UNEXPECTED_HANGUP, The connection was closed unexpectedly.. Device Client will retry connection until it is successfully connected to the core.
2024-07-15T13:54:04.323Z [INFO] {SharedCrtResourceManager.cpp}: MQTT connection resumed with return code: 0
2024-07-15T13:54:05.239Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection interrupted with error: libaws-c-mqtt: AWS_ERROR_MQTT_UNEXPECTED_HANGUP, The connection was closed unexpectedly.. Device Client will retry connection until it is successfully connected to the core.

[ramg44]

@HarshGandhi-AWS i see below error for disconnection,

{
"clientId": "greengrass-poc-jetson",
"timestamp": 1721057131716,
"eventType": "disconnected",
"clientInitiatedDisconnect": false,
"sessionIdentifier": "d57e62be-4b4a-8aca-f14dea14ac40",
"principalIdentifier": "da63b3f6370a70a68548ee90b4dcca1c603a46b64435afc0535",
"disconnectReason": "DUPLICATE_CLIENTID",
"versionNumber": 78
}

[ramg44]

there were two certificated one with greengrass and the other with thing i created. deactivated one. Then it started working