As a PDS operator, I want to know the health of the registry API service

[jimmie]

This indication will be based on a new API endpoint specific to indicating the health of the service. The primary purpose of this is to provide a comprehensive assessment of if the service is healthy, to be used as the healthcheck for the service listener.

Sending a request to this endpoint will provide a summary of the following:

• Opensearch connectivity
• Springboot metrics, if possible
• Non-sensitive config details: active remotes (in CCS context), active indices (in new MT context)
• JVM information: Runtime.freeMemory(), Runtime.maxMemory(), Runtime.version()

Failures or certain levels of the above info will result in a non-200 return code.

The determined state of health will be:

• returned in a JSON response payload. This will be disregarded by the listener group healthcheck but will provide key information to operators if invoked directly
• recorded in the service log file (i.e. stdout/stderr which is recorded in CloudWatch logs)

Once available, we will need to update the terraform scripts to include this endpoint in the listener healthcheck definition.

Acceptance Criteria

Given a nominal running Registry API and OpenSearch registry
When I perform a query of the healthcheck/ endpoint
Then I expect to receive a 200 response and metadata indicating a successful running application

Given a running Registry API and OpenSearch registry, with an off-nominal state for the Registry API
When I perform a query of the healthcheck/ endpoint
Then I expect to receive TBD response code(s) and applicable metadata

Given a running Registry API and Registry (OpenSearch), with an off-nominal state for the Registry (OpenSearch)
When I perform a query of the healthcheck/ endpoint
Then I expect to receive TBD response code(s) and applicable metadata

Sub-tasks

• Develop Postman test cases for new healthcheck endpoint #350

[jimmie]

Derived from registry-api #297

[jordanpadams]

@jimmie can we also be sure to include a check that there is non-zero results returned? basically just making sure there is actually data in the registry too, not just that everything is running.

[jimmie]

Sure, we can do that - maybe get a count of documents and include that in the return payload? Note that this will not constitute a failure on the part of the healthcheck (i.e. there may be zero documents but if everything else checks out OK a 200 will still be returned) since the intention for this is to convey to ECS/Fargate of whether the task needs to be recycled.

[tloubrieu-jpl]

@jimmie what are you thinking of, regarding the springboot metrics ?

[jimmie]

@tloubrieu-jpl I am not 100% sure, I did a quick (very quick) scan and saw hints that there may be some useful information but it may prove to be very difficult to access. I was hoping for something like # of requests, response rates, etc to include in the response payload but maybe I'm being too optimistic.

[tloubrieu-jpl]

Thanks @jimmie I was seeing the springboot can provide a specific URL for those metrics, maybe we could expose it separatelly, maybe with a logn/passowrd protection. It is not critical and not part of this ticket anyway.

[jordanpadams]

@jimmie can you help with the off-nominal response code(s) expected for the 2 failure acceptance criteria described in the original ticket above? Or if there are other acceptance criteria you think are worth noting (e.g. 404 vs 418 vs 501 vs ...)

[tloubrieu-jpl]

Waiting for @jimmie to create a PR (I pinged him on slack today)

[jordanpadams]

endpoint done. tests needed added to icebox