Observability Patterns

[kkuzemka]

Establish Observability Patterns

The Federated Gateway service needs established observability patterns to help users understand how the service is operating.

User Stories

• As a developer/infrastructure engineer I want to view a centralized set of service logs.
• As a developer/infrastructure engineer I want to request a health endpoint and see the status of the service.
• As a developer/infrastructure engineer I want to trace request through the Gateway.

Deliverable

• Logger pattern created with .env log level configuration.
• Documented logging strategy for service.
• Creation of a health API endpoint that returns service status.
• Implementation of a local tracing solution/tool that can monitor where (Gateway vs service) in the request things slow or fail using Apollo's built in tracing features. Needs to be enabled via a .env variable.

Acceptance Criteria

• When starting the service container I should see some sort of logging in the container logs.
• With a running service container I should be able to access the service's health endpoint and get a 200 status.
• Followed Infrastructure Principles Handbook on observability.

[nnnnat]

@ticean & @rosshadden I've added a user story and a naive deliverable for distributed tracing in the Gateway. Looking to yall in defining acceptance criteria and guiding implementation since this is a little outside of my skill set.

[nnnnat]

@griggheo wanted to loop you in on this issue and the tracing conversation

[griggheo]

Thanks Nat, I'll pitch in when necessary but this looks good so far.

[ticean]

Do we require "distributed tracing"? Would local tracing be enough to get started?

The gql server libs might offer a local tracing option out of the box? Is it enough to see counts of requests, fails, how long they are taking, byte size, etc? I think we can get all of this with local tracing. This kind of tracing wouldn't give insight into why those backend requests took a long time. We'd have to investigate that separately.

Distributed tracing implies an external tracing server (ex Jaeger). We'd also need to implement that kind of tracing in the backend apps to get the "distributed" benefits.

[griggheo]

Yes, I think it makes sense to start with local tracing vs a more heavy-duty distributed tracing solution.

[nnnnat]

@ticean no i don't think we need to require distributed tracing for this and looks like we might be able to leverage the built in Apollo tracing tools to do local tracing.
https://github.com/apollographql/apollo-server/tree/master/packages/apollo-tracing
https://www.apollographql.com/docs/apollo-server/federation/metrics/
https://www.apollographql.com/docs/apollo-server/api/apollo-server/#constructoroptions-apolloserver

[kkuzemka]

Will need to create individual more granular tickets