Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tt 590 custom log targets #787

Closed
wants to merge 11 commits into from
Closed

Tt 590 custom log targets #787

wants to merge 11 commits into from

Conversation

Tofel
Copy link
Contributor

@Tofel Tofel commented Nov 28, 2023

This PR introduces the ability to easy add new logging targets while adding two of them: Loki and local files.

It is possible to use any number of logging targets concurrently. By default no logging targets, they need to be specified using env variable LOGWATCH_LOG_TARGETS, which currently accepts two values: file and loki (case insensitive).

LogWatch's API has not changed, but now the constructor accepts variadic options that can be used to override default implementations for log target handlers (useful for example for unit testing if we don't want to create any files or send anything to Loki but still make sure that correct handler was called expected number of times).

The constructor also executes some log target validations and operations:

is there a known handler for every requested log target?
deactivate known handlers that weren't requested/set via environment variables
Also, Loki client is now lazy initalised the first time it is requested instead of being always initialised in the constructor (even if streaming logs to Loki was not requested).

Last, but not least... file logs are also streamed instead of being saved at the end of test, which means that in case of container failure we will still have some logs which might help to debug failure cause.

When a test ends each active handler prints information about log location.

It is recommended to connect/disconnect LogWatch to containers using PostStarts/PostStops hooks instead of doing it manually.

Known issues existing in previous versions:

slow containers might result in panic when log producer cannot connect to a container within 5 seconds (context timeout hardcoded in testcontainers-go, there's a testcontainers/testcontainers-go#1278 that adds retries (although it seems from my experiments that if log producer fails to start in 5 seconds there's something off with the container and retrying won't help anyway)
if the test is too short logs might not be sent to Loki before test finishes (we should consider adding a way to flush them on shutdown)
Breaking changes:

env variable TEST_LOG_COLLECT is no longer used. To indicate that you want to collect logs simply add targets to LOGWATCH_LOG_TARGETS or leave it empty to skip log collection
New env variables:

GRAFANA_URL -- URL to Grafana instance that is connected to Loki instance to which logs where streamed
GRAFANA_DATASOURCE -- Loki datasource id that will be used when building the URL
What's missing:

tests for new functionality
validating whether Grafana-related env vars were set
creating/reading a global execution id that will be send together with logs and allow us to filter them in Grafana
building of Grafana Dashboard URL instead of Explore URL (as is currently implemented)
Example of changes to chainlink repo that uses testcontainers-go hooks to start and stop listening to logs can be found smartcontractkit/chainlink#11007.

@Tofel Tofel closed this Nov 28, 2023
@Tofel Tofel deleted the tt_590_custom_log_targets branch November 28, 2023 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant