Pluggable logging

[ramonacat]

This ticket is for the CNCF summer mentorship 2022

---

Describe the problem you are trying to solve

Tremor is an event processing system that can - among other things - process logs and metrics. We would like tremor to have a facility to handle its logs through its own pipelines (similar to the pluggable metrics experience), instead of the current state, where the logs have to be received externally.
For production users this would allow:

• tailoring tremor to different environments,
• simplified production configuration
  • today in one of production environments Tremor writes its logs to a file, which is then read by other systems, that parse and ship them back to tremor. With pluggable logging this dance is avoided, making the whole system cheaper and easier to maintain.
• removing the dependency on external tools.

Describe the solution you'd like

We would like to have an internal connector, which will be a source for the internal tremor logs.

Goals

• Tremor logs can travel through the internal tremor pipelines

[ghost]

Hi, I'm Damilola Olayinka, a Computer Science undergraduate. I got to know about Tremor through the LFX Mentorship programme and I found it interesting. I would love to work on this issue but i am having some difficulties understanding the project as I am new to this. Would you mind sharing some resources which i can go through prior to submitting my application for the program. @Agares @darach

[wraymo]

Hi, I’m Rui, a master student in computer engineering. Currently I'm working on log compression and I’m learning Rust. I’m also very interested in this issue but have the same problem as Damilola. Could you provide some guidance?

[Licenser]

Hi @yimikao, hi @wraymo thank you two for your interest, and sorry for the late reply. Absolutely, there are a few starting points that can be of interest.

The metrics guide

Note this is still in preview as we're flashing out the documentation for the upcoming release, but in the Tremor metrics section, we show how one can consume metrics from tremor itself to post-process them. In very short, this project is about doing the same with logs.

The metrics connector source

If you prefer reading sources over documentation that could be a good starting point, as before, this is how we do metrics and while it's not going to be "the same" it'll probably give a feel about the direction for what we're aiming for.

log4rs docs

This is the library for logging we're currently using so digging a bit in there about how to integrate with it could be helpful.

the community chat

While less about the technical part we have a good community on discord, with a channel dedicated to mentoring where you can ask questions, chat with prior mentees, and all that :)

I hope that answers some of the questions.