[BUG] Consolidate default log formats in flytekit and the backend

[eapolinario]

Describe the bug

We currently use 2 slightly different logging formats in flytekit and the backend components.

This forces users to maintain multiple parsers to be able to ingest Flyte logs.

Expected behavior

We need to be backwards compatible, so at the very least we should expose the logging format used by flytekit.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[andrusha]

Wouldn't it be better to use Flyte-like logging in Flytekit instead?

It'll be a breaking change for library consumers, but then also Flytekit logging format is a mix of plaintext and json without ability to override it from the Python root logger. This seem to be more problematic to parse than straight json coming from Flyte itself and I would rather standardize on that one.

[eapolinario]

@andrusha , the key here is that this would be a breaking change. We have anecdotal evidence that there are Flyte deployments out there where people went this route of parsing the 2 formats to ingest logs.

Having said that, this should be quite simple implement. One idea is to have an environment variable that is checked prior to setting the logging format here. Contributions are welcome!

[VPraharsha03]

I would like to work on this, can you please assign me ? @samhita-alla

[samhita-alla]

@VPraharsha03, are you working on creating a PR for this issue?

[github-actions]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable.
Thank you for your contribution and understanding! 🙏