Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase Airflow logging coverage, review existing logs and add missing essential logs #43180

Open
1 of 2 tasks
omkar-foss opened this issue Oct 18, 2024 · 1 comment
Open
1 of 2 tasks

Increase Airflow logging coverage, review existing logs and add missing essential logs #43180

omkar-foss opened this issue Oct 18, 2024 · 1 comment
Labels
area:logging kind:feature Feature Requests kind:meta High-level information important to the community

Comments

@omkar-foss
Copy link
Collaborator

omkar-foss commented Oct 18, 2024 •
edited
Loading

Description

As per users' feedback in the Airflow Debugging Survey 2024, about 42% of respondents mentioned that logs are missing or non-existent in some cases.

Use case/motivation

Goal for this issue is to review the existing Airflow logs coverage and enhance it by adding logs wherever appropriately required.

Related issues

Parent Issue: #40975

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@omkar-foss omkar-foss added kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet labels Oct 18, 2024
@omkar-foss omkar-foss changed the title Increase Airflow logging coverage, review existing logs and add where missing Increase Airflow logging coverage, review existing logs and add missing essential logs Oct 18, 2024
@dosubot dosubot bot added the area:logging label Oct 18, 2024
@omkar-foss omkar-foss added kind:meta High-level information important to the community and removed needs-triage label for new issues that we didn't triage yet labels Oct 23, 2024
@Dev-iL
Copy link
Contributor

Dev-iL commented Oct 29, 2024

A good example of an issue with logging is when (fatal) errors from workers are not being forwarded to the scheduler, and it's very hard to trace what's actually wrong. In cases when a worker is exhibiting an early enough error (being unable to load the full Airflow machinery that records logs locally and forwards them to the scheduler/webserver), displaying a stack trace captured on the worker would help immensely.

Ref: #42136

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:logging kind:feature Feature Requests kind:meta High-level information important to the community
Projects
Debugging Improvements - Airflow 3
Status: Planning
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Dev-iL @omkar-foss