Create a basic guiding doc - How to Debug Your Airflow Deployment

[omkar-foss]

Description

Create a basic guiding doc "How to Debug Your Airflow Deployment" that can guide users towards debugging their Airflow deployment. This doc can be based on general Airflow best practices + Agans' 9 rules of debugging:

1. Understand the system
2. Make it fail
3. Quit thinking and look
4. Divide and conquer
5. Change one thing at a time
6. Keep an audit trail
7. Check the plug
8. Get a fresh view
9. If you didn't fix it, it ain't fixed

The guide shall express above rules using Airflow examples, such that the users can read the guide and understand how to approach similar Airflow debugging scenarios.

Use case/motivation

David A. Wheeler's Article (2004): https://dwheeler.com/essays/debugging-agans.html

Related issues

Parent Issue: #40975

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Dev-iL]

Relevant discussion on Slack with some ideas on how the outputs of this should be structured and how to set the readers' expectations.

Key ideas raised therein (since the Slack thread will be unreachable eventually):

• "How to think like airflow debugger": some kind of best practices, where we see an example of a problem and lead to an issue or PR where we describe how you could approach looking at the issue.
• We should not aim to provide a “comprehensive” guide - or anything that will point our users to “I have not found how to debug this particular problem - you must add it as this is the “official” guide. It’s nearly impossible to provide even remotely comprehensive debugging guidelines for all the kinds of deployments our users might have - and they should know that they should likely build their own “debugging recipes” specific for their deployments. It should be clear we are providing “guidelines” and “hints” - but not “recipes and end-2-end solutions how to debug”.
• We should clearly set the boundaries and expectations (of the task), as this is easy to aim a bit too high for that.
• The target of this effort will be the Troubleshooting page. We can structure it with a TOC on top, filled with examples, possibly split into categories, where each example has 3 sections:
  • symptoms: what the user sees in the logs (such as error messages), ui notifications, etc.
  • diagnosis: an explanation of what likely caused this (misconfiguration, unaddressed removed deprecations, ...)
  • suggestions: this might have concrete steps for fixing the issue, links to other issues, further diagnosis suggestions, etc.
• We may take inspiration from the Installation of Airflow page that has sections like "What you are expected to handle" (what you should expect from yourself) and "What Apache Airflow Community provides for that method" (what you should expect from the community).