Status of testing of Apache Airflow 2.2.1rc2

[potiuk]

Airflow 2.2.1rc2 testing

We have a kind request for all the contributors to the latest Apache Airflow RC 2.2.1rc2.

Could you please help us to test the RC versions of Airflow?

Please let us know in the comment if the issue is addressed in the latest RC.

• Document changes related to PR #16634 (#18644): @Jorricks
• Don't install SQLAlchemy/Pendulum adapters for other DBs (#18745): @JavierLopezT @ashb
  Linked issues:
  • ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: cannot allocate memory in static TLS block (#17546)
• Don't bake ENV and _cmd into tmp config for non-sudo (#18772): @ashb @khalidmammadov
  Linked issues:
  • Remove duplicate line of code for configuration copy (#18723)
• Add Callbacks Section to Logging & Monitoring (#18842): @john-jac
• Add information about keepalives for managed Postgres (#18850): @hpatel-higi @potiuk
  Linked issues:
  • Scheduler stops running: psycopg2.OperationalError: SSL SYSCALL error: EOF detected (#18846)
• Fix occassional deadlock on MSSQL test DagMaker cleanup (#18857): @potiuk
• Group PATCH DAGrun together with other DAGRun endpoints (#18885): @BasPH
• Fix catchup by limiting queued dagrun creation using max_active_runs (#18897): @ephraimbuddy
• Check python version before starting triggerer (#18926): @jedcunningham
• Try to move "dangling" rows in upgradedb (#18953): @uranusjr @sbialkowski-pixel
  Linked issues:
  • Migrate from 2.1.4 to 2.2.0 (#18894)
• Fix XCom.delete error in Airflow 2.2.0 (#18956): @jordanjeremy
• Row lock TI query in SchedulerJob._process_executor_events (#18975): @ephraimbuddy
• Fixes #18943 null execution date on insert to task_fail violating NOT NULL (#18979): @robinedwards
  Linked issues:
  • IntegrityError inserting into task_fail table with null execution_date from TI.handle_failure_with_callback (#18943)
• Sentry before send fallback (#18980): @leonsmith @joshzana
  Linked issues:
  • Missing value for new before_send config causes airflow.exceptions.AirflowConfigException: section/key [sentry/before_send] not found in config (#18963)
• Upgrade old DAG/task param format when deserializing from the DB (#18986): @ashb
• fail backfill command before loading DAGs if missing args (#18994): @SamWheating
• Workaround libstdcpp TLS error (#19010): @potiuk
• Allow Param to support a default value of None (#19034): @jedcunningham
• Warn about unsupported Python 3.10 (#19060): @mik-laj
• Rename trigger page label to Logical Date (#19061): @uranusjr
• Remove incorrect type comment in Swagger2Specification._set_defaults classmethod (#19065): @brain-buster
• Relax packaging requirement (#19087): @uranusjr @andrewgodwin
  Linked issues:
  • Add "packaging" to core requirements (#18122)
• Change ds, ts, etc. back to use logical date (#19088): @uranusjr
• Ensure task state doesn't change when marked as failed/success/skipped (#19095): @ephraimbuddy
• add detailed information to logging when a dag or a task finishes. (#19097): @steveyz-astro
• Hide tooltip when next run is none (#19112): @bbovenzi
• Prevent scheduler crash when serialized dag is missing (#19113): @ephraimbuddy
• Fix queued dag runs changes catchup=False behaviour (#19130): @robinedwards
• Add test for interval timetable catchup=False (#19145): @uranusjr @robinedwards
  Linked issues:
  • Fix queued dag runs changes catchup=False behaviour (#19130)
• Crerate TI context with data interval compat layer (#19148): @uranusjr @andreychernih
  Linked issues:
  • TypeError: unsupported operand type(s) for +=: 'NoneType' and 'datetime.timedelta' (#19143)
• Faster db migration to Airflow 2.2 (#19166): @jedcunningham
• Add TriggererJob to jobs check command (#19179): @mik-laj
• Clearing ti.next_method and ti.next_kwargs on task finish. (#19183): @ReadytoRocc @jedcunningham
  Linked issues:
  • Deferred TI's next_method is not cleared when Clearing a Successful Task (#19120)
  • Deferred TI's next_method and next_kwargs not cleared on retries (#18146)
• Fix airflow jobs check cmd for TriggererJob (#19185): @mik-laj
• Add DagRun.logical_date as a property (#19198): @uranusjr
• Fix Unexpected commit error in schedulerjob (#19213): @ephraimbuddy
  Linked issues:
  • Ensure task state doesn't change when marked as failed/success/skipped (#19095)

Thanks to all who contributed to the release (probably not a complete list!):
@ephraimbuddy @JavierLopezT @jedcunningham @sbialkowski-pixel @Jorricks @khalidmammadov @john-jac @brain-buster @andrewgodwin @joshzana @SamWheating @mik-laj @leonsmith @ReadytoRocc @ashb @robinedwards @potiuk @bbovenzi @jordanjeremy @steveyz-astro @BasPH @hpatel-higi @uranusjr @andreychernih

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[raphaelauv]

python3 -m pip install apache-airflow[postgres]==2.2.1rc2 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.1rc2/constraints-3.9.txt"

log

ERROR: Cannot install apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    flask-appbuilder 3.3.2 depends on Flask-OpenID<2 and >=1.2.5
    The user requested (constraint) flask-openid==1.2.5

[potiuk]

python3 -m pip install apache-airflow[postgres]==2.2.1rc2 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.1rc2/constraints-3.9.txt"

log

ERROR: Cannot install apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    flask-appbuilder 3.3.2 depends on Flask-OpenID<2 and >=1.2.5
    The user requested (constraint) flask-openid==1.2.5

This error makes no sense >=1.2.5 is ok for 1.2.5.

1. Are you sjure you used the right Python version?
2. Can you upgrade PIP to latest version

@uranusjr - I have no time to take a look at it soon, can you please take a look? I think we had similar case before that wrong conflict was reported

[kaxil]

Yeah the error is weird !!

[raphaelauv]

using python 3.9 ( pip version is 21.3.1 )

but for 2.2.1rc1 I do not have the problem

python3 -m pip install apache-airflow[postgres]==2.2.1rc1 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.1rc1/constraints-3.9.txt"

[uranusjr]

The final error message is bad, but an error is emitted that indicates the real issue:

Collecting Flask-OpenID==1.2.5
  Downloading Flask-OpenID-1.2.5.tar.gz (43 kB)
     |████████████████████████████████| 43 kB 699 kB/s
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/uranusjr/Documents/play/airflow-221rc2-flask-openid/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-iqrcgtl5/flask-openid_b6dd0035aaa24038a0cbc5a5b9b113c7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-iqrcgtl5/flask-openid_b6dd0035aaa24038a0cbc5a5b9b113c7/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-ydevayvc
       cwd: /tmp/pip-install-iqrcgtl5/flask-openid_b6dd0035aaa24038a0cbc5a5b9b113c7/
  Complete output (1 lines):
  error in Flask-OpenID setup command: use_2to3 is invalid.
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/d1/a2/9d1fba3287a65f81b9d1c09c4f7cb16f8ea4988b1bc97ffea0d60983338f/Flask-OpenID-1.2.5.tar.gz#sha256=5a8ffe1c8c0ad1cc1f5030e1223ea27f8861ee0215a2a58a528cc61379e5ccab (from https://pypi.org/simple/flask-openid/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

So this is the same use_2to3 issue we had previously. The constraints file should pin Flask-OpenID to 1.3.0 (which has a pre-built wheel available) instead of 1.2.5.

[potiuk]

So this is the same use_2to3 issue we had previously. The constraints file should pin Flask-OpenID to 1.3.0 (which has a pre-built wheel available) instead of 1.2.5.

Fixing it!

[potiuk]

Thanks @uranusjr I knew I saw it before!

[potiuk]

Resolved - the constraint tag was wrongly set (it was set to be the same as 2.1.0b1. It should work now @raphaelauv !

[raphaelauv]

@potiuk yes now it work , sorry for not pointing to the reel error , thanks @uranusjr

[potiuk]

Cool! Also Docker Images are rebuilt and pushed now with the updated constraints. All should be good for testing 🚀

[potiuk]

Just tested - the #19010 - works fine - LD_PRELOAD is propely set in the image 👍

[robinedwards]

#18943 is fixed

[robinedwards]

I just discovered #19252 not sure if its a blocker though

[kaxil]

Yeah that sounds like a regression unfortunately :( -- was it working in 2.2.0 @robinedwards ?

[robinedwards]

No it doesn't work in 2.2.0 either

[kaxil]

Aah ok -- in that case it is not a regression so not a blocker. Let's fix it for 2.2.2

[leonsmith]

#18980 looks good 👍

[jordanjeremy]

confirmed that #18956 is fixed

[jedcunningham]

Tested #18926, #19034, #19166, #19183, #19179, #19185.

I also partially tested #19088, though I didn't tick this one off. I only verified manual runs logged into their own files which only uses ts.

[potiuk]

I also ticked off all the "docs/test only" changes (should have been removed from the issue in the first place which we can do in the future).

I guess we have few more hours to test the remaining ones (if possible) :)

[jedcunningham]

Thanks everyone for testing! Apache Airflow 2.2.1 (based on RC2) has been accepted.

I'll continue with the release process, and the release announcement will follow shortly.