Status of testing of Apache Airflow 2.5.1rc1

[pierrejeambrun]

We have a kind request for all the contributors to the latest Apache Airflow RC 2.5.1rc1.

Could you please help us to test the RC versions of Airflow?

Please let us know in the comment if the issue is addressed in the latest RC.

• Allow generation of connection URI to work when no conn type (#26765): @dstandish
• Fix sla checks being skipped when the preceding task instance misses the sla (#26968): @doiken
• Fix ti._try_number for deferred and up_for_reschedule tasks (#26993): @shokosanma @bbovenzi @tanelk
  Linked issues:
  • can't see failed sensor task log on webpage (#26960)
  • Latest log not shown in grid view for deferred task (#27955)
• Additional info about Segmentation Violation in LocalTaskJob (#27381): @Taragolis #28975
• Handle DAG disappearing mid-flight when dag verification happens (#27720): @ramziyassine @potiuk
  Linked issues:
  • AirflowException Crashing the Scheduler During the scheduling loop (_verify_integrity_if_dag_changed) (#27622)
• dagrun, next_dagruns_to_examine, add MySQL index hint (#27821): @michaelmicheal
• Make sure we can get out of a faulty scheduler state (#27834): @stijndehaes
• Don't log CLI actions if db not initialized (#27851): @dstandish
• fix: current_state method on TaskInstance doesn't filter by map_index (#27898): @xlanor @tirkarthi
  Linked issues:
  • current_state method of TaskInstance fails for mapped task instance (#27864)
• Order TIs by map_index (#27904): @tanelk
• Fix deadlock when chaining multiple empty mapped tasks (#27964): @ephraimbuddy
• Simplify dataset subgraph logic (#27987): @bbovenzi
• Prevent double loading of providers from local paths (#27988): @ashb
• Handle bad zip files nicely when parsing DAGs. (#28011): @potiuk
• Use asserts instead of exceptions for executor not started (#28019): @dstandish
• Return list of tasks that will be queued (#28066): @atrbgithub
• Resolve false warning about calling conf.get on moved item (#28075): @dstandish
• Bump decode-uri-component from 0.2.0 to 0.2.2 in /airflow/www (#28080): @dependabot[bot]
• Add automated version replacement in example dag indexes (#28090): @potiuk
• Rewrite kerberos security integration and unit tests (#28092): @Taragolis
• Add airflow.api.auth.backend.session to backend sessions in compose (#28094): @norm @potiuk @azraelxuemo
  Linked issues:
  • Change the default auth backend to session (#21640)
  • close (#28091)
• Don't emit FutureWarning when code not calling old key (#28109): @dstandish
• Make BaseJob.most_recent_job favor "running" jobs (#28119): @kosteev
  Linked issues:
  • Scheduler heartbeat warning message in Airflow UI displaying that scheduler is down sometimes incorrect (#28118)
• Improve run/task grid view actions (#28130): @atrbgithub @bbovenzi
  Linked issues:
  • Return list of tasks that will be queued (#28066)
• Ensure the dagbag_size metric decreases when files are deleted (#28135): @ashb
• Do not warn when airflow dags tests command is used (#28138): @potiuk @MatrixManAtYrService
  Linked issues:
  • IndexError in airflow dags test re: scheduling delay stats (#28106)
• Ensure that warnings from @dag decorator are reported in dag file (#28153): @ashb
• Fix next run datasets error (#28165): @bbovenzi
• Move Integration tests to separate package under tests (#28170): @potiuk
• Make invalid characters exception more readable (#28181): @uranusjr
• Add custom pickling hooks to LazyXComAccess (#28191): @AndrewTsao @uranusjr
  Linked issues:
  • Dynamic task context fails to be pickled (#28146)
• Fix scheduler orm DetachedInstanceError when find zombies in standalone dag processor mode (#28198): @BobDu
• Merge redis and rabbitmq integration into celery (#28205): @potiuk
• Integration tests are separated into separate command and CI job (#28207): @potiuk
• Make live logs reading work for "other" k8s executors (#28213): @dstandish
• Make arguments 'offset' and 'length' not required (#28234): @malthe
• Fix db clean warnings (#28243): @jedcunningham
• Convert test_task_command to Pytest and unquarantine tests in it (#28247): @potiuk
• Trigger gevent monkeypatching via environment variable (#28283): @webster-chainalysis @potiuk
  Linked issues:
  • Can't read S3 remote logs when using gevent/eventlent webserver workers. (#8212)
• Add SIGUSR2 handler for LocalTaskJob and workers to aid debugging (#28309): @ashb
• Add setters for on_*_callbacks to MappedOperator (#28313): @cccs-seb
• Fix the sql syntax in merge_data (#28353): @avicol
• Correctly select a mapped task's "previous" task (#28379): @uranusjr @internetcoffeephone
  Linked issues:
  • Dynamic task mapping does not correctly handle depends_on_past (#28296)
• Maintain manual scroll position in task logs (#28386): @bbovenzi
• Show mapped task groups in graph (#28392): @bbovenzi
• Dont show task/run durations when there is no start_date (#28395): @bbovenzi
• A manual run can't look like a scheduled one (#28397): @saulbein @uranusjr
  Linked issues:
  • Triggering a DAG with the same run_id as a scheduled one causes the scheduler to crash (#27818)
• Separate callModal from dag.js (#28410): @bbovenzi
• Fix calendar view for CronTriggerTimeTable dags (#28411): @bbovenzi
• Re-enable Plyvel on ARM as it now builds cleanly (#28443): @potiuk
• Add ensure_ascii=False in trigger dag run API (#28451): @akakakakakaa
• Fix bad pods pickled in executor_config (#28454): @dstandish
• Ensure correct log dir in file task handler (#28477): @pingzh
• Add retry to purge_inactive_dag_warnings (#28481): @michaelmicheal
  Linked issues:
  • purge_inactive_dag_warnings Unhandled Exception in DagFileProcessorManager (#28480)
• Move MyPY plugins of ours to dev folder (#28498): @potiuk
• Fix selective checks handling error tracebacks in CI (#28514): @potiuk
• Fix redirection for selective checks for all workflows (#28518): @potiuk
  Linked issues:
  • Fix selective checks handling error tracebacks in CI (#28514)
• Add VERBOSE equal to false for all just changed selective checks (#28519): @potiuk
  Linked issues:
  • Fix redirection for selective checks for all workflows (#28518)
• Consider previous run in CronTriggerTimetable (#28532): @uranusjr
• Cleanup and do housekeeping with plugin examples (#28537): @potiuk
• Emit warnings for conf.get* from the right source location (#28543): @ashb
• Add back join to zombie query that was dropped in #28198 (#28544): @jedcunningham
• navbar, cap dropdown size, and add scroll bar (#28561): @michaelmicheal
  Linked issues:
  • Issues with Custom Menu Items on Smaller Windows (#28483)
• Move TI setNote endpoints under TaskInstance (#28566): @jedcunningham
• Guard not-yet-expanded ti in trigger rule dep (#28592): @uranusjr
• Fix Connection.get_extra type (#28594): @bravmi
• Fix UI caret direction (#28624): @vchiapaikeo
• Change Architecture and OperatingSystem classes into Enums (#28627): @potiuk
• Add doc-strings and small improvement to email util (#28634): @stamixthereal
• Nest header blocks in divs to fix dagid copy nit on dag.html (#28643): @vchiapaikeo
• Fix description of output redirection for access_log for gunicorn (#28672): @potiuk
• Handle ConnectionReset exception in Executor cleanup (#28685): @nguyenmphu @maxnathaniel
  Linked issues:
  • Scheduler pod hang when K8s API call fail (#28328)
• Row-lock TIs to be removed during mapped task expansion (#28689): @ephraimbuddy
• Allow XComArgs for external_task_ids of ExternalTaskSensor (#28692): @vchiapaikeo
• Fix "airflow tasks render" cli command for mapped task instances (#28698): @ephraimbuddy
• Sensor mention in taskflow concepts (#28708): @csm10495
• Bump json5 from 1.0.1 to 1.0.2 in /airflow/www (#28715): @dependabot[bot]
• Limit SQLAlchemy to below 2.0 (#28725): @potiuk
• Update CSRF token to expire with session (#28730): @malthe @maxnathaniel
  Linked issues:
  • CSRF token should be expire with session (#28356)
• Only patch single label when adopting pod (#28776): @jedcunningham
• Fix UIAlert should_show when AUTH_ROLE_PUBLIC set (#28781): @vchiapaikeo
• Remove swagger-ui extra from connexion to resolve CVE-2019-17495 (#28788): @JGoldman110 @hughlunnon
  Linked issues:
  • CVE-2019-17495 for swagger-ui (#28381)
• Fix masking of non-sensitive environment variables (#28802): @ephraimbuddy

Thanks to all who contributed to the release (probably not a complete list!):
@avicol @akakakakakaa @stamixthereal @shokosanma @kosteev @ephraimbuddy @vchiapaikeo @ramziyassine @hughlunnon @ashb @saulbein @JGoldman110 @norm @AndrewTsao @BobDu @xlanor @stijndehaes @jedcunningham @nguyenmphu @bbovenzi @MatrixManAtYrService @cccs-seb @webster-chainalysis @malthe @Taragolis @pingzh @dstandish @doiken @azraelxuemo @michaelmicheal @uranusjr @tirkarthi @bravmi @csm10495 @potiuk @maxnathaniel @internetcoffeephone @atrbgithub @dependabot[bot] @tanelk

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[stamixthereal]

#28634 has been tested and working as expected

[Taragolis]

#27381 - Test that now user get info in task logs if Segmentation Fault detected during LocalJob execution
#28092 - This one only related to internal stuff, like CI and tests stability

[vchiapaikeo]

#28624 and #28643 - UI changes look good
#28692 - Ran a test dag and is working as expected

[avicol]

#28353 Is fixed in the 2.5.1rc1

[csm10495]

#28708 still looks good.

[saulbein]

#28397 errors out as intended, looks good.

[potiuk]

All Good in my changes (I left one change for @webster-chainalys to confirm :)

• Handle DAG disappearing mid-flight when dag verification happens #27720 is difficult to trigger but this is just exception handling and I checked it is present in the sources and it's really and edge case
• Handle bad zip files nicely when parsing DAGs. #28011 works fine - tested it with REALLY BAD zip file :)
• Add automated version replacement in example dag indexes #28090 - doc change only
• Add airflow.api.auth.backend.session to backend sessions in compose #28094 - doc/docker-compose change only (I checked it is updated)
• Do not warn when airflow dags tests command is used #28138 - works. no warning generated
• Move Integration tests to separate package under tests #28170 - just a structure change for tests
• Merge redis and rabbitmq integration into celery #28205 - just a structure change for tests
• Integration tests are separated into separate command and CI job #28207 - just a structure change for tests
• Convert test_task_command to Pytest and unquarantine tests in it #28247 - test fix only
• Trigger gevent monkeypatching via environment variable #28283 - I will defer to @webster-chainalysis to test if it works for them
• Re-enable Plyvel on ARM as it now builds cleanly #28443 - ARM image has plyvel
• Move MyPY plugins of ours to dev folder #28498 - test only change
• Fix selective checks handling error tracebacks in CI #28514 Fix redirection for selective checks for all workflows #28518 Add VERBOSE equal to false for all just changed selective checks #28519 - CI only changes
• Cleanup and do housekeeping with plugin examples #28537 - doc change, checked that empty_plugin is there.
• Change Architecture and OperatingSystem classes into Enums #28627 - internal refactor only. Seems to detect architecture properly
• Fix description of output redirection for access_log for gunicorn #28672 - proper help printed
• Limit SQLAlchemy to below 2.0 #28725 - proper metadata in wheel: Requires-Dist: sqlalchemy (<2.0,>=1.4)

[JGoldman110]

#28788 - Swagger UI looks good

[hughlunnon]

@JGoldman110 agreed, #28788 is resolved

[doiken]

#26968 looks good.

[internetcoffeephone]

#28379 looks good.

[nguyenmphu]

#28328 looks good.

[Taragolis]

• Show mapped task groups in graph #28392 looks like this PR break showing Dynamic Task Mapping which not in Task Group, details in Empty Graph view if DAG contains Dynamic Task Mapping #28988

[pierrejeambrun]

A bug has been found in rc1 that necessitate rc2, please test rc2 here: #29026