Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-36129][autoscaler] Autoscaler is compatible with Flink 1.20 #869

Merged
merged 2 commits into from
Aug 23, 2024

Conversation

1996fanrui
Copy link
Member

@1996fanrui 1996fanrui commented Aug 22, 2024

What is the purpose of the change

AggregatedMetric added the skew field in Flink 1.20, it caused the AggregatedMetric of old version cannot parse the AggregatedMetric of 1.20.

Following is the root exception:

Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "skew" (class org.apache.flink.runtime.rest.messages.job.metrics.AggregatedMetric), not marked as ignorable (5 known properties: "min", "id", "max", "avg", "sum"])
at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: java.util.ArrayList[0]->org.apache.flink.runtime.rest.messages.job.metrics.AggregatedMetric["skew"])

Brief change log

  • [FLINK-36129] Supporting Flink 1.20 and removing Flink 1.15
    • Here is the CI result of first commit.
    • We can see Unrecognized field "skew" related error, it works after the second commit.
  • [FLINK-36129][autoscaler] Autoscaler is compatible with Flink 1.20
    • Copy AggregatedMetric to this repo, and removing it after upgrading maven dependency to 1.20.
    • AggregatedMetric is copied into 3 modules: flink-autoscaler, flink-kubernetes-operator and flink-autoscaler-standalone
    • Copying it to flink-autoscaler, because flink-autoscaler is using AggregatedMetric.
    • Copying it to flink-kubernetes-operator and flink-autoscaler-standalone, because we hope the copied AggregatedMetric to replace the AggregatedMetric of maven dependency.

Verifying this change

Added the CI to run with 1.20.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changes to the CustomResourceDescriptors: no
  • Core observer or reconciler logic that is regularly executed: no

Documentation

  • Does this pull request introduce a new feature? no

@1996fanrui 1996fanrui force-pushed the 36129/1-20 branch 2 times, most recently from 5465871 to 2765ebb Compare August 22, 2024 06:48
@1996fanrui 1996fanrui marked this pull request as draft August 22, 2024 06:48
@1996fanrui 1996fanrui marked this pull request as ready for review August 22, 2024 08:15
Copy link
Member Author

@1996fanrui 1996fanrui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @gyfora , I found autoscaler doesn't work with flink 1.20. This PR did 2 changes:

  1. Operator supports flink 1.20, and enable CI for 1.20 (disable CI for 1.15)
  2. Fix autoscaler compatibility issue with 1.20

I explained them in detail in the PR description.

@1996fanrui 1996fanrui requested a review from gyfora August 22, 2024 08:19
Copy link
Member Author

@1996fanrui 1996fanrui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gyfora for the quick review, I will merge it if no more comment within 24 hours.

@1996fanrui 1996fanrui merged commit b26a0f5 into apache:main Aug 23, 2024
223 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants