[FLINK-33803] Remove deprecated lastReconciledSpec meta generation in favour of observedGeneration

[angelachenn]

What is the purpose of the change

Followup to https://issues.apache.org/jira/browse/FLINK-33803 (#755)

The observedGeneration field in the FlinkDeployment CRD status is used to persist the generation that was last acted on by the operator. At the end of a successful reconciliation, the operator sets the observedGeneration field to the current generation number (from the CRD metadata).

This PR removes internal usages of the lastReconciledSpec metadata (and generation) that were kept in the initial PR to avoid downstream breakage. observedGeneration replaces this semantically. Removing the deprecated fields is to end up with a cleaner codebase.

Note: the removed fields were nested within the lastReconciledSpec String, thus there is no documentation to be updated (please advise if I missed something).

Brief change log

• There is no more ObjectMeta metadata attribute (containing generation) within the ReconciliationMetadata class
  • that field will not exist to the operator within FlinkDeploymentStatus w.r.t. reconciliation
• removed the metadata field (from the CRD metadata) during deserialization before passing the content into a ReconciliationMetadata instance such that its constructor does not mutate the resources
  • if not, Jackson throws a JsonMappingException: Unrecognized field 'metadata' (class org.apache.flink.kubernetes.operator.api.reconciler.ReconciliationMetadata), not marked as ignorable, causing pods to fail.

Verifying this change

This change added tests and can be verified as follows:

• Modified existing tests that checked for specWithMeta.getMeta().getMetadata().getGeneration() to assert against status.getObservedGeneration() instead (tests pass)
• Manually verified the change by running a cluster with a FlinkDeployment custom resource that depends on the operator (with 2 JobManagers & 2 TaskManagers), and checking its CRD, verifying that fields are mapped accordingly.

Logs

Before:

k get flinkdeployment flink --context=<context> -n <ns>  -o yaml | yq .status.reconciliationStatus.lastReconciledSpec | jq .resource_metadata
{
  "apiVersion": "flink.apache.org/v1beta1",
  "metadata": {
    "generation": 5
  },
  "firstDeployment": false
}

k get flinkdeployment flink --context=<context> -n <ns>  -o yaml | yq .status.observedGeneration
5

After:

k get flinkdeployment flink --context=<context> -n <ns>  -o yaml | yq .status.reconciliationStatus.lastReconciledSpec | jq .resource_metadata                                                                                                                                                      
{
  "apiVersion": "flink.apache.org/v1beta1",
  "firstDeployment": true
}

k get flinkdeployment flink --context=<context> -n <ns>  -o yaml | yq .status.observedGeneration
5

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changes to the CustomResourceDescriptors: yes
• Core observer or reconciler logic that is regularly executed: yes

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[gyfora]

hey @angelachenn , are you still working on this change?

[angelachenn]

hey @gyfora, thanks for checking in, and apologies for the lack of updates. I had to shift my focus to some higher-priority tasks and have also been on vacation. I do intend to keep working on this change and will be back in office later next week.

[gyfora]

@angelachenn no hurries I was just curious if you were planning to continue at some point :)
Thank you!

[angelachenn]

hey @gyfora, my updated PR is ready for review, I've addressed your comment, tweaked some small things and also added a test for backwards compatibility. All checks pass on my forked branch & I've also tested the changes locally, although please advise if I've missed anything. thank you!