Fixed the DSPA status update on reconciliation #651

hbelmiro · 2024-05-16T18:25:07Z

The issue resolved by this Pull Request:

Resolves https://issues.redhat.com/browse/RHOAIENG-6287

Description of your changes:

This PR refactors controllers.DSPAReconciler#Reconcile so it updates the DSPA status correctly.

Testing instructions

Scenario 1 - Successful deployment

Deploy config/samples/v2/dspa-simple to the kubeflow namespace.
Wait until all the pods get started.

oc get pods -n kubeflow
NAME                                                      READY   STATUS    RESTARTS   AGE
ds-pipeline-metadata-envoy-sample-5857b74974-psjcd        2/2     Running   0          49s
ds-pipeline-metadata-grpc-sample-79995d8b76-8rg8c         1/1     Running   0          49s
ds-pipeline-persistenceagent-sample-69c64c4cf-8bgvl       1/1     Running   0          50s
ds-pipeline-sample-7975c7f99-b5fl5                        2/2     Running   0          50s
ds-pipeline-scheduledworkflow-sample-bc6f4c55d-prfbz      1/1     Running   0          50s
ds-pipeline-ui-sample-867bf74ff9-kp8f2                    2/2     Running   0          49s
ds-pipeline-workflow-controller-sample-757696fcff-4864z   1/1     Running   0          49s
mariadb-sample-5455fd4c74-plnl9                           1/1     Running   0          69s
minio-sample-5d58bf78f9-4mcnr                             1/1     Running   0          69s

Check the DSPA status:

oc get datasciencepipelinesapplication.datasciencepipelinesapplications.opendatahub.io/sample -o yaml -n kubeflow | yq .status

You should see a status similar to the following:

conditions:
  - lastTransitionTime: "2024-05-16T18:22:36Z"
    message: Database connectivity successfully verified
    reason: DatabaseAvailable
    status: "True"
    type: DatabaseAvailable
  - lastTransitionTime: "2024-05-16T18:22:36Z"
    message: Object Store connectivity successfully verified
    reason: ObjectStoreAvailable
    status: "True"
    type: ObjectStoreAvailable
  - lastTransitionTime: "2024-05-16T18:22:48Z"
    message: Component [ds-pipeline-sample] is minimally available.
    reason: MinimumReplicasAvailable
    status: "True"
    type: APIServerReady
  - lastTransitionTime: "2024-05-16T18:22:43Z"
    message: Component [ds-pipeline-persistenceagent-sample] is minimally available.
    reason: MinimumReplicasAvailable
    status: "True"
    type: PersistenceAgentReady
  - lastTransitionTime: "2024-05-16T18:22:41Z"
    message: Component [ds-pipeline-scheduledworkflow-sample] is minimally available.
    reason: MinimumReplicasAvailable
    status: "True"
    type: ScheduledWorkflowReady
  - lastTransitionTime: "2024-05-16T18:22:48Z"
    message: All components are ready.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Ready

Scenario 2 - Deployment failure

Deploy config/samples/v2/dspa-simple to the an-even-longer-veeeeeeeeeeeeeeery-long-data-science-project namespace.
Wait until the deployment fails with the following message:

ERROR	Reconciler error	{"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"sample","namespace":"an-even-longer-veeeeeeeeeeeeeeery-long-data-science-project"}, "namespace": "an-even-longer-veeeeeeeeeeeeeeery-long-data-science-project", "name": "sample", "reconcileID": "30ac4943-f7ca-464c-9e42-7a729bb89b30", "error": "Route.route.openshift.io \"ds-pipeline-sample\" is invalid: spec.host: Invalid value: \"ds-pipeline-sample-an-even-longer-veeeeeeeeeeeeeeery-long-data-science-project.apps.hbelmiro-10.dev.datahub.redhat.com\": must be no more than 63 characters"}

Check the DSPA status:

oc get datasciencepipelinesapplication.datasciencepipelinesapplications.opendatahub.io/sample -o yaml -n an-even-longer-veeeeeeeeeeeeeeery-long-data-science-project | yq .status

You should see a status similar to the following:

conditions:
  - lastTransitionTime: "2024-05-16T17:59:15Z"
    message: Database connectivity successfully verified
    reason: DatabaseAvailable
    status: "True"
    type: DatabaseAvailable
  - lastTransitionTime: "2024-05-16T17:59:14Z"
    message: Object Store connectivity successfully verified
    reason: ObjectStoreAvailable
    status: "True"
    type: ObjectStoreAvailable
  - lastTransitionTime: "2024-05-16T17:59:15Z"
    message: 'Route.route.openshift.io "ds-pipeline-sample" is invalid: spec.host: Invalid value: "ds-pipeline-sample-an-even-longer-veeeeeeeeeeeeeeery-long-data-science-project.apps.hbelmiro-10.dev.datahub.redhat.com": must be no more than 63 characters'
    reason: FailingToDeploy
    status: "False"
    type: APIServerReady
  - lastTransitionTime: "2024-05-16T17:58:54Z"
    message: ""
    reason: Unknown
    status: Unknown
    type: PersistenceAgentReady
  - lastTransitionTime: "2024-05-16T17:58:54Z"
    message: ""
    reason: Unknown
    status: Unknown
    type: ScheduledWorkflowReady
  - lastTransitionTime: "2024-05-16T17:58:54Z"
    message: "Route.route.openshift.io \"ds-pipeline-sample\" is invalid: spec.host: Invalid value: \"ds-pipeline-sample-an-even-longer-veeeeeeeeeeeeeeery-long-data-science-project.apps.hbelmiro-10.dev.datahub.redhat.com\": must be no more than 63 characters \n \n \n"
    reason: MinimumReplicasAvailable
    status: "False"
    type: Ready

Checklist

The commits are squashed in a cohesive manner and have meaningful messages.
Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
The developer has manually tested the changes and verified that the changes work

dsp-developers · 2024-05-16T18:27:30Z

A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-651
An OCP cluster where you are logged in as cluster admin is required.

To use this image run the following:

cd $(mktemp -d)
git clone git@github.com:opendatahub-io/data-science-pipelines-operator.git
cd data-science-pipelines-operator/
git fetch origin pull/651/head
git checkout -b pullrequest 07e8e3b476ed8cf6cf2c9b58dcec2b3e799dfb86
oc new-project opendatahub
make deploy IMG="quay.io/opendatahub/data-science-pipelines-operator:pr-651"

More instructions here on how to deploy and test a Data Science Pipelines Application.

VaniHaripriya · 2024-05-22T18:48:39Z

@hbelmiro Tested both the scenarios and it worked as expected.
But I see that ds-pipeline-ui pod is crashing due to which kfp-ui is not able to retrieve pipelines.

Scenario1 Result:

conditions:
  - lastTransitionTime: "2024-05-22T17:43:33Z"
    message: Database connectivity successfully verified
    reason: DatabaseAvailable
    status: "True"
    type: DatabaseAvailable
  - lastTransitionTime: "2024-05-22T17:43:33Z"
    message: Object Store connectivity successfully verified
    reason: ObjectStoreAvailable
    status: "True"
    type: ObjectStoreAvailable
  - lastTransitionTime: "2024-05-22T17:43:44Z"
    message: Component [ds-pipeline-sample] is minimally available.
    reason: MinimumReplicasAvailable
    status: "True"
    type: APIServerReady
  - lastTransitionTime: "2024-05-22T17:43:39Z"
    message: Component [ds-pipeline-persistenceagent-sample] is minimally available.
    reason: MinimumReplicasAvailable
    status: "True"
    type: PersistenceAgentReady
  - lastTransitionTime: "2024-05-22T17:43:39Z"
    message: Component [ds-pipeline-scheduledworkflow-sample] is minimally available.
    reason: MinimumReplicasAvailable
    status: "True"
    type: ScheduledWorkflowReady
  - lastTransitionTime: "2024-05-22T17:43:44Z"
    message: All components are ready.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Ready

Scenario2 Result :

$ oc get datasciencepipelinesapplication.datasciencepipelinesapplications.opendatahub.io/sample -o yaml -n an-even-longer-veeeeeeeeeeeeeeery-long-data-science-project | yq .status
conditions:
  - lastTransitionTime: "2024-05-22T17:42:03Z"
    message: Database connectivity successfully verified
    reason: DatabaseAvailable
    status: "True"
    type: DatabaseAvailable
  - lastTransitionTime: "2024-05-22T17:42:03Z"
    message: Object Store connectivity successfully verified
    reason: ObjectStoreAvailable
    status: "True"
    type: ObjectStoreAvailable
  - lastTransitionTime: "2024-05-22T17:42:03Z"
    message: 'Route.route.openshift.io "ds-pipeline-sample" is invalid: spec.host: Invalid value: "ds-pipeline-sample-an-even-longer-veeeeeeeeeeeeeeery-long-data-science-project.apps.vmudadla.dev.datahub.redhat.com": must be no more than 63 characters'
    reason: FailingToDeploy
    status: "False"
    type: APIServerReady
  - lastTransitionTime: "2024-05-22T17:42:03Z"
    message: ""
    reason: Unknown
    status: Unknown
    type: PersistenceAgentReady
  - lastTransitionTime: "2024-05-22T17:42:03Z"
    message: ""
    reason: Unknown
    status: Unknown
    type: ScheduledWorkflowReady
  - lastTransitionTime: "2024-05-22T17:42:03Z"
    message: "Route.route.openshift.io \"ds-pipeline-sample\" is invalid: spec.host: Invalid value: \"ds-pipeline-sample-an-even-longer-veeeeeeeeeeeeeeery-long-data-science-project.apps.vmudadla.dev.datahub.redhat.com\": must be no more than 63 characters \n \n \n"
    reason: MinimumReplicasAvailable
    status: "False"
    type: Ready

VaniHaripriya · 2024-05-22T19:46:19Z

I re-created the kubeflow namespace and tested out scenario1. I don't see any errors in kfp-ui now. I confirm that both the scenarios are working as expected.

controllers/dspastatus/dspa_status.go

controllers/dspipeline_controller.go

controllers/dspastatus/dspa_status.go

Signed-off-by: hbelmiro <helber.belmiro@gmail.com>

dsp-developers · 2024-05-23T19:14:17Z

Change to PR detected. A new PR build was completed.
A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-651

controllers/dspipeline_controller.go

HumairAK · 2024-05-27T17:14:33Z

Tested and works as intended:

Also tested tearing down pods and DSPA status continues to update as before

great work @hbelmiro !

HumairAK · 2024-05-27T17:14:39Z

/lgtm
/approve

openshift-ci · 2024-05-27T17:14:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: HumairAK

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [HumairAK]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot added the do-not-merge/work-in-progress label May 16, 2024

hbelmiro marked this pull request as ready for review May 16, 2024 18:42

openshift-ci bot removed the do-not-merge/work-in-progress label May 16, 2024

openshift-ci bot requested review from DharmitD and gmfrasca May 16, 2024 18:42

HumairAK assigned HumairAK and unassigned HumairAK May 22, 2024

HumairAK requested review from HumairAK and removed request for gmfrasca May 22, 2024 18:19

HumairAK reviewed May 22, 2024

View reviewed changes

hbelmiro added 4 commits May 23, 2024 11:33

RHOAIENG-6278 - Fixed DSPA status update on reconciliation

081fca6

Signed-off-by: hbelmiro <helber.belmiro@gmail.com>

Changed SetDatabaseNotReady and SetObjStoreNotReady to be more versatile

f5ae58c

Signed-off-by: hbelmiro <helber.belmiro@gmail.com>

BuildTrueCondition exported

9cc7336

Signed-off-by: hbelmiro <helber.belmiro@gmail.com>

Renamed methods

7a46a7b

Signed-off-by: hbelmiro <helber.belmiro@gmail.com>

hbelmiro force-pushed the RHOAIENG-6287 branch from 07e8e3b to 7a46a7b Compare May 23, 2024 19:11

HumairAK reviewed May 27, 2024

View reviewed changes

controllers/dspipeline_controller.go Show resolved Hide resolved

openshift-ci bot assigned HumairAK May 27, 2024

openshift-ci bot added the lgtm label May 27, 2024

openshift-ci bot added the approved label May 27, 2024

HumairAK merged commit 5eacdd6 into opendatahub-io:main May 27, 2024
5 of 6 checks passed

hbelmiro deleted the RHOAIENG-6287 branch May 27, 2024 17:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed the DSPA status update on reconciliation #651

Fixed the DSPA status update on reconciliation #651

hbelmiro commented May 16, 2024

dsp-developers commented May 16, 2024

VaniHaripriya commented May 22, 2024 •

edited

Loading

VaniHaripriya commented May 22, 2024

dsp-developers commented May 23, 2024

HumairAK commented May 27, 2024

HumairAK commented May 27, 2024

openshift-ci bot commented May 27, 2024

Fixed the DSPA status update on reconciliation #651

Fixed the DSPA status update on reconciliation #651

Conversation

hbelmiro commented May 16, 2024

The issue resolved by this Pull Request:

Description of your changes:

Testing instructions

Scenario 1 - Successful deployment

Scenario 2 - Deployment failure

Checklist

dsp-developers commented May 16, 2024

VaniHaripriya commented May 22, 2024 • edited Loading

VaniHaripriya commented May 22, 2024

dsp-developers commented May 23, 2024

HumairAK commented May 27, 2024

HumairAK commented May 27, 2024

openshift-ci bot commented May 27, 2024

VaniHaripriya commented May 22, 2024 •

edited

Loading