Tests might fail if update notification message is displayed in output

[rm3l]

/kind bug
/area testing

Noticed this while investigating failures on #6500:

cmd_add_binding_test

Expected
      <string>: apiVersion: binding.operators.coreos.com/v1alpha1
      kind: ServiceBinding
      metadata:
        creationTimestamp: null
        name: aname
      spec:
        application:
          group: apps
          kind: Deployment
          name: app
          version: v1
        bindAsFiles: true
        detectBindingResources: true
        services:
        - group: postgresql.k8s.enterprisedb.io
          id: aname
          kind: Cluster
          name: cluster-sample
          resource: clusters
          version: v1
      status:
        secret: ""
      
      ---
      A newer version of odo (v3.7.0) is available,
      visit https://github.com/redhat-developer/odo/releases to update.
      If you wish to disable this notification, run:
      odo preference set UpdateNotification false
      ---
      
  to be equivalent to
      <string>: apiVersion: binding.operators.coreos.com/v1alpha1
      kind: ServiceBinding
      metadata:
        creationTimestamp: null
        name: aname
      spec:
        application:
          group: apps
          kind: Deployment
          name: app
          version: v1
        bindAsFiles: true
        detectBindingResources: true
        services:
        - group: postgresql.k8s.enterprisedb.io
          id: aname
          kind: Cluster
          name: cluster-sample
          resource: clusters
          version: v1
      status:
        secret: ""
      
  In [It] at: /go/odo_1/tests/integration/cmd_add_binding_test.go:53

...
Summarizing 2 Failures:
  [FAIL] odo add binding command tests [It] should create a binding using the workload parameter and naming strategy
  /go/odo_1/tests/integration/cmd_add_binding_test.go:86
  [FAIL] odo add binding command tests [It] should create a binding using the workload parameter
  /go/odo_1/tests/integration/cmd_add_binding_test.go:53

Ran 343 of 611 Specs in 1188.567 seconds
FAIL! -- 341 Passed | 2 Failed | 0 Pending | 268 Skipped

cmd_devfile_build_images_test

    [odo] build -t localhost:5000/devfile-nodejs-deploy:0.1.0 -f /tmp/221699204/Dockerfile /tmp/221699204
    [odo] 
 ✓  Building image locally [5ms]
    [odo] 
    [odo] ---
    [odo] A newer version of odo (v3.7.0) is available,
    [odo] visit https://github.com/redhat-developer/odo/releases to update.
    [odo] If you wish to disable this notification, run:
    [odo] odo preference set UpdateNotification false
    [odo] ---
    Setting current dir to: /go/odo_1/tests/integration
    Deleting dir: /tmp/221699204
    Deleting dir: /tmp/3623255060
  << End Captured GinkgoWriter Output

  Expected
      <string>: odo preference set UpdateNotification false
  to be equivalent to
      <string>: build -t localhost:5000/devfile-nodejs-deploy:0.1.0 -f /tmp/221699204/Dockerfile /tmp/221699204
  In [It] at: /go/odo_1/tests/integration/cmd_devfile_build_images_test.go:125
...
Summarizing 4 Failures:
  [FAIL] odo devfile build-images command tests label nocluster when using a devfile.yaml containing an Image component with a build context [It] should build image via Docker if build context references PROJECT_SOURCE env var [nocluster]
  /go/odo_1/tests/integration/cmd_devfile_build_images_test.go:125
  [FAIL] odo devfile build-images command tests label nocluster when using a devfile.yaml containing an Image component with a build context [It] should build image via Podman if build context references PROJECT_SOURCE env var [nocluster]
  /go/odo_1/tests/integration/cmd_devfile_build_images_test.go:125
  [FAIL] odo devfile build-images command tests label nocluster when using a devfile.yaml containing an Image component with no build context [It] should build image via Podman by defaulting build context to devfile path [nocluster]
  /go/odo_1/tests/integration/cmd_devfile_build_images_test.go:169
  [FAIL] odo devfile build-images command tests label nocluster when using a devfile.yaml containing an Image component with no build context [It] should build image via Docker by defaulting build context to devfile path [nocluster]
  /go/odo_1/tests/integration/cmd_devfile_build_images_test.go:169

Ran 113 of 611 Specs in 79.630 seconds
FAIL! -- 109 Passed | 4 Failed | 0 Pending | 498 Skipped

Any logs, error output, etc?

All the tests below seem to have failed for the same reason:

• https://s3.eu-de.cloud-object-storage.appdomain.cloud/odo-tests-openshift-logs/pr-6500-openshift-tests-4093.txt
• https://s3.eu-de.cloud-object-storage.appdomain.cloud/odo-tests-openshift-logs/pr-6500-nocluster-tests-1043.txt
• https://s3.eu-de.cloud-object-storage.appdomain.cloud/odo-tests-openshift-logs/pr-6500-kubernetes-tests-4057.txt
• https://s3.eu-de.cloud-object-storage.appdomain.cloud/odo-tests-openshift-logs/pr-6500-windows-tests-3292.txt

[feloy]

As I understand, this is because the PR's branch was based on a commit before the bump to 3.7.0.
This should not happen with a branch based on the same version

[rm3l]

As I understand, this is because the PR's branch was based on a commit before the bump to 3.7.0. This should not happen with a branch based on the same version

Yes, indeed. We should not have the issue with an up-to-date branch. But it is interesting that it highlighted the issue, I think, as it is not clear (from the contributor's perspective) that the PR branch is outdated.
For better resilience, I was thinking of either disabling the UpdateNotifiction preference for the tests where it is not useful, or making the checks on the output less strict.

[feloy]

Yes, I agree that making the checks on the output less strict would help.