fix: printout errors even if we have a timeout #5628

jmrodri · 2022-04-03T03:41:22Z

Signed-off-by: jesus m. rodriguez jesusr@redhat.com

Description of the change:
Handle DeadlineExceeded errors by allowing test output to be printed even during context timeouts

Motivation for the change:
context timeouts hide all test results

Fixes: #5415
Checklist

If the pull request includes user-facing changes, extra documentation is required:

Add a new changelog fragment in changelog/fragments (see changelog/fragments/00-template.yaml)
Add or update relevant sections of the docs website in website/content/en/docs

jmrodri · 2022-04-03T03:48:07Z

before this change, timeout would simply result in the context error showing:

$ operator-sdk scorecard ./bundle --wait-time 5s
Error: error running tests context deadline exceeded
Usage:
  operator-sdk scorecard [flags]

Flags:
  -c, --config string            path to scorecard config file
  -h, --help                     help for scorecard
      --kubeconfig string        kubeconfig path
  -L, --list                     Option to enable listing which tests are run
  -n, --namespace string         namespace to run the test images in
  -o, --output string            Output format for results. Valid values: text, json, xunit (default "text")
  -l, --selector string          label selector to determine which tests are run
  -s, --service-account string   Service account to use for tests (default "default")
  -x, --skip-cleanup             Disable resource cleanup after tests are run
  -b, --storage-image string     Storage image to be used by the Scorecard pod (default "docker.io/library/busybox@sha256:c71cb4f7e8ececaffb34037c2637dc86820e4185100e18b4d02d613a9bd772af")
  -t, --test-output string       Test output directory. (default "test-output")
  -u, --untar-image string       Untar image to be used by the Scorecard pod (default "registry.access.redhat.com/ubi8@sha256:910f6bc0b5ae9b555eb91b88d28d568099b060088616eba2867b07ab6ea457c7")
  -w, --wait-time duration       seconds to wait for tests to complete. Example: 35s (default 30s)

Global Flags:
      --plugins strings   plugin keys to be used for this subcommand execution
      --verbose           Enable verbose logging

FATA[0005] error running tests context deadline exceeded

jmrodri · 2022-04-03T03:48:44Z

after this change, the test results will printout

$ operator-sdk scorecard ./bundle --wait-time 5s                                                                                                
--------------------------------------------------------------------------------                                                                                                              
Image:      quay.io/operator-framework/scorecard-test:v1.18.0                                                                                                                                 
Entrypoint: [scorecard-test olm-crds-have-resources]                                                                                                                                          
Labels:                                                                                                                                                                                       
        "suite":"olm"                                                                                                                                                                         
        "test":"olm-crds-have-resources-test"                                                                                                                                                 
Results:                                                                                                                                                                                      
        Name: olm-crds-have-resources                                                                                                                                                         
        State: fail                                                                                                                                                                           
                                                                                                                                                                                              
        Errors:                                                                                                                                                                               
                Owned CRDs do not have resources specified                                                                                                                                    
        Log:                                                                                                                                                                                  
                Loaded ClusterServiceVersion: memcached-operator.v0.0.1                                                                                                                       
                                                                                                                                                                                              
                                                                                                                                                                                              
--------------------------------------------------------------------------------                                                                                                              
Image:      quay.io/operator-framework/scorecard-test:v1.18.0                                                                                                                                 
Entrypoint: [scorecard-test basic-check-spec]                                                                                                                                                 
Labels:                                                                                                                                                                                       
        "suite":"basic"                                                                                                                                                                       
        "test":"basic-check-spec-test"                                                                                                                                                        
Results:                                                                                                                                                                                      
        State: fail                                                                                                                                                                           
                                                                                                                                                                                              
        Errors:                                                                                                                                                                               
                timed out waiting for the condition                                                                                                                                           
                                                                                                                                                                                              
                                                                                                                                                                                              
--------------------------------------------------------------------------------                                                                                                              
Image:      quay.io/operator-framework/scorecard-test:v1.18.0                                                                                                                                 
Entrypoint: [scorecard-test olm-spec-descriptors]                                                                                                                                             
Labels:                                                                                                                                                                                       
        "suite":"olm"                                                                                                                                                                         
        "test":"olm-spec-descriptors-test"                                                                                                                                                    
Results:                                                                                                                                                                                      
        State: fail                       
                                                                                                                                                                                              
        Errors:
                timed out waiting for the condition


--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.18.0
Entrypoint: [scorecard-test olm-bundle-validation]
Labels:
        "suite":"olm"
        "test":"olm-bundle-validation-test"
Results:
        State: fail

        Errors:
                timed out waiting for the condition


--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.18.0
Entrypoint: [scorecard-test olm-status-descriptors]
Labels:
        "suite":"olm"
        "test":"olm-status-descriptors-test"
Results:
        State: fail

        Errors:
                timed out waiting for the condition


--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.18.0
Entrypoint: [scorecard-test olm-crds-have-validation]
Labels:
        "suite":"olm"
        "test":"olm-crds-have-validation-test"
Results:
        State: fail

        Errors:
                timed out waiting for the condition


Error: error running tests context deadline exceeded
Usage:
  operator-sdk scorecard [flags]

Flags:
  -c, --config string            path to scorecard config file
  -h, --help                     help for scorecard
      --kubeconfig string        kubeconfig path
  -L, --list                     Option to enable listing which tests are run
  -n, --namespace string         namespace to run the test images in
  -o, --output string            Output format for results. Valid values: text, json, xunit (default "text")
  -l, --selector string          label selector to determine which tests are run
  -s, --service-account string   Service account to use for tests (default "default")
  -x, --skip-cleanup             Disable resource cleanup after tests are run
  -b, --storage-image string     Storage image to be used by the Scorecard pod (default "docker.io/library/busybox@sha256:c71cb4f7e8ececaffb34037c2637dc86820e4185100e18b4d02d613a9bd772af")
  -t, --test-output string       Test output directory. (default "test-output")
  -u, --untar-image string       Untar image to be used by the Scorecard pod (default "registry.access.redhat.com/ubi8@sha256:910f6bc0b5ae9b555eb91b88d28d568099b060088616eba2867b07ab6ea457c7
")
  -w, --wait-time duration       seconds to wait for tests to complete. Example: 35s (default 30s)

Global Flags:
      --plugins strings   plugin keys to be used for this subcommand execution
      --verbose           Enable verbose logging

FATA[0005] error running tests context deadline exceeded

Signed-off-by: jesus m. rodriguez <jesusr@redhat.com>

camilamacedo86

/lgtm

rashmigottipati

Just one nit.

/lgtm

rashmigottipati · 2022-04-07T17:34:05Z

internal/cmd/operator-sdk/scorecard/cmd.go

@@ -233,6 +233,12 @@ func (c *scorecardCmd) run() (err error) {

 		scorecardTests, err = o.Run(ctx)
 		if err != nil {
+			// if we got a timeout; printout the test results if there are any
+			if err == context.DeadlineExceeded {


Nit:

Suggested change

if err == context.DeadlineExceeded {

if errors.Is(err, context.DeadlineExceeded) {

Signed-off-by: jesus m. rodriguez <jesusr@redhat.com> Signed-off-by: Venkat Ramaraju <vramaraj@redhat.com>

…ework#5628)" This reverts commit b389906.

openshift-ci bot requested review from joelanford and theishshah April 3, 2022 03:41

jmrodri force-pushed the issue5415 branch from 6b84502 to 56d4d36 Compare April 3, 2022 03:46

fix: printout errors even if we have a timeout

c57b644

Signed-off-by: jesus m. rodriguez <jesusr@redhat.com>

jmrodri force-pushed the issue5415 branch from 56d4d36 to c57b644 Compare April 3, 2022 03:58

jmrodri mentioned this pull request Apr 3, 2022

Error: error running tests context deadline exceeded is missing the root cause #5415

Closed

camilamacedo86 approved these changes Apr 6, 2022

View reviewed changes

openshift-ci bot assigned camilamacedo86 Apr 6, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 6, 2022

rashmigottipati approved these changes Apr 7, 2022

View reviewed changes

openshift-ci bot assigned rashmigottipati Apr 7, 2022

rashmigottipati reviewed Apr 7, 2022

View reviewed changes

jmrodri merged commit fda35b2 into operator-framework:master Apr 7, 2022

VenkatRamaraju pushed a commit to VenkatRamaraju/operator-sdk that referenced this pull request May 18, 2022

Revert "fix: printout errors even if we have a timeout (operator-fram…

95085ae

…ework#5628)" This reverts commit b389906.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: printout errors even if we have a timeout #5628

fix: printout errors even if we have a timeout #5628

jmrodri commented Apr 3, 2022 •

edited

Loading

jmrodri commented Apr 3, 2022

jmrodri commented Apr 3, 2022 •

edited

Loading

camilamacedo86 left a comment

rashmigottipati left a comment •

edited

Loading

rashmigottipati Apr 7, 2022

	if err == context.DeadlineExceeded {
	if errors.Is(err, context.DeadlineExceeded) {

fix: printout errors even if we have a timeout #5628

fix: printout errors even if we have a timeout #5628

Conversation

jmrodri commented Apr 3, 2022 • edited Loading

jmrodri commented Apr 3, 2022

jmrodri commented Apr 3, 2022 • edited Loading

camilamacedo86 left a comment

Choose a reason for hiding this comment

rashmigottipati left a comment • edited Loading

Choose a reason for hiding this comment

rashmigottipati Apr 7, 2022

Choose a reason for hiding this comment

jmrodri commented Apr 3, 2022 •

edited

Loading

jmrodri commented Apr 3, 2022 •

edited

Loading

rashmigottipati left a comment •

edited

Loading