Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: error running tests context deadline exceeded is missing the root cause #5415

Closed
camilamacedo86 opened this issue Nov 26, 2021 · 3 comments · Fixed by #5628
Closed
Assignees
Labels
scorecard Issue relates to the scorecard subcomponent
Milestone

Comments

@camilamacedo86
Copy link
Contributor

camilamacedo86 commented Nov 26, 2021

Bug Report

Error: error running tests context deadline exceeded is missing the root cause

What did you do?

CVP has been using Scorecard and facing:

+ echo 'Running the operator-sdk scorecard test using the basic configuration, json output and storing it in the artifacts directory'
Running the operator-sdk scorecard test using the basic configuration, json output and storing it in the artifacts directory
+ operator-sdk scorecard --config /tmp/config/scorecard-basic-config.yml --namespace oo-qrfcx --kubeconfig /tmp/kubeconfig-293377029 --verbose --output json test-operator
time="2021-11-26T00:54:33Z" level=debug msg="Debug logging is set"
Error: error running tests context deadline exceeded

What did you expect to see?

The root cause of the issue in the error message
Ideally, if possible as well a suggestion over how to solve such as: Use the options --storage-image for the image ... and --untar-image for ... in order to ... (if that is the same root cause of #5306)

What did you see instead? Under which circumstances?

Only error running tests context deadline exceeded which shows that occurs when is not possible to download "docker.io/busybox:1.33.0". See: #4886

@camilamacedo86
Copy link
Contributor Author

Hi @jmrodri @tlwu2013 could we get help with this one?

@varshaprasad96 varshaprasad96 added the scorecard Issue relates to the scorecard subcomponent label Nov 29, 2021
@varshaprasad96 varshaprasad96 modified the milestones: v1.16.0, v1.17.0 Nov 29, 2021
@jberkhahn jberkhahn modified the milestones: v1.17.0, v1.18.0 Jan 31, 2022
@asmacdo asmacdo modified the milestones: v1.18.0, 1.19.0 Feb 2, 2022
@jmrodri
Copy link
Member

jmrodri commented Mar 16, 2022

Looking at the code when there is a timeout, we check the timeout error.
https://github.com/operator-framework/operator-sdk/blob/master/internal/scorecard/scorecard.go#L98-L102

	select {
	case <-ctx.Done():
		err = ctx.Err()
	default:
	}

So we had a context timeout and captured the error. It does look like we may still have testOutput. We then short circuit because there was an error from the o.Run.

https://github.com/operator-framework/operator-sdk/blob/master/internal/cmd/operator-sdk/scorecard/cmd.go#L234-L237

		scorecardTests, err = o.Run(ctx)
		if err != nil {
			return fmt.Errorf("error running tests %w", err)
		}

If there are no errors, we will usually get to printOutput.

https://github.com/operator-framework/operator-sdk/blob/master/internal/cmd/operator-sdk/scorecard/cmd.go#L240-L242

if err := c.printOutput(scorecardTests); err != nil {
		log.Fatal(err)
	}

I think we should also see if there are results in scorecardTests and add that to the output when there is an error during o.Run. Maybe something like:

		scorecardTests, err = o.Run(ctx)
		if err != nil && err != context.DeadlineExceeded {
			return fmt.Errorf("error running tests %w", err)
		}

Or we could ALWAYS print out the results regardless of error.

@jmrodri
Copy link
Member

jmrodri commented Apr 3, 2022

So I posted a PR #5628 that prints out the output. @camilamacedo86 let me know if this seems okay. Basically if there is a timeout, we will print the test results then return an error to cobra so that we still error out. This should give any information that may have occurred during a test failure.

$ operator-sdk scorecard ./bundle --wait-time 5s                                                                                                
--------------------------------------------------------------------------------                                                                                                              
Image:      quay.io/operator-framework/scorecard-test:v1.18.0                                                                                                                                 
Entrypoint: [scorecard-test olm-crds-have-resources]                                                                                                                                          
Labels:                                                                                                                                                                                       
        "suite":"olm"                                                                                                                                                                         
        "test":"olm-crds-have-resources-test"                                                                                                                                                 
Results:                                                                                                                                                                                      
        Name: olm-crds-have-resources                                                                                                                                                         
        State: fail                                                                                                                                                                           
                                                                                                                                                                                              
        Errors:                                                                                                                                                                               
                Owned CRDs do not have resources specified                                                                                                                                    
        Log:                                                                                                                                                                                  
                Loaded ClusterServiceVersion: memcached-operator.v0.0.1                                                                                                                       
                                                                                                                                                                                              
                                                                                                                                                                                              
--------------------------------------------------------------------------------                                                                                                              
Image:      quay.io/operator-framework/scorecard-test:v1.18.0                                                                                                                                 
Entrypoint: [scorecard-test basic-check-spec]                                                                                                                                                 
Labels:                                                                                                                                                                                       
        "suite":"basic"                                                                                                                                                                       
        "test":"basic-check-spec-test"                                                                                                                                                        
Results:                                                                                                                                                                                      
        State: fail                                                                                                                                                                           
                                                                                                                                                                                              
        Errors:                                                                                                                                                                               
                timed out waiting for the condition                                                                                                                                           
                                                                                                                                                                                              
                                                                                                                                                                                              
--------------------------------------------------------------------------------                                                                                                              
Image:      quay.io/operator-framework/scorecard-test:v1.18.0                                                                                                                                 
Entrypoint: [scorecard-test olm-spec-descriptors]                                                                                                                                             
Labels:                                                                                                                                                                                       
        "suite":"olm"                                                                                                                                                                         
        "test":"olm-spec-descriptors-test"                                                                                                                                                    
Results:                                                                                                                                                                                      
        State: fail                       
                                                                                                                                                                                              
        Errors:
                timed out waiting for the condition


--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.18.0
Entrypoint: [scorecard-test olm-bundle-validation]
Labels:
        "suite":"olm"
        "test":"olm-bundle-validation-test"
Results:
        State: fail

        Errors:
                timed out waiting for the condition


--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.18.0
Entrypoint: [scorecard-test olm-status-descriptors]
Labels:
        "suite":"olm"
        "test":"olm-status-descriptors-test"
Results:
        State: fail

        Errors:
                timed out waiting for the condition


--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.18.0
Entrypoint: [scorecard-test olm-crds-have-validation]
Labels:
        "suite":"olm"
        "test":"olm-crds-have-validation-test"
Results:
        State: fail

        Errors:
                timed out waiting for the condition


Error: error running tests context deadline exceeded
Usage:
  operator-sdk scorecard [flags]

Flags:
  -c, --config string            path to scorecard config file
  -h, --help                     help for scorecard
      --kubeconfig string        kubeconfig path
  -L, --list                     Option to enable listing which tests are run
  -n, --namespace string         namespace to run the test images in
  -o, --output string            Output format for results. Valid values: text, json, xunit (default "text")
  -l, --selector string          label selector to determine which tests are run
  -s, --service-account string   Service account to use for tests (default "default")
  -x, --skip-cleanup             Disable resource cleanup after tests are run
  -b, --storage-image string     Storage image to be used by the Scorecard pod (default "docker.io/library/busybox@sha256:c71cb4f7e8ececaffb34037c2637dc86820e4185100e18b4d02d613a9bd772af")
  -t, --test-output string       Test output directory. (default "test-output")
  -u, --untar-image string       Untar image to be used by the Scorecard pod (default "registry.access.redhat.com/ubi8@sha256:910f6bc0b5ae9b555eb91b88d28d568099b060088616eba2867b07ab6ea457c7
")
  -w, --wait-time duration       seconds to wait for tests to complete. Example: 35s (default 30s)

Global Flags:
      --plugins strings   plugin keys to be used for this subcommand execution
      --verbose           Enable verbose logging

FATA[0005] error running tests context deadline exceeded 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scorecard Issue relates to the scorecard subcomponent
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants