Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Che-Theia build is failed sporadically within the HappyPath test jobs #18575

Closed
22 tasks
azatsarynnyy opened this issue Dec 9, 2020 · 5 comments
Closed
22 tasks

Che-Theia build is failed sporadically within the HappyPath test jobs #18575

azatsarynnyy opened this issue Dec 9, 2020 · 5 comments
Labels
e2e-test/failure Issues that is related to a test failures reported by our CI platform and our QE. kind/bug Outline of a bug - must adhere to the bug report template. severity/blocker Causes system to crash and be non-recoverable or prevents Che developers from working on Che code.

Comments

@azatsarynnyy
Copy link
Member

azatsarynnyy commented Dec 9, 2020 •
edited by dmytro-ndp
Loading

Describe the bug

After two days of shutdown of HappyPath test jobs, we noticed today it started failing sporadically on Che-Theia PRs.

While there's a PR where HappyPath tests are passed successfully eclipse-che/che-theia#947 (comment),
there're several PRs with the failed HappyPath test jobs: eclipse-che/che-theia#945 (comment), eclipse-che/che-theia#948 (comment), eclipse-che/che-theia#778 (comment).

The error is related to an image build. It's the same in all the cases:

16:14:47 Step 27/38 : RUN find /che-theia-build -exec sh -c "chgrp 0 {}; chmod g+rwX {}" \; 2>log.txt &&     find /che-theia-build/plugins -name "*.sh" | xargs chmod +x
16:14:47  ---> Running in 7b1faecc48c6
16:15:10 BusyBox v1.31.1 () multi-call binary.
16:15:10 
16:15:10 Usage: chmod [-Rcvf] MODE[,MODE]... FILE...
16:15:10 
16:15:10 Each MODE is one or more of the letters ugoa, one of the
16:15:10 symbols +-= and one or more of the letters rwxst
16:15:10 
16:15:10 	-R	Recurse
16:15:10 	-c	List changed files
16:15:10 	-v	List all files
16:15:10 	-f	Hide errors
16:15:10 The command '/bin/sh -c find /che-theia-build -exec sh -c "chgrp 0 {}; chmod g+rwX {}" \; 2>log.txt &&     find /che-theia-build/plugins -name "*.sh" | xargs chmod +x' returned a non-zero code: 123
16:15:10 Build step 'Execute shell' marked build as failure

Failing command: https://github.com/eclipse/che-theia/blob/fdf6fee29b1bc0f8aa0286a99f188fe54dcafe3c/dockerfiles/theia/Dockerfile#L119

Successful build flow:

Step 27/38 : RUN find /che-theia-build -exec sh -c "chgrp 0 {}; chmod g+rwX {}" \; 2>log.txt &&     find /che-theia-build/plugins -name "*.sh" | xargs chmod +x
 ---> Running in 78e5548bdfb4
Removing intermediate container 78e5548bdfb4
 ---> 5a4fbf01748a
Step 28/38 : FROM node:12.20.0-alpine3.12 as runtime
 ---> 7d016f811d40
Step 29/38 : ENV USE_LOCAL_GIT=true     HOME=/home/theia     SHELL=/bin/bash     THEIA_DEFAULT_PLUGINS=local-dir:///default-theia-plugins     LOCAL_GIT_DIRECTORY=/usr     GIT_EXEC_PATH=/usr/libexec/git-core     PORT_PLUGIN_EXCLUDE_3130=TRUE     YARN_FLAGS=""
 ---> Running in e25a4d60679b
Removing intermediate container e25a4d60679b
 ---> c21fd1076d21
Step 30/38 : EXPOSE 3100 3130
 ---> Running in bee002d6d349
Removing intermediate container bee002d6d349
 ---> a1ffe9b61ea3
Step 31/38 : COPY --from=build-result /che-theia-build/plugins /default-theia-plugins
 ---> 68c1550fabf6
Step 32/38 : RUN apk add --update --no-cache sudo git bzip2 which bash curl openssh openssh-keygen less sshpass lsblk
 ---> Running in 92e8c41954c2
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/27) Installing ncurses-terminfo-base (6.2_p20200523-r0)Successful build flow:
Step 27/38 : RUN find /che-theia-build -exec sh -c "chgrp 0 {}; chmod g+rwX {}" \; 2>log.txt &&     find /che-theia-build/plugins -name "*.sh" | xargs chmod +x
 ---> Running in 78e5548bdfb4
Removing intermediate container 78e5548bdfb4
 ---> 5a4fbf01748a
Step 28/38 : FROM node:12.20.0-alpine3.12 as runtime
 ---> 7d016f811d40
Step 29/38 : ENV USE_LOCAL_GIT=true     HOME=/home/theia     SHELL=/bin/bash     THEIA_DEFAULT_PLUGINS=local-dir:///default-theia-plugins     LOCAL_GIT_DIRECTORY=/usr     GIT_EXEC_PATH=/usr/libexec/git-core     PORT_PLUGIN_EXCLUDE_3130=TRUE     YARN_FLAGS=""
 ---> Running in e25a4d60679b
Removing intermediate container e25a4d60679b
 ---> c21fd1076d21
Step 30/38 : EXPOSE 3100 3130
 ---> Running in bee002d6d349
Removing intermediate container bee002d6d349
 ---> a1ffe9b61ea3
Step 31/38 : COPY --from=build-result /che-theia-build/plugins /default-theia-plugins
 ---> 68c1550fabf6
Step 32/38 : RUN apk add --update --no-cache sudo git bzip2 which bash curl openssh openssh-keygen less sshpass lsblk
 ---> Running in 92e8c41954c2
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/27) Installing ncurses-terminfo-base (6.2_p20200523-r0)
...

Che version

  • latest
  • nightly
  • other: please specify

Steps to reproduce

Expected behavior

Runtime

  • kubernetes (include output of kubectl version)
  • Openshift (include output of oc version)
  • minikube (include output of minikube version and kubectl version)
  • minishift (include output of minishift version and oc version)
  • docker-desktop + K8S (include output of docker version and kubectl version)
  • other: (please specify)

Screenshots

Installation method

  • chectl
    • provide a full command that was used to deploy Eclipse Che (including the output)
    • provide an output of chectl version command
  • OperatorHub
  • I don't know

Environment

  • my computer
    • Windows
    • Linux
    • macOS
  • Cloud
    • Amazon
    • Azure
    • GCE
    • other (please specify)
  • other: please specify

Eclipse Che Logs

Additional context
@azatsarynnyy azatsarynnyy added the kind/bug Outline of a bug - must adhere to the bug report template. label Dec 9, 2020
@dmytro-ndp dmytro-ndp added the e2e-test/failure Issues that is related to a test failures reported by our CI platform and our QE. label Dec 9, 2020
@che-bot che-bot added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Dec 9, 2020
@azatsarynnyy azatsarynnyy added the severity/blocker Causes system to crash and be non-recoverable or prevents Che developers from working on Che code. label Dec 9, 2020
@azatsarynnyy
Copy link
Member Author

I'm marking it as a blocker since we are not able to merge any Che-Theia PR.
The last PR where the tests passed today was eclipse-che/che-theia#947 (comment)

@azatsarynnyy
Copy link
Member Author

Looks like it can't call chmod
https://github.com/eclipse/che-theia/blob/f0586a8b28f3b0ca8254cdb39f45b703bcc88c41/dockerfiles/theia/Dockerfile#L119

@azatsarynnyy
Copy link
Member Author

It's also interesting that the same build has no issues when running on GitHub infra, with Actions.

@azatsarynnyy
Copy link
Member Author

It seems the initial error has gone 🤷‍♂️
And now Jenkins can't clone the repo:
https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/che-pr-tests/view/K8S/job/Che-Theia-PR-check-E2E-Happy-path-tests-against-K8S__reserved/2085/console

21:30:22 ERROR: Error cloning remote repo 'origin'
21:30:22 hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/eclipse/che-theia +refs/heads/*:refs/remotes/origin/*" returned status code 128:
21:30:22 stdout: 
21:30:22 stderr: fatal: unable to access 'https://github.com/eclipse/che-theia/': Failed connect to github.com:443; Connection timed out
21:30:22 
21:30:22 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
21:30:22 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
21:30:22 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
21:30:22 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:420)
21:30:22 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:629)
21:30:22 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
21:30:22 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
21:30:22 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
21:30:22 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
21:30:22 	at hudson.remoting.Request$2.run(Request.java:369)
21:30:22 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
21:30:22 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
21:30:22 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
21:30:22 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
21:30:22 	at java.lang.Thread.run(Thread.java:748)
21:30:22 	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to minikube-rhel7-24gb-26611
21:30:22 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
21:30:22 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
21:30:22 		at hudson.remoting.Channel.call(Channel.java:955)
21:30:22 		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
21:30:22 		at sun.reflect.GeneratedMethodAccessor1176.invoke(Unknown Source)
21:30:22 		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
21:30:22 		at java.lang.reflect.Method.invoke(Method.java:498)
21:30:22 		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
21:30:22 		at com.sun.proxy.$Proxy129.execute(Unknown Source)
21:30:22 		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1146)
21:30:22 		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
21:30:22 		at hudson.scm.SCM.checkout(SCM.java:504)
21:30:22 		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
21:30:22 		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
21:30:22 		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
21:30:22 		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
21:30:22 		at hudson.model.Run.execute(Run.java:1798)
21:30:22 		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
21:30:22 		at hudson.model.ResourceController.execute(ResourceController.java:97)
21:30:22 		at hudson.model.Executor.run(Executor.java:429)
21:30:22 ERROR: Error cloning remote repo 'origin'

@azatsarynnyy
Copy link
Member Author

It looks like everything is stabilized now. The tests are passes successfully on several Che-Theia PRs:

Most likely, it was a glitch on CI side. So, closing it.

@amisevsk amisevsk removed the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Dec 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
e2e-test/failure Issues that is related to a test failures reported by our CI platform and our QE. kind/bug Outline of a bug - must adhere to the bug report template. severity/blocker Causes system to crash and be non-recoverable or prevents Che developers from working on Che code.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dmytro-ndp @azatsarynnyy @amisevsk @che-bot