Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed run for Docker image build push varcall_py36 #703

Closed
ashwini06 opened this issue Aug 12, 2021 · 2 comments · Fixed by #706
Closed

Failed run for Docker image build push varcall_py36 #703

ashwini06 opened this issue Aug 12, 2021 · 2 comments · Fixed by #706

Comments

@ashwini06
Copy link
Contributor

Describe the bug
Upon merging the PR into develop branch, the jobs for building the docker image will be run. One of the jobs Docker image build push (varcall_py36) was failed. More details https://github.com/Clinical-Genomics/BALSAMIC/actions/runs/1119405249

Version (please complete the following information):
7.2.6

Additional context
Check this earlier merged PR where this problem might have been started

https://github.com/Clinical-Genomics/BALSAMIC/runs/3144150768
https://github.com/Clinical-Genomics/BALSAMIC/runs/3219860390
@khurrammaqbool
Copy link
Collaborator

Log from latest merge (#699) to develop is below, which shows that the operation timed out at six hours for running command RUN /bin/sh varcall_py36.sh varcall_py36 && conda clean --all --yes

5h 59m 46s
Run docker/build-push-action@v2
  with:
    push: true
    tags: clinicalgenomics/balsamic:develop-varcall_py36
    build-args: CONTAINER_NAME=varcall_py36
    load: false
    no-cache: false
    pull: false
    github-token: ***
Docker info
  /usr/bin/docker version
  Client:
   Version:           20.10.7+azure
   API version:       1.41
   Go version:        go1.13.15
   Git commit:        f0df35096d5f5e6b559b42c7fde6c65a2909f7c5
   Built:             Mon May 31 09:40:07 2021
   OS/Arch:           linux/amd64
   Context:           default
   Experimental:      true
  
  Server:
   Engine:
    Version:          20.10.7+azure
    API version:      1.41 (minimum version 1.12)
    Go version:       go1.13.15
    Git commit:       b0f5bc36fea9dfb9672e1e9b1278ebab797b9ee0
    Built:            Tue Jun  1 22:34:42 2021
    OS/Arch:          linux/amd64
    Experimental:     false
   containerd:
    Version:          1.4.9+azure
    GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
   runc:
    Version:          1.0.1
    GitCommit:        4144b63817ebcc5b358fc2c8ef95f7cddd709aa7
   docker-init:
    Version:          0.19.0
    GitCommit:        
  /usr/bin/docker info
  Client:
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Build with BuildKit (Docker Inc., 0.6.0+azure)
  
  Server:
   Containers: 1
    Running: 1
    Paused: 0
    Stopped: 0
   Images: 18
   Server Version: 20.10.7+azure
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Cgroup Version: 1
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: e25210fe30a0a703442421b0f60afac609f950a3
   runc version: 4144b63817ebcc5b358fc2c8ef95f7cddd709aa7
   init version: 
   Security Options:
    apparmor
    seccomp
     Profile: default
   Kernel Version: 5.4.0-1055-azure
   Operating System: Ubuntu 18.04.5 LTS
   OSType: linux
   Architecture: x86_64
   CPUs: 2
   Total Memory: 6.791GiB
   Name: fv-az54-994
   ID: UEOF:IJI2:FMBG:LZGQ:BIVO:JKKM:GWQA:XTRO:AJV6:DIX3:S4ZN:EVOP
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: ***
   Registry: https://index.docker.io/v1/
   Labels:
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false
  
  WARNING: No swap limit support
/usr/bin/docker buildx build --build-arg CONTAINER_NAME=varcall_py36 --tag clinicalgenomics/balsamic:develop-varcall_py36 --iidfile /tmp/docker-build-push-I32XVq/iidfile --secret id=GIT_AUTH_TOKEN,src=/tmp/docker-build-push-I32XVq/tmp-3224-sFyMe2a765g0 --push https://github.com/Clinical-Genomics/BALSAMIC.git#260def3fc7d6b05a849b5b042b57cb4b6a33f5e8
#1 [internal] load git source https://github.com/Clinical-Genomics/BALSAMIC.git#260def3fc7d6b05a849b5b042b57cb4b6a33f5e8
#1 0.016 hint: Using 'master' as the name for the initial branch. This default branch name
#1 0.016 hint: is subject to change. To configure the initial branch name to use in all
#1 0.016 hint: of your new repositories, which will suppress this warning, call:
#1 0.016 hint: 
#1 0.016 hint: 	git config --global init.defaultBranch <name>
#1 0.016 hint: 
#1 0.016 hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
#1 0.016 hint: 'development'. The just-created branch can be renamed via this command:
#1 0.016 hint: 
#1 0.016 hint: 	git branch -m <name>
#1 0.017 Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
#1 0.022 fatal: Not a valid object name 260def3fc7d6b05a849b5b042b57cb4b6a33f5e8^{commit}
#1 6.801 From https://github.com/Clinical-Genomics/BALSAMIC
#1 6.801  * [new branch]      develop                 -> origin/develop
#1 6.801  * [new branch]      feat/ascat_rules        -> origin/feat/ascat_rules
#1 6.801  * [new branch]      feat/balsamic_container -> origin/feat/balsamic_container
#1 6.801  * [new branch]      feat/github_actions_ref_20210719 -> origin/feat/github_actions_ref_20210719
#1 6.802  * [new branch]      feat/update_snakemake_6_5_3 -> origin/feat/update_snakemake_6_5_3
#1 6.802  * [new branch]      jupyter_notebook        -> origin/jupyter_notebook
#1 6.802  * [new branch]      master                  -> origin/master
#1 6.802  * [new branch]      qc_check                -> origin/qc_check
#1 6.802  * [new branch]      refactor/annotate_snakemakerules -> origin/refactor/annotate_snakemakerules
#1 6.802  * [new branch]      refactor/rule_cnvkit_paired -> origin/refactor/rule_cnvkit_paired
#1 6.802  * [new branch]      refactor/umi_variantcalling_snakemake_rules -> origin/refactor/umi_variantcalling_snakemake_rules
#1 6.803  * [new branch]      release/8_0_0           -> origin/release/8_0_0
#1 6.803  * [new tag]         v7.2.6                  -> v7.2.6
#1 6.845  * [new tag]         1.1.0                   -> 1.1.0
#1 6.845  * [new tag]         1.9.0                   -> 1.9.0
#1 6.845  * [new tag]         2.6.1                   -> 2.6.1
#1 6.845  * [new tag]         2.9.0                   -> 2.9.0
#1 6.845  * [new tag]         3.0.1                   -> 3.0.1
#1 6.845  * [new tag]         v2.9.6                  -> v2.9.6
#1 6.845  * [new tag]         v2.9.7                  -> v2.9.7
#1 6.845  * [new tag]         v2.9.8                  -> v2.9.8
#1 6.845  * [new tag]         v3.0.1                  -> v3.0.1
#1 6.846  * [new tag]         v3.1.1                  -> v3.1.1
#1 6.846  * [new tag]         v3.1.2                  -> v3.1.2
#1 6.846  * [new tag]         v3.1.3                  -> v3.1.3
#1 6.846  * [new tag]         v3.1.4                  -> v3.1.4
#1 6.847  * [new tag]         v3.2.0                  -> v3.2.0
#1 6.847  * [new tag]         v3.2.1                  -> v3.2.1
#1 6.847  * [new tag]         v3.2.2                  -> v3.2.2
#1 6.847  * [new tag]         v3.2.3                  -> v3.2.3
#1 6.847  * [new tag]         v3.3.0                  -> v3.3.0
#1 6.847  * [new tag]         v3.3.1                  -> v3.3.1
#1 6.847  * [new tag]         v4.0.0                  -> v4.0.0
#1 6.847  * [new tag]         v4.0.1                  -> v4.0.1
#1 6.847  * [new tag]         v4.1.0                  -> v4.1.0
#1 6.847  * [new tag]         v4.2.0                  -> v4.2.0
#1 6.848  * [new tag]         v4.2.1                  -> v4.2.1
#1 6.848  * [new tag]         v4.2.2                  -> v4.2.2
#1 6.848  * [new tag]         v4.2.3                  -> v4.2.3
#1 6.848  * [new tag]         v4.2.4                  -> v4.2.4
#1 6.848  * [new tag]         v4.3.0                  -> v4.3.0
#1 6.848  * [new tag]         v4.4.0                  -> v4.4.0
#1 6.848  * [new tag]         v4.5.0                  -> v4.5.0
#1 6.848  * [new tag]         v5.0.0                  -> v5.0.0
#1 6.848  * [new tag]         v5.0.1                  -> v5.0.1
#1 6.848  * [new tag]         v5.1.0                  -> v5.1.0
#1 6.849  * [new tag]         v6.0.0                  -> v6.0.0
#1 6.849  * [new tag]         v6.0.1                  -> v6.0.1
#1 6.849  * [new tag]         v6.0.2                  -> v6.0.2
#1 6.850  * [new tag]         v6.0.3                  -> v6.0.3
#1 6.850  * [new tag]         v6.0.4                  -> v6.0.4
#1 6.850  * [new tag]         v6.1.0                  -> v6.1.0
#1 6.850  * [new tag]         v6.1.1                  -> v6.1.1
#1 6.851  * [new tag]         v6.1.2                  -> v6.1.2
#1 6.851  * [new tag]         v7.0.0                  -> v7.0.0
#1 6.851  * [new tag]         v7.0.1                  -> v7.0.1
#1 6.853  * [new tag]         v7.0.2                  -> v7.0.2
#1 6.853  * [new tag]         v7.1.0                  -> v7.1.0
#1 6.853  * [new tag]         v7.1.1                  -> v7.1.1
#1 6.858  * [new tag]         v7.1.10                 -> v7.1.10
#1 6.858  * [new tag]         v7.1.2                  -> v7.1.2
#1 6.858  * [new tag]         v7.1.3                  -> v7.1.3
#1 6.858  * [new tag]         v7.1.4                  -> v7.1.4
#1 6.858  * [new tag]         v7.1.5                  -> v7.1.5
#1 6.859  * [new tag]         v7.1.6                  -> v7.1.6
#1 6.859  * [new tag]         v7.1.7                  -> v7.1.7
#1 6.859  * [new tag]         v7.1.8                  -> v7.1.8
#1 6.859  * [new tag]         v7.1.9                  -> v7.1.9
#1 6.859  * [new tag]         v7.2.0                  -> v7.2.0
#1 6.859  * [new tag]         v7.2.1                  -> v7.2.1
#1 6.859  * [new tag]         v7.2.2                  -> v7.2.2
#1 6.860  * [new tag]         v7.2.3                  -> v7.2.3
#1 6.860  * [new tag]         v7.2.4                  -> v7.2.4
#1 6.860  * [new tag]         v7.2.5                  -> v7.2.5
#1 7.879 Submodule 'ascatNgs' (https://github.com/cancerit/ascatNgs) registered for path 'ascatNgs'
#1 7.883 Cloning into '/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2/fs/ascatNgs'...
#1 8.956 Submodule path 'ascatNgs': checked out '0d83b28d3837d1d001c1dceefcd20b71aa351225'
#1 DONE 9.0s

#2 [internal] load metadata for docker.io/continuumio/miniconda3:4.9.2-alpine
#2 ...

#3 [auth] continuumio/miniconda3:pull token for registry-1.docker.io
#3 DONE 0.0s

#2 [internal] load metadata for docker.io/continuumio/miniconda3:4.9.2-alpine
#2 DONE 0.3s

#4 [1/5] FROM docker.io/continuumio/miniconda3:4.9.2-alpine@sha256:82bd96b0e95188e152d137f6c9834ea731bfc78e5c5f27b3c90f2be31e9e61d8
#4 resolve docker.io/continuumio/miniconda3:4.9.2-alpine@sha256:82bd96b0e95188e152d137f6c9834ea731bfc78e5c5f27b3c90f2be31e9e61d8 done
#4 DONE 0.0s

#4 [1/5] FROM docker.io/continuumio/miniconda3:4.9.2-alpine@sha256:82bd96b0e95188e152d137f6c9834ea731bfc78e5c5f27b3c90f2be31e9e61d8
#4 sha256:261bf2ca9be03f107cd0389e51acc63c6fae81ceaa25de64f68ad9301ef95ef5 5.72MB / 5.72MB 0.1s done
#4 sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 2.80MB / 2.80MB 0.1s done
#4 extracting sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 0.1s done
#4 sha256:6cb18017795d33446d8fbed8b7373e868da0c9c0504ebe8109f770c94714897a 8.39MB / 52.98MB 0.2s
#4 sha256:6cb18017795d33446d8fbed8b7373e868da0c9c0504ebe8109f770c94714897a 19.92MB / 52.98MB 0.3s
#4 extracting sha256:261bf2ca9be03f107cd0389e51acc63c6fae81ceaa25de64f68ad9301ef95ef5
#4 sha256:6cb18017795d33446d8fbed8b7373e868da0c9c0504ebe8109f770c94714897a 33.55MB / 52.98MB 0.5s
#4 extracting sha256:261bf2ca9be03f107cd0389e51acc63c6fae81ceaa25de64f68ad9301ef95ef5 0.2s done
#4 sha256:6cb18017795d33446d8fbed8b7373e868da0c9c0504ebe8109f770c94714897a 52.98MB / 52.98MB 0.6s
#4 sha256:6cb18017795d33446d8fbed8b7373e868da0c9c0504ebe8109f770c94714897a 52.98MB / 52.98MB 0.9s done
#4 extracting sha256:6cb18017795d33446d8fbed8b7373e868da0c9c0504ebe8109f770c94714897a
#4 extracting sha256:6cb18017795d33446d8fbed8b7373e868da0c9c0504ebe8109f770c94714897a 2.1s done
#4 DONE 3.0s

#5 [2/5] RUN apk add --no-cache bash gcc git
#5 0.083 fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
#5 2.659 fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
#5 2.797 (1/20) Installing ncurses-terminfo-base (6.2_p20200523-r0)
#5 2.808 (2/20) Installing ncurses-libs (6.2_p20200523-r0)
#5 2.821 (3/20) Installing readline (8.0.4-r0)
#5 2.834 (4/20) Installing bash (5.0.17-r0)
#5 2.855 Executing bash-5.0.17-r0.post-install
#5 2.858 (5/20) Installing libstdc++ (9.3.0-r2)
#5 2.882 (6/20) Installing binutils (2.34-r2)
#5 2.972 (7/20) Installing gmp (6.2.0-r0)
#5 2.985 (8/20) Installing isl (0.18-r0)
#5 3.011 (9/20) Installing libgomp (9.3.0-r2)
#5 3.020 (10/20) Installing libatomic (9.3.0-r2)
#5 3.028 (11/20) Installing libgphobos (9.3.0-r2)
#5 3.085 (12/20) Installing mpfr4 (4.0.2-r4)
#5 3.097 (13/20) Installing mpc1 (1.1.0-r1)
#5 3.106 (14/20) Installing gcc (9.3.0-r2)
#5 4.091 (15/20) Installing ca-certificates (20191127-r4)
#5 4.110 (16/20) Installing nghttp2-libs (1.41.0-r0)
#5 4.119 (17/20) Installing libcurl (7.78.0-r0)
#5 4.133 (18/20) Installing expat (2.2.9-r1)
#5 4.143 (19/20) Installing pcre2 (10.35-r0)
#5 4.158 (20/20) Installing git (2.26.3-r0)
#5 4.350 Executing busybox-1.31.1-r19.trigger
#5 4.358 Executing ca-certificates-20191127-r4.trigger
#5 4.394 Executing glibc-bin-2.32-r0.trigger
#5 4.398 /usr/glibc-compat/sbin/ldconfig: /usr/glibc-compat/lib/ld-linux-x86-64.so.2 is not a symbolic link
#5 4.398 
#5 4.409 OK: 159 MiB in 37 packages
#5 DONE 7.4s

#6 [3/5] COPY BALSAMIC/containers/varcall_py36/varcall_py36.yaml ./varcall_py36.yaml
#6 DONE 0.0s

#7 [4/5] COPY BALSAMIC/containers/varcall_py36/varcall_py36.sh ./varcall_py36.sh
#7 DONE 0.0s

#8 [5/5] RUN /bin/sh varcall_py36.sh varcall_py36 && conda clean --all --yes
#8 0.486 Collecting package metadata (repodata.json): ...working... done
#8 41.29 Solving environment: ...working... 
Error: The operation was canceled.

@khurrammaqbool khurrammaqbool linked a pull request Aug 16, 2021 that will close this issue
Fix/All containers´ performance improvements and fix for varcall_py36 container build #706
Merged
3 tasks
@khurrammaqbool
Copy link
Collaborator

Although the issue coincided with the time some pull requests were merged to develop branch, the problem resided with containers trying to resolve conflicts and in case it was not possible for conda to resolve conflicts in time, container build timed-out. This also caused other containers to take more time to build and push. These issues were fixed in #706 by adding channels and versions to each package for all conda containers. This significantly reduced container build time resulting in performance improvements and, in some cases, it may reduce container size. With these improvements, all packages, now, are required to have channel and it is highly recommended to add version information with resolved conflicts. Finally, this will also help make seamless package updates.

@hassanfa hassanfa mentioned this issue Aug 25, 2021
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@khurrammaqbool @ashwini06