05.Version cache - docker dpkg caching support #12005

Kalimuthu-Velappan · 2022-09-08T01:33:59Z

This feature caches all the deb files during docker build and stores them
into version cache.

It loads the cache file if already exists in the version cache and copies the extracted
deb file from cache file into Debian cache path( /var/cache/apt/archives).

The apt-install always installs the deb file from the cache if exists, this
avoid unnecessary package download from the repo and speeds up the overall build.

The cache file is selected based on the SHA value of version dependency
files.

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

lgtm-com · 2022-09-08T02:17:09Z

This pull request introduces 1 alert when merging a7949e321850d73be472e832a750aeb9d102cf16 into 38cc35f - view on LGTM.com

new alerts:

1 for Modification of parameter with default

Kalimuthu-Velappan · 2022-09-11T17:05:00Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-09-11T17:05:01Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-09-11T17:05:13Z

Azure Pipelines successfully started running 1 pipeline(s).

adyeung · 2022-09-14T23:15:11Z

@xumia @liushilongbuaa submitter has taken the time to split the original code PR to smaller submissions for review, pls help take a look

zhangyanzhao · 2022-11-11T22:53:05Z

@xumia @liushilongbuaa can you please review the PR? Thanks.

xumia · 2022-11-17T04:42:12Z

Makefile.work

@@ -217,15 +230,21 @@ SLAVE_TAG = $(shell \
 	(cat $(SLAVE_DIR)/Dockerfile.user \
 		 $(SLAVE_DIR)/Dockerfile \
 		 $(SLAVE_DIR)/buildinfo/versions/versions-* \
-		 .git/HEAD \
+		.git/HEAD 2>/dev/null \


.git/HEAD

PR build will not have chance to use cache, can we remove it?

/_work/27/s$ cat .git/HEAD
ref: refs/heads/master-12001

In current PR build, it will use the - as the name.

xumia · 2022-11-17T05:04:17Z

src/sonic-build-hooks/scripts/utils.sh

+    #echo ${!lock_fd}
+    if ! flock -x -w ${timeout} "${!lock_fd}" ; then
+        echo "ERROR: Lock timeout trying to access ${filename}.flock";
+        exit 1;


How about printing to stderr

echo "ERROR: Lock timeout trying to access ${filename}.flock" 1>&2;

xumia · 2022-11-17T05:20:36Z

src/sonic-build-hooks/scripts/pre_run_buildinfo

+	APT::Keep-Downloaded-Packages "true";
+	EOF
+
+fi


Please remove the empty line

xumia · 2022-11-17T05:22:07Z

src/sonic-build-hooks/scripts/pre_run_buildinfo

+# Extract the cache inside the docker build.
+if [ -f ${PKG_CACHE_FILE_NAME} ]; then
+	tar -C ${PKG_CACHE_PATH} -xvf  ${PKG_CACHE_FILE_NAME}
+	test -e ${PKG_CACHE_PATH}/deb && cp ${PKG_CACHE_PATH}/deb/* /var/cache/apt/archives/ 


Please remove the additional space character at the end of line 40, thanks.

src/sonic-build-hooks/scripts/post_run_buildinfo

It defines and passes the necessary version cache environment variables to the caching framework. It adds the utils script for shared cache file access. It also adds the post-cleanup logic for cleaning the unwanted files from the docker/image after the version cache creation.

Added DBGOPT Make variable to enable the cache framework scripts in trace mode. This option takes the part name of the script to enable the particular shell script in trace mode. Multiple shell script names can also be given. Eg: make DBGOPT="image|docker" Added verbose mode to dump the version merge details during build/dry-run mode. Eg: scripts/versions_manager.py freeze -v \ 'dryrun|cmod=docker-swss|cfile=versions-deb|cname=all|stage=sub|stage=add'

This feature caches all the deb files during docker build and stores them into version cache. It loads the cache file if already exists in the version cache and copies the extracted deb file from cache file into Debian cache path( /var/cache/apt/archives). The apt-install always installs the deb file from the cache if exists, this avoid unnecessary package download from the repo and speeds up the overall build. The cache file is selected based on the SHA value of version dependency files.

mssonicbld · 2023-02-10T13:24:20Z

@Kalimuthu-Velappan PR conflicts with 202211 branch

saiarcot895 · 2023-03-13T19:37:54Z

@Kalimuthu-Velappan can this be cherry-picked to the 202211 and 202205 branches? This will save about 100MB of disk space, which will be immensely helpful on low-storage platforms.

yxieca · 2023-03-13T20:26:08Z

@Kalimuthu-Velappan this change is desired for 202205 branch to help reduce image size and installation footprint. Please help create PR for 202205 branch.

mssonicbld · 2023-03-15T16:17:43Z

@Kalimuthu-Velappan PR conflicts with 202205 branch

…)" This reverts commit 0dc22bd.

Revert "05.Version cache - docker dpkg caching support (sonic-net#12005)" This reverts commit 0dc22bd.

…-net#12005)"" This reverts commit 4be0805.

This feature caches all the deb files during docker build and stores them into version cache. It loads the cache file if already exists in the version cache and copies the extracted deb file from cache file into Debian cache path( /var/cache/apt/archives). The apt-install always installs the deb file from the cache if exists, this avoid unnecessary package download from the repo and speeds up the overall build. The cache file is selected based on the SHA value of version dependency files. Why I did it How I did it How to verify it * 03.Version-cache - framework environment settings It defines and passes the necessary version cache environment variables to the caching framework. It adds the utils script for shared cache file access. It also adds the post-cleanup logic for cleaning the unwanted files from the docker/image after the version cache creation. * 04.Version cache - debug framework Added DBGOPT Make variable to enable the cache framework scripts in trace mode. This option takes the part name of the script to enable the particular shell script in trace mode. Multiple shell script names can also be given. Eg: make DBGOPT="image|docker" Added verbose mode to dump the version merge details during build/dry-run mode. Eg: scripts/versions_manager.py freeze -v \ 'dryrun|cmod=docker-swss|cfile=versions-deb|cname=all|stage=sub|stage=add' * 05.Version cache - docker dpkg caching support This feature caches all the deb files during docker build and stores them into version cache. It loads the cache file if already exists in the version cache and copies the extracted deb file from cache file into Debian cache path( /var/cache/apt/archives). The apt-install always installs the deb file from the cache if exists, this avoid unnecessary package download from the repo and speeds up the overall build. The cache file is selected based on the SHA value of version dependency files.

Kalimuthu-Velappan requested review from qiluo-msft, xumia and lguohan as code owners September 8, 2022 01:33

zhangyanzhao requested a review from liushilongbuaa September 14, 2022 06:18

Kalimuthu-Velappan force-pushed the DOCKER_DPKG_CACHE_FRAMEWORK branch from a7949e3 to 1c3ee38 Compare October 12, 2022 13:15

Kalimuthu-Velappan mentioned this pull request Oct 19, 2022

02.Version cache - docker cache build framework #12001

Merged

7 tasks

zhangyanzhao mentioned this pull request Nov 11, 2022

SONiC Build Improvements sonic-net/SONiC#942

Merged

xumia reviewed Nov 17, 2022

View reviewed changes

src/sonic-build-hooks/scripts/post_run_buildinfo Outdated Show resolved Hide resolved

Kalimuthu-Velappan force-pushed the DOCKER_DPKG_CACHE_FRAMEWORK branch from 715bbc5 to 68ed10e Compare November 23, 2022 13:43

Kalimuthu-Velappan force-pushed the DOCKER_DPKG_CACHE_FRAMEWORK branch from 68ed10e to b041acb Compare December 1, 2022 04:16

Kalimuthu-Velappan added 3 commits December 5, 2022 07:50

Kalimuthu-Velappan force-pushed the DOCKER_DPKG_CACHE_FRAMEWORK branch from b041acb to 6660b29 Compare December 5, 2022 15:52

Kalimuthu-Velappan mentioned this pull request Dec 8, 2022

Build Enhancements sonic-net/SONiC#941

Closed

xumia approved these changes Dec 12, 2022

View reviewed changes

xumia merged commit 0dc22bd into sonic-net:master Dec 12, 2022

This was referenced Dec 12, 2022

03.Version-cache - framework environment settings #12003

Closed

04.Version cache - debug framework #12004

Closed

kv-y mentioned this pull request Dec 14, 2022

[Build] fix make clean #12713

Merged

kv-y mentioned this pull request Jan 2, 2023

[Build] remove duplicate code from dockers cleanup code #13224

Closed

kv-y mentioned this pull request Jan 23, 2023

06.Version-cache - Environment settings #13024

Open

8 tasks

StormLiangMS added the Approved for 202211 Branch label Feb 10, 2023

mssonicbld added the Cherry Pick Conflict_202211 label Feb 10, 2023

yxieca added the Request for 202205 Branch label Mar 13, 2023

yxieca added the Approved for 202205 Branch label Mar 15, 2023

mssonicbld added the Cherry Pick Conflict_202205 label Mar 15, 2023

yxieca removed Cherry Pick Conflict_202205 Request for 202205 Branch Approved for 202205 Branch labels Mar 24, 2023

oleksandrivantsiv added a commit to oleksandrivantsiv/sonic-buildimage that referenced this pull request Apr 11, 2023

Revert "05.Version cache - docker dpkg caching support (sonic-net#12005…

4be0805

…)" This reverts commit 0dc22bd.

oleksandrivantsiv added a commit to oleksandrivantsiv/sonic-buildimage that referenced this pull request Apr 11, 2023

[build] fix compilation issue after revert.

a8e9478

Revert "05.Version cache - docker dpkg caching support (sonic-net#12005)" This reverts commit 0dc22bd.

oleksandrivantsiv added a commit to oleksandrivantsiv/sonic-buildimage that referenced this pull request Apr 11, 2023

Revert "Revert "05.Version cache - docker dpkg caching support (sonic…

8427897

…-net#12005)"" This reverts commit 4be0805.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

05.Version cache - docker dpkg caching support #12005

05.Version cache - docker dpkg caching support #12005

Kalimuthu-Velappan commented Sep 8, 2022

lgtm-com bot commented Sep 8, 2022

Kalimuthu-Velappan commented Sep 11, 2022

mssonicbld commented Sep 11, 2022

azure-pipelines bot commented Sep 11, 2022

adyeung commented Sep 14, 2022

zhangyanzhao commented Nov 11, 2022

xumia Nov 17, 2022

Kalimuthu-Velappan Nov 23, 2022

xumia Nov 17, 2022

Kalimuthu-Velappan Nov 23, 2022

xumia Nov 17, 2022

Kalimuthu-Velappan Nov 23, 2022

xumia Nov 17, 2022

Kalimuthu-Velappan Nov 23, 2022

mssonicbld commented Feb 10, 2023

saiarcot895 commented Mar 13, 2023

yxieca commented Mar 13, 2023

mssonicbld commented Mar 15, 2023

05.Version cache - docker dpkg caching support #12005

05.Version cache - docker dpkg caching support #12005

Conversation

Kalimuthu-Velappan commented Sep 8, 2022

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

lgtm-com bot commented Sep 8, 2022

Kalimuthu-Velappan commented Sep 11, 2022

mssonicbld commented Sep 11, 2022

azure-pipelines bot commented Sep 11, 2022

adyeung commented Sep 14, 2022

zhangyanzhao commented Nov 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mssonicbld commented Feb 10, 2023

saiarcot895 commented Mar 13, 2023

yxieca commented Mar 13, 2023

mssonicbld commented Mar 15, 2023