06.Version-cache - Environment settings #13024

Kalimuthu-Velappan · 2022-12-12T08:27:33Z

ARCH flags are not passed correctly to version cache framework.
Added required environment variables to the cache framework.

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Kalimuthu-Velappan · 2022-12-20T02:41:31Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-12-20T02:41:41Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-12-20T02:41:56Z

Azure Pipelines successfully started running 1 pipeline(s).

- ARCH flags are not passed correctly to version cache framework. - Added required environment variables to the cache framework. - Fixed the uncleaned cache file copy code from version script due to merge.

xumia · 2022-12-21T12:36:15Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2022-12-21T12:36:27Z

Azure Pipelines successfully started running 1 pipeline(s).

xumia · 2022-12-21T12:36:29Z

Need to rerun it for armhf agent pool change.

Kalimuthu-Velappan · 2022-12-22T15:48:30Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-12-22T15:48:32Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-12-22T15:48:46Z

Azure Pipelines successfully started running 1 pipeline(s).

kv-y · 2023-01-23T10:19:29Z

@Kalimuthu-Velappan @xumia

In December new commits for vcache were merged and now we have 2 sonic-slave-images after build finished.

sonic-slave-bullseye              bc6ac16e263   38cab04607b5   47 seconds ago   6.03GB
tmp-sonic-slave-bullseye          bc6ac16e263   4bd80d159d7d   51 seconds ago   6.03GB

Some cleanup scripts can failed because they expect slave images have pattern sonic-slave-*:

sonic-buildimage/.azure-pipelines/template-clean-sonic-slave.yml

Line 7 in 260a2ec

images=$(docker images 'sonic-slave-*' -a -q)

Should we fix this?

Kalimuthu-Velappan · 2023-01-23T14:32:40Z

Thanks for flagging this. I will check and fix this.

…

On Mon, Jan 23, 2023 at 3:49 PM Konstantin Vasin ***@***.***> wrote: @Kalimuthu-Velappan <https://github.com/Kalimuthu-Velappan> @xumia <https://github.com/xumia> In December new commits for vcache was merged and now we have 2 sonic-slave-images after build finished. sonic-slave-bullseye bc6ac16e263 38cab04607b5 47 seconds ago 6.03GB tmp-sonic-slave-bullseye bc6ac16e263 4bd80d159d7d 51 seconds ago 6.03GB Some cleanup scripts can failed because they expect slave images have pattern sonic-slave-*: https://github.com/sonic-net/sonic-buildimage/blob/260a2ec3e7c671b716529b0faf9040fc924ecddc/.azure-pipelines/template-clean-sonic-slave.yml#L7 Should we fix this? — Reply to this email directly, view it on GitHub <#13024 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2UC2SYJMNHSV5KLIPU7ALWTZLLZANCNFSM6AAAAAAS3UH5LI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

kv-y · 2023-01-23T16:01:03Z

Also probably need to fix make clean.
My temporary solution from #12713 doesn't work anymore after #12005 was merged.

make clean
....
rm: cannot remove 'target/versions/dockers/docker-router-advertiser': Is a directory
make: *** [slave.mk:1512: target/docker-router-advertiser.gz-clean] Error 1
make[1]: *** [Makefile.work:567: clean] Error 2

k-v1 · 2023-03-17T11:38:57Z

rsync is installed now for every build to every docker image:

sonic-buildimage/src/sonic-build-hooks/scripts/pre_run_buildinfo

Line 44 in 96817c4

DISTRO=${DISTRO} apt-get update && apt-get install -y rsync

but we remove rsync from docker containers only if vcache is enabled:

sonic-buildimage/src/sonic-build-hooks/scripts/post_run_cleanup

Lines 10 to 14 in 96817c4

    
           if [ ! -z "$(get_version_cache_option)" ]; then 
        
           	#Delete the rsync package files 
        
           	if [[ ! ${IMAGENAME} =~ -slave- ]]; then 
        
           		/usr/bin/apt-get purge -y --auto-remove rsync

I think need to fix it and install rsync only if vcache is enabled.

Also auto-generated code is inserted in Dockerfiles above line:
COPY ["sources.list.{{ CONFIGURED_ARCH }}", "/etc/apt/sources.list"]
As a result we install rsync not from selected debian mirror but from default mirror (deb.debian.org) used in debian:bullseye docker image.

k-v1 · 2023-05-11T01:19:33Z

For make clean command issue I created this PR [Build] Fix make clean command #14406
To remove tmp-sonic-slave docker image we can -f flag here:

sonic-buildimage/scripts/collect_docker_version_files.sh

Line 48 in 73914e7

docker rmi tmp-${DOCKER_IMAGE_TAG}
For rsync issue this PR was merged: [Build] don't install rsync if vcache is disabled #14407
But still need to test how we install rsync when vcache is enabled.
For some Dockerfiles command apt-get update && apt-get install -y rsync is inserted above this step:

sonic-buildimage/dockers/docker-base-bullseye/Dockerfile.j2

Line 30 in 73914e7

COPY ["sources.list.{{ CONFIGURED_ARCH }}", "/etc/apt/sources.list"]

As a result we do apt-get update and install rsync from default debian mirror like deb.debian.org.
After that we change debian mirror to correct and install other packages from correct mirror.
This bug probably should also be fixed.

k-v1 · 2023-05-22T20:26:21Z

scripts/collect_docker_version_files.sh


 	# Select version files for SHA calculation
-	VERSION_FILES="${SRC_VERSION_PATH}/dockers/${DOCKER_IMAGE_NAME}/versions-*-${DISTRO}-${ARCH} ${SRC_VERSION_PATH}/default/versions-*"
+	VERSION_FILES="${SRC_VERSION_PATH}/dockers/${DOCKER_IMAGE_NAME/-dbg/}/versions-*-${DISTRO}-${ARCH} ${SRC_VERSION_PATH}/default/versions-*"


If I try to build broadcom I get cat: 'files/build/versions/dockers/sonic-slave-bullseye/versions-*-bullseye-amd64': No such file or directory when calculate sha1sum hash below.
It's because version file for amd64 doesn't have arch suffix:
versions-deb-bullseye versions-deb-bullseye-arm64 versions-deb-bullseye-armhf versions-py3
Should we fix this?

Kalimuthu-Velappan requested review from qiluo-msft, xumia and lguohan as code owners December 12, 2022 08:27

xumia previously approved these changes Dec 12, 2022

View reviewed changes

Kalimuthu-Velappan force-pushed the ARCH_SETTINGS branch from 90a8b6c to a15eeca Compare December 13, 2022 10:16

06.Version-cache - Environment settings

14f23fa

- ARCH flags are not passed correctly to version cache framework. - Added required environment variables to the cache framework. - Fixed the uncleaned cache file copy code from version script due to merge.

Kalimuthu-Velappan force-pushed the ARCH_SETTINGS branch from a15eeca to 14f23fa Compare December 21, 2022 04:48

liat-grozovik dismissed xumia’s stale review via 14f23fa April 13, 2023 11:18

k-v1 mentioned this pull request May 11, 2023

06.Version Cache - Support for wget #14612

Merged

8 tasks

k-v1 reviewed May 22, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

06.Version-cache - Environment settings #13024

06.Version-cache - Environment settings #13024

Kalimuthu-Velappan commented Dec 12, 2022

Kalimuthu-Velappan commented Dec 20, 2022

mssonicbld commented Dec 20, 2022

azure-pipelines bot commented Dec 20, 2022

xumia commented Dec 21, 2022

azure-pipelines bot commented Dec 21, 2022

xumia commented Dec 21, 2022

Kalimuthu-Velappan commented Dec 22, 2022

mssonicbld commented Dec 22, 2022

azure-pipelines bot commented Dec 22, 2022

kv-y commented Jan 23, 2023 •

edited

Loading

Kalimuthu-Velappan commented Jan 23, 2023 via email

kv-y commented Jan 23, 2023

k-v1 commented Mar 17, 2023

k-v1 commented May 11, 2023

k-v1 May 22, 2023

06.Version-cache - Environment settings #13024

Are you sure you want to change the base?

06.Version-cache - Environment settings #13024

Conversation

Kalimuthu-Velappan commented Dec 12, 2022

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Kalimuthu-Velappan commented Dec 20, 2022

mssonicbld commented Dec 20, 2022

azure-pipelines bot commented Dec 20, 2022

xumia commented Dec 21, 2022

azure-pipelines bot commented Dec 21, 2022

xumia commented Dec 21, 2022

Kalimuthu-Velappan commented Dec 22, 2022

mssonicbld commented Dec 22, 2022

azure-pipelines bot commented Dec 22, 2022

kv-y commented Jan 23, 2023 • edited Loading

Kalimuthu-Velappan commented Jan 23, 2023 via email

kv-y commented Jan 23, 2023

k-v1 commented Mar 17, 2023

k-v1 commented May 11, 2023

k-v1 May 22, 2023

Choose a reason for hiding this comment

kv-y commented Jan 23, 2023 •

edited

Loading