[build] add support for 2 stage rootfs build #15924

Yakiv-Huryk · 2023-07-20T14:19:31Z

This adds optimization for the SONiC image build by splitting the final build step into two stages. It allows running the first stage in parallel, improving build time.

The optimization is enabled via new rules/config flag ENABLE_RFS_SPLIT_BUILD (disabled by default)

Why I did it

To improve a build time.

Work item tracking

Microsoft ADO (number only):

How I did it

Added a logic to run build_debian.sh in two stages, transferring the progress via a new build artifact.

How to verify it

make ENABLE_RFS_SPLIT_BUILD=y SONIC_BUILD_JOBS=32 target/<IMAGE_NAME>.bin

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Yakiv-Huryk · 2023-07-20T14:20:10Z

HLD:
sonic-net/SONiC#1430

Yakiv-Huryk · 2023-08-11T14:29:40Z

/azpw run

mssonicbld · 2023-08-11T14:29:42Z

/AzurePipelines run

azure-pipelines · 2023-08-11T14:29:47Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

Yakiv-Huryk · 2023-08-11T15:19:10Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2023-08-11T15:19:12Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2023-08-11T15:19:22Z

Azure Pipelines successfully started running 1 pipeline(s).

build_debian.sh

k-v1 · 2023-08-17T13:39:36Z

make clean and make distclean commands don't remove this squashfs file from target directory.
Is it expected behaviour? Maybe we need to add clean rule for this target too.
But unlike other targets owner of this file is root.

slave.mk

k-v1 · 2023-08-18T22:07:04Z

Please build with and without this feature and compare file /etc/sonic/sonic_version.yml
You probably get different files.

sonic-buildimage/slave.mk

Lines 1341 to 1343 in 0bd8c3b

    
           export components="$(foreach component,$(notdir $^),\ 
        
           	$(shell [[ ! -z '$($(component)_VERSION)' && ! -z '$($(component)_NAME)' ]] && \ 
        
           		echo $($(component)_NAME)==$($(component)_VERSION)))"

sonic-buildimage/build_debian.sh

Line 591 in 0bd8c3b

    
           j2 files/build_templates/sonic_version.yml.j2 | sudo tee $FILESYSTEM_ROOT/etc/sonic/sonic_version.yml

sonic-buildimage/files/build_templates/sonic_version.yml.j2

Lines 23 to 28 in 0bd8c3b

    
           {% if components is defined -%} 
        
           {% for component in components.split() | unique -%} 
        
           {% set name, version = component.split('==') -%} 
        
           {{ name }}: {{ version }} 
        
           {% endfor -%} 
        
           {% endif -%}

Possible solution is to generate file sonic_version.yml later (after 1st stage)

qiluo-msft · 2023-08-20T00:30:51Z

slave.mk

+SONIC_TARGET_LIST += $(addprefix $(TARGET_PATH)/, $(SONIC_RFS_TARGETS))
+endif
+
+$(addprefix $(TARGET_PATH)/, $(SONIC_RFS_TARGETS)) : $(TARGET_PATH)/% : \


SONIC_RFS_TARGETS

Could you make sure the new target SONIC_RFS_TARGETS is reused if we build multiple vendors image in series?

We also have reproducible mechanism in rules/*.dep files. Is it possible to define this target's dep file, so the target is reused as cached for other later build jobs?

@xumia to review. #Closed

The target is reused. I've also added a cache: a880d53
Since the targets are defined dynamically, I could not add the '.dep' file. Instead, the relevant variables are also set dynamically.

liat-grozovik · 2023-08-27T06:34:35Z

who ever is reviewing this PR, we have a target to finish this review and enjoy the approved time of build in the next week.
who ever is tagged and wish to provide feedback please do it asap.
@Yakiv-Huryk @xumia @Kalimuthu-Velappan

k-v1 · 2023-08-27T10:15:12Z

who ever is reviewing this PR, we have a target to finish this review and enjoy the approved time of build in the next week.

Still need to fix one issue I mentioned before.
#15924 (comment)

Also @qiluo-msft suggested to use cache for 1st-stage squashfs file. Need feedback from @Yakiv-Huryk is it possible or not.

Kalimuthu-Velappan · 2023-08-29T08:34:11Z

slave.mk

 SONIC_RFS_TARGETS= $(foreach installer, $(SONIC_INSTALLERS), $(call rfs_get_installer_dependencies,$(installer)))
+
+ifeq ($(ENABLE_RFS_SPLIT_BUILD),y)


Try to avoid the if..else statement inside the build_debian.sh file.
Can you split the build_debian.sh into two separate files?

build_debian_rootfs.sh => which contains all the standard squashfs packages

build_debian_sonic.sh => Includes all the sonic packages into existing squashfs and creates the dockerfs.

I'm not sure I understand why splitting the build_debian.sh is better. Also, with the current design, the feature is optional and configurable (via ENABLE_RFS_SPLIT_BUILD). If we split the build_debian.sh, the current way to build will no longer be available.

Can you please help to understand why the if..else is better avoided?

If we split the build_debian.sh, the current way to build will no longer be available.

I think this feature (RFS_SPLIT_BUILD) should be enabled by default in the future.
Two versions are hard to maintain.
In this case, splitting into two separate files is a good solution.

But probably we can merge this PR as a temporary solution and test builds with enabled option for a while.

Kalimuthu-Velappan · 2023-08-29T08:37:39Z

slave.mk

@@ -1573,7 +1575,9 @@ SONIC_CLEAN_TARGETS += $(addsuffix -clean,$(addprefix $(TARGET_PATH)/, \
 		       $(SONIC_DOCKER_IMAGES) \
 		       $(SONIC_DOCKER_DBG_IMAGES) \
 		       $(SONIC_SIMPLE_DOCKER_IMAGES) \
-		       $(SONIC_INSTALLERS)))
+		       $(SONIC_INSTALLERS) \
+		       $(SONIC_RFS_TARGETS)))


Can you add caching support for rootfs preparation in squashfs file system?
You can refer DPKG caching framework for deb packages.

Added in a880d53

slave.mk

k-v1 · 2023-08-31T17:00:15Z

@Yakiv-Huryk
1.
I'm trying to build make target/sonic-broadcom.bin__broadcom__rfs.squashfs with enabled DPKG cache, but file sonic-broadcom.bin__broadcom__rfs.squashfs.flags is empty.
If you open these files for other packages you can find they contain something like broadcom amd64 bullseye bullseye.

So need to check these lines in rules/functions:

$(1)_DEP_FLAGS = $(SONIC_COMMON_FLAGS_LIST)
$(1)_DEP_FILES = $(SONIC_COMMON_BASE_FILES_LIST) build_debian.sh onie-image.conf

Maybe $(SONIC_COMMON_FLAGS_LIST) and $(SONIC_COMMON_BASE_FILES_LIST) are not defined here.

Many files from /files directory are copied into the rootfs at 1st stage.
But if we change these files then we still use old version of sonic-broadcom.bin__broadcom__rfs.squashfs from DPKG cache directory.

Need to fix these issues or merge this PR without DPKG cache. We can add DPKG cache support for rootfs later after additional testing. But anyway for caching need review from @xumia or @Kalimuthu-Velappan. I'm not very familiar with these feature.

All other issues I mentioned earlier are fixed.

Yakiv-Huryk · 2023-09-01T16:41:38Z

@k-v1
Fixed the # 1.

Regarding the first stage dependencies (including many files from /files):
I've collected a list and added it to the _DEP_FILES.
However, it's not a robust solution since there is no way to track what is used in the build_debian.sh, and the list should be manually maintained which is not optimal.
@xumia @Kalimuthu-Velappan do you have any ideas/suggestions on how to handle this?

rules/config

Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

* add define_rfs_target function to define cache-related variables * add SONIC_RFS_TARGETS handling to Makefile.cache Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

…uild Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

* cache-related variables are now defined in Makefile.cache * removed define_rfs_target * add a list of files that are used in the first stage of RFS build Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

* removed ENABLE_RFS_SPLIT_BUILD variable Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

This is to address the fact that the build_debian_base_system.sh uses $TARGET/baseimage, so running two build_debian_base_system.sh in parallel is a race. * add DEPENDENT_RFS (analogous to DEPENDENT_MACHINE) * rearrange the build so that the installer's MACHINE RFS and DEPENDENT_MACHINE RFS are not built in parallel Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

* add all the files from files/sshd,dhcp,image_config Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

* add more files to RFS_DEB_FILES * improved umount in build_debian.sh (taken from sonic-net#16672) * explicit RFS_SPLIT_LAST_STAGE=n when building the first stage Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

saiarcot895 · 2023-10-02T20:54:04Z

Makefile.cache

+	$(shell git ls-files files/apparmor) \
+	$(shell git ls-files files/apt) \
+	$(wildcard files/sshd/*) \
+	$(wildcard files/dhcp/*) \


Can these be changed to git ls-files as well?

Sure, done.

Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

Yakiv-Huryk · 2023-10-04T13:47:09Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2023-10-04T13:47:11Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2023-10-04T13:47:25Z

Azure Pipelines successfully started running 1 pipeline(s).

Yakiv-Huryk · 2023-10-05T14:06:42Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2023-10-05T14:06:45Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2023-10-05T14:07:01Z

Azure Pipelines successfully started running 1 pipeline(s).

liat-grozovik · 2023-10-10T16:28:46Z

@saiarcot895 could you please take further look now that the comments were addressed?

liat-grozovik · 2023-10-11T06:34:25Z

@Yakiv-Huryk many thanks for this great achievements.
Thanks for all the rest who provided feedback and made it even better

skg-net · 2024-02-06T00:29:38Z

@Yakiv-Huryk Can you please update the Quality Metric (Alpha/Beta/GA) for the feature either in this PR comments or in HLD itself based on https://github.com/sonic-net/SONiC/blob/master/doc/SONiC%20feature%20quality%20definition.md
Thanks

k-v1 · 2024-02-08T16:11:42Z

@skg-net

Original feature was implemented and merged to master.
But @qiluo-msft suggested to use DPKG cache for 1st stage rootfs file.
These improvements for DPKG cache were also implemented and merged in this PR but disabled because they contain some bugs (#16944).

I suggested additional fixes and improvements (#17100).
But my PR is for Debian bullseye. Now SONiC is based on Debian bookworm and need to update my PR.
I can try to update my PR but I'm not sure should I waste my time on this because my other PRs for build time improvements and DPKG cache have not been merged (e.g. #16959, #15735, #17644).

Other possible solution if @Yakiv-Huryk provides fixes by himself (e.g. partially based on #17100).
I can review his PR in this case.

Yakiv-Huryk · 2024-02-15T12:29:43Z

@Yakiv-Huryk Can you please update the Quality Metric (Alpha/Beta/GA) for the feature either in this PR comments or in HLD itself based on https://github.com/sonic-net/SONiC/blob/master/doc/SONiC%20feature%20quality%20definition.md Thanks

It’s GA. Since it’s a build improvement, it doesn't really fit the Feature Quality Definition. There is no flag to enable/disable it and no sonic-mgmt tests.

Yakiv-Huryk requested review from qiluo-msft, xumia and lguohan as code owners July 20, 2023 14:19

Yakiv-Huryk mentioned this pull request Jul 20, 2023

Add HLD for RFS split build optimization sonic-net/SONiC#1430

Merged

vivekrnv approved these changes Jul 21, 2023

View reviewed changes

k-v1 reviewed Aug 15, 2023

View reviewed changes

build_debian.sh Outdated Show resolved Hide resolved

k-v1 reviewed Aug 17, 2023

View reviewed changes

build_debian.sh Outdated Show resolved Hide resolved

k-v1 reviewed Aug 17, 2023

View reviewed changes

slave.mk Outdated Show resolved Hide resolved

qiluo-msft reviewed Aug 20, 2023

View reviewed changes

Kalimuthu-Velappan reviewed Aug 29, 2023

View reviewed changes

slave.mk Outdated Show resolved Hide resolved

Yakiv-Huryk force-pushed the rfs-split branch from 8e1e685 to 4f6bb10 Compare August 30, 2023 19:21

qiluo-msft reviewed Sep 4, 2023

View reviewed changes

rules/config Outdated Show resolved Hide resolved

Yakiv-Huryk force-pushed the rfs-split branch from dd87b35 to b12c093 Compare September 8, 2023 16:29

qiluo-msft previously approved these changes Sep 10, 2023

View reviewed changes

Yakiv-Huryk dismissed qiluo-msft’s stale review via f27faea September 21, 2023 16:05

Yakiv-Huryk added 8 commits October 2, 2023 09:44

[build] add debootstrap dependency to RFS, update the "clean" rule

9b0ceef

Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

[build] add cache support for RFS

cab2fb6

* add define_rfs_target function to define cache-related variables * add SONIC_RFS_TARGETS handling to Makefile.cache Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

[build] move sonic_version.yml generation to the second part of RFS b…

dd4b8d1

…uild Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

[build] fix cache support for RFS

b69a7c0

* cache-related variables are now defined in Makefile.cache * removed define_rfs_target * add a list of files that are used in the first stage of RFS build Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

[build] enable RFS split build by default

e30b32f

* removed ENABLE_RFS_SPLIT_BUILD variable Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

[build] update RFS cache deb files (RFS_DEP_FILES)

122cd6b

* add all the files from files/sshd,dhcp,image_config Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

[build] RFS split improvements

b6f9006

* add more files to RFS_DEB_FILES * improved umount in build_debian.sh (taken from sonic-net#16672) * explicit RFS_SPLIT_LAST_STAGE=n when building the first stage Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

Yakiv-Huryk force-pushed the rfs-split branch from 2d638f0 to b6f9006 Compare October 2, 2023 13:44

dgsudharsan requested review from saiarcot895 and k-v1 October 2, 2023 17:00

k-v1 approved these changes Oct 2, 2023

View reviewed changes

saiarcot895 reviewed Oct 2, 2023

View reviewed changes

[build] use git ls-files instead of wildcard for RFS cache

ca16676

Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>

dgsudharsan requested a review from saiarcot895 October 5, 2023 15:05

saiarcot895 approved these changes Oct 10, 2023

View reviewed changes

liat-grozovik approved these changes Oct 11, 2023

View reviewed changes

liat-grozovik merged commit 6cb8893 into sonic-net:master Oct 11, 2023

Staphylo mentioned this pull request Oct 19, 2023

Add build option to reduce final image size #16729

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[build] add support for 2 stage rootfs build #15924

[build] add support for 2 stage rootfs build #15924

Yakiv-Huryk commented Jul 20, 2023

Yakiv-Huryk commented Jul 20, 2023

Yakiv-Huryk commented Aug 11, 2023

mssonicbld commented Aug 11, 2023

azure-pipelines bot commented Aug 11, 2023

Yakiv-Huryk commented Aug 11, 2023

mssonicbld commented Aug 11, 2023

azure-pipelines bot commented Aug 11, 2023

k-v1 commented Aug 17, 2023

k-v1 commented Aug 18, 2023 •

edited

Loading

qiluo-msft Aug 20, 2023 •

edited

Loading

Yakiv-Huryk Aug 30, 2023

liat-grozovik commented Aug 27, 2023

k-v1 commented Aug 27, 2023

Kalimuthu-Velappan Aug 29, 2023 •

edited

Loading

Yakiv-Huryk Aug 30, 2023

k-v1 Aug 31, 2023

Kalimuthu-Velappan Aug 29, 2023

Yakiv-Huryk Aug 30, 2023

k-v1 commented Aug 31, 2023

Yakiv-Huryk commented Sep 1, 2023

saiarcot895 Oct 2, 2023

Yakiv-Huryk Oct 3, 2023

Yakiv-Huryk commented Oct 4, 2023

mssonicbld commented Oct 4, 2023

azure-pipelines bot commented Oct 4, 2023

Yakiv-Huryk commented Oct 5, 2023

mssonicbld commented Oct 5, 2023

azure-pipelines bot commented Oct 5, 2023

liat-grozovik commented Oct 10, 2023

liat-grozovik commented Oct 11, 2023

skg-net commented Feb 6, 2024

k-v1 commented Feb 8, 2024

Yakiv-Huryk commented Feb 15, 2024

		SONIC_RFS_TARGETS= $(foreach installer, $(SONIC_INSTALLERS), $(call rfs_get_installer_dependencies,$(installer)))

		ifeq ($(ENABLE_RFS_SPLIT_BUILD),y)

[build] add support for 2 stage rootfs build #15924

[build] add support for 2 stage rootfs build #15924

Conversation

Yakiv-Huryk commented Jul 20, 2023

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Yakiv-Huryk commented Jul 20, 2023

Yakiv-Huryk commented Aug 11, 2023

mssonicbld commented Aug 11, 2023

azure-pipelines bot commented Aug 11, 2023

Yakiv-Huryk commented Aug 11, 2023

mssonicbld commented Aug 11, 2023

azure-pipelines bot commented Aug 11, 2023

k-v1 commented Aug 17, 2023

k-v1 commented Aug 18, 2023 • edited Loading

qiluo-msft Aug 20, 2023 • edited Loading

Choose a reason for hiding this comment

Yakiv-Huryk Aug 30, 2023

Choose a reason for hiding this comment

liat-grozovik commented Aug 27, 2023

k-v1 commented Aug 27, 2023

Kalimuthu-Velappan Aug 29, 2023 • edited Loading

Choose a reason for hiding this comment

Yakiv-Huryk Aug 30, 2023

Choose a reason for hiding this comment

k-v1 Aug 31, 2023

Choose a reason for hiding this comment

Kalimuthu-Velappan Aug 29, 2023

Choose a reason for hiding this comment

Yakiv-Huryk Aug 30, 2023

Choose a reason for hiding this comment

k-v1 commented Aug 31, 2023

Yakiv-Huryk commented Sep 1, 2023

saiarcot895 Oct 2, 2023

Choose a reason for hiding this comment

Yakiv-Huryk Oct 3, 2023

Choose a reason for hiding this comment

Yakiv-Huryk commented Oct 4, 2023

mssonicbld commented Oct 4, 2023

azure-pipelines bot commented Oct 4, 2023

Yakiv-Huryk commented Oct 5, 2023

mssonicbld commented Oct 5, 2023

azure-pipelines bot commented Oct 5, 2023

liat-grozovik commented Oct 10, 2023

liat-grozovik commented Oct 11, 2023

skg-net commented Feb 6, 2024

k-v1 commented Feb 8, 2024

Yakiv-Huryk commented Feb 15, 2024

k-v1 commented Aug 18, 2023 •

edited

Loading

qiluo-msft Aug 20, 2023 •

edited

Loading

Kalimuthu-Velappan Aug 29, 2023 •

edited

Loading