-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[build] Check versions availability in rules/*.mk at init and use apt-get source instead of dget or wget in Makefiles #13381
Open
guillaumelambert
wants to merge
19
commits into
sonic-net:master
Choose a base branch
from
Orange-OpenSource:apt_source
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- create an a apt alternate configuration directory during 'make init' - use 'apt-get source' instead of dget in makefiles - adapt packages versions accordingly in rules/XX.mk files SONiC images compilation is currently designed to be performed with a non-root user under Ubuntu systems. As a result, there is no guarantee that debian repos are declared in /etc/apt/sources.list on the Ubuntu host system used. Although for this reason 'apt-get source' can not be directly used (out of the box) to that purpose, the build process recompiles packages for various GNU+Linux Debian distributions. And the packages sources used are retrieved from more or less official repositories since the current solution in SONiC is to use dget and URLs from various locations. As discussed in the following URL https://serverfault.com/questions/447457/use-apt-get-source-on-a-debian-repo-without-using-etc-apt-source-list this is the simplest solution but it has several drawbacks from a maintenance and security standpoint, for example : * maintaining manually these URLS since they can be removed by external sites owners sonic-net#13281 * potential loose authentication of packages sources sonic-net#13302 A more complex solution is prosposed (on the same URL) and adress these drawbacks by using an alternate apt configuration directory that is writable by non-root user. It allows here to: * identify packages sources with official Debian Keyring GPG signatures * use 'apt-get source <pkgsrc_name>' to retrieve packages without maintaining several specific URLs * in a next step use 'apt-cache showsrc <pkgsrc_name>' to retrieve a list of available versions to ckeck packages availability during the 'make init' phase Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
- add a script to check packages sources versions declared in rules/isc-dhcp.mk - call this script durint the 'make init' phase Files with the extension .mk in "rules/" folder are parsed several times during the 'make configure' phase. Thus, it is preferable to check and optionnaly set the package versions declared there before during the previous 'make init' phase. This will prevent from 'apt-get source' calls failures. Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
…ation - declare bash and debootstrap in automatic version detection script i.e. scripts/check_rules_mk_files_versions.sh - add a notice in script comments - adapt consequently corresponding rules/*.mk and src/*/Makefile files i.e rules/bash.mk rules/debootstrap.mk src/debootstrap/Makefile Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
…nfiguration - declare kdump-tools and lm-sensors in scripts/check_rules_mk_files_versions.sh - adapt consequently corresponding rules/*.mk files Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
…sions.sh Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
guillaumelambert
requested review from
qiluo-msft,
xumia and
lguohan
as code owners
January 16, 2023 19:43
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
…les_versions.sh Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
guillaumelambert
force-pushed
the
apt_source
branch
from
January 17, 2023 07:12
437a82e
to
a40317c
Compare
Builds themselves were OK. |
guillaumelambert
added a commit
to guillaumelambert/sonic-buildimage
that referenced
this pull request
Apr 12, 2023
- use dget instead of wget in some Makefiles - switch related HTTP URL to HTTPS and official debian mirror when possible TODO: migrate azure debian mirror to debian official ones (this might require upgrading some debian packages version in rules/*.mk) NB: A more complete solution with apt-get is proposed in sonic-net#13381 SONiC images compilation is currently designed to be performed with a non-root user under Ubuntu systems. And the build process recompiles packages for various GNU+Linux Debian distributions. As a result, there is no guarantee that debian repos are declared in /etc/apt/sources.list on the Ubuntu host system used. And 'apt-get source' can not be directly used out-of-the-box. Currently, the packages sources are retrieved from more or less official URLs and locations with dget and wget. As discussed in the following URL https://serverfault.com/questions/447457/use-apt-get-source-on-a-debian-repo-without-using-etc-apt-source-list dget is the simplest solution. wget has several drawbacks from a maintenance and security standpoint: - retrieving multiple URLs - loose authentication of packages sources especially when retrieved with HTTP instead of HTTPS This is potentially vulnerable to MITM and can induce malicious code injection. dget by default verifies package signature and can prevent most of such MITM attacks Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
8 tasks
guillaumelambert
added a commit
to Orange-OpenSource/sonic-buildimage
that referenced
this pull request
Apr 20, 2023
- use dget instead of wget in some Makefiles - switch related HTTP URL to HTTPS and official debian mirror when possible TODO: migrate azure debian mirror to debian official ones (this might require upgrading some debian packages version in rules/*.mk) NB: A more complete solution with apt-get is proposed in sonic-net#13381 SONiC images compilation is currently designed to be performed with a non-root user under Ubuntu systems. And the build process recompiles packages for various GNU+Linux Debian distributions. Though, there is no guarantee that debian repos are declared in /etc/apt/sources.list on the Ubuntu host system used. As a result, 'apt-get source' can not be directly used out-of-the-box. Currently, the packages sources are retrieved from more or less official URLs and locations with dget and wget. As discussed in the following URL https://serverfault.com/questions/447457/use-apt-get-source-on-a-debian-repo-without-using-etc-apt-source-list dget is the simplest solution. wget has several drawbacks from a maintenance and security standpoint: - retrieving multiple URLs - loose authentication of packages sources especially when retrieved with HTTP instead of HTTPS This is potentially vulnerable to MITM and can induce malicious code injection. dget by default verifies package signature and can prevent most of such MITM attacks Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Available versions can be retrieved with 'apt-cache showsrc'
If target version is no more available, there is no point to pursue the build.
Cf [build] Fix isc-dhcp full version in rules.mk #13288
SONiC images compilation is currently designed to be performed with a
non-root user under Ubuntu systems.
And the build process recompiles packages for various GNU+Linux Debian
distributions.
As a result, there is no guarantee that debian repos are declared in
/etc/apt/sources.list on the Ubuntu host system used.
And 'apt-get source' can not be directly used out-of-the-box.
Currently, the packages sources are retrieved from more or less official URLs
and locations with dget and wget.
As discussed in the following URL
https://serverfault.com/questions/447457/use-apt-get-source-on-a-debian-repo-without-using-etc-apt-source-list
dget is the simplest solution but this approach has several drawbacks from a
maintenance and security standpoint, for example :
sites owners
[Build] Failed to build isc dhcp #13281
A more complex solution is proposed (on the same URL at serverfault) and address these
drawbacks by using an alternate apt configuration directory that is writable
by non-root user. This PR leverages this solution and allows here to:
several specific URLs
list of available versions to ckeck packages availability during the
'make init' phase
If no candidate version is available, the script will exit with an error code and stops the whole compilation process.
This will prevent starting a long build process condemned to fail and thus spare precious time and computing resources.
Issue #13281
Which release branch to backport (provide reason below if selected)
potentially all
Description for the changelog
[build] Check versions availability in rules/*.mk at init and use apt-get source instead of dget or wget in Makefiles
A picture of a cute animal (not mandatory but encouraged)