Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[build] Check versions availability in rules/*.mk at init and use apt-get source instead of dget or wget in Makefiles #13381

Open
wants to merge 19 commits into
base: master
Choose a base branch
from

Conversation

guillaumelambert
Copy link
Contributor

@guillaumelambert guillaumelambert commented Jan 16, 2023

  • create an a apt alternate configuration directory during 'make init'
  • use 'apt-get source' instead of dget and wget in makefiles
  • adapt packages versions accordingly in rules/XX.mk files
  • perform versions check at the make init stage to avoid time and resources waste
    Available versions can be retrieved with 'apt-cache showsrc'
    If target version is no more available, there is no point to pursue the build.
    Cf [build] Fix isc-dhcp full version in rules.mk #13288

SONiC images compilation is currently designed to be performed with a
non-root user under Ubuntu systems.
And the build process recompiles packages for various GNU+Linux Debian
distributions.
As a result, there is no guarantee that debian repos are declared in
/etc/apt/sources.list on the Ubuntu host system used.
And 'apt-get source' can not be directly used out-of-the-box.
Currently, the packages sources are retrieved from more or less official URLs
and locations with dget and wget.
As discussed in the following URL
https://serverfault.com/questions/447457/use-apt-get-source-on-a-debian-repo-without-using-etc-apt-source-list
dget is the simplest solution but this approach has several drawbacks from a
maintenance and security standpoint, for example :

  • maintaining manually these URLS since they can be removed by external
    sites owners
    [Build] Failed to build isc dhcp #13281
  • loose authentication of packages sources retrieved with HTTP instead of HTTPS potentially vulnerable to MITM at least for wget

A more complex solution is proposed (on the same URL at serverfault) and address these
drawbacks by using an alternate apt configuration directory that is writable
by non-root user. This PR leverages this solution and allows here to:

  • identify packages sources with official Debian Keyring GPG signatures
  • use 'apt-get source <pkgsrc_name>' to retrieve packages without maintaining
    several specific URLs
  • use 'apt-cache showsrc <pkgsrc_name>' to retrieve a
    list of available versions to ckeck packages availability during the
    'make init' phase
    If no candidate version is available, the script will exit with an error code and stops the whole compilation process.
    This will prevent starting a long build process condemned to fail and thus spare precious time and computing resources.

Issue #13281

Which release branch to backport (provide reason below if selected)

potentially all

Description for the changelog

[build] Check versions availability in rules/*.mk at init and use apt-get source instead of dget or wget in Makefiles

A picture of a cute animal (not mandatory but encouraged)

                      .".
                     /  |
                    /  /
                   / ,"
       .-------.--- /
      "._ __.-/ o. o\  
         "   (    Y  )
              )     /
             /     (
            /       Y
        .-"         |
       /  _     \    \ 
      /    `. ". ) /' )
     Y       )( / /(,/
    ,|      /     )
   ( |     /     /
    " \_  (__   (__        [nabis]
        "-._,)--._,)

- create an a apt alternate configuration directory during 'make init'
- use 'apt-get source' instead of dget in makefiles
- adapt packages versions accordingly in rules/XX.mk files

SONiC images compilation is currently designed to be performed with a
non-root user under Ubuntu systems.
As a result, there is no guarantee that debian repos are declared in
/etc/apt/sources.list on the Ubuntu host system used.
Although for this reason 'apt-get source' can not be directly used
(out of the box) to that purpose, the build process recompiles packages
for various GNU+Linux Debian distributions.
And the packages sources used are retrieved from more or less official
repositories since the current solution in SONiC is to use dget and URLs
from various locations.
As discussed in the following URL
https://serverfault.com/questions/447457/use-apt-get-source-on-a-debian-repo-without-using-etc-apt-source-list
this is the simplest solution but it has several drawbacks from a
maintenance and security standpoint, for example :
* maintaining manually these URLS since they can be removed by external
  sites owners
sonic-net#13281
* potential loose authentication of packages sources
sonic-net#13302

A more complex solution is prosposed (on the same URL) and adress these
drawbacks by using an alternate apt configuration directory that is writable
by non-root user. It allows here to:
* identify packages sources with official Debian Keyring GPG signatures
* use 'apt-get source <pkgsrc_name>' to retrieve packages without maintaining
  several specific URLs
* in a next step use 'apt-cache showsrc <pkgsrc_name>' to retrieve a
  list of available versions to ckeck packages availability during the
  'make init' phase

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
- add a script to check packages sources versions declared in
  rules/isc-dhcp.mk
- call this script durint the 'make init' phase

Files with the extension .mk in "rules/" folder are parsed several times
during the 'make configure' phase.
Thus, it is preferable to check and optionnaly set the package versions
declared there before during the previous 'make init' phase.
This will prevent from 'apt-get source' calls failures.

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
…ation

- declare bash and debootstrap in automatic version detection script
  i.e. scripts/check_rules_mk_files_versions.sh
- add a notice in script comments
- adapt consequently corresponding rules/*.mk and src/*/Makefile files
  i.e rules/bash.mk rules/debootstrap.mk src/debootstrap/Makefile

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
…nfiguration

- declare kdump-tools and lm-sensors
  in scripts/check_rules_mk_files_versions.sh
- adapt consequently corresponding rules/*.mk files

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
…sions.sh

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
…les_versions.sh

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
@guillaumelambert
Copy link
Contributor Author

Builds themselves were OK.
It seems previous CI run was KO because there was an issue in the queue to lock one of the testbed.
https://dev.azure.com/mssonic/build/_build/results?buildId=203972&view=logs&j=90badf3f-7a57-5c0e-d9bd-d2e7a4fde36a&t=bc4195ad-8694-52a1-21a3-01785e725d8d

guillaumelambert added a commit to guillaumelambert/sonic-buildimage that referenced this pull request Apr 12, 2023
- use dget instead of wget in some Makefiles
- switch related HTTP URL to HTTPS and official debian mirror when
  possible

TODO: migrate azure debian mirror to debian official ones (this might
require upgrading some debian packages version in rules/*.mk)
NB: A more complete solution with apt-get is proposed in
sonic-net#13381

SONiC images compilation is currently designed to be performed with a
non-root user under Ubuntu systems.
And the build process recompiles packages for various GNU+Linux Debian
distributions.
As a result, there is no guarantee that debian repos are declared in
/etc/apt/sources.list on the Ubuntu host system used.
And 'apt-get source' can not be directly used out-of-the-box.
Currently, the packages sources are retrieved from more or less official URLs
and locations with dget and wget.

As discussed in the following URL
https://serverfault.com/questions/447457/use-apt-get-source-on-a-debian-repo-without-using-etc-apt-source-list
dget is the simplest solution.

wget has several drawbacks from a maintenance and security standpoint:

- retrieving multiple URLs
- loose authentication of packages sources especially when retrieved with HTTP instead of HTTPS
  This is potentially vulnerable to MITM and can induce malicious code injection.

dget by default verifies package signature and can prevent most of such MITM attacks

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
guillaumelambert added a commit to Orange-OpenSource/sonic-buildimage that referenced this pull request Apr 20, 2023
- use dget instead of wget in some Makefiles
- switch related HTTP URL to HTTPS and official debian mirror when
  possible

TODO: migrate azure debian mirror to debian official ones (this might
require upgrading some debian packages version in rules/*.mk)
NB: A more complete solution with apt-get is proposed in
sonic-net#13381

SONiC images compilation is currently designed to be performed with a
non-root user under Ubuntu systems.
And the build process recompiles packages for various GNU+Linux Debian
distributions.
Though, there is no guarantee that debian repos are declared in
/etc/apt/sources.list on the Ubuntu host system used.
As a result, 'apt-get source' can not be directly used out-of-the-box.
Currently, the packages sources are retrieved from more or less official URLs
and locations with dget and wget.

As discussed in the following URL
https://serverfault.com/questions/447457/use-apt-get-source-on-a-debian-repo-without-using-etc-apt-source-list
dget is the simplest solution.

wget has several drawbacks from a maintenance and security standpoint:

- retrieving multiple URLs
- loose authentication of packages sources especially when retrieved with HTTP instead of HTTPS
  This is potentially vulnerable to MITM and can induce malicious code injection.

dget by default verifies package signature and can prevent most of such MITM attacks

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant