Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SBOM Software Bill Of Materials #4050

Closed
solofoA45 opened this issue Nov 10, 2023 · 26 comments
Closed

SBOM Software Bill Of Materials #4050

solofoA45 opened this issue Nov 10, 2023 · 26 comments
Labels
documentation Improvements or additions to documentation

Comments

@solofoA45
Copy link

For security and compliance concerns, it would be good to have a list
of dependencies for example to assess which security vulnerabilities affect Xpra:
https://en.wikipedia.org/wiki/Software_supply_chain

While this is rather clear for linux (RPM) packages, this is less
clear for windows packages and HTML5 client packages.

  • For windows packages it seems not clear which MSYS packages were used to build the packages and their versions
  • For HTML5 client, the libraries (aurora,...) seem to be included in html5/js/lib/ but not origin, version, patches,...

Is there already a way to get these informations?

@totaam totaam added the documentation Improvements or additions to documentation label Nov 13, 2023
@totaam
Copy link
Collaborator

totaam commented Nov 13, 2023

Here are some starting points and initial thoughts.

This should tie into the Usage : Security documentation since some issues can be mitigated or bypassed using a tight configuration.

The closest thing we have to an overview of the dependencies is here: https://github.com/Xpra-org/xpra/blob/master/docs/Build/Dependencies.md (and includes some pretty diagrams)

MacOS

The MacOS builds are by far the easiest to track since we define and build every single library ourselves.
The list can be found by checking out the build repository at the date matching that of the release:
https://github.com/Xpra-org/gtk-osx-build
There are a few extra packages in there that are only used on MacOS for packaging (ie: xar, etc), those are not shipped.

MS Windows

The full MSYS2 package list on the build system at time of writing has 179 packages:

$ pacman -Qe | wc -l
179

There are also python dependencies which do not have corresponding MINGW packages and are installed via pip:

#these need to be converted to PKGBUILD:
$PACMAN --noconfirm --needed -S ${XPKG}python-pip ${XPKG}python-pycryptodome ${XPKG}python-keyring ${XPKG}python-idna openssl-devel
for x in browser-cookie3 pylsqpack aioquic pyaes pbkdf2 pytools; do
pip3 install $x
done

Then there are also some manual steps:
echo "to package the EXE, install verpatch:"
for:

  • packaging tools
  • printing libraries
  • documentation packaging
  • theming
  • CUDA

Packages we do not update as often as others

There can be many reasons why some packages are not updated as regularly as others:

  • incessant package releases (often causing breakage): setuptools, zeroconf
  • buggy releases (see above)
  • avoiding the risk of breaking things (ie: build and packaging tools)
  • harder to update (non automatic updates via pip): pip3 list --outdated
  • unsupported build environments: some projects no longer support python2, or ancient versions of python3, etc - and this has a cascading effect
  • effectively abandoned (ie: pyu2f): should we remove them?

Opaque packages:

  • libyuv : google repository with no actual releases - who really knows what's going on there?

3.1

Some issues are magnified in 3.1:

On the whole, I don't think that it is reasonable to expect the 3.1.x to have the same level of maintenance as current versions.
CUDA

@totaam
Copy link
Collaborator

totaam commented Nov 24, 2023

We should probably split the dependencies into categories - this is probably too many:

  • filter out build time dependencies that aren't actually used (just installed as a transitive dependency of something else)
  • build time dependencies that aren't shipped (ie: gcc though libgcc might be?, bash, etc)
  • shipped but without the faulty component (ie: libtiff CVE was for an executable we do not include, not the libtiff library we do ship)
  • shipped but unused (ie: unused ffmpeg codecs: 20bb5f0, other parsers)
  • shipped and used, but not exploitable (ie: requires a specific format or API call argument)
  • runtime and used but can be disabled (ie: --video-decoders=none)
  • runtime, used and cannot be disabled (ie: GTK, cairo, etc)
  • runtime, network facing - the danger zone: lz4, rencodeplus, python interpreter.
    etc.

@totaam
Copy link
Collaborator

totaam commented Dec 25, 2023

The MS Windows dependencies can be recorded in xpra/build_info.py.
We can get most of the packages from pacman - and perhaps trim most of the build time dependencies as those aren't very relevant?
The python pip dependencies are going to be a pain.

@totaam
Copy link
Collaborator

totaam commented Dec 26, 2023

@totaam
Copy link
Collaborator

totaam commented Jan 11, 2024

Both MacOS and MS Windows builds will now record the libraries and python modules present on the build system when the installer is generated.
This will include dependencies we don't really care about: build tools, libraries we don't bundle, etc. But it is safer to record too much than too little, and filtering was hard and would also have required constant fine tuning.

The feature for the html5 client is now tracked here: Xpra-org/xpra-html5#277


Next up:

  • use a better format: preserve the package names in full using string keys rather than a variable name with prefix
  • cleanup the macos pip3 freeze output - what is the checksum for?
  • query more details for python projects using https://pypi.org/pypi/$PROJECT/json
  • use https://cyclonedx.org/use-cases/ file format?
  • make it easier to access this information, include this together with the exe / dmg packages in the download area?
  • tie it to the dependency page?
  • some kind of automated tracking?

@totaam
Copy link
Collaborator

totaam commented Mar 8, 2024

Another tricky one to handle is pdfium-binaries releases - this release page does show a line that says something like "This version was built with branch chromium/6337 of PDFium".
The archive containing the DLL we need also contains a VERSION file:

$ cat pdfium/VERSION 
MAJOR=124
MINOR=0
BUILD=6337
PATCH=0

The easiest way might be to create a "fake" pacman PKGBUILD for it.

@totaam
Copy link
Collaborator

totaam commented Mar 30, 2024

The new script that I am working on would flag:
liblzma-5.dll as belonging to /mingw64/bin/liblzma-5.dll is owned by mingw-w64-x86_64-xz 5.6.1-2.
(current DLL should be fine but it was previously owned by the vulnerable version of xz-utils: 5.6.1-1)

Fixed in:
msys2/MSYS2-packages@eb7abbb
First vulnerable version was added in:
msys2/MSYS2-packages@d153a09
So any builds between 2024-02-25 and today are shipping the vulnerable library.

On the plus side, the exploit seems to target a specific function in openssh - with glibc, and we don't use openssh by default, and no glibc, and not as a server... So no need to panic.


Good links on the subject:

@totaam
Copy link
Collaborator

totaam commented Apr 3, 2024

Forgot another packages missing from MSYS2 that we should contribute upstream:

echo "you may also want to install libspng for faster PNG support"

Trivial to install: meson build && ninja install.

@totaam
Copy link
Collaborator

totaam commented May 17, 2024

cyclonedx-python-lib: This Python package provides data models, validators and more, to help you create/render/read CycloneDX documents.

totaam added a commit that referenced this issue Oct 19, 2024
too much is better than too little,
use structured data so we can more easily access it in Python without making assumptions
@totaam
Copy link
Collaborator

totaam commented Oct 20, 2024

As of 5d21b1b, a default build will add a lot of metadata in xpra/build_info.py:

build = {'user': 'Windows 10 Test', 'on': 'Win10-Build-VM', 'date': '2024-10-20', 'time': '23:10', 'machine': 'AMD64', 'cpu': 'Intel64 Family 6 Model 94 Stepping 3, GenuineIntel', 'bit': '64bit', 'os': 'Microsoft Windows 10 Pro', 'type': '', 'python': '3.11.10', 'cython': '3.0.11', 'compiler': 'gcc (Rev1, Built by MSYS2 project) 14.2.0', 'nvcc': '', 'linker': 'GNU ld (GNU Binutils) 2.43.1'}
libs = {'apr': '1.7.5-1', 'apr-util': '1.6.3-2', 'asciidoc': '10.2.1-2', 'autoconf-wrapper': '20240607-1', ...

(only the first 4 packages are show here since there are literally hundreds of them, as we record all the packages installed on the build system - whether they are used or not, which is impossible to ascertain)

Then also:

BUILD_OPTIONS={'build': True, 'clean': True, 'cuda': False, 'desktop_logon': False, 'docs': False, 'fixups': True, 'html5': False, 'install': True, 'installer': False, 'light': True, 'manual': False, 'msi': False, 'numpy': False, 'openssh': False, 'openssl': False, 'paexec': False, 'putty': False, 'run': True, 'sbom': True, 'service': False, 'sign': True, 'tests': False, 'verbose': False, 'verpatch': True, 'zip': False, 'zip_modules': True}

dll_sbom={'lib/gdk-pixbuf-2.0/2.10.0/loaders/io-wmf.dll': ('mingw-w64-x86_64-libwmf', '0.2.13-1'), 'lib/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-ani.dll': ('mingw-w64-x86_64-gdk-pixbuf2', '2.42.12-3'), ...

Again, far too many to show here.
This one is smarter and only includes packages that we actually ship.
Only a few DLLs aren't identified yet, which we need to submit upstream: pdfium, libspng, win32com.

This data can be converted to a JSON SBOM using cyclonedx - and maybe this should be a separate build arctifact?

@totaam
Copy link
Collaborator

totaam commented Oct 21, 2024

@solofoA45 good news is that the scripting part is mostly done and working well, the bits that still need doing stick out from the script output:

$ python3.11 ./packaging/MSWindows/BUILD.py
* 21:42:25 Collecting version information
    Python 3.11.10 (main, Sep 10 2024, 13:02:13)  [GCC 14.2.0 64 bit (AMD64)]
    Xpra-Light 6.3-r36810
    using 4 cpus
* 21:42:31 locating `signtool`
* 21:42:31 Cleaning output directories and generated files
* 21:42:34 Building Cython modules
* 21:43:27 Generating installation directory
* 21:44:43 Fixups: paths, etc
* 21:44:44 Fixup GStreamer
* 21:44:44 Fixup DLLs
* 21:45:18 Deleting unnecessary DLLs
* 21:45:18 Removing unnecessary Python modules
Warning: glob 'lib/backports' did not match any files!
Warning: glob 'lib/yaml' did not match any files!
* 21:45:18 Removing unnecessary files
* 21:45:21 Removing unnecessary PIL plugins
* 21:45:21 Removing empty directories
* 21:45:22 cuda: False
Warning: glob 'lib/pycuda*' did not match any files!
Warning: glob 'lib/curand*' did not match any files!
* 21:45:22 numpy: False
Warning: 'dist/lib/numpy' does not exist
Warning: glob 'lib/libopenblas*' did not match any files!
Warning: glob 'lib/libgfortran*' did not match any files!
Warning: glob 'lib/libquadmath*' did not match any files!
* 21:45:22 zipping up some Python modules
Warning: glob 'lib/unittest' did not match any files!
Warning: glob 'lib/pynvml' did not match any files!
Warning: glob 'lib/ldap' did not match any files!
Warning: glob 'lib/ldap3' did not match any files!
* 21:45:28 Deleting unnecessary `share/` files
* 21:45:28 Removing empty icon directories
* 21:45:28 Adding EXE manifests
* 21:45:28 Generating gdk pixbuf loaders cache
* 21:45:28 Generating icons and theme cache
* 21:45:28 Recording SBOM
Warning: no package data found for 'lib/libspng-0.dll'
Warning: no package data found for 'pdfium.dll'
Warning: no package data found for 'C:/msys64/mingw64/lib/python3.11/site-packages/aioquic'
Warning: no package data found for 'C:/msys64/mingw64/lib/python3.11/site-packages/pylsqpack'
181     dist
* 21:48:25 Creating ZIP file:
72      Xpra-Light-x86_64_6.3-r36810.zip
* 21:48:44 Creating the installer using InnoSetup
51      Xpra-Light-x86_64_Setup_6.3-r36810.exe
* 21:49:28 Signing EXE
* 21:49:29 Running the new installer

And I think that this should be relatively straightforward to backport this code to older branches:

  • it doesn't get used on anything but MS Windows
  • it only adds data to build info and sbom files, so this is unlikely to break anything

@totaam
Copy link
Collaborator

totaam commented Oct 21, 2024

Here's the beginning of the raw SBOM data for a full build (this is parseable as python source):

# 307 SBOM path entries:
{
'lib/gdk-pixbuf-2.0/2.10.0/loaders/io-wmf.dll': \
    (19975, '39f9911e319826a956a6a858dca4ad5d406e580db8eff3d760245b4b26435540', 'libwmf', '0.2.13-1'),
'lib/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-ani.dll': \
    (26756, '9ed6fb05b84163c24ee2ecf3036b3a23a0f88e2bd29f45bbf1e97cfc079e7216', 'gdk-pixbuf2', '2.42.12-3'),
'lib/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-avif.dll': \
    (25362, '6f99f0d702fc085fadf9d2a9033f0c8349c0b2d73019e5e32a4ffcaa50b3a0bf', 'libavif', '1.1.1-1'),
'lib/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-bmp.dll': \
    (27093, 'bfbba366556397a66c82844a2fea656baa303ab501c112b1139143623a392d13', 'gdk-pixbuf2', '2.42.12-3'), 
...
# 134 'mingw-w64-x86_64-' packages:
('aom', 'atk', 'boost', 'brotli', 'bzip2', 'cairo', 'crypto++', 'curl', 'cyrus-sasl', 'dav1d', 'dbus', 'dbus-glib', 'double-conversion', 'expat', 'faac', 'faad2', 'flac', 'fontconfig', 'freetype', 'fribidi', 'gcc-libgfortran', 'gcc-libs', 'gdk-pixbuf2', 'gettext-runtime', 'glib2', 'gmp', 'gnutls', 'gobject-introspection-runtime', 'graphite2', 'gss', 'gst-plugins-bad', 'gst-plugins-bad-libs', 'gst-plugins-base', 'gst-plugins-good', 'gst-plugins-ugly', 'gstreamer', 'gtk3', 'harfbuzz', 'highway', 'icu', 'jbigkit', 'lame', 'lcms2', 'lerc', 'libavif', 'libb2', 'libcroco', 'libdatrie', 'libdeflate', 'libepoxy', 'libffi', 'libgcrypt', 'libgpg-error', 'libheif', 'libiconv', 'libidn', 'libidn2', 'libimagequant', 'libjpeg-turbo', 'libjxl', 'libogg', 'libpng', 'libpsl', 'libraqm', 'librsvg', 'libsodium', 'libsoup', 'libssh2', 'libtasn1', 'libthai', 'libtiff', 'libunistring', 'libvorbis', 'libvpx', 'libwebp', 'libwinpthread-git', 'libwmf', 'libx264', 'libxml2', 'libyaml', 'libyuv', 'lz4', 'md4c', 'mpdecimal', 'mpg123', 'nettle', 'nghttp2', 'nghttp3', 'openblas', 'openh264', 'openjpeg2', 'openjph', 'openldap', 'openssl', 'opus', 'orc', 'p11-kit', 'pango', 'pcre2', 'pixman', 'python', 'python-bcrypt', 'python-cairo', 'python-cffi', 'python-charset-normalizer', 'python-cryptography', 'python-importlib-metadata', 'python-importlib_resources', 'python-ldap', 'python-netifaces', 'python-numpy', 'python-oauthlib', 'python-pillow', 'python-py-cpuinfo', 'python-pycryptodome', 'python-pycryptodomex', 'python-pynacl', 'python-pyopengl-accelerate', 'python-pyparsing', 'python-pywin32', 'python-watchdog', 'python-winkerberos', 'python-zeroconf', 'qrencode', 'qt6-base', 'rav1e', 'shishi', 'sqlite3', 'svt-av1', 'wavpack', 'xxhash', 'xz', 'zlib', 'zstd')

@totaam
Copy link
Collaborator

totaam commented Oct 22, 2024

Some of the missing MSYS2 packages have been submitted upstream:

This one I will probably keep in my own repo as I don't think that will be acceptable for upstream:

Which only leaves CUDA.
It makes sense to special case it since we already have a add_cuda function and cuda switch.
They ship their own metadata file which we'll need to parse:

$ cat /z/version.json 
{
   "cuda" : {
      "name" : "CUDA SDK",
      "version" : "12.6.0"
   },
...

totaam added a commit that referenced this issue Oct 22, 2024
use our own code for getting the disk usage
totaam added a commit that referenced this issue Oct 24, 2024
this has to be done before zipping up the python modules
totaam added a commit that referenced this issue Nov 9, 2024
so we don't depend on too many extra packages that are never actually used (ie: 'libheif')
also fix the pixbuf filtering, which never deleted any actual loaders (oops)
totaam added a commit that referenced this issue Nov 9, 2024
totaam added a commit that referenced this issue Nov 9, 2024
so we can avoid all the dependencies this requires: keyring and lots of jaraco modules, just for parsing http headers - really?
@totaam
Copy link
Collaborator

totaam commented Nov 9, 2024

Remaining warnings for light builds during the cyclonedx export step that still need to be investigated:

Warning: 'mingw-w64-x86_64-c-ares' not found, dependency of curl
Warning: 'mingw-w64-x86_64-ca-certificates' not found, dependency of curl
Warning: 'mingw-w64-x86_64-nettle>=3.6' not found, dependency of gnutls
Warning: 'mingw-w64-x86_64-p11-kit>=0.23.1' not found, dependency of gnutls
Warning: 'mingw-w64-x86_64-glib-networking' not found, dependency of libsoup
Warning: 'mingw-w64-x86_64-sqlite3' not found, dependency of libsoup
Warning: 'mingw-w64-x86_64-libsystre' not found, dependency of ncurses
Warning: 'mingw-w64-x86_64-wineditline' not found, dependency of pcre2
Warning: 'mingw-w64-x86_64-mpdecimal' not found, dependency of python
Warning: 'mingw-w64-x86_64-sqlite3' not found, dependency of python
Warning: 'mingw-w64-x86_64-python-decorator' not found, dependency of python-gssapi
Warning: 'mingw-w64-x86_64-gss' not found, dependency of python-gssapi

totaam added a commit that referenced this issue Nov 10, 2024
and generate the cache
totaam added a commit that referenced this issue Nov 10, 2024
totaam added a commit that referenced this issue Nov 10, 2024
(rather than special casing them)
totaam added a commit that referenced this issue Nov 10, 2024
* do bundle gss / kerberos so even light clients can use it to authenticate,
* don't bundle 'duktape'
totaam added a commit that referenced this issue Nov 10, 2024
* record DLL found in pkcs11 subdir and don't try to use this subdir as a python module,
* record 'decorator' by hand,
* document the details on the 'mpdecimal' unnecessary dependency warning,
* skip more optional deps
* always remove extra python modules when making 'light' builds, even if not zipping the modules
@totaam
Copy link
Collaborator

totaam commented Nov 10, 2024

With the latest dependency updates and tweaks, only a few corner cases remain for the "light" builds:

  • pyvda - needs a PKGBUILD
  • some mingw packages still need sorting out (either ignored or bundled in): c-ares, nettle, p11-kit, wineditline

@totaam
Copy link
Collaborator

totaam commented Nov 10, 2024

To make things even more "interesting", MSYS2 just switched from Python 3.11 to Python 3.12.
So the 6.3-r36873 beta builds are still using Python 3.11.10, but any later builds will be based on Python 3.12.x.

Good news is that the upgrade just worked.
All that was needed afterwards - as expected:

totaam added a commit that referenced this issue Nov 17, 2024
@totaam
Copy link
Collaborator

totaam commented Nov 17, 2024

The export-sbom step of the new build script can be executed separately:

packaging/MSWindows/BUILD.py export-sbom

It requires:

  • Python 3.12 must be installed in C:\Program Files\Python312
  • Add cyclonedx to it: Python.exe -m pip install cyclonedx-bom

totaam added a commit that referenced this issue Nov 23, 2024
@totaam
Copy link
Collaborator

totaam commented Nov 23, 2024

There is now a link from the download page to the corresponding sbom files. (only for the beta releases for now, others will be added later)

totaam added a commit that referenced this issue Nov 23, 2024
also fix type hints and return the correct data type
totaam added a commit that referenced this issue Nov 23, 2024
* match v5.x setup.py flags
* no manifests in this branch
* add numeric version no (easier than reverse computing it)
@totaam
Copy link
Collaborator

totaam commented Nov 23, 2024

Caveats:

  • "Full" builds still trigger a bunch of warnings
  • pdfium, pycuda, browser_cookie3, etc... should be packaged better

Closing.

@totaam totaam closed this as completed Nov 23, 2024
totaam added a commit that referenced this issue Nov 23, 2024
@totaam totaam changed the title [Q] Software Bill Of Materials? SBOM Software Bill Of Materials Nov 29, 2024
@totaam
Copy link
Collaborator

totaam commented Nov 29, 2024

With the latest fixes, the only missing piece is a PKGBUILD for pycuda.

@totaam
Copy link
Collaborator

totaam commented Nov 30, 2024

For pycuda, see #2400 (comment)

@totaam
Copy link
Collaborator

totaam commented Dec 1, 2024

This also works for arm64 builds: #3941

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

2 participants