Unable to classify complex licenses

[yaronkaikov]

What happened:
I am generating an SBOM report with the command syft <docker image> --config syft.yaml -o cyclonedx-json@1.4

syft.yaml configuration:

golang:
   # search for go package licences by retrieving the package from a network proxy
   # SYFT_GOLANG_SEARCH_REMOTE_LICENSES env var
   search-remote-licenses: true

java:
   maven-url: "https://repo1.maven.org/maven2"
   max-parent-recursive-depth: 5
   # enables Syft to use the network to fill in more detailed information about artifacts
   # currently this enables searching maven-url for license data
   # when running across pom.xml files that could have more information, syft will
   # explicitly search maven for license information by querying the online pom when this is true
   # this option is helpful for when the parent pom has more data,
   # that is not accessible from within the final built artifact
   use-network: true

after generating the report, we noticed that we have some missing licenses for the following packages (see attached file) -
missing_licenses - nightly.csv

What you expected to happen:

Have no missing licenses

Steps to reproduce the issue:

Generate SBOM report using the command: syft docker.io/scylladb/scylla-nightly:latest --config syft.yaml -o cyclonedx-json@1.4

Anything else we need to know?:

Environment:

• Output of syft version:

bash-5.2# /opt/sbom/syft version
Application: syft
Version:    1.17.0
BuildDate:  2024-11-21T14:39:38Z
GitCommit:  a8d4202d77b6b31e75ce5af09a8b03ad14e533d3
GitDescription: v1.17.0
Platform:   linux/amd64
GoVersion:  go1.22.9
Compiler:   gc
bash-5.2# 

• OS (e.g: cat /etc/os-release or similar):

bash-5.2# cat /etc/os-release 
NAME="Fedora Linux"
VERSION="41 (Container Image)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=41
VERSION_CODENAME=""
PLATFORM_ID="platform:f41"
PRETTY_NAME="Fedora Linux 41 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:41"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f41/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=41
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=41
SUPPORT_END=2025-12-15
VARIANT="Container Image"
VARIANT_ID=container

[kzantow]

Hi @yaronkaikov, I had a look at this; it looks like there are a number of different things going on.

The first couple of packages are go modules.

github.com/dennwc/ioctl -- It looks like Syft is doing the most accurate thing here: this was published at v1.0.0, which did not include a LICENSE file, this was added later. From Syft's perspective this version does not have a license, even though you could look at github and expect it was what is contained in the LICENSE file. We could possibly add some sort of github lookup here if we were unable to find a license in the distribution, but we're only looking at the specific files for the version found today.

github.com/prometheus/node_exporter -- this uncovered an issue where some failures were preventing looking up licenses online at all, I've fixed this with PR #3549.

The operating system entry is a synthetic package based on the distro information Syft found, we should figure out how to determine the correct license to include. Any ideas here are welcome!

The remainder fall into 2 categories: python and debian packages.

The python packages (pkg:pypi/...) may not have licenses declared in the METADATA file, but instead have a LICENSE file of some sort on the filesystem. Syft does not currently read these files nor does it have remote license resolution for these, but we'd love to add handling for one or both of these! There is some more information in #2624

The debian packages (pkg:deb/...) should have license information provided, as most packages show... but it seems as though Syft isn't able to ascertain licenses from certain copyright files -- the first couple I checked are quite complicated amalgamations of lots of text of licenses and other stuff, so I suspect the library we are using for identification just isn't able to identify these with any sort of confidence. Happy for any suggestions how to interpret /usr/share/doc/libpython3-stdlib/copyright and /usr/share/doc/libcrypt1/copyright, for example; what would you expect to see for these files?

[yaronkaikov]

@kzantow I have noticed that the latest release 1.19.0 contains the fix, but the license is still empty, so either i am doing something wrong, or the fix doesn't solve the issue

Is there a way in the syft.conf file to manually get package info with license?

[kzantow]

I created a new go.mod with github.com/prometheus/node_exporter and purposely gave it a bad go mod directory. When I scan that with the latest Syft (SYFT_GOLANG_LOCAL_MOD_CACHE_DIR=/asdf syft . -o json --enrich all), I see:

    {     
      "id": "56ae66a38ee0a4ce",
      "name": "github.com/prometheus/node_exporter",
      "version": "v1.8.2",
      "type": "go-module",
      "foundBy": "go-module-file-cataloger",
      "locations": [
        {
          "path": "/go.mod",
          "accessPath": "/go.mod",
          "annotations": {
            "evidence": "primary"
          }
        }
      ],
      "licenses": [
        {
          "value": "Apache-2.0",
          "spdxExpression": "Apache-2.0",
          "type": "concluded",
          "urls": [
            "https://proxy.golang.org/github.com/prometheus/node_exporter/@v/v1.8.2.zip#github.com/prometheus/node_exporter@v1.8.2/LICENSE"
          ],
          "locations": []
        }   
      ],  
      "language": "go",

... so it looks like the go license issue should be fixed. The others packages mentioned have different reasons the licenses were not included: the Python package does not read the license files (tracked already in #2624). And the Deb issue, I think is due to these packages having complex license text, which Syft is unable to classify with the current license classification library. I've reopened this issue and retitled it to better reflect the current status.

Please let me know if I've misunderstood anything; and any further examples/information you could provide about specifics we could look at could help to get this fixed!