Why not use ORT built-in SPDX report?

[nicorikken]

I'm looking into license scanning solutions and I already found ORT as a batteries-included solution around scancode-toolkit. This repo was linked as an example for a github action, which is still a todo there.

Looking into your solution I noticed you use your own spdx-builder. Why did you choose a roll-your-own solution, rather than the built-in spdx report from ORT? Was this a timing issue that it was not yet available in ORT?

Also: thanks for developing this in public, it is an interesting solution that has some nice ideas like pushing the SPDX documents to a central repo.

[JeroenKnoops]

We're doing research on the matter, and we found that in order to fully understand the world of SBOM, we need to do some investigation ourselves.

We're choosing another approach to getting the SBOM. We're using ORT for getting the dependency structures for projects (as input for SPDX-builder. SPDX-builder uses the License-scanner with scancode to get more detailed information about the actual licenses in the source. This is a deployed service which does this in the background with a way of concluding the licenses with a ui.. difference between declared / detected and concluded licenses.
The SPDX-builder tool outputs a SPDX file which can be imported into Bom-Bar where you can check against policies depending on the deploy context.

THIS IS ALL EXPERIMENTAL AND NOT USED IN PRODUCTION

We're also comparing the spdx outputs with blackduck (SPDX-builder can produce a SPDX file from a blackduck output) and other tools.

In Philips we have a lot of different type of projects / setups. So using this setup we've learned a lot about how you can create a SPDX file from an existing project.

We found that it's very difficult to have a tool give the right SBOM if it does not know the context of a project. Often results from a tool need to be checked / filtered by an application expert on what's really in the code.
That's why we're now experimenting with shifting the burden of delivering a proper (easy to generate and readable) tree of dependencies for a project to the application expert.
Almost any package tool can produce a list of dependencies during buildtime. Something like: mvn dependency:tree for maven. See https://github.com/philips-software/spdx-builder/blob/develop/src/main/resources/treeformats.yml
If you have a project with multiple technologies in it, you can combine these outputs and feed them into SPDX-builder. SPDX-builder will send the information to another service called Bom-base which is a knowledge base. Bom-base will replace the functionality of License-scanner. It will search all kind of sources for the truth. It uses ClearlyDefined to get the source. It uses Scancode to get the licenses. It uses harvesters to get information from the package managers like pypi. And so on.
The data is send to SPDX-builder and SPDX-builder will create a nice SPDX document of it, which you can use in your checks.

The key difference is that the scan environment only needs to know it's own build environment. No more detect scanners with a certain version of dotnet in it which perform some magic. No more magic detection of how a developer setup his python project, which is often not standarised. We need to involve the application expert anyways, so let's make him co-owner of the SBOM by asking her what dependencies there are in the project. We convert them to the latest purl spec or whatever is necessary.

Again, this is all experimental and we're testing it with a lot of different internal projects. We want to minimize the impact on the projects and we want to create various views on the SBOM for different people. Legal should see about conflicts in compatible licenses.. Security should see about vulnerabilities. An Export Control officer should see if some 'dangerous' packages are used and the developer should have a feeling the tools are supporting her work instead of adding work to her already busy agenda.

More on the Tree-based part: https://github.com/philips-software/spdx-builder/blob/develop/docs/usage_with_tree.md

We're planning to deploy OSS versions of the tools so we can also demo this to others.

[nicorikken]

Thanks for the detailed response. It is interesting. I'm also looking into a setup where the 'application experts' are responsible for generating and submitting the SPDX files. Byt your idea of still scanning on a central level sounds interesting. The infrastructure becomes somewhat more complex (moving parts at the teams and at the central level), but it seems to make sense. Anyway, thanks for your work in this area, it is inspiring!