Add an SPDX file to the repository to streamline license and security reviews by user organizations #42102

SamuraiAku · 2021-09-03T05:59:14Z

Summary

This PR proposes to add an SPDX file to the repository. This file would describe all of the source code required to build Julia that exists outside of the main repository, in a human and machine readable format.

Background

The Software Package Data Exchange (SPDX) is a standard maintained by the Linux Foundation. It's reason for existence is best summarized in section 1.3 of the specification:

Companies and organizations (collectively “Organizations”) are widely using and reusing open source and other software packages. Accurate identification of software is key for many supply chain processes. Vulnerability remediation starts with knowing the details of which version of software is in use on a system. Compliance with the associated licenses requires a set of analysis activities and due diligence that each Organization performs independently, which may include a manual and/or automated scan of software and identification of associated licenses followed by manual verification. Software development teams across the globe use the same open source packages, but little infrastructure exists to facilitate collaboration on the analysis or share the results of these analysis activities. As a result, many groups are performing the same work leading to duplicated efforts and redundant information. The SPDX working group seeks to create a data exchange format so that information about software packages and related content may be collected and shared in a common format with the goal of saving time and improving data accuracy.

The SPDX homepage is https://spdx.dev
A helpful overview of the use of SPDX can be found at https://spdx.dev/resources/use/
The SPDX specification can be found at https://spdx.dev/specifications/
An SPDX file can be validated with an online tool provided by the SPDX organization (https://tools.spdx.org/app/validate/)
The SPDX specification was recently ratified as an ISO standard (ISO/IEC 5962:2021)

Justification for this PR

Today Julia is the underdog and anything that reduces barriers to adoption is a benefit to the project. The review of open source software by organizations, such as described above, is one such barrier. This type of review is commonly required for every point release that the organization wishes to use.

Julia is a large and quite complex project pulling code from 23 external repositories as part of the build (see THIRDPARTY.md for a full list). Julia also maintains 8 of the standard libraries in separate repositories.

When you describe Julia like that it's quite clear that performing a proper review of Julia takes considerable effort. They have to go into each of the external repositories, check the license, see if anything important has changed since the last review, write this all down and then figure out if Julia is acceptable to their organization.

Given the amount of work involved you can understand why some organizations might give up on approving Julia on a regular basis and say that other more widely used tools such as Matlab, Python, and R are "good enough" for their needs.

The file THIRDPARTY.md does contain much of the information that the reviewer will want, but it is in an ad-hoc format that is awkward for a human reviewer unfamiliar with Julia and is not suitable for automated processing.

Adding an SPDX document to Julia has the potential to streamline the review process leading to faster, more frequent approval of Julia versions in organizations. Basically this document does a lot of the grunt work for the reviewer. It provides the reviewer with a standardized, human-readable, machine-scannable list of all of the external software components (packages in SPDX parlance) used in Julia, what their licenses are, where they are downloaded from and how they are used in Julia (relationships in SPDX parlance). Additional information and notes for each package can be added as needed. It also makes it easy for a reviewer to compare the current SPDX file with the file from the previously approved version of Julia and see if anything important has changed.

I certainly don't see how it could hurt the project to have the file present.

Tracking Version Information

The SPDX package for Julia itself includes the version number.

I have deliberately not included version info for all the software packages Julia pulls in from other repositories. I don't see how that information could be reliably kept up-to-date by hand and thus it is best not to include it at all. Instead I have added informational text in the field sourceInfo pointing the reader to the appropriate makefile where the version information is contained. This way even when package versions are updated the SPDX file does not need to be changed.

There are ways to incorporate the SPDX file into the build process so that we can guarantee the accuracy of the document, including version number, but that's best left for another PR

Updating the SPDX Document for each release

An SPDX file supports several file formats. I have chosen to use JSON because tools are widely available for this format, including for Julia, making it easy to update the fields.

Each time a new version of Julia is released just a few fields will need to be updated in this file, assuming no changes to the external software packages that need to be captured.

documentNamespace: A new UUID to be incorporated into the string. This field provides a unique identifier for this version of the SPDX file.
creationInfo.created: Note that the SPDX specification is very specific about the date_time format to be used.
packages[1].versionInfo : The Julia version number. Update with each new release
packages[1].downloadLocation: Update to point to the correct version of the Julia source code
packages[1].copyrightText: Update if needed

Status at initial posting

To kick off this discussion, I have included only a few packages to demonstrate how an SPDX file works. The packages were chosen to show all the different types of packages and what is similar and different about their description in SPDX. These packages are

Julia: A description of the main repository
Pkg: Part of the standard library
OpenBLAS: An external dependency
libuv: An external dependency that is forked and maintained by the Julia project
zlib: A library distributed with Julia. I'm not sure what this means, but I'm guessing that it is used only by the installer, hence the DISTRIBUTION_ARTIFACT relationship. Can someone confirm if that is correct?
Patchelf: A build tool

This file has been validated by the SPDX group's online tool as conforming to the specification.

Next steps

Putting together a complete SPDX file is a lot of work. I would prefer to not continue that work until a consensus has been reached that the document presented here is acceptable and that approval of this PR would be likely when the work and reviews are complete.

…sts 6 packages

julia.spdx.json

goneall

Suggesting a change to the license for patchelf.

julia.spdx.json

goneall

LGTM

SamuraiAku · 2021-09-20T05:55:55Z

That's all the stdlibs added.

…that the LLVM license changed from UIUC to Apache back in v8. Corrected the license reference in THIRDPARTY.md which pointed to the v6 license and is no longer correct.

SamuraiAku · 2021-09-25T05:20:48Z

Added the rest of the Base and Core external dependencies.

SamuraiAku · 2021-10-04T22:47:29Z

@goneall can you have a look at my package entry for libgit2? This package has a special exception to the GPL and I had to create my own license info field for it. Does that look right to you? And do you know if there is an exception in the license list I should be using instead? I didn't see anything that looked right, but I it's good to double-check. Thanks.

…n repository.

goneall · 2021-10-07T22:10:41Z

@SamuraiAku The current SPDX spec license expression documented in Annex D only allows standard exception ID's. LicenseRef style ID's are not allowed.

The way to handle this situation is to create a LicenseRef for then entire license including the exception. In this case, it would be something like LicenseRef-GPL2-only-with-libgit2-exception. The license text for this license ref would be the either the text as found in the library or GPL-2.0 license texted concatenated with the exception text.

Note that there are some discussions in progress to allow the use of LicenseRefs for exception in the SPDX 3.0 spec.

SamuraiAku · 2021-11-13T07:20:41Z

Changed my mind on SuiteSparse. Julia uses too many modules to make it worthwhile to make each one a package in SPDX. The copyright text field is very large, but the licensing fields make sense I think.

SamuraiAku · 2021-11-28T02:42:26Z

I think that's everything. The SPDX file describes every external dependency listed in THIRDPARTY.md, plus all the external stdlibs. Next step is to review all the entries and add a script under contrib for updating the document for each release.

@ViralBShah I have a question about Zlib and 7-Zip. They are listed separately in THIRDPARTY.md where it says "Julia bundles the following external programs and libraries". I presume that means that they are used by Julia in a different manner than the other dependencies. I wasn't sure what that was, so I guessed that these two are RUNTIME_DEPENDENCY_OF Julia while all the other non-build tool dependencies are BUILD_DEPENDENCY_OF Julia. Can you comment?

ViralBShah · 2021-12-02T16:08:09Z

I believe zlib is a build dependency and 7-zip is a runtime dependency.

SamuraiAku · 2021-12-10T06:48:24Z

For review, I've read back the SPDX file in this PR and created a table of the most important information fields with the following code

using JSON  
using TypedTables  
using PrettyTables  
pkgNames= Vector{String}()
pkgLicenses= Vector{String}()
pkgHomePages= Vector{String}()
pkgDownloads= Vector{String}()
pkgRelationshipsToJulia= Vector{String}()
spdxData= JSON.parsefile("./julia.spdx.json")
for pkg in spdxData["packages"]
    if pkg["name"] != "Julia"
        push!(pkgNames, pkg["name"])
        push!(pkgLicenses, pkg["licenseConcluded"])
        push!(pkgHomePages, pkg["homepage"])
        push!(pkgDownloads, pkg["downloadLocation"])
        for rel in spdxData["relationships"]
            if rel["spdxElementId"] == pkg["SPDXID"]
                push!(pkgRelationshipsToJulia, rel["relationshipType"])
            end
        end
    end
end
reviewTable= Table(Name= pkgNames, License= pkgLicenses, DepType= pkgRelationshipsToJulia, HomePage= pkgHomePages, Dowload= pkgDownloads)
open("SPDXreviewTable.md", "w") do f
    pretty_table(f, reviewTable; tf= tf_markdown)
end

Name	License	DepType	HomePage	Dowload
Pkg.jl	MIT	BUILD_DEPENDENCY_OF	https://julialang.org	git+https://github.com/JuliaLang/Pkg.jl.git
Statistics.jl	MIT	BUILD_DEPENDENCY_OF	https://julialang.org	git+https://github.com/JuliaLang/Statistics.jl.git
libCURL.jl	MIT	BUILD_DEPENDENCY_OF	https://julialang.org	git+https://github.com/JuliaWeb/LibCURL.jl.git
Downloads.jl	MIT	BUILD_DEPENDENCY_OF	https://julialang.org	git+https://github.com/JuliaLang/Downloads.jl.git
ArgTools.jl	MIT	BUILD_DEPENDENCY_OF	https://julialang.org	git+https://github.com/JuliaIO/ArgTools.jl.git
Tar.jl	MIT	BUILD_DEPENDENCY_OF	https://julialang.org	git+https://github.com/JuliaIO/Tar.jl.git
NetworkOptions.jl	MIT	BUILD_DEPENDENCY_OF	https://julialang.org	git+https://github.com/JuliaLang/NetworkOptions.jl.git
SuiteSparse.jl	MIT	BUILD_DEPENDENCY_OF	https://julialang.org	git+https://github.com/JuliaLang/SuiteSparse.jl.git
SHA.jl	MIT	BUILD_DEPENDENCY_OF	https://julialang.org	git+https://github.com/JuliaCrypto/SHA.jl.git
dSFMT	BSD-3-Clause	BUILD_DEPENDENCY_OF	https://github.com/MersenneTwister-Lab/dSFMT	git+https://github.com/MersenneTwister-Lab/dSFMT.git
OpenLibm	MIT	BUILD_DEPENDENCY_OF	https://julialang.org	git+https://github.com/JuliaMath/openlibm.git
GMP	LGPL-3.0-or-later	BUILD_DEPENDENCY_OF	https://gmplib.org/	https://gmplib.org/download/gmp/
libgit2	LicenseRef-GPL-2.0-only-with-libgit2-exception	BUILD_DEPENDENCY_OF	https://libgit2.org	git+https://github.com/libgit2/libgit2.git
curl	curl	BUILD_DEPENDENCY_OF	https://curl.se	git+https://github.com/curl/curl.git
libssh2	BSD-3-Clause	BUILD_DEPENDENCY_OF	https://www.libssh2.org	git+https://github.com/libssh2/libssh2.git
mbedtls	Apache-2.0	BUILD_DEPENDENCY_OF	https://tls.mbed.org	git+https://github.com/ARMmbed/mbedtls.git
mpfr	LGPL-3.0-or-later	BUILD_DEPENDENCY_OF	https://www.mpfr.org/	https://www.mpfr.org/
OpenBLAS	BSD-3-Clause	BUILD_DEPENDENCY_OF	https://www.openblas.net	git+https://github.com/xianyi/OpenBLAS.git
LAPACK	BSD-3-Clause	BUILD_DEPENDENCY_OF	https://netlib.org/	https://www.netlib.org/lapack/
PCRE	BSD-3-Clause	BUILD_DEPENDENCY_OF	https://www.pcre.org	https://ftp.pcre.org/pub/pcre/
LibSuiteSparse	GPL-2.0-or-later	BUILD_DEPENDENCY_OF	https://people.engr.tamu.edu/davis/suitesparse.html	git+https://github.com/DrTimothyAldenDavis/SuiteSparse.git
LibBlasTrampoline	MIT	BUILD_DEPENDENCY_OF	https://github.com/JuliaLinearAlgebra	git+https://github.com/JuliaLinearAlgebra/libblastrampoline.git
NGHTTP2	MIT	BUILD_DEPENDENCY_OF	https://nghttp2.org	git+https://github.com/nghttp2/nghttp2.git
libunwind	MIT	BUILD_DEPENDENCY_OF	http://www.nongnu.org/libunwind/	git+https://github.com/libunwind/libunwind.git
libuv	MIT	BUILD_DEPENDENCY_OF	https://libuv.org	git+https://github.com/JuliaLang/libuv.git
llvm	Apache-2.0 WITH LLVM-exception	BUILD_DEPENDENCY_OF	https://llvm.org	git+https://github.com/JuliaLang/llvm-project.git
utf8proc	MIT	BUILD_DEPENDENCY_OF	https://github.com/JuliaStrings/utf8proc	git+https://github.com/JuliaLang/utf8proc.git
7-Zip	LGPL-3.0-or-later	RUNTIME_DEPENDENCY_OF	https://www.7-zip.org	https://sourceforge.net/projects/p7zip/files/p7zip
zlib	Zlib	BUILD_DEPENDENCY_OF	https://zlib.net	git+https://github.com/madler/zlib.git
patchelf	GPL-3.0-or-later	BUILD_TOOL_OF	https://nixos.org/patchelf.html	git+https://github.com/NixOS/patchelf.git
objconv	GPL-3.0-or-later	BUILD_TOOL_OF	https://www.agner.org/optimize/#objconv	https://www.agner.org/optimize/objconv.zip
libwhich	MIT	BUILD_TOOL_OF	https://github.com/vtjnash/libwhich	git+https://github.com/vtjnash/libwhich.git

ViralBShah · 2021-12-10T11:09:03Z

Looks great. Let's have a few more eyeballs on this.

…file.

SamuraiAku · 2021-12-21T07:27:47Z

Added a script in contrib that allows for easy updating of the SPDX document with each release.

I think this is ready to merge.

ViralBShah · 2021-12-21T09:32:39Z

Addresses #35042 partially.

ViralBShah · 2022-01-07T08:12:47Z

I suggest moving the julia.spdx.json file to contrib. Any objection to that?

SamuraiAku · 2022-01-07T18:39:19Z

I would keep it at the top level. It's equivalent to THIRDPARTY.md, in a standardized format. Automated scanners may or may not find it if we move it down.

On a related note, is there anything we should do to make sure that the contrib/updateSPDX.jl script is run with every release? It makes the most sense to me to run the script in the same commit that updates VERSION.

vtjnash · 2022-01-07T22:59:07Z

Add it to the release process checklist (release-candidate) in the Makefile. Though it looks like @KristofferC or @staticfloat should update parts of those directions, since we don't upload the files manually anymore.

… reviews by user organizations (JuliaLang#42102) * Add an SPDX file to the repository. * New script contrib/updateSPDX.jl . Ran the script to update the SPDX file.

nalimilan · 2022-08-24T12:04:12Z

THIRDPARTY.md lists a few source files which are part of Julia but use licenses other than MIT. Shouldn't they be listed in the SPDX file? Otherwise it's misleading to say that Julia itself is MIT. I'd even argue that information about these files should be moved from THIRDPARTY.md to LICENSE.md, as they are not third-party libraries at all.

SamuraiAku · 2022-08-26T06:56:08Z

Adding the files and code snippets to the SPDX file should be done, but doing it properly seemed too hard for a first attempt. With the external dependencies it’s fairly easy to browse the deps directory and see what changed and update accordingly. Additional licenses in the source code have to be tracked manually which makes any updates or deletions more painful. But it can be done.

On overall licensing of Julia, unless it’s discovered that some GPL has been included in the Julia source, which I don’t think has been done, then all the Julia source would I think be considered MIT since it’s more permissive that the other licenses present and full credit is given. But the Julia binary is GPL since some of the external dependencies are GPL. Not an issue since all code is published. Only if someone wanted to roll proprietary improvements into Julia source and redistribute would that be an issue.

nalimilan · 2022-08-26T07:41:44Z

Unfortunately, the problem is precisely that MIT is more permissive than licenses used by at least some of the source files listed in THIRDPARTY.md. For example, the Zlib license adds the restriction that "The origin of this software must not be misrepresented" (it's not completely clear what this implies...). The LLVM licence clearly requires that "Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimers.". But the MIT license doesn't require that, so if Julia includes that file and yet claims that the whole codebase is MIT, it doesn't give the correct information for people who want to redistribute it. The fact that we have an SPDX file that doesn't mention these licences is dangerous for them.

EDIT: BTW, I think we are also not respecting these licences when building Julia tarballs, as they don't include a copy of the text of these licenses (only an hyperlink in THIRDPARTY.md), contrary to what they require. We should probably copy them to a directory.

ViralBShah · 2022-08-26T14:38:11Z

I think we need to open this as a separate issue. Agree on adding the text of the licenses if that is required by those licenses, and adding to the SPDX file. I believe the MIT license also requires it, since it has the clause: "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software." Of course we do include the MIT license.

StefanKarpinski · 2022-09-28T15:50:45Z

What needs to be done here? This issue is closed but the last few comments imply that some issue remains.

ViralBShah · 2022-09-28T16:51:54Z

Milan pointed out some things above that are part of Julia source and not third party.

Add an SPDX file to the repository. Incomplete at this point, only li…

4ed2221

…sts 6 packages

StefanKarpinski reviewed Sep 9, 2021

View reviewed changes

julia.spdx.json Show resolved Hide resolved

goneall suggested changes Sep 9, 2021

View reviewed changes

julia.spdx.json Outdated Show resolved Hide resolved

Correction to patchelf License in SPDX doc

1f2474f

goneall approved these changes Sep 10, 2021

View reviewed changes

SamuraiAku added 11 commits September 14, 2021 21:33

Merge remote-tracking branch 'origin/master' into add_SPDX

58efbd0

Merge remote-tracking branch 'origin/master' into add_SPDX

f776fa8

Add Statistics stdlib to SPDX file

4b6b2a4

Add LibCURL.jl stdlib to the SPDX file

47c2f77

Minor update to names of a few stdlib packages

c33e315

Add Downloads stlib to SPDX

6a70bdd

Add ArgTools stdlib to SPDX document

1bb78cb

Add stdlib Tar to the SPDX file

3a7844d

Added stdlib NetworkOptions to the SPDX document

db11989

Add SuiteSparse.jl to SPDX file

e19d580

Add SHA.jl to the SPDX document

06a8325

SamuraiAku added 3 commits September 20, 2021 21:01

Add libunwind to the SPDX doc

608e1cd

Add LLVM to SPDX file. In the course of adding this item, discovered …

39d2ff1

…that the LLVM license changed from UIUC to Apache back in v8. Corrected the license reference in THIRDPARTY.md which pointed to the v6 license and is no longer correct.

Add utf8proc to the SPDX document

a6cc2d3

SamuraiAku added 4 commits September 30, 2021 22:13

Add dSFMT to the SPDX file

858bda1

Add openlibm to the SPDX file

d6b1586

Add GMP to SPDX document

40a9c6e

Add libgit2 to the SPDX document

9909120

SamuraiAku added 3 commits October 5, 2021 22:20

Merge remote-tracking branch 'origin/master' into add_SPDX

8f08c1c

Add curl to SPDX file

f064cfd

Updates to LLVM in SPDX. Julia has forked LLVM and maintains their ow…

cd36d62

…n repository.

SamuraiAku added 2 commits November 10, 2021 22:53

Add NGHTTP2 to the SPDX document

14664b0

Merge remote-tracking branch 'origin/master' into add_SPDX

6ebd574

SamuraiAku added 3 commits November 24, 2021 23:35

Add objconv to the SPDX file

48e7898

Add libwhich to the SPDX doc

c5556db

Add 7zip to SPDX doc, plus cleanup in a few spots

e07cd46

A few corrections to the SPDX doc

dab807d

Fix zlib relationship in SPDX doc

0e06a72

New script contrib/updateSPDX.jl . Ran the script to update the SPDX …

7d7ec12

…file.

SamuraiAku changed the title ~~RFC: Add an SPDX file to the repository to streamline license and security reviews by user organizations~~ Add an SPDX file to the repository to streamline license and security reviews by user organizations Dec 21, 2021

SamuraiAku marked this pull request as ready for review December 21, 2021 07:25

ViralBShah merged commit 7cd1da3 into JuliaLang:master Dec 21, 2021

SamuraiAku mentioned this pull request Jan 8, 2022

Addition to release checklist in makefile #43709

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an SPDX file to the repository to streamline license and security reviews by user organizations #42102

Add an SPDX file to the repository to streamline license and security reviews by user organizations #42102

SamuraiAku commented Sep 3, 2021 •

edited

Loading

goneall left a comment

goneall left a comment

SamuraiAku commented Sep 20, 2021

SamuraiAku commented Sep 25, 2021

SamuraiAku commented Oct 4, 2021

goneall commented Oct 7, 2021

SamuraiAku commented Nov 13, 2021

SamuraiAku commented Nov 28, 2021

ViralBShah commented Dec 2, 2021

SamuraiAku commented Dec 10, 2021

ViralBShah commented Dec 10, 2021

SamuraiAku commented Dec 21, 2021

ViralBShah commented Dec 21, 2021

ViralBShah commented Jan 7, 2022

SamuraiAku commented Jan 7, 2022

vtjnash commented Jan 7, 2022 •

edited

Loading

nalimilan commented Aug 24, 2022

SamuraiAku commented Aug 26, 2022

nalimilan commented Aug 26, 2022 •

edited

Loading

ViralBShah commented Aug 26, 2022

StefanKarpinski commented Sep 28, 2022

ViralBShah commented Sep 28, 2022

Add an SPDX file to the repository to streamline license and security reviews by user organizations #42102

Add an SPDX file to the repository to streamline license and security reviews by user organizations #42102

Conversation

SamuraiAku commented Sep 3, 2021 • edited Loading

Summary

Background

Justification for this PR

Tracking Version Information

Updating the SPDX Document for each release

Status at initial posting

Next steps

goneall left a comment

Choose a reason for hiding this comment

goneall left a comment

Choose a reason for hiding this comment

SamuraiAku commented Sep 20, 2021

SamuraiAku commented Sep 25, 2021

SamuraiAku commented Oct 4, 2021

goneall commented Oct 7, 2021

SamuraiAku commented Nov 13, 2021

SamuraiAku commented Nov 28, 2021

ViralBShah commented Dec 2, 2021

SamuraiAku commented Dec 10, 2021

ViralBShah commented Dec 10, 2021

SamuraiAku commented Dec 21, 2021

ViralBShah commented Dec 21, 2021

ViralBShah commented Jan 7, 2022

SamuraiAku commented Jan 7, 2022

vtjnash commented Jan 7, 2022 • edited Loading

nalimilan commented Aug 24, 2022

SamuraiAku commented Aug 26, 2022

nalimilan commented Aug 26, 2022 • edited Loading

ViralBShah commented Aug 26, 2022

StefanKarpinski commented Sep 28, 2022

ViralBShah commented Sep 28, 2022

SamuraiAku commented Sep 3, 2021 •

edited

Loading

vtjnash commented Jan 7, 2022 •

edited

Loading

nalimilan commented Aug 26, 2022 •

edited

Loading