Implement test to detect non-open source licenses

[MichaelSimons]

This is related to #2359. Source-build should have it's own test to catch/prevent non-open source licensed code.

[MichaelSimons]

[Triage] This is not a trivial task. There are a lot of intricacies in scanning for non-permissive licenses and the scanning itself will be time intensive.

From @omajid: Existing tools to scan: https://github.com/nexB/scancode-toolkit

[omajid]

Debian has some recommendations too https://wiki.debian.org/CopyrightReviewTools

[mthalman]

For scancode toolkit, if there end up being custom licenses we've identified as disallowed, we may be able to register them to be detected by the tool. See Humanizr/sample-aspnetmvc#1 (comment)

[mthalman]

@omajid - Since you've made use Scancode Toolkit, I wanted to ask a question on what settings you use for that? Have you been able to run that tool across the whole VMR in one command? I'm trying to do so and it's been running for 4 hrs now and I suspect there's an issue that is causing it to never finish exit.

[omajid]

I haven't used scancode directly. We have an internal service that wraps it, so I haven't touched the exact configuration used.

Looking at the log from a recent run, I see:

[2023-08-02 00:24:57,407] [INFO] Start to scan source licenses with Scancode...
[2023-08-02 14:07:39,739] [ERROR] dotnet-004acac.tar.gz-extract/src/razor/NOTICE.txt: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,750] [ERROR] dotnet-004acac.tar.gz-extract/src/roslyn/src/Compilers/CSharp/Test/Semantic/Semantics/NullableReferenceTypesTests.cs: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,751] [ERROR] dotnet-004acac.tar.gz-extract/src/runtime/src/tests/JIT/jit64/opt/cse/HugeArray1.cs: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,751] [ERROR] dotnet-004acac.tar.gz-extract/src/runtime/src/tests/JIT/jit64/opt/cse/hugeexpr1.cs: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,752] [ERROR] dotnet-004acac.tar.gz-extract/src/runtime/src/tests/JIT/Regression/VS-ia64-JIT/V1.2-M02/b28158/b28158.il: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,753] [ERROR] dotnet-004acac.tar.gz-extract/src/source-build-reference-packages/src/targetPacks/ILsrc/microsoft.netcore.app.ref/5.0.0/ref/net5.0/System.Runtime.il: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,754] [ERROR] dotnet-004acac.tar.gz-extract/src/source-build-reference-packages/src/targetPacks/ILsrc/microsoft.netcore.app.ref/6.0.0/ref/net6.0/System.Runtime.il: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,754] [ERROR] dotnet-004acac.tar.gz-extract/src/source-build-reference-packages/src/targetPacks/ILsrc/microsoft.netcore.app.ref/7.0.0/ref/net7.0/System.Runtime.il: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,755] [ERROR] dotnet-004acac.tar.gz-extract/src/source-build-reference-packages/src/targetPacks/ILsrc/netstandard.library/2.0.1/build/netstandard2.0/ref/netstandard.il: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,756] [ERROR] dotnet-004acac.tar.gz-extract/src/source-build-reference-packages/src/targetPacks/ILsrc/netstandard.library/2.0.3/build/netstandard2.0/ref/netstandard.il: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,757] [ERROR] dotnet-004acac.tar.gz-extract/src/source-build-reference-packages/src/targetPacks/ILsrc/netstandard.library.ref/2.1.0/ref/netstandard2.1/netstandard.il: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,758] [ERROR] dotnet-004acac.tar.gz-extract/src/source-build-reference-packages/src/targetPacks/ILsrc/netstandard.library.ref/2.1.0/ref/netstandard2.1/netstandard.xml: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:39,759] [ERROR] dotnet-004acac.tar.gz-extract/src/source-build-reference-packages/src/textOnlyPackages/src/microsoft.private.intellisense/7.0.0-preview-20221010.1/IntellisenseFiles/netstandard/1033/netstandard.xml: [u'ERROR: for scanner: licenses:\nERROR: Processing interrupted: timeout after 300 seconds.']
[2023-08-02 14:07:40,019] [INFO] Done

Looks like it takes ~14 hours....

[mthalman]

Looks like it takes ~14 hours....

(Cries in compute cost.)

[mthalman]

Fixed by dotnet/installer#17442