Linux (OpenJDK 10) | Windows (Oracle JDK 9) | JitPack (OpenJDK 8) |
---|---|---|
Interact with us! |
---|
The OSS Review Toolkit (ORT) assists with verifying Free and Open Source Software license compliance by checking a project's source code and its dependencies.
From a bird's eye, it works by analyzing the project's build system for dependencies, downloading the source code of the dependencies, scanning all source code for license information, and summarizing the results.
The different tools that make up ORT are designed as libraries (for programmatic use), with a minimal command line interface (for scripted use).
The toolkit consists of the following tools:
- Analyzer - determines dependencies of a project. Supports multiple package managers and sub-projects. No changes to the projects are required.
- Downloader - fetches the source code referred to by the Analyzer result.
- Scanner - wraps existing license / copyright scanners to detect findings in local source code directories.
- Evaluator - evaluates license findings an created customizable results or follow-up actions using a rules DSL based on Kotlin.
- Reporter - presents results in various formats like visual reports, compliance documents or Bill-Of-Materials (BOMs) to easily identify dependencies, licenses, copyrights or policy violations.
The following tools are planned but not yet available:
- Advisor - retrieves security advisories based on the Analyzer result.
- Documenter - generates the final outcome of the review process incl. legal conclusions, e.g. annotated SPDX files that can be included into the distribution.
Preliminary binary artifacts for ORT are currently available via JitPack. Please note that due to limitations with the JitPack build environment, the reporter is not able to create the Web App report.
Install the following basic prerequisites:
- Git (any recent version will do).
Then clone this repository. If you intend to run tests, you need to clone with submodules by running
git clone --recurse-submodules
. If you have already cloned non-recursively, you can initialize submodules afterwards
by running git submodule update --init --recursive
.
Install the following basic prerequisites:
- Docker (and ensure its daemon is running).
Change into the created directory and run docker/build.sh
.
Install these additional prerequisites:
- OpenJDK 8 or Oracle JDK 8u161 or later (not the JRE as you need the
javac
compiler); also remember to set theJAVA_HOME
environment variable accordingly. - For the Web App reporter:
Change into the created directory and run ./gradlew installDist
(on the first run this will bootstrap Gradle and
download all required dependencies).
ORT can now be run using
./cli/build/install/ort/bin/ort --help
Note that if you make any changes to ORT's source code, you would have to regenerate the distribution using either the Build using Docker or Build natively steps above.
To avoid that, you can also build and run ORT in one go (if you have the prerequisites from the Build natively section installed):
./gradlew cli:run --args="--help"
Note that in this case the working directory used by ORT is that of the cli
project, not directory gradlew
is
located in (see gradle/gradle#6074).
Like for building ORT from sources you have the option to run ORT from a Docker image (which comes with all runtime dependencies) or to run ORT natively (in which case some additional requirements need to be fulfilled).
Run docker/run.sh "<DOCKER_ARGS>" <ORT_ARGS>
where <DOCKER_ARGS>
are passed to docker run
(and need to be quoted
if spaces are contained) and <ORT_ARGS>
are passed to ORT. You typically use <DOCKER_ARGS>
to mount the project
directory to scan into the running container to let ORT access it, for example:
docker/run.sh "-v /workspace:/project" --info analyze -f JSON -i /project -o /project/ort/analyzer
First of all, make sure that the locale of your system is set to en_US.UTF-8
as using other locales might lead to
issues with parsing the output of some external tools.
Then install any missing external command line tools as listed by
./cli/build/install/ort/bin/ort requirements
or
./gradlew cli:run --args="requirements"
Then run ORT like
./cli/build/install/ort/bin/ort --info analyze -f JSON -i /project -o /project/ort/analyzer
or
./gradlew cli:run --args="--info analyze -f JSON -i /project -o /project/ort/analyzer"
A basic ORT pipeline (using the analyzer, scanner and reporter) can easily be run on Jenkins CI by using the Jenkinsfile in a (declarative) pipeline job.
Please see GettingStarted.md for an introduction to the individual tools.
Please see Configuration.md for details about the ORT configuration.
The Analyzer is a Software Composition Analysis (SCA) tool that determines the dependencies of software projects inside
the specified input directory (-i
). It does so by querying the detected package managers; no modifications to your
existing project source code, like applying build system plugins, are necessary for that to work. The tree of transitive
dependencies per project is written out as part of an
OrtResult in YAML (or
JSON, see -f
) format to a file named analyzer-result.yml
in the specified output directory (-o
). The output file
exactly documents the status quo of all package-related meta-data. It can be further processed or manually edited before
passing it to one of the other tools.
Currently, the following package managers are supported:
- Bower (JavaScript)
- Bundler (Ruby)
- Cargo (Rust)
- Conan (C / C++)
- dep (Go)
- DotNet (.NET, with currently some limitations)
- Glide (Go)
- Godep (Go)
- Gradle (Java)
- Maven (Java)
- NPM (Node.js)
- NuGet (.NET, with currently some limitations)
- Composer (PHP)
- PIP (Python)
- Pipenv (Python)
- Pub (Dart / Flutter)
- SBT (Scala)
- Stack (Haskell)
- Yarn (Node.js)
Taking an ORT result file with an analyzer result as the input (-a
), the Downloader retrieves the source code of all
contained packages to the specified output directory (-o
). The Downloader takes care of things like normalizing URLs
and using the appropriate VCS tool to checkout source code from version control.
Currently, the following Version Control Systems are supported:
This tool wraps underlying license / copyright scanners with a common API so all supported scanners can be used in the
same way to easily run them and compare their results. If passed an ORT result file with an analyzer result (-a
), the
Scanner will automatically download the sources of the dependencies via the Downloader and scan them afterwards.
Currently, the following license scanners are supported:
For a comparison of some of these, see this Bachelor Thesis.
In order to not download or scan any previously scanned sources again, the Scanner can use a storage backend to store scan results for later reuse.
By default the Scanner stores scan results on the local file system in the current user's home directory (i.e.
~/.ort/scanner/scan-results
) for later reuse. The storage directory can be customized by passing an ORT configuration
file (-c
) that contains a respective local file storage configuration:
ort {
scanner {
fileBasedStorage {
backend {
localFileStorage {
directory = "/tmp/ort/scan-results"
}
}
}
}
}
Any HTTP file server can be used to store scan results. Custom headers can be configured to provide authentication credentials. For example, to use Artifactory to store scan results, use the following configuration:
ort {
scanner {
fileBasedStorage {
backend {
httpFileStorage {
url = "https://artifactory.domain.com/artifactory/repository/scan-results"
headers {
X-JFrog-Art-Api = "api-token"
}
}
}
}
}
}
To use PostgreSQL to store scan results, use the following configuration:
ort {
scanner {
postgresStorage {
url = "jdbc:postgresql://example.com:5444/database"
schema = "schema"
username = "username"
password = "password"
}
}
}
The scanner creates a table called scan_results
and stores the data in a
jsonb column.
The Evaluator is used to perform custom license policy checks on scan results. The rules to check against are implemented via scripting. Currently, Kotlin script with a dedicated DSL is used for that, but support for other scripting languages can be added as well. See no_gpl_declared.kts for a very simple example of a rule written in Kotlin script which verifies that no dependencies that declare the GPL are used.
The reporter generates human-readable reports from the scan result file generated by the scanner (-s
). It is designed
to support multiple output formats.
Currently, the following report formats are supported (reporter names are case-insensitive):
- CycloneDX BOM (
-f CycloneDx
) - Excel sheet (
-f Excel
) - NOTICE file (
-f Notice
) - Static HTML (
-f StaticHtml
) - Web App (
-f WebApp
)
ORT is written in Kotlin and uses Gradle as the build system, with Kotlin script instead of Groovy as the DSL.
When developing on the command line, use the committed Gradle wrapper to bootstrap Gradle in the configured version and execute any given tasks. The most important tasks for this project are:
Task | Purpose |
---|---|
assemble | Build the JAR artifacts for all projects |
detekt | Run static code analysis on all projects |
test | Run unit tests for all projects |
funTest | Run functional tests for all projects |
installDist | Build all projects and install the start scripts for distribution |
All contributions need to pass the detekt
, test
and funTest
checks before they can be merged.
For IDE development we recommend the IntelliJ IDEA Community Edition which can directly import the Gradle build files. After cloning the project's source code recursively, simply run IDEA and use the following steps to import the project.
-
From the wizard dialog: Select Import Project.
From a running IDEA instance: Select File -> New -> Project from Existing Sources...
-
Browse to ORT's source code directory and select either the
build.gradle.kts
or thesettings.gradle.kts
file. -
In the Import Project from Gradle dialog select Use auto-import and leave all other settings at their defaults.
To set up a basic run configuration for debugging, navigate to Main.kt
in the cli
module and look for the
fun main(args: Array<String>)
function. In the gutter next to it, a green "Play" icon should be displayed. Click on it
and select Run 'com.here.ort.Main'
to run the entry point, which implicitly creates a run configuration. Double-check
that running ORT without any arguments will simply show the command line help in IDEA's Run tool window. Finally, edit
the created run configuration to your needs, e.g. by adding an argument and options to run a specific ORT sub-command.
Copyright (C) 2017-2019 HERE Europe B.V.
See the LICENSE file in the root of this project for license details.