Reachability (#1372)

.gitignore

@@ -29,10 +29,17 @@ fossa.telemetry.json
 # Integration Tests
 integration-test/artifacts/
 
+# macOs Finder
+.DS_Store
+
 # One offs
 test/App/Fossa/VSI/DynLinked/testdata/hello_standard*
 test/App/Fossa/VSI/DynLinked/testdata/hello_setuid*
 
 # Rust
 
 target/
+
+# Include targets in test
+!test/reachability/testdata/maven-default/target
+!test/reachability/testdata/maven-build-config/dist

Changelog.md

@@ -1,5 +1,9 @@
 # FOSSA CLI Changelog
 
+## v3.9.2
+- Maven: Adds reachability analysis [#1372](https://github.com/fossas/fossa-cli/pull/1377)
+- Gradle: Adds reachability analysis [#1377](https://github.com/fossas/fossa-cli/pull/1377)
+
 ## v3.9.1
 - `--detect-dynamic`: Safely ignores scenarios in ldd output parsing where we run into not found error. ([#1376](https://github.com/fossas/fossa-cli/pull/1376)) 
 

docs/contributing/reachability.md

@@ -0,0 +1,134 @@
+# Reachability
+
+### What is Reachability?
+Reachability Analysis is a security offering designed to enhance FOSSA's security analysis by providing context on vulnerable packages. It alleviates the constraints of traditional CVE assessments through the static analysis of application and dependency code, confirming the presence of vulnerable call paths. 
+
+### Limitations
+- Reachability currently supports all Maven and Gradle projects dynamically analyzed by fossa-cli. 
+- The target jar of the project must exist, prior to the analysis. If the jar artifact is not present, or FOSSA CLI fails to
+associate this jar with project, FOSSA CLI will not perform reachability analysis.
+- Reachability requires that `java` is present in PATH, and `java` version must be greater than `1.8` (jdk8+).
+
+For example, 
+- if you are using maven, you should run `mvn package` to ensure jar artifact exists, prior to running `fossa analyze`
+- if you are using gradle, you should run `gradlew build` to ensure jar artifact exists, prior to running `fossa analyze`
+
+### Maven Analysis
+
+For Maven projects, FOSSA CLI performs an analysis to infer dependencies. If FOSSA CLI identifies a complete dependency graph, which may include both direct and transitive dependencies, it attempts to infer the built JAR file for reachability analysis. It looks for `./target/{artifact}-{version}.jar` from the POM directory. If the POM file provides `build.directory` or `build.finalName` attributes, they are used instead of the default target jar path. For this reason, perform `fossa analyze` after the project of interest is built (g.g. `mvn package`), and target artifact exists in the directory.
+
+### Gradle Analysis
+
+For Gradle projects, FOSSA CLI invokes `./gradlew -I jsonpaths.gradle jsonPaths`. Where [jsonpaths.gradle](./../../scripts/jarpaths.gradle) is gradle script, which uses `java` plugin, and `jar` task associated with gradle to infer path of the built jar file. If neither of those are present, FOSSA CLI won't be able to identify jar artifacts for analysis.
+
+### How do I debug reachability from `fossa-cli`?
+
+```bash
+fossa analyze --debug
+
+cat fossa.debug.json | jq '.bundleReachabilityRaw'
+[
+  {
+    "callGraphAnalysis": {
+      "value": [
+        {
+          "parsedJarContent": {
+            "kind": "ContentRaw",
+            "value": "some value"
+          },
+          "parsedJarPath": "/Users/dev/example/example-projects/maven/example-maven-project/target/example-artifact-1.1.jar"
+        }
+      ],
+      "kind": "JarAnalysis"
+    },
+    "srcUnitDependencies": [
+      "mvn+com.fasterxml.jackson.core:jackson-databind$2.13.0",
+      "mvn+joda-time:joda-time$2.10.14",
+      "mvn+com.fasterxml.jackson.core:jackson-annotations$2.13.0",
+      "mvn+com.fasterxml.jackson.core:jackson-core$2.13.0"
+    ],
+    "srcUnitManifest": "/Users/dev/example/example-projects/maven/example-maven-project/",
+    "srcUnitName": "/Users/dev/example/example-projects/maven/example-maven-project/",
+    "srcUnitOriginPaths": [
+      "pom.xml"
+    ],
+    "srcUnitType": "maven"
+  },
+  {
+    "callGraphAnalysis": {
+      "tag": "NoCallGraphAnalysis"
+    },
+    "srcUnitDependencies": [],
+    "srcUnitManifest": "manifest",
+    "srcUnitName": "name",
+    "srcUnitOriginPaths": [],
+    "srcUnitType": "type"
+  }
+]
+
+
+cat fossa.debug.json | jq '.bundleReachabilityEndpoint'
+{
+ # content uploaded to endpoint
+}
+```
+
+<!-- 
+## How do I debug reachability from endpoint?
+
+```bash
+# get what we sent to endpoint
+cat fossa.debug.json | jq '.bundleReachabilityEndpoint' > rawReachabilityJob.json
+
+# run job in explain mode
+yarn repl
+explainReachability('rawReachabilityJob.json')
+
+# [1] I was provided 'rawReachabilityJob.json'
+# [2] I'm parsing file: 'rawReachabilityJob.json'
+# [3] I found [X] reachability units
+# [4] Working on [0] reachability unit
+# -- 
+# {
+#   ....  
+# }
+#
+```
+-->
+
+## F.A.Q. 
+
+1. What data from my codebase is uploaded to endpoint?
+
+We upload call graph of (caller, and callee) relationships, in which
+caller and callee are fully qualified symbol name. We do not upload source code.
+
+Here is example of caller, callee relationship that is uploaded to endpoint.
+
+```txt
+M:com.example.app.utils.ContextReader:<init>() (O)java.lang.Object:<init>()
+M:com.example.app.utils.ContextReader:parseWithCtx(java.net.URL) (O)java.io.File:<init>(java.lang.String)
+M:com.example.app.utils.ContextReader:parseWithCtx(java.net.URL) (S)com.google.common.io.Files:toString(java.io.File,java.nio.charset.Charset)
+M:com.example.app.utils.ContextReader:parseWithCtx(java.net.URL) (O)org.dom4j.jaxb.JAXBReader:<init>(java.lang.String)
+M:com.example.app.utils.ContextReader:parseWithCtx(java.net.URL) (M)org.dom4j.jaxb.JAXBReader:read(java.net.URL)
+M:com.example.app.App:<init>() (O)java.lang.Object:<init>()
+M:com.example.app.App:main(java.lang.String[]) (O)java.net.URI:<init>(java.lang.String)
+M:com.example.app.App:main(java.lang.String[]) (M)java.net.URI:toURL()
+M:com.example.app.App:main(java.lang.String[]) (S)com.example.app.App:parse(java.net.URL)
+M:com.example.app.App:main(java.lang.String[]) (M)java.io.PrintStream:println(java.lang.Object)
+M:com.example.app.App:main(java.lang.String[]) (S)com.example.app.utils.ContextReader:parseWithCtx(java.net.URL)
+M:com.example.app.App:main(java.lang.String[]) (M)java.io.PrintStream:println(java.lang.Object)
+M:com.example.app.App:parse(java.net.URL) (O)org.dom4j.io.SAXReader:<init>()
+M:com.example.app.App:parse(java.net.URL) (M)org.dom4j.io.SAXReader:read(java.net.URL)
+```
+
+You can inspect the data by running:
+
+```bash
+; fossa analyze --output --debug # --output to not communicate with endpoint
+; gunzip fossa.debug.json.gz     # extract produced debug bundle
+
+# content in .bundleReachabilityRaw is uploaded
+# to endpoint for reachability analysis.
+; cat fossa.debug.json | jq '.bundleReachabilityRaw'
+```

integration-test/Analysis/FixtureUtils.hs

@@ -10,12 +10,14 @@ module Analysis.FixtureUtils (
   TestC,
   performDiscoveryAndAnalyses,
   getArtifact,
+  testRunnerWithLogger,
+  withResult,
 ) where
 
 import App.Fossa.Analyze.Types (AnalyzeProject (analyzeProject))
 import App.Fossa.Config.Analyze (ExperimentalAnalyzeConfig (ExperimentalAnalyzeConfig), GoDynamicTactic (GoModulesBasedTactic))
 import App.Types (OverrideDynamicAnalysisBinary)
-import Control.Carrier.Debug (ignoreDebug)
+import Control.Carrier.Debug (IgnoreDebugC, ignoreDebug)
 import Control.Carrier.Diagnostics (DiagnosticsC, runDiagnostics)
 import Control.Carrier.Finally (FinallyC, runFinally)
 import Control.Carrier.Lift (Lift, sendIO)
@@ -109,6 +111,7 @@ data FixtureArtifact = FixtureArtifact
 type TestC m =
   ExecIOC
     $ ReadFSIOC
+    $ IgnoreDebugC
     $ DiagnosticsC
     $ LoggerC
     $ ReaderC OverrideDynamicAnalysisBinary
@@ -124,6 +127,7 @@ testRunnerWithLogger f env =
   f
     & runExecIOWithinEnv env
     & runReadFSIO
+    & ignoreDebug
     & runDiagnostics
     & withDefaultLogger SevWarn
     & runReader (mempty :: OverrideDynamicAnalysisBinary)

integration-test/Reachability/UploadSpec.hs

@@ -0,0 +1,179 @@
+{-# LANGUAGE QuasiQuotes #-}
+{-# LANGUAGE TemplateHaskell #-}
+
+module Reachability.UploadSpec (spec) where
+
+import Analysis.FixtureUtils (FixtureEnvironment (..), TestC, testRunnerWithLogger, withResult)
+import App.Fossa.Analyze.Project (ProjectResult (..))
+import App.Fossa.Analyze.Types (
+  AnalysisScanResult (..),
+  DiscoveredProjectIdentifier (..),
+  DiscoveredProjectScan (..),
+ )
+import App.Fossa.Reachability.Jar (callGraphFromJar)
+import App.Fossa.Reachability.Types (
+  CallGraphAnalysis (..),
+  ContentRef (..),
+  ParsedJar (..),
+  SourceUnitReachability (..),
+ )
+import App.Fossa.Reachability.Upload (
+  analyzeForReachability,
+  callGraphOf,
+ )
+import Data.ByteString.Lazy qualified as LB
+import Data.Foldable (for_)
+import Data.String.Conversion (toText)
+import Data.Text (Text)
+import Data.Text.Encoding qualified as TL
+import Diag.Result (Result (..))
+import Graphing (empty)
+import Path (
+  Abs,
+  Dir,
+  File,
+  Path,
+  Rel,
+  mkRelDir,
+  mkRelFile,
+  (</>),
+ )
+import Path.IO qualified as PIO
+import Test.Hspec (Spec, describe, it, runIO, shouldBe)
+import Text.RawString.QQ (r)
+import Types (
+  DiscoveredProjectType (MavenProjectType),
+  GraphBreadth (..),
+ )
+
+java8 :: FixtureEnvironment
+java8 = NixEnv ["jdk8"]
+
+java11 :: FixtureEnvironment
+java11 = NixEnv ["jdk11"]
+
+java17 :: FixtureEnvironment
+java17 = NixEnv ["jdk17"]
+
+java21 :: FixtureEnvironment
+java21 = NixEnv ["jdk21"]
+
+run :: FixtureEnvironment -> TestC IO a -> IO (Result a)
+run env act = testRunnerWithLogger act env
+
+spec :: Spec
+spec = describe "Reachability" $ do
+  describe "callGraphFromJar" $ do
+    jarFile <- runIO sampleJarFile
+    for_
+      [ java8
+      , java11
+      , java17
+      , java21
+      ]
+      $ \env -> do
+        it ("should compute in java: " <> show env) $ do
+          let expected = Just (sampleJarParsed jarFile)
+          res <- run env $ callGraphFromJar jarFile
+
+          withResult res $ \_ res' -> res' `shouldBe` expected
+
+  describe "callGraphOf" $ do
+    projDir <- (</> sampleMavenProjectDir) <$> runIO PIO.getCurrentDir
+    jarFile <- (</> sampleMavenProjectJar) <$> runIO PIO.getCurrentDir
+
+    it "should retrieve call graph" $ do
+      let expected = Just (mavenCompleteScanUnit projDir jarFile)
+      resp <- run java8 $ callGraphOf (mavenCompleteScan projDir)
+      withResult resp $ \_ res -> res `shouldBe` expected
+
+  describe "analyzeForReachability" $ do
+    projDir <- (</> sampleMavenProjectDir) <$> runIO PIO.getCurrentDir
+    jarFile <- (</> sampleMavenProjectJar) <$> runIO PIO.getCurrentDir
+
+    it "should return analyzed reachability unit" $ do
+      let expected = [mavenCompleteScanUnit projDir jarFile]
+      let analysisResult =
+            AnalysisScanResult
+              [mavenCompleteScan projDir]
+              successNothing
+              successNothing
+              successNothing
+              successNothing
+              successNothing
+
+      analyzed <- run java8 $ analyzeForReachability analysisResult
+      withResult analyzed $ \_ analyzed' -> analyzed' `shouldBe` expected
+
+sampleMavenProjectDir :: Path Rel Dir
+sampleMavenProjectDir = $(mkRelDir "test/Reachability/testdata/maven-default/")
+
+sampleMavenProjectJar :: Path Rel File
+sampleMavenProjectJar = $(mkRelFile "test/Reachability/testdata/maven-default/target/project-1.0.0.jar")
+
+mavenCompleteScan :: Path Abs Dir -> DiscoveredProjectScan
+mavenCompleteScan dir = mkDiscoveredProjectScan MavenProjectType dir Complete
+
+mavenCompleteScanUnit :: Path Abs Dir -> Path Abs File -> SourceUnitReachability
+mavenCompleteScanUnit projDir jarFile =
+  mkReachabilityUnit
+    projDir
+    [ ParsedJar
+        jarFile
+        (ContentRaw sampleJarParsedContent')
+    ]
+
+mkDiscoveredProjectScan :: DiscoveredProjectType -> Path Abs Dir -> GraphBreadth -> DiscoveredProjectScan
+mkDiscoveredProjectScan projectType dir breadth =
+  Scanned
+    (DiscoveredProjectIdentifier dir projectType)
+    ( Success
+        []
+        (ProjectResult projectType dir empty breadth mempty)
+    )
+
+sampleJarFile :: IO (Path Abs File)
+sampleJarFile = do
+  cwd <- PIO.getCurrentDir
+  pure (cwd </> $(mkRelFile "test/Reachability/testdata/sample.jar"))
+
+successNothing :: Result (Maybe a)
+successNothing = Success [] Nothing
+
+mkReachabilityUnit :: Path Abs Dir -> [ParsedJar] -> SourceUnitReachability
+mkReachabilityUnit dir jars =
+  SourceUnitReachability
+    "maven"
+    (toText dir)
+    (toText dir)
+    []
+    []
+    (JarAnalysis jars)
+
+sampleJarParsed :: Path Abs File -> ParsedJar
+sampleJarParsed path =
+  ParsedJar
+    { parsedJarPath = path
+    , parsedJarContent = ContentRaw sampleJarParsedContent'
+    }
+
+sampleJarParsedContent :: Text
+sampleJarParsedContent =
+  [r|C:vuln.project.sample.App java.lang.Object
+C:vuln.project.sample.App java.net.URI
+C:vuln.project.sample.App java.lang.System
+C:vuln.project.sample.App vuln.project.sample.App
+C:vuln.project.sample.App java.io.PrintStream
+C:vuln.project.sample.App org.dom4j.io.SAXReader
+C:vuln.project.sample.App java.lang.Exception
+C:vuln.project.sample.App org.dom4j.DocumentException
+M:vuln.project.sample.App:<init>() (O)java.lang.Object:<init>()
+M:vuln.project.sample.App:main(java.lang.String[]) (O)java.net.URI:<init>(java.lang.String)
+M:vuln.project.sample.App:main(java.lang.String[]) (M)java.net.URI:toURL()
+M:vuln.project.sample.App:main(java.lang.String[]) (S)vuln.project.sample.App:parse(java.net.URL)
+M:vuln.project.sample.App:main(java.lang.String[]) (M)java.io.PrintStream:println(java.lang.Object)
+M:vuln.project.sample.App:parse(java.net.URL) (O)org.dom4j.io.SAXReader:<init>()
+M:vuln.project.sample.App:parse(java.net.URL) (M)org.dom4j.io.SAXReader:read(java.net.URL)|]
+
+sampleJarParsedContent' :: LB.ByteString
+sampleJarParsedContent' = LB.fromStrict . TL.encodeUtf8 $ sampleJarParsedContent