Add new command to "diff" two BOM versions and produce JSON Patch output (RFC 6902)

.vscode/settings.json

@@ -36,6 +36,7 @@
             "Freetype",
             "Gladman",
             "Globus",
+            "gojsondiff",
             "gojsonschema",
             "gomod",
             "GTPL",
@@ -47,6 +48,7 @@
             "Jaxen",
             "JDBM",
             "JDOM",
+            "jsondiff",
             "jwangsadinata",
             "Jython",
             "LARAVEL",
@@ -60,6 +62,7 @@
             "multimap",
             "myservices",
             "NOASSERTION",
+            "nosec",
             "NTIA",
             "Nyffenegger",
             "OPENCHAIN",

README.md

@@ -34,16 +34,18 @@ The utility supports the following commands:
 - [Installation](#installation)
 - [Running](#running)
 - [Commands](#commands)
-  - [Exit codes](#exit-codes): `0` == no error, `1` == app.
-  - [Helpful flags](#helpful-flags)
-  - [license](#license)
-    - [list](#license-list-subcommand) subcommand
-    - [policy](#license-policy-subcommand) subcommand
-  - [query](#query)
-  - [resource](#resource)
-  - [schema](#schema)
-  - [vulnerability](#vulnerability)
-  - [validate](#validate)
+  - [General information](#general-command-information)
+    - [Exit codes](#exit-codes): (e.g., `0`: none, `1`: application, `2`: validation)
+    - [Persistent flags](#persistent-flags) (e.g., `--format`, `--quiet`, `--where`)
+  - [`license` command](#license)
+    - [list](#license-list-subcommand) subcommand: lists all license information found in the BOM
+    - [policy](#license-policy-subcommand) subcommand: lists configurable license usage policies
+  - [`query` command](#query): extract JSON objects and fields from a BOM using SQL-like queries
+  - [`resource` command](#resource): list resource information by type (e.g., components, services)
+  - [`schema` command](#schema): list supported BOM formats, versions, variants
+  - [`validate` command](#validate): BOM against declared or required schema
+  - [`vulnerability` command](#vulnerability): lists vulnerability summary information included in the BOM or VEX
+  - [`diff` command](#diff): *experimental*: shows the delta between two BOM versions
 - [Design considerations](#design-considerations)
 - [Development](#development)
   - [Prerequisites](#prerequisites)
@@ -110,7 +112,9 @@ For convenience, links to each command's section are here:
 - [validate](#validate)
 - [help](#help)
 
-### Exit codes
+### General command information
+
+#### Exit codes
 
 All commands return a numeric exit code (i.e., a POSIX exit code) for use in automated processing where `0` indicates success and a non-zero value indicates failure of some kind designated by the number.
 
@@ -120,7 +124,7 @@ The SBOM Utility always returns one of these 3 codes to accommodate logic in BAS
 - `1`= application error
 - `2`= validation error
 
-#### Example: exit code
+##### Example: exit code
 
 This example uses the `schema` list command to verify its exit code:
 
@@ -802,9 +806,9 @@ The following flags can be used to improve performance when formatting error out
 
 Use the `--error-limit x` flag to reduce the formatted error result output to the first `x` errors.  By default, only the first 10 errors are output with an informational messaging indicating `x/y` errors were shown.
 
-##### `--error-colorize` flag
+##### `--colorize` flag
 
-Use the `--error-colorize=true|false` flag to not add color formatting to error result output.  By default, formatted error output is colorized to help with human readability; for automated use, it can be turned off.
+Use the `--colorize=true|false` flag to add/remove color formatting to error result output.  By default, formatted error output is colorized to help with human readability; for automated use, it can be turned off.
 
 #### Validate Examples
 
@@ -984,6 +988,60 @@ CVE-2020-25649           611      CVSSv31: 7.5 (high), CVSSv31: 8.2 (high), CVSS
 
 ---
 
+### Diff
+
+This *experimental* command will compare two BOMs and return the delta (or "diff") in JSON (diff-patch format) or text.
+
+**Notes**
+
+- This command is undergoing analysis and tests which are exposing some underlying issues around "moved" objects in dependent diff-patch packages that may not be fixable and have no alternatives.
+  - *Specifically, the means by which "moved" objects are assigned "similarity" scores appears flawed in the case of JSON.*
+  - *Additionally, some of the underlying code relies upon Go maps which do not preserve key ordering.*
+
+#### Diff supported output formats
+
+Use the `--format` flag on the to choose one of the supported output formats:
+
+- txt (default), json
+
+#### Diff Examples
+
+##### Example: Add, delete and modify
+
+```bash
+./sbom-utility diff -i test/diff/json-array-order-change-with-add-and-delete-base.json -r test/diff/json-array-order-change-with-add-and-delete-delta.json --quiet --format txt --colorize=true
+```
+
+```bash
+ {
+   "licenses": [
+     0: {
+       "license": {
+-        "id": "Apache-1.0"
++        "id": "GPL-2.0"
+       }
+     },
+-+    2=>1: {
+-+      "license": {
+-+        "id": "GPL-3.0-only"
+-+      }
+-+    },
+     2: {
+       "license": {
+         "id": "GPL-3.0-only"
+       }
+     },
+     3: {
+       "license": {
+         "id": "MIT"
+       }
+     }
+   ]
+ }
+```
+
+---
+
 ### Help
 
 The utility supports the `help` command for the root command as well as any supported commands
@@ -1034,7 +1092,7 @@ In the future, we envision support for additional kinds of BOMs (e.g., Hardware
 
 #### Prerequisites
 
-- Go v1.18 or higher: see [https://go.dev/doc/install](https://go.dev/doc/install)
+- Go v1.20.1 or higher: see [https://go.dev/doc/install](https://go.dev/doc/install)
 - `git` client: see [https://git-scm.com/downloads](https://git-scm.com/downloads)
 
 #### Building
@@ -1253,7 +1311,7 @@ go test github.com/CycloneDX/sbom-utility/cmd -v --quiet
 run an individual test within the `cmd` package:
 
 ```bash
-go test github.com/CycloneDX/sbom-utility/cmd -v -run TestCdx13MinRequiredBasic
+go test github.com/CycloneDX/sbom-utility/cmd -v -run TestValidateCdx14MinRequiredBasic
 ```
 
 #### Debugging go tests

cmd/diff.go

@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package cmd
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"strings"
+
+	"github.com/CycloneDX/sbom-utility/utils"
+	diff "github.com/mrutkows/go-jsondiff"
+	"github.com/mrutkows/go-jsondiff/formatter"
+	"github.com/spf13/cobra"
+)
+
+// Command help formatting
+const (
+	FLAG_DIFF_OUTPUT_FORMAT_HELP = "format output using the specified type"
+)
+
+var DIFF_OUTPUT_SUPPORTED_FORMATS = MSG_SUPPORTED_OUTPUT_FORMATS_HELP +
+	strings.Join([]string{FORMAT_TEXT, FORMAT_JSON}, ", ")
+
+// validation flags
+const (
+	FLAG_DIFF_FILENAME_REVISION       = "input-revision"
+	FLAG_DIFF_FILENAME_REVISION_SHORT = "r"
+	MSG_FLAG_INPUT_REVISION           = "input filename for the revised file to compare against the base file"
+	MSG_FLAG_DIFF_COLORIZE            = "Colorize diff text output (true|false); default false"
+)
+
+func NewCommandDiff() *cobra.Command {
+	var command = new(cobra.Command)
+	command.Use = CMD_USAGE_DIFF
+	command.Short = "Report on differences between two BOM files using RFC 6902 format"
+	command.Long = "Report on differences between two BOM files using RFC 6902 format"
+	command.Flags().StringVarP(&utils.GlobalFlags.OutputFormat, FLAG_FILE_OUTPUT_FORMAT, "", FORMAT_TEXT,
+		FLAG_DIFF_OUTPUT_FORMAT_HELP+DIFF_OUTPUT_SUPPORTED_FORMATS)
+	command.Flags().StringVarP(&utils.GlobalFlags.DiffFlags.RevisedFile,
+		FLAG_DIFF_FILENAME_REVISION,
+		FLAG_DIFF_FILENAME_REVISION_SHORT,
+		"", // no default value (empty)
+		MSG_FLAG_INPUT_REVISION)
+	command.Flags().BoolVarP(&utils.GlobalFlags.DiffFlags.Colorize, FLAG_COLORIZE_OUTPUT, "", false, MSG_FLAG_DIFF_COLORIZE)
+	command.RunE = diffCmdImpl
+	command.PreRunE = func(cmd *cobra.Command, args []string) (err error) {
+		// Test for required flags (parameters)
+		err = preRunTestForFiles(cmd, args)
+
+		return
+	}
+	return command
+}
+
+func preRunTestForFiles(cmd *cobra.Command, args []string) error {
+	getLogger().Enter()
+	defer getLogger().Exit()
+	getLogger().Tracef("args: %v", args)
+
+	// Make sure the base (input) file is present and exists
+	baseFilename := utils.GlobalFlags.InputFile
+	if baseFilename == "" {
+		return getLogger().Errorf("Missing required argument(s): %s", FLAG_FILENAME_INPUT)
+	} else if _, err := os.Stat(baseFilename); err != nil {
+		return getLogger().Errorf("File not found: `%s`", baseFilename)
+	}
+
+	// Make sure the revision file is present and exists
+	revisedFilename := utils.GlobalFlags.DiffFlags.RevisedFile
+	if revisedFilename == "" {
+		return getLogger().Errorf("Missing required argument(s): %s", FLAG_DIFF_FILENAME_REVISION)
+	} else if _, err := os.Stat(revisedFilename); err != nil {
+		return getLogger().Errorf("File not found: `%s`", revisedFilename)
+	}
+
+	return nil
+}
+
+func diffCmdImpl(cmd *cobra.Command, args []string) (err error) {
+	getLogger().Enter(args)
+	defer getLogger().Exit()
+
+	// Create output writer
+	outputFile, writer, err := createOutputFile(utils.GlobalFlags.OutputFile)
+	getLogger().Tracef("outputFile: `%v`; writer: `%v`", outputFile, writer)
+
+	// use function closure to assure consistent error output based upon error type
+	defer func() {
+		// always close the output file
+		if outputFile != nil {
+			err = outputFile.Close()
+			if err != nil {
+				return
+			}
+			getLogger().Infof("Closed output file: `%s`", utils.GlobalFlags.OutputFile)
+		}
+	}()
+
+	err = Diff(utils.GlobalFlags)
+
+	return
+}
+
+func Diff(flags utils.CommandFlags) (err error) {
+	getLogger().Enter()
+	defer getLogger().Exit()
+
+	// create locals
+	format := utils.GlobalFlags.OutputFormat
+	baseFilename := utils.GlobalFlags.InputFile
+	outputFilename := utils.GlobalFlags.OutputFile
+	outputFormat := utils.GlobalFlags.OutputFormat
+	deltaFilename := utils.GlobalFlags.DiffFlags.RevisedFile
+	deltaColorize := utils.GlobalFlags.DiffFlags.Colorize
+
+	// Create output writer
+	outputFile, output, err := createOutputFile(outputFilename)
+
+	// use function closure to assure consistent error output based upon error type
+	defer func() {
+		// always close the output file
+		if outputFile != nil {
+			err = outputFile.Close()
+			getLogger().Infof("Closed output file: `%s`", utils.GlobalFlags.OutputFile)
+		}
+	}()
+
+	getLogger().Infof("Reading file (--input-file): `%s` ...", baseFilename)
+	// #nosec G304 (suppress warning)
+	bBaseData, errReadBase := ioutil.ReadFile(baseFilename)
+	if errReadBase != nil {
+		getLogger().Debugf("%v", bBaseData[:255])
+		err = getLogger().Errorf("Failed to ReadFile '%s': %s\n", utils.GlobalFlags.InputFile, err.Error())
+		return
+	}
+
+	getLogger().Infof("Reading file (--input-revision): `%s` ...", deltaFilename)
+	// #nosec G304 (suppress warning)
+	bRevisedData, errReadDelta := ioutil.ReadFile(deltaFilename)
+	if errReadDelta != nil {
+		getLogger().Debugf("%v", bRevisedData[:255])
+		err = getLogger().Errorf("Failed to ReadFile '%s': %s\n", utils.GlobalFlags.InputFile, err.Error())
+		return
+	}
+
+	// Compare the base with the revision
+	differ := diff.New()
+	getLogger().Infof("Comparing files: `%s` (base) to `%s` (revised) ...", baseFilename, deltaFilename)
+	d, err := differ.Compare(bBaseData, bRevisedData)
+	if err != nil {
+		err = getLogger().Errorf("Failed to Compare data: %s\n", err.Error())
+	}
+
+	// Output the result
+	var diffString string
+	if d.Modified() {
+		getLogger().Infof("Outputting listing (`%s` format)...", format)
+		switch outputFormat {
+		case FORMAT_TEXT:
+			var aJson map[string]interface{}
+			err = json.Unmarshal(bBaseData, &aJson)
+
+			if err != nil {
+				err = getLogger().Errorf("json.Unmarshal() failed '%s': %s\n", utils.GlobalFlags.InputFile, err.Error())
+				return
+			}
+
+			config := formatter.AsciiFormatterConfig{
+				ShowArrayIndex: true,
+			}
+			config.Coloring = deltaColorize
+			formatter := formatter.NewAsciiFormatter(aJson, config)
+			diffString, err = formatter.Format(d)
+		case FORMAT_JSON:
+			formatter := formatter.NewDeltaFormatter()
+			diffString, err = formatter.Format(d)
+			// Note: JSON data files MUST ends in a newline as this is a POSIX standard
+		default:
+			// Default to Text output for anything else (set as flag default)
+			getLogger().Warningf("Diff output format not supported for `%s` format.", format)
+		}
+
+		fmt.Fprintf(output, "%s\n", diffString)
+
+	} else {
+		getLogger().Infof("No deltas found. baseFilename: `%s`, revisedFilename=`%s` match.",
+			utils.GlobalFlags.InputFile,
+			utils.GlobalFlags.DiffFlags.RevisedFile)
+	}
+
+	return
+}