Improve BOM processing performance

[nscuro]

Description

This refactors BomUploadProcessingTask to improve its efficiency and consistency.

Notable changes:

• Components and services are now de-duplicated in-memory based on their ComponentIdentity before interacting with the database
  • Parsing and converting of the BOM is more performant because it doesn't require interaction with external services (e.g. database)
  • De-duplication takes BOM refs used for the dependency graph into consideration. If a component or service "falls victim" to de-duplication, its BOM ref is re-wired to point to the component it was deemed to duplicate instead. This keeps the dependency graph intact.
  • Addresses various issues in the original logic, e.g.
    • DependencyTrack hangs when uploading a large SBOM to a project a second time dependency-track#1905 (regression test included in this PR)
    • Error while uploading same SBOM Second time dependency-track#2131
    • Believe this is an issue ... findings change with each scan of same SBOM  dependency-track#2519 (regression test included in this PR)
    • Incompatibility with SBOM from @cyclonedx/cyclonedx-npm dependency-track#2265
• When correlating components from the uploaded BOM with components in the database, the correlation query is now more strict
  • See Believe this is an issue ... findings change with each scan of same SBOM  dependency-track#2519 (comment) for more context
• Processing now runs in a single database transaction
  • Contrasting previous logic which used many small transactions
  • Now, if BOM processing fails half-way, we don't leave broken state behind
  • Changes are regularly flushed to the database over the course of the transaction

Addressed Issue

Closes DependencyTrack/hyades#635

Additional Details

But does it perform better? Uh, yeah.

I compared this PR with the current snapshot build of hyades-apiserver, using the Hyades docker-compose.yml. Other Hyades services besides the API server have been turned off for this test.

I uploaded the same 1000 BOMs to each version. The high-level results are as follows:

API Server version	BOM Upload Start	BOM Upload End	BOM Processing End
This PR	13:55:00	13:59:06	13:59:07
Current snapshot	14:07:45	14:18:44	14:22:23

The API server version built from this PR took just about 4min for the entire operation, whereas the current snapshot took 15min.

System / JVM Statistics

Overall resource footprint is down. Less CPU and Heap usage.

As the Postgres container is running on the same machine, the System CPU usage graph being lower for this PR is a good sign as well. Less stress is being put on the database server.

This PR

snapshot

Database / ORM Statistics

Because all operations of BOM processing now run in a single transaction, the total number of transaction is noticeably down. Also, only the transactional connection pool is utilized. It can be observed that only at the very beginning there are a few connections in pending state.

In contrast, in the snapshot version, both connection pools are heavily utilized, and there's a constantly high number of pending connections on the non-transactional pool. The number of transactions is also insanely high.

This PR

snapshot

BOM Processing Duration

The average duration of BOM processing stays constantly in the single-digit second range for this PR, whereas for snapshot it even reaches up to almost 2min.

This PR

snapshot

Kafka Records Produced

Another nice indicator that things have improved: The number of records produced to Kafka per second have noticeably increased. This is just a side effect of BOM processing completing faster, but still nice to observe.

This PR

snapshot

Internal Event System

The laptop I am testing on has 16 CPU cores. DT will thus use a worker thread pool of up to 64 threads. Worker pool threads are responsible for processing BOMs, thus their usage is a good indicator of how performant the processing is.

With this PR, the number of workers used just barely reaches 20 shorty. Also, there were no Events Queued the entire time, indicating that BOMs were processed almost as soon as they were uploaded.

With snapshot, the pool was entirely saturated the whole time. And still events were queueing up, eventually reaching 250 yet-to-be-processed events.

This PR

snapshot

Checklist

• I have read and understand the contributing guidelines
• This PR fixes a defect, and I have provided tests to verify that the fix is effective
• This PR implements an enhancement, and I have provided tests to verify that it works as intended
• This PR introduces changes to the database model, and I have added corresponding update logic
• This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

[nscuro]

Note: This PR should be squash-merged so it's easier to revert if we have to.

[VithikaS]

It looks great!

[nscuro]

@VithikaS Could you please re-approve, given everything still looks OK? Had to resolve some merge conflicts by rebasing.

[mehab]

lgtm