Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accurate deps #498

Merged
merged 37 commits into from
Oct 27, 2023
Merged
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
11fc031
Add `cargo-metadata` as a dependency
Shnatsel Oct 25, 2023
6a85dba
Drop logging configuration for Cargo internals
Shnatsel Oct 25, 2023
79a5c52
Drop #[deny] directives for warnings and lints. These should not be u…
Shnatsel Oct 25, 2023
1c29f67
Implement querying `cargo metadata`
Shnatsel Oct 25, 2023
85225dc
Drop some more #[deny] directives that would break production builds
Shnatsel Oct 25, 2023
a358ce4
WIP conversion of create_sboms() to cargo-metadata
Shnatsel Oct 25, 2023
f7d240c
convert create_sboms() and create_bom() to cargo-metadata
Shnatsel Oct 26, 2023
9163729
Convert the rest of the generator functions to cargo-metadata
Shnatsel Oct 26, 2023
efc37f2
Comment out toplevel/all dependency filtering for now
Shnatsel Oct 26, 2023
b3f9ad4
BEHOLD, IT COMPILES
Shnatsel Oct 26, 2023
dbd3367
Purge the last reference to Cargo internals
Shnatsel Oct 26, 2023
7b52d4a
Remove Cargo from the dependency tree
Shnatsel Oct 26, 2023
c398423
Re-enable reading config from Cargo.toml, even though this seems like…
Shnatsel Oct 26, 2023
7342217
Do not assert any specific error message on invalid Cargo.toml in tes…
Shnatsel Oct 26, 2023
0bdd508
Add a comment explaining that the configuration does not work correct…
Shnatsel Oct 26, 2023
e519aef
cargo fmt
Shnatsel Oct 26, 2023
1b7372e
Reimplement filtering of top-level dependencies
Shnatsel Oct 26, 2023
edc3583
Also reimplement all_dependencies() with cargo-metadata
Shnatsel Oct 26, 2023
e0a4cb5
Make top_level_dependencies() consistent with all_dependencies() in o…
Shnatsel Oct 26, 2023
d61cc75
Re-enable selection of direct deps only or all deps
Shnatsel Oct 26, 2023
0623190
Drop unused import
Shnatsel Oct 26, 2023
ade4077
Include the root package in the output of top_level_dependencies() an…
Shnatsel Oct 26, 2023
39b3426
Fix a test that was creating an invalid dependency without a lib targ…
Shnatsel Oct 26, 2023
416914d
Suppress the warning about resolve being unused for now
Shnatsel Oct 26, 2023
6a731f5
Omit the toplevel package from the list of components
Shnatsel Oct 26, 2023
b13338e
top_level_dependencies(): exclude dev-dependencies
Shnatsel Oct 26, 2023
25eb095
Split filtering out dev-dependencies into its own function
Shnatsel Oct 26, 2023
f14932d
Split stripping dev-dependencies from a Node into a helper function
Shnatsel Oct 26, 2023
5c1ae6d
rename 'member' variable to 'root' for clarity
Shnatsel Oct 26, 2023
5f4ebd5
Implement BFS to filter out dev-dependencies and dependencies of othe…
Shnatsel Oct 26, 2023
1f20310
Make BFS actually run - fix the root element already being present
Shnatsel Oct 26, 2023
a592eee
Clarify comment
Shnatsel Oct 26, 2023
f6a8203
drop a fulfilled TODO
Shnatsel Oct 26, 2023
d707809
Merge branch 'main' into accurate-deps
Shnatsel Oct 26, 2023
e0c720c
Fix a typo
Shnatsel Oct 27, 2023
1089b28
Fix typo in comment
Shnatsel Oct 27, 2023
14e1026
apply style suggestion
Shnatsel Oct 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 65 additions & 14 deletions cargo-cyclonedx/src/generator.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,10 @@ use crate::toml::config_from_file;
use crate::toml::ConfigError;

use cargo_metadata;
use cargo_metadata::DependencyKind;
use cargo_metadata::Metadata as CargoMetadata;
use cargo_metadata::Node;
use cargo_metadata::NodeDep;
use cargo_metadata::Package;
use cargo_metadata::PackageId;

Expand Down Expand Up @@ -388,44 +390,93 @@ pub enum GeneratorError {
}

fn top_level_dependencies(
member: &PackageId,
root: &PackageId,
packages: &PackageMap,
resolve: &ResolveMap,
) -> (PackageMap, ResolveMap) {
log::trace!("Adding top-level dependencies to SBOM");
let direct_dep_ids = resolve[member].dependencies.as_slice();

// Only include packages that have dependency kinds other than "Development"
let root_node = strip_dev_dependencies(&resolve[root]);

let mut pkg_result = PackageMap::new();
pkg_result.insert(member.to_owned(), packages[member].to_owned());
for id in direct_dep_ids {
pkg_result.insert(id.to_owned(), packages[id].to_owned());
// Record the root package, then its direct non-dev dependencies
pkg_result.insert(root.to_owned(), packages[root].to_owned());
for id in &root_node.dependencies {
pkg_result.insert((*id).to_owned(), packages[id].to_owned());
}

let mut resolve_result = ResolveMap::new();
resolve_result.insert(member.to_owned(), resolve[member].clone());
for id in direct_dep_ids {
// Clear all dependencies, pretend there is only one level
for id in &root_node.dependencies {
// Clear all depedencies, pretend there is only one level
let mut node = resolve[id].clone();
node.deps = Vec::new();
node.dependencies = Vec::new();
resolve_result.insert(id.to_owned(), node);
resolve_result.insert((*id).to_owned(), node);
}
// Insert the root node at the end now that we're done iterating over it
resolve_result.insert(root.to_owned(), root_node);
Shnatsel marked this conversation as resolved.
Show resolved Hide resolved

(pkg_result, resolve_result)
}

fn all_dependencies(
member: &PackageId,
root: &PackageId,
packages: &PackageMap,
resolve: &ResolveMap,
) -> (PackageMap, ResolveMap) {
log::trace!("Adding all dependencies to SBOM");

// FIXME: run BFS to filter out irrelevant dependencies,
// such as dev dependencies that do not affect the final binary
// or dependencies of other packages in the workspace
// Note: using Vec (without deduplication) can theoretically cause quadratic memory usage,
// but since `Node` does not implement `Ord` or `Hash` it's hard to deduplicate them.
// These are all pointers and there's not a lot of them, it's highly unlikely to be an issue in practice.
// We can work around this by using a map instead of a set if need be.
let mut current_queue: Vec<&Node> = vec![&resolve[root]];
let mut next_queue: Vec<&Node> = Vec::new();

let mut out_resolve = ResolveMap::new();

// Run breadth-first search (BFS) over the dependency graph
// to determine which nodes are actually depended on by our package
// (not other packages) and to remove dev-dependencies
while !current_queue.is_empty() {
for node in current_queue.drain(..) {
// If we haven't processed this node yet...
if !out_resolve.contains_key(&node.id) {
// Add the node to the output
out_resolve.insert(node.id.to_owned(), strip_dev_dependencies(node));
// Queue its dependencies for the next BFS loop iteration
next_queue.extend(non_dev_dependencies(&node.deps).map(|dep| &resolve[&dep.pkg]));
}
}
std::mem::swap(&mut current_queue, &mut next_queue);
}

// Remove everything from `packages` that doesn't appear in the `resolve` we've built
let out_packages = packages
.iter()
.filter(|(id, _pkg)| out_resolve.contains_key(id))
.map(|(id, pkg)| (id.to_owned(), pkg.to_owned()))
.collect();

(out_packages, out_resolve)
}

fn strip_dev_dependencies(node: &Node) -> Node {
let mut node = node.clone();
node.deps = non_dev_dependencies(&node.deps).cloned().collect();
node.dependencies = node.deps.iter().map(|d| d.pkg.to_owned()).collect();
node
}

(packages.clone(), resolve.clone())
/// Filters out dependencies only used for development, and not affecting the final binary.
/// These are specified under `[dev-dependencies]` in Cargo.toml.
fn non_dev_dependencies(input: &[NodeDep]) -> impl Iterator<Item = &NodeDep> {
input.iter().filter(|p| {
p.dep_kinds
.iter()
.any(|dep| dep.kind != DependencyKind::Development)
})
}

/// Contains a generated SBOM and context used in its generation
Expand Down
Loading