Auto merge of #7977 - ehuss:unit-graph, r=alexcrichton

Add unit-graph JSON output.

This adds a `--unit-graph` flag that will emit a JSON object of Cargo's internal build graph.  See unstable.md for more details.

The primary motivator is to provide an accurate picture of which features are set. With the new feature resolver it is not possible to properly represent the features in the `cargo metadata` structure, because features are no longer unified.  Also, features selected depend on the command, and exactly which packages are being built.  To handle that in `cargo metadata`, it would need to add a "mode" flag, and a superset of flags for all build commands (test, check, build, etc.). To me that seemed like a difficult path to take.

This may also be helpful for making visualizations of the true dependencies.  `cargo metadata` doesn't show the intra-package dependencies like build scripts or test units, and walking the `cargo metadata` graph correctly isn't always obvious.

This initial concept exposes almost all of the fields. That may be a little too much, but I imagine we could always trim it before stabilizing.  This structure also has a high risk of being unstable, since it has a good chance of changing form in the future.  I figure that can be addressed with documentation emphasizing that it may change and we may not always provide backwards-compatibility (though we will try if it is not too much burden).

This could also potentially be extended in the future to include things like artifact paths, or "freshness", if we'd like to.

src/bin/cargo/commands/bench.rs

@@ -44,6 +44,7 @@ pub fn cli() -> App {
             "no-fail-fast",
             "Run all benchmarks regardless of failure",
         ))
+        .arg_unit_graph()
         .after_help(
             "\
 The benchmark filtering argument BENCHNAME and all the arguments following the

src/bin/cargo/commands/build.rs

@@ -41,6 +41,7 @@ pub fn cli() -> App {
         .arg_manifest_path()
         .arg_message_format()
         .arg_build_plan()
+        .arg_unit_graph()
         .after_help(
             "\
 All packages in the workspace are built if the `--workspace` flag is supplied. The

src/bin/cargo/commands/check.rs

@@ -33,6 +33,7 @@ pub fn cli() -> App {
         .arg_target_dir()
         .arg_manifest_path()
         .arg_message_format()
+        .arg_unit_graph()
         .after_help(
             "\
 If the `--package` argument is given, then SPEC is a package ID specification

src/bin/cargo/commands/doc.rs

@@ -30,6 +30,7 @@ pub fn cli() -> App {
         .arg_target_dir()
         .arg_manifest_path()
         .arg_message_format()
+        .arg_unit_graph()
         .after_help(
             "\
 By default the documentation for the local package and all dependencies is

src/bin/cargo/commands/run.rs

@@ -24,6 +24,7 @@ pub fn cli() -> App {
         .arg_target_dir()
         .arg_manifest_path()
         .arg_message_format()
+        .arg_unit_graph()
         .after_help(
             "\
 If neither `--bin` nor `--example` are given, then if the package only has one

src/bin/cargo/commands/rustc.rs

@@ -29,6 +29,7 @@ pub fn cli() -> App {
         .arg_target_dir()
         .arg_manifest_path()
         .arg_message_format()
+        .arg_unit_graph()
         .after_help(
             "\
 The specified target for the current package (or package specified by SPEC if

src/bin/cargo/commands/rustdoc.rs

@@ -33,6 +33,7 @@ pub fn cli() -> App {
         .arg_target_dir()
         .arg_manifest_path()
         .arg_message_format()
+        .arg_unit_graph()
         .after_help(
             "\
 The specified target for the current package (or package specified by SPEC if

src/bin/cargo/commands/test.rs

@@ -54,6 +54,7 @@ pub fn cli() -> App {
         .arg_target_dir()
         .arg_manifest_path()
         .arg_message_format()
+        .arg_unit_graph()
         .after_help(
             "\
 The test filtering argument TESTNAME and all the arguments following the

src/cargo/core/compiler/build_config.rs

@@ -22,6 +22,8 @@ pub struct BuildConfig {
     pub force_rebuild: bool,
     /// Output a build plan to stdout instead of actually compiling.
     pub build_plan: bool,
+    /// Output the unit graph to stdout instead of actually compiling.
+    pub unit_graph: bool,
     /// An optional override of the rustc process for primary units
     pub primary_unit_rustc: Option<ProcessBuilder>,
     pub rustfix_diagnostic_server: RefCell<Option<RustfixDiagnosticServer>>,
@@ -79,6 +81,7 @@ impl BuildConfig {
             message_format: MessageFormat::Human,
             force_rebuild: false,
             build_plan: false,
+            unit_graph: false,
             primary_unit_rustc: None,
             rustfix_diagnostic_server: RefCell::new(None),
         })

src/cargo/core/compiler/compile_kind.rs

@@ -9,7 +9,7 @@ use std::path::Path;
 /// compilations, where cross compilations happen at the request of `--target`
 /// and host compilations happen for things like build scripts and procedural
 /// macros.
-#[derive(PartialEq, Eq, Hash, Debug, Clone, Copy, PartialOrd, Ord, Serialize)]
+#[derive(PartialEq, Eq, Hash, Debug, Clone, Copy, PartialOrd, Ord)]
 pub enum CompileKind {
     /// Attached to a unit that is compiled for the "host" system or otherwise
     /// is compiled without a `--target` flag. This is used for procedural
@@ -41,6 +41,18 @@ impl CompileKind {
     }
 }
 
+impl serde::ser::Serialize for CompileKind {
+    fn serialize<S>(&self, s: S) -> Result<S::Ok, S::Error>
+    where
+        S: serde::ser::Serializer,
+    {
+        match self {
+            CompileKind::Host => None::<&str>.serialize(s),
+            CompileKind::Target(t) => Some(t.name).serialize(s),
+        }
+    }
+}
+
 /// Abstraction for the representation of a compilation target that Cargo has.
 ///
 /// Compilation targets are one of two things right now:

src/cargo/core/compiler/context/mod.rs

@@ -16,7 +16,7 @@ use super::custom_build::{self, BuildDeps, BuildScriptOutputs, BuildScripts};
 use super::fingerprint::Fingerprint;
 use super::job_queue::JobQueue;
 use super::layout::Layout;
-use super::unit_dependencies::{UnitDep, UnitGraph};
+use super::unit_graph::{UnitDep, UnitGraph};
 use super::{BuildContext, Compilation, CompileKind, CompileMode, Executor, FileFlavor};
 
 mod compilation_files;

src/cargo/core/compiler/fingerprint.rs

@@ -202,7 +202,7 @@ use serde::de;
 use serde::ser;
 use serde::{Deserialize, Serialize};
 
-use crate::core::compiler::unit_dependencies::UnitDep;
+use crate::core::compiler::unit_graph::UnitDep;
 use crate::core::{InternedString, Package};
 use crate::util;
 use crate::util::errors::{CargoResult, CargoResultExt};

src/cargo/core/compiler/links.rs

@@ -1,4 +1,4 @@
-use super::unit_dependencies::UnitGraph;
+use super::unit_graph::UnitGraph;
 use crate::core::{PackageId, Resolve};
 use crate::util::errors::CargoResult;
 use std::collections::{HashMap, HashSet};

src/cargo/core/compiler/mod.rs

@@ -15,6 +15,7 @@ pub mod standard_lib;
 mod timings;
 mod unit;
 pub mod unit_dependencies;
+pub mod unit_graph;
 
 use std::env;
 use std::ffi::{OsStr, OsString};
@@ -38,7 +39,7 @@ pub use self::job::Freshness;
 use self::job::{Job, Work};
 use self::job_queue::{JobQueue, JobState};
 use self::output_depinfo::output_depinfo;
-use self::unit_dependencies::UnitDep;
+use self::unit_graph::UnitDep;
 pub use crate::core::compiler::unit::{Unit, UnitInterner};
 use crate::core::manifest::TargetSourcePath;
 use crate::core::profiles::{Lto, PanicStrategy, Profile};

src/cargo/core/compiler/unit_dependencies.rs

@@ -15,6 +15,7 @@
 //! (for example, with and without tests), so we actually build a dependency
 //! graph of `Unit`s, which capture these properties.
 
+use crate::core::compiler::unit_graph::{UnitDep, UnitGraph};
 use crate::core::compiler::Unit;
 use crate::core::compiler::{BuildContext, CompileKind, CompileMode};
 use crate::core::dependency::DepKind;
@@ -27,25 +28,6 @@ use crate::CargoResult;
 use log::trace;
 use std::collections::{HashMap, HashSet};
 
-/// The dependency graph of Units.
-pub type UnitGraph<'a> = HashMap<Unit<'a>, Vec<UnitDep<'a>>>;
-
-/// A unit dependency.
-#[derive(Debug, Clone, Hash, Eq, PartialEq, PartialOrd, Ord)]
-pub struct UnitDep<'a> {
-    /// The dependency unit.
-    pub unit: Unit<'a>,
-    /// The purpose of this dependency (a dependency for a test, or a build
-    /// script, etc.).
-    pub unit_for: UnitFor,
-    /// The name the parent uses to refer to this dependency.
-    pub extern_crate_name: InternedString,
-    /// Whether or not this is a public dependency.
-    pub public: bool,
-    /// If `true`, the dependency should not be added to Rust's prelude.
-    pub noprelude: bool,
-}
-
 /// Collection of stuff used while creating the `UnitGraph`.
 struct State<'a, 'cfg> {
     bcx: &'a BuildContext<'a, 'cfg>,

src/cargo/core/compiler/unit_graph.rs

@@ -0,0 +1,121 @@
+use crate::core::compiler::Unit;
+use crate::core::compiler::{CompileKind, CompileMode};
+use crate::core::profiles::{Profile, UnitFor};
+use crate::core::{nightly_features_allowed, InternedString, PackageId, Target};
+use crate::util::CargoResult;
+use std::collections::HashMap;
+use std::io::Write;
+
+/// The dependency graph of Units.
+pub type UnitGraph<'a> = HashMap<Unit<'a>, Vec<UnitDep<'a>>>;
+
+/// A unit dependency.
+#[derive(Debug, Clone, Hash, Eq, PartialEq, PartialOrd, Ord)]
+pub struct UnitDep<'a> {
+    /// The dependency unit.
+    pub unit: Unit<'a>,
+    /// The purpose of this dependency (a dependency for a test, or a build
+    /// script, etc.).
+    pub unit_for: UnitFor,
+    /// The name the parent uses to refer to this dependency.
+    pub extern_crate_name: InternedString,
+    /// Whether or not this is a public dependency.
+    pub public: bool,
+    /// If `true`, the dependency should not be added to Rust's prelude.
+    pub noprelude: bool,
+}
+
+const VERSION: u32 = 1;
+
+#[derive(serde::Serialize)]
+struct SerializedUnitGraph<'a> {
+    version: u32,
+    units: Vec<SerializedUnit<'a>>,
+    roots: Vec<usize>,
+}
+
+#[derive(serde::Serialize)]
+struct SerializedUnit<'a> {
+    pkg_id: PackageId,
+    target: &'a Target,
+    profile: &'a Profile,
+    platform: CompileKind,
+    mode: CompileMode,
+    features: &'a Vec<InternedString>,
+    #[serde(skip_serializing_if = "std::ops::Not::not")] // hide for unstable build-std
+    is_std: bool,
+    dependencies: Vec<SerializedUnitDep>,
+}
+
+#[derive(serde::Serialize)]
+struct SerializedUnitDep {
+    index: usize,
+    extern_crate_name: InternedString,
+    // This is only set on nightly since it is unstable.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    public: Option<bool>,
+    // This is only set on nightly since it is unstable.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    noprelude: Option<bool>,
+    // Intentionally not including `unit_for` because it is a low-level
+    // internal detail that is mostly used for building the graph.
+}
+
+pub fn emit_serialized_unit_graph(
+    root_units: &[Unit<'_>],
+    unit_graph: &UnitGraph<'_>,
+) -> CargoResult<()> {
+    let is_nightly = nightly_features_allowed();
+    let mut units: Vec<(&Unit<'_>, &Vec<UnitDep<'_>>)> = unit_graph.iter().collect();
+    units.sort_unstable();
+    // Create a map for quick lookup for dependencies.
+    let indices: HashMap<&Unit<'_>, usize> = units
+        .iter()
+        .enumerate()
+        .map(|(i, val)| (val.0, i))
+        .collect();
+    let roots = root_units.iter().map(|root| indices[root]).collect();
+    let ser_units = units
+        .iter()
+        .map(|(unit, unit_deps)| {
+            let dependencies = unit_deps
+                .iter()
+                .map(|unit_dep| {
+                    // https://github.com/rust-lang/rust/issues/64260 when stabilized.
+                    let (public, noprelude) = if is_nightly {
+                        (Some(unit_dep.public), Some(unit_dep.noprelude))
+                    } else {
+                        (None, None)
+                    };
+                    SerializedUnitDep {
+                        index: indices[&unit_dep.unit],
+                        extern_crate_name: unit_dep.extern_crate_name,
+                        public,
+                        noprelude,
+                    }
+                })
+                .collect();
+            SerializedUnit {
+                pkg_id: unit.pkg.package_id(),
+                target: unit.target,
+                profile: &unit.profile,
+                platform: unit.kind,
+                mode: unit.mode,
+                features: &unit.features,
+                is_std: unit.is_std,
+                dependencies,
+            }
+        })
+        .collect();
+    let s = SerializedUnitGraph {
+        version: VERSION,
+        units: ser_units,
+        roots,
+    };
+
+    let stdout = std::io::stdout();
+    let mut lock = stdout.lock();
+    serde_json::to_writer(&mut lock, &s)?;
+    write!(lock, "\n")?;
+    Ok(())
+}

src/cargo/core/profiles.rs

@@ -374,6 +374,7 @@ impl Profiles {
     /// times).
     pub fn get_profile_run_custom_build(&self, for_unit_profile: &Profile) -> Profile {
         let mut result = Profile::default();
+        result.name = for_unit_profile.name;
         result.root = for_unit_profile.root;
         result.debuginfo = for_unit_profile.debuginfo;
         result.opt_level = for_unit_profile.opt_level;
@@ -578,10 +579,11 @@ pub enum ProfileRoot {
 
 /// Profile settings used to determine which compiler flags to use for a
 /// target.
-#[derive(Clone, Copy, Eq, PartialOrd, Ord)]
+#[derive(Clone, Copy, Eq, PartialOrd, Ord, serde::Serialize)]
 pub struct Profile {
     pub name: InternedString,
     pub opt_level: InternedString,
+    #[serde(skip)] // named profiles are unstable
     pub root: ProfileRoot,
     pub lto: Lto,
     // `None` means use rustc default.
@@ -743,8 +745,21 @@ pub enum Lto {
     Named(InternedString),
 }
 
+impl serde::ser::Serialize for Lto {
+    fn serialize<S>(&self, s: S) -> Result<S::Ok, S::Error>
+    where
+        S: serde::ser::Serializer,
+    {
+        match self {
+            Lto::Bool(b) => b.to_string().serialize(s),
+            Lto::Named(n) => n.serialize(s),
+        }
+    }
+}
+
 /// The `panic` setting.
-#[derive(Clone, Copy, PartialEq, Eq, Debug, Hash, PartialOrd, Ord)]
+#[derive(Clone, Copy, PartialEq, Eq, Debug, Hash, PartialOrd, Ord, serde::Serialize)]
+#[serde(rename_all = "lowercase")]
 pub enum PanicStrategy {
     Unwind,
     Abort,

src/cargo/ops/cargo_compile.rs

@@ -30,6 +30,7 @@ use std::sync::Arc;
 
 use crate::core::compiler::standard_lib;
 use crate::core::compiler::unit_dependencies::build_unit_dependencies;
+use crate::core::compiler::unit_graph;
 use crate::core::compiler::{BuildConfig, BuildContext, Compilation, Context};
 use crate::core::compiler::{CompileKind, CompileMode, RustcTargetData, Unit};
 use crate::core::compiler::{DefaultExecutor, Executor, UnitInterner};
@@ -490,6 +491,11 @@ pub fn compile_ws<'a>(
         &std_roots,
     )?;
 
+    if bcx.build_config.unit_graph {
+        unit_graph::emit_serialized_unit_graph(&units, &unit_dependencies)?;
+        return Ok(Compilation::new(&bcx, build_config.requested_kind)?);
+    }
+
     let ret = {
         let _p = profile::start("compiling");
         let cx = Context::new(config, &bcx, unit_dependencies, build_config.requested_kind)?;

src/cargo/util/command_prelude.rs

@@ -150,6 +150,10 @@ pub trait AppExt: Sized {
         ))
     }
 
+    fn arg_unit_graph(self) -> Self {
+        self._arg(opt("unit-graph", "Output build graph in JSON (unstable)").hidden(true))
+    }
+
     fn arg_new_opts(self) -> Self {
         self._arg(
             opt(
@@ -438,11 +442,17 @@ pub trait ArgMatchesExt {
         build_config.message_format = message_format.unwrap_or(MessageFormat::Human);
         build_config.requested_profile = self.get_profile_name(config, "dev", profile_checking)?;
         build_config.build_plan = self._is_present("build-plan");
+        build_config.unit_graph = self._is_present("unit-graph");
         if build_config.build_plan {
             config
                 .cli_unstable()
                 .fail_if_stable_opt("--build-plan", 5579)?;
         };
+        if build_config.unit_graph {
+            config
+                .cli_unstable()
+                .fail_if_stable_opt("--unit-graph", 8002)?;
+        }
 
         let opts = CompileOptions {
             config,