Add Nix cataloger

[wagoodman]

This builds on the original work from @juliosueiras on #1107 . The changes include:

• pulling the latest changes from main in
• refactoring to use resolver.AllLocations
• moving the set of owned locations to the pkg.NixStoreMetadata and leaving the pkg.Locations to still indicate the locations where the existence of the package is evident (just the nix store path directory, not the contents)
• refactored the nix store path parsing and added unit tests
• added integration tests

I'm not very familiar with the Nix ecosystem so review from someone more familiar with Nix would be very helpful! (cc: @flokli if you have the time 🙏 )

Details

I want to be clear on how this is working and what the consequences are. This implementation looks at only what directories are found in /nix/store and will construct packages for each output directory found (which means derivations are not included). This means that packages that have multiple outputs will seemingly be unrelated. This is because there is not enough metadata locally in the scan target to determine what outputs should be related.

For example, if you install python37 from nixpkgs, you will see two outputs, out (the default, so has no name) and debug:

$ nix-env -iA nixpkgs.python37
$ ls -al /nix/store | grep 'python3-3.7'
dr-xr-xr-x 3 root root     4096 Jan  1  1970 13nr17z3x46xknz27gn9p77wxnll2sxl-python3-3.7.16-debug
dr-xr-xr-x 7 root root     4096 Jan  1  1970 a1gzdpyn8bbp192cavnxl6sa3snjx563-python3-3.7.16
-r--r--r-- 1 root root    11859 Jan  1  1970 y72x7nz63cw29lzy239acwcc7gswljn2-python3-3.7.16.drv

When running syft, you will see two python3@3.7.16 packages, one for each output:

$ syft -o json ... | jq '.artifacts[] | select(.name == "python3")'
{
  "id": "5225a10fc861ef0e",
  "name": "python3",
  "version": "3.7.16",
  "type": "nix",
  "foundBy": "nix-store-cataloger",
  "locations": [
    {
      "path": "/nix/store/a1gzdpyn8bbp192cavnxl6sa3snjx563-python3-3.7.16",
      "layerID": "sha256:01c6ae491e3fc339b93ccf883079fba571df6ddfdaa3ed5c2ac91578ce85a1f2"
    }
  ],
  "licenses": [],
  "language": "",
  "cpes": [
    "cpe:2.3:a:python3:python3:3.7.16:*:*:*:*:*:*:*"
  ],
  "purl": "pkg:nix/python3@3.7.16?outputhash=a1gzdpyn8bbp192cavnxl6sa3snjx563",
  "metadataType": "NixStoreMetadata",
  "metadata": {
    "outputHash": "a1gzdpyn8bbp192cavnxl6sa3snjx563",
    "files": [ ... snip ... ]
},
{
  "id": "8c1c788327e1f52e",
  "name": "python3",
  "version": "3.7.16",
  "type": "nix",
  "foundBy": "nix-store-cataloger",
  "locations": [
    {
      "path": "/nix/store/13nr17z3x46xknz27gn9p77wxnll2sxl-python3-3.7.16-debug",
      "layerID": "sha256:01c6ae491e3fc339b93ccf883079fba571df6ddfdaa3ed5c2ac91578ce85a1f2"
    }
  ],
  "licenses": [],
  "language": "",
  "cpes": [
    "cpe:2.3:a:python3:python3:3.7.16:*:*:*:*:*:*:*"
  ],
  "purl": "pkg:nix/python3@3.7.16?output=debug&outputhash=13nr17z3x46xknz27gn9p77wxnll2sxl",
  "metadataType": "NixStoreMetadata",
  "metadata": {
    "outputHash": "13nr17z3x46xknz27gn9p77wxnll2sxl",
    "output": "debug",
    "files": [ ... snip ... ]
}

The pURLS are nicely distinct:

• "pkg:nix/python3@3.7.16?outputhash=a1gzdpyn8bbp192cavnxl6sa3snjx563",
• "pkg:nix/python3@3.7.16?output=debug&outputhash=13nr17z3x46xknz27gn9p77wxnll2sxl"

But the package names are not unique, which is technically correct, but it would be more ideal to try and relate or merge these packages in the future.

Closes #462

[github-actions]

Benchmark Test Results

Benchmark results from the latest changes vs base branch

goos: linux%0Agoarch: amd64%0Apkg: github.com/anchore/syft/test/integration%0Acpu: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz%0A                                                          │ ./.tmp/benchmark-c6a7c19.txt │%0A                                                          │            sec/op            │%0AImagePackageCatalogers/alpmdb-cataloger-2                                   11.68m ± 19%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                             843.4µ ±  1%25%0AImagePackageCatalogers/python-package-cataloger-2                           3.025m ±  2%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                   684.9µ ±  1%25%0AImagePackageCatalogers/javascript-package-cataloger-2                       353.7µ ±  1%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                   502.6µ ±  1%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                   485.6µ ±  3%25%0AImagePackageCatalogers/java-cataloger-2                                     11.16m ± 16%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                     9.223µ ±  7%25%0AImagePackageCatalogers/apkdb-cataloger-2                                    546.7µ ±  2%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                         18.08µ ±  1%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                              944.1µ ±  1%25%0AImagePackageCatalogers/portage-cataloger-2                                  342.0µ ±  4%25%0AImagePackageCatalogers/nix-store-cataloger-2                                222.6µ ±  2%25%0AImagePackageCatalogers/sbom-cataloger-2                                     110.3µ ±  1%25%0AImagePackageCatalogers/binary-cataloger-2                                   196.0µ ±  3%25%0Ageomean                                                                     449.6µ%0A%0A                                                          │ ./.tmp/benchmark-c6a7c19.txt │%0A                                                          │             B/op             │%0AImagePackageCatalogers/alpmdb-cataloger-2                                   5.062Mi ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                             123.9Ki ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                           946.6Ki ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                   155.9Ki ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                       90.80Ki ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                   144.6Ki ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                   170.2Ki ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                     2.723Mi ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                     1.555Ki ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                    129.2Ki ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                         3.133Ki ± 0%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                              314.6Ki ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                  77.17Ki ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                36.07Ki ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                     13.57Ki ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                   29.91Ki ± 0%25%0Ageomean                                                                     101.7Ki%0A%0A                                                          │ ./.tmp/benchmark-c6a7c19.txt │%0A                                                          │          allocs/op           │%0AImagePackageCatalogers/alpmdb-cataloger-2                                    86.71k ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                              2.049k ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                            15.48k ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                    3.458k ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                        1.205k ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                    2.646k ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                    3.759k ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                      38.26k ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                       40.00 ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                     3.438k ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                           101.0 ± 0%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                               5.011k ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                   1.539k ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                  671.0 ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                       392.0 ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                     872.0 ± 0%25%0Ageomean                                                                      2.062k

[wagoodman]

JSON schema diff:

# $ diff schema/json/schema-7.0.1.json schema/json/schema-7.0.2.json
848a849,868
>     "NixStoreMetadata": {
>       "properties": {
>         "hash": {
>           "type": "string"
>         },
>         "output": {
>           "type": "string"
>         },
>         "files": {
>           "items": {
>             "type": "string"
>           },
>           "type": "array"
>         }
>       },
>       "type": "object",
>       "required": [
>         "hash"
>       ]
>     },
1010a1031,1033
>             },
>             {
>               "$ref": "#/$defs/NixStoreMetadata"

[flokli]

bin is just the name of the output, it's not part of the version. There can be multiple outputs with the same "version number", but different output names. I'd probably just drop the optional -{bin,lib,dev,doc,devdoc} (these are the most common ones) output name from the version number - or is there a notion of multiple package outputs in syft?

[wagoodman]

Question: is glibc-2.34 really the "package" and the single package has multiple outputs? If that's the case, there is more work to be done -- the cataloger should be merging these into a single package, and the metadata for each needs to account for expressing multiple outputs (and surrounding metadata).

[flokli]

This is the bin output for the glibc 2.34 build. It includes binaries produced by the build, like bin/getent. The glibc .so file would be in a separate output (lib or the default output, where the prefix is omitted).

It makes sense to distinguish between these outputs (so the info should not be merged), and it also makes sense to keep the output hash as an identifier.

We can have multiple glibc-2.34 in the same "closure" / filesystem (let's say one carrying a patch, the other not), and we can ship the bin outputs or not, depending on whether we want getent or not.

If there's a vulnerability, the output_path allows distinguishing between the patched version or unpatched version, and knowing which outputs you ship also helps determining whether you might be affected by a certain vulnerability and it's limited to the CLI parser, not the library.

I'm having a hard time calling this package, because it's nothing bundled up, and these are the output paths produced by a build. After évaluation, Nix uses "derivations" as build recipes. Each derivation is a single build, and it may produce multiple outputs (with output paths).

[wagoodman]

It sounds like the best way forward is to collect these definitions into a single package, where the metadata would be able to track the multiple outputs:

{
  "name": "glibc"
  "version": "2.34"
  ...
  "metadata": {
    "outputs": [
        {
          "output": "man"
          "outputhash": "cfmmhxh0c5nxnag1bixdg7d3g2c9wan2"
          "files": [...]
        }
        {
          "output": "bin"
          "outputhash": "h0cnbmfcn93xm5dg2x27ixhag1cwndga"
          "files": [...]
        }
      ]
  }
}

In this way there is still a single logical glibc@2.34 package and grype downstream would be able to look at the outputhash for each output on the metadata.

If there's a vulnerability, the output_path allows distinguishing between the patched version or unpatched version

it sounds like the output_path would be really important to try and capture on the metadata directly then? Just to double check... is this output path the path to the store (e.g. /nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo)? Or is this referring to another value?

I'm having a hard time calling this package, because it's nothing bundled up, and these are the output paths produced by a build.

I can see what you're saying here (I think). I started with the assumption of what you might find on search.nixos.org ... say for glibc https://search.nixos.org/packages?channel=22.11&show=glibc&from=0&size=1&sort=relevance&type=packages&query=glibc

I feel this orients the results in terms of the logical name + version and outputs available. I've been equating the name + version as the idea of a singular logical package and the list of outputs installed as something that is attached metadata to that (single) package. I see what you're saying with nothing is being bundled up but syft doesn't necessarily require that packaging ecosystems do any bundling of builds/files at all.

Is this a kosher way to think about "packages" in the Nix ecosystem? Or said another way, could representing singular packages with possibly multiple outputs be inaccurate or misleading in downstream analysis?

[flokli]

it sounds like the output_path would be really important to try and capture on the metadata directly then? Just to double check... is this output path the path to the store (e.g. /nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo)? Or is this referring to another value?

Let me pick a better example:

If I take a current checkout of nixpkgs (4bb072f0a8b267613c127684e099a70e1f6ff106), there's multiple outputs of the openssl package, described in the "build recipe" (.drv):

$ nix repl .
nix-repl> openssl
«derivation /nix/store/3jbv3scpmj2kxhshfz544jq37dzywbss-openssl-3.0.8.drv»

nix-repl> openssl.outputs
[ "bin" "dev" "out" "man" "doc" "debug" ]

nix-repl> :b openssl

This derivation produced the following outputs:
  bin -> /nix/store/05qna7girdhnzx38jkhr4yyh7gmnizv3-openssl-3.0.8-bin
  debug -> /nix/store/spa8fwjwd6r4n6d4gqg4s1ylijmkc0x7-openssl-3.0.8-debug
  dev -> /nix/store/ggsmy3rwsr6q6ri45ck3jdqnz9cf6dm6-openssl-3.0.8-dev
  doc -> /nix/store/bsabnghysgdmfhjlj2pal3i25q9w2brf-openssl-3.0.8-doc
  man -> /nix/store/4ac38nfgh4whllggv4zwp14va4mfirr7-openssl-3.0.8-man
  out -> /nix/store/s8vg2h8xzqmjd72f3g3p1jqy2lbbapc6-openssl-3.0.8

So, all these output paths are produced by the same build recipe (/nix/store/3jbv3scpmj2kxhshfz544jq37dzywbss-openssl-3.0.8.drv).

• The bin output of that derivation has an output hash of 05qna7girdhnzx38jkhr4yyh7gmnizv3 (nixbase32 encoding).
• The debug output has an output hash of spa8fwjwd6r4n6d4gqg4s1ylijmkc0x7
• The dev output has an output hash of ggsmy3rwsr6q6ri45ck3jdqnz9cf6dm6
• ...

If I now go back in time to a previous nixpkgs checkout (2 weeks before, 1df7332c722cc8d235b979cfa8f1bbe949b722fb), I still have the same openssl version in that case, but the build recipe is a different one (and due to this, all the hashes).

$ nix repl .
nix-repl> openssl   
«derivation /nix/store/y43ka1zh6icrf515hc5qzb643wsscxmd-openssl-3.0.8.drv»

nix-repl> openssl.outputs
[ "bin" "dev" "out" "man" "doc" "debug" ]

nix-repl> :b openssl

This derivation produced the following outputs:
  bin -> /nix/store/b1rd646j6yc22qnhiydb5a1hi5hc4dky-openssl-3.0.8-bin
  debug -> /nix/store/8zw06pv5v5l7ggklyls77n5fw03vgdr7-openssl-3.0.8-debug
  dev -> /nix/store/gs1841l5x53q3r5l196ld9kxq45pvf82-openssl-3.0.8-dev
  doc -> /nix/store/42krirjplr0a4z3w2myhrawg5kbgmy40-openssl-3.0.8-doc
  man -> /nix/store/rrz6hdzxsxg3yadk02gcypib36bq2zqm-openssl-3.0.8-man
  out -> /nix/store/fgrj06y1x83fwh8hqgg02v9abc7a7b65-openssl-3.0.8

Now the binoutput hash is b1rd646j6yc22qnhiydb5a1hi5hc4dky.

Both these packages can coexist in /nix/store.

In more complicated closures / "container contents", it's very well possible for, let's say, two slightly different flavors of libraries to be linked against two different binaries.

Key takeaways:

• just the "name" (openssl-3.0.8) doesn't uniquely identify a package, the outputhash is crucial
• The output hash uniquely identifies an output, as soon as the build recipe (or any of its dependencies) changes it'll differ
• Just by looking at the paths, without any additional database (that's what syft is doing) it's not possible to map / group certain output paths to the .drv / build recipe producing it.

[wagoodman]

Thanks for the example / input here -- much appreciated! I think the takeaways make sense given the fact that syft is guessing from a directory structure instead of package metadata information. In the future I feel it would be ideal to improve this in some way to make it more obvious which outputs are related... but I don't see a safe way to do this given the information available and the non-guarentees on the sqlite cache that's there. Thanks for helping to navigate these traps @flokli !

Ok, back to your question (I tooks us on a little bit of a tangent):

bin is just the name of the output, it's not part of the version. There can be multiple outputs with the same "version number", but different output names. I'd probably just drop the optional -{bin,lib,dev,doc,devdoc} (these are the most common ones) output name from the version number - or is there a notion of multiple package outputs in syft?

This function first extracts the version as if it were a semver, then later in processing (in parseNixStorePath) the last non-numeric, dash-delimited string is assumed to be the nix output. That is, the package ultimately reflects your suggestion (to not include output names in the version).

There isn't a notion of a "package output" in syft, however, what it will look like to most users is that there are multiple instances of the package name-version installed, and only when they look at the .metadata or .purl will they see the specific output this package represents.

[wagoodman]

@flokli Thanks a lot for the review 🙌 . There are a couple questions I had, the biggest one being this one #1696 (comment) .

[wagoodman]

As a bit of a side note, this thread was interesting and helpful in understanding more about the future of Nix in terms of capturing the right details in an SBOM and how that would relate to vulnerability analysis https://discourse.nixos.org/t/the-future-of-the-vulnerability-roundups/22424 .

[wagoodman]

@flokli I think this is ready for merging, but I wanted to hold off and get your take first. Are there any other changes relative to the Nix ecosystem that should be considered before considering this done? (or any nuance that I might have missed in your previous comments that is not addressed?)

[flokli]

@wagoodman still looks about right. I'd probably make it clear somewhere that this only looks at the store paths to do all the cataloguing, in case we add other ways to convey more information.

[raboof]

Cool addition! I took the liberty of starting a Discourse thread about the use of purl for Nix packages at https://discourse.nixos.org/t/package-urls-purl-for-nix-packages