Add file catalogers to selection configuration

[wagoodman]

Description

This PR adds file catalogers to the cataloger selection configuration:

$ syft cataloger list
Default selections: 1
  • 'all'
Selection expressions: 0
┌───────────────────────────┬───────────────────────┐
│ FILE CATALOGER            │ TAGS                  │
├───────────────────────────┼───────────────────────┤
│ file-content-cataloger    │ content, file         │
│ file-digest-cataloger     │ digest, file          │
│ file-executable-cataloger │ binary-metadata, file │
│ file-metadata-cataloger   │ file, file-metadata   │
└───────────────────────────┴───────────────────────┘
┌────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────┐
│ PACKAGE CATALOGER                      │ TAGS                                                                      │
├────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
(same as today...)
└────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────┘

To ensure configuration is backwards compatible, file is added to the list of default tags, no matter if the user attempts to override this:

# try and set the default set to only "image", leaving out "file" implicitly
syft cataloger list --override-default-catalogers image
[0000]  WARN adding 'file' tag to the default cataloger selection, to override add '-file' to the cataloger selection request
...

Once the user specifies -file the warning does not appear:

# explicitly remove catalogers with 'file' tag
$ syft cataloger list --override-default-catalogers image --select-catalogers -file     
Default selections: 1
  • 'image'
Selection expressions: 1
  • '-file' (remove)
No file catalogers selected

(show package cataloger list, just as today...)

Expressions in the past were applied to all package cataloger tasks, and there was one group of tasks where selections could be applied. This PR adds the ability to apply the selection request to different groups of tasks, applying only the tags and names which a task in the group responds to. Any tags or names that don't apply to tasks within a group are ignored. Any tags or names that were in the original request but were not used against any group results in an error (stop from cataloging). This effectively splits the original user selection request dynamically amongst the group of tasks.

What does this mean in terms of syntax rules? The syntax for cataloger selection does not change. This means that package catalogers and file catalogers can be operated on independently without risk of affecting the one another.

Take for example:

syft python:latest --select-catalogers python
...

Today this uses the default tag (image) and subselects to package catalogers that have the python tag and file cataloging stays intact (as evidence of "file metadata", "file digests" and "executables"):

...
 ✔ Cataloged contents                                                                                                                                                                                           11663d4f2372df9c95bc55815c98527fd9f7267062c95b37424f485b2cc9df39
   ├── ✔ Packages                        [2 packages]  
   ├── ✔ File metadata                   [4 locations]  
   ├── ✔ File digests                    [4 files]  
   └── ✔ Executables                     [1,425 executables]  
NAME       VERSION  TYPE     
mercurial  6.3.2    python    
pip        24.2     python  

Without operating on groups independently and splitting the request, we'd see the same command result in NOT running any file catalogers:

...
 ✔ Cataloged contents                                                                                                                                                                                           11663d4f2372df9c95bc55815c98527fd9f7267062c95b37424f485b2cc9df39
   └── ✔ Packages                        [2 packages]  
NAME       VERSION  TYPE     
mercurial  6.3.2    python    
pip        24.2     python  

Instead we want a much more explicit expression to remove file cataloging:

syft python:latest --select-catalogers 'python,-file'

This is made possible by always injecting file as one of the default tags used for all cataloger selections.

• Partially addresses Setting file.metadata.selection to none still results in files in the SBOM #2989

Type of change

• New feature (non-breaking change which adds functionality)

Checklist:

• I have added unit tests that cover changed behavior
• I have tested my code in common scenarios and confirmed there are no regressions
• I have added comments to my code, particularly in hard-to-understand sections

[willmurphyscode]

I'd like to understand a little more context here. It seems like SYFT_FILE_METADATA_SELECTION=none has the same effect on main today as passing --select-catalogers -file will after this. Is that correct?

[wagoodman]

That is correct. I also think it's pretty non-obvious to folks that SYFT_FILE_METADATA_SELECTION affects more than just the file metadata cataloger. This PR addresses only this.

[willmurphyscode]

This seems a bit like it says Warning: doing what Syft always did, but now we feel weird about it. Does this need to be a log.Warn?

Can we just document somewhere that some amount of file cataloging is on unless removed?

[wagoodman]

what this is trying to convey is what to do next when there is surprising behavior. The rule of thumb for logging here is to only use warn when we have something to direct the user to do to correct a situation. This feels like the right spot for that since we're taking an action to be backwards compatible but could be surprising to folks that already have the sub-selection operation down.

To your point though, I should also update the readme with this context (but I think the warning is good to have too).

[spiffcs]

@wagoodman I think the README needs a quick update per your previous comment - or is that something to add into the wiki post merge?

FWIW I agree with keeping the warning here.

[willmurphyscode]

Under what invocations do we expect the warning to print? --override-default-catalogers ...? --select-catalogers java?

It looks like the warning is printed on overriding the default catalogers now, but not otherwise?

[willmurphyscode]

I have sort of a general question:

Currently, file cataloger selection is controlled by SYFT_FILE_METADATA_SELECTION, which can be all or owned-by-package or none, where none seems to do the same thing as --select-catalogers -file would in this PR. However, it's possible to make contradictory selections, and people may already have .file.metadata_selection set in their Syft configs. How is the cataloger selection meant to interact with that?

[wagoodman]

Good point -- though it's similar to having a package cataloger configuration to enable data enrichment then de-selecting the cataloger. I think we should allow the action to occur (override configuration with cataloger selection) but we could log-warn in these kinds of situations.

[popey]

Note from live stream to take on board feedback from #3468 also.

[spiffcs]

🟢 - I think @willmurphyscode had a comment that you responded to regarding a README update, but this LGTM and is important for me to pull in as a part of the refactor for license content PR so 👍