Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: apk product/vendor generation for old metadata #1635

Merged
merged 1 commit into from
Mar 1, 2023

Conversation

westonsteimel
Copy link
Contributor

This fixes some instances where the improved APK CPE generation logic caused regressions for older alpine package APK metadata. It now generates multiple "upstream" candidates with both name and package type which reduces the amount of duplicated code in the apk cpe gen logic. This also improves the handling of stream version packages, so now we can correctly identify packages such as ruby3.2-rexml as the rexml ruby gem.

This fixes some instances where the improved APK CPE generation
logic caused regressions for older alpine package APK metadata.
It now generates multiple "upstream" candidates with both name
and package type which reduces the amount of duplicated code in
the apk cpe gen logic.  This also improves the handling of stream
version packages, so now we can correctly identify packages such
as ruby3.2-rexml as the rexml ruby gem.

Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
@westonsteimel westonsteimel requested a review from a team March 1, 2023 11:45
@github-actions
Copy link

github-actions bot commented Mar 1, 2023

Benchmark Test Results

Benchmark results from the latest changes vs base branch
goos: linux
goarch: amd64
pkg: github.com/anchore/syft/test/integration
cpu: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
                                                          │ ./.tmp/benchmark-dad3dfd.txt │
                                                          │            sec/op            │
ImagePackageCatalogers/alpmdb-cataloger-2                                   11.72m ±  6%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                             845.8µ ± 12%
ImagePackageCatalogers/python-package-cataloger-2                           3.006m ±  1%
ImagePackageCatalogers/php-composer-installed-cataloger-2                   647.2µ ±  1%
ImagePackageCatalogers/javascript-package-cataloger-2                       349.3µ ±  1%
ImagePackageCatalogers/dpkgdb-cataloger-2                                   465.9µ ±  2%
ImagePackageCatalogers/rpm-db-cataloger-2                                   436.7µ ±  1%
ImagePackageCatalogers/java-cataloger-2                                     10.78m ±  7%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                     7.976µ ±  2%
ImagePackageCatalogers/apkdb-cataloger-2                                    785.1µ ±  1%
ImagePackageCatalogers/go-module-binary-cataloger-2                         17.90µ ±  1%
ImagePackageCatalogers/dotnet-deps-cataloger-2                              951.1µ ±  2%
ImagePackageCatalogers/portage-cataloger-2                                  282.1µ ±  1%
ImagePackageCatalogers/sbom-cataloger-2                                     103.7µ ±  1%
ImagePackageCatalogers/binary-cataloger-2                                   144.4µ ±  0%
geomean                                                                     451.7µ

                                                          │ ./.tmp/benchmark-dad3dfd.txt │
                                                          │             B/op             │
ImagePackageCatalogers/alpmdb-cataloger-2                                   5.060Mi ± 0%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                             141.9Ki ± 0%
ImagePackageCatalogers/python-package-cataloger-2                           946.7Ki ± 0%
ImagePackageCatalogers/php-composer-installed-cataloger-2                   155.9Ki ± 0%
ImagePackageCatalogers/javascript-package-cataloger-2                       95.96Ki ± 0%
ImagePackageCatalogers/dpkgdb-cataloger-2                                   144.7Ki ± 0%
ImagePackageCatalogers/rpm-db-cataloger-2                                   170.7Ki ± 0%
ImagePackageCatalogers/java-cataloger-2                                     2.722Mi ± 0%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                     1.523Ki ± 0%
ImagePackageCatalogers/apkdb-cataloger-2                                    206.1Ki ± 0%
ImagePackageCatalogers/go-module-binary-cataloger-2                         3.102Ki ± 0%
ImagePackageCatalogers/dotnet-deps-cataloger-2                              314.0Ki ± 0%
ImagePackageCatalogers/portage-cataloger-2                                  75.52Ki ± 0%
ImagePackageCatalogers/sbom-cataloger-2                                     13.06Ki ± 0%
ImagePackageCatalogers/binary-cataloger-2                                   21.19Ki ± 0%
geomean                                                                     110.7Ki

                                                          │ ./.tmp/benchmark-dad3dfd.txt │
                                                          │          allocs/op           │
ImagePackageCatalogers/alpmdb-cataloger-2                                    86.71k ± 0%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                              2.159k ± 0%
ImagePackageCatalogers/python-package-cataloger-2                            15.49k ± 0%
ImagePackageCatalogers/php-composer-installed-cataloger-2                    3.458k ± 0%
ImagePackageCatalogers/javascript-package-cataloger-2                        1.253k ± 0%
ImagePackageCatalogers/dpkgdb-cataloger-2                                    2.646k ± 0%
ImagePackageCatalogers/rpm-db-cataloger-2                                    3.759k ± 0%
ImagePackageCatalogers/java-cataloger-2                                      38.27k ± 0%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                       40.00 ± 0%
ImagePackageCatalogers/apkdb-cataloger-2                                     4.988k ± 0%
ImagePackageCatalogers/go-module-binary-cataloger-2                           101.0 ± 0%
ImagePackageCatalogers/dotnet-deps-cataloger-2                               5.010k ± 0%
ImagePackageCatalogers/portage-cataloger-2                                   1.487k ± 0%
ImagePackageCatalogers/sbom-cataloger-2                                       392.0 ± 0%
ImagePackageCatalogers/binary-cataloger-2                                     649.0 ± 0%
geomean                                                                      2.243k

Type Type
}

func (m ApkMetadata) UpstreamCandidates() (candidates []UpstreamCandidate) {
Copy link
Contributor

@wagoodman wagoodman Mar 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more of a preference than anything else -- since UpstreamCandidates (and the former Upstream) is only used for the benefit of CPE generation the more ideal spot is where CPE generation is being done, in syft/pkg/cataloger/common/cpe. This also has the benefit of being unexported so it can't be used in the API.

What are your thoughts about this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I would have liked that; however, it's also needed for the purl.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I guess I could move it under syft/pkg/cataloger/common/cpe and then import it in apkdb?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would still have to be exported though, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed the usage in purl processing. I take back my comment then. I think in the future this will be in a better position to refactor: later we'll be moving the CPE generation logic back to where packages are created (near the catalogers) which means that we could remove this from the metadata object at that time.

@westonsteimel westonsteimel merged commit 8e1205f into main Mar 1, 2023
@westonsteimel westonsteimel deleted the fix-apk-ruby-packages branch March 1, 2023 14:58
@kzantow kzantow added the bug Something isn't working label Mar 2, 2023
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
This fixes some instances where the improved APK CPE generation
logic caused regressions for older alpine package APK metadata.
It now generates multiple "upstream" candidates with both name
and package type which reduces the amount of duplicated code in
the apk cpe gen logic.  This also improves the handling of stream
version packages, so now we can correctly identify packages such
as ruby3.2-rexml as the rexml ruby gem.

Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants