Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tie dynamic linking all together #818

Merged
merged 26 commits into from
Mar 7, 2022

Conversation

jssblck
Copy link
Member

@jssblck jssblck commented Feb 23, 2022

Overview

Ties dynamic linking together.
Also removes APK support, as we found that to have unacceptable performance when using syft.

We'll make a ticket to revisit APK support when/if users request it.

Acceptance criteria

Support reporting dynamically linked dependencies.

Testing plan

Manual testing (todo: fill this in)

Testing on Ubuntu

I set up an ubuntu docker container which could build the CLI.
I then ran the following command:

cabal run fossa -- analyze ./vendor-bins/ --experimental-analyze-dynamic-deps /bin/ls

I verified that the results appear as expected. The FOSSA UI shows the following results:
Screen Shot 2022-02-24 at 4 54 22 PM

I verified the results by running ldd and looking up each package manually:

root@69f93939bff1:/opt/projects/fossa-cli# ldd /bin/ls
	linux-vdso.so.1 (0x0000ffff98f57000)
	libselinux.so.1 => /lib/aarch64-linux-gnu/libselinux.so.1 (0x0000ffff98ebc000)
	libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff98d49000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffff98f27000)
	libpcre2-8.so.0 => /lib/aarch64-linux-gnu/libpcre2-8.so.0 (0x0000ffff98cbb000)
	libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffff98ca7000)
	libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff98c77000)

root@69f93939bff1:/opt/projects/fossa-cli# dpkg -S /lib/aarch64-linux-gnu/libselinux.so.1
libselinux1:arm64: /lib/aarch64-linux-gnu/libselinux.so.1

root@69f93939bff1:/opt/projects/fossa-cli# dpkg -S /lib/aarch64-linux-gnu/libc.so.6
libc6:arm64: /lib/aarch64-linux-gnu/libc.so.6

root@69f93939bff1:/opt/projects/fossa-cli# dpkg -S /lib/aarch64-linux-gnu/libpcre2-8.so.0
dpkg-query: no path found matching pattern /lib/aarch64-linux-gnu/libpcre2-8.so.0

root@69f93939bff1:/opt/projects/fossa-cli# dpkg -S /lib/aarch64-linux-gnu/libdl.so.2
libc6:arm64: /lib/aarch64-linux-gnu/libdl.so.2

root@69f93939bff1:/opt/projects/fossa-cli# dpkg -S /lib/aarch64-linux-gnu/libpthread.so.0
libc6:arm64: /lib/aarch64-linux-gnu/libpthread.so.0

Testing on Fedora

I set up a Fedora installation inside Parallels Desktop. I then set up the Haskell toolchain and built the CLI.

I then ran the following command:

cabal run fossa -- analyze ./vendor-bins/ --experimental-analyze-dynamic-deps /bin/ls

I verified that the results appear as expected. The FOSSA UI displays the following results:
Screen Shot 2022-03-01 at 2 50 12 PM

I verified the results by running ldd and looking up each package manually:

[parallels@fedora fossa-cli]$ ldd /bin/ls
	linux-vdso.so.1 (0x0000ffff866af000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x0000ffff865fd000)
	libcap.so.2 => /lib64/libcap.so.2 (0x0000ffff865dc000)
	libc.so.6 => /lib64/libc.so.6 (0x0000ffff86451000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffff86672000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x0000ffff863b0000)
	libdl.so.2 => /lib64/libdl.so.2 (0x0000ffff8638f000)
[parallels@fedora fossa-cli]$ rpm -qf /lib64/libselinux.so.1
libselinux-3.2-1.fc34.aarch64
[parallels@fedora fossa-cli]$ rpm -qf /lib64/libcap.so.2
libcap-2.48-2.fc34.aarch64
[parallels@fedora fossa-cli]$ rpm -qf /lib64/libc.so.6
glibc-2.33-21.fc34.aarch64
[parallels@fedora fossa-cli]$ rpm -qf /lib64/libpcre2-8.so.0
pcre2-10.36-4.fc34.aarch64
[parallels@fedora fossa-cli]$ rpm -qf /lib64/libdl.so.2
glibc-2.33-21.fc34.aarch64

Risks

Not very risky. Adds a new additional source unit.

References

Closes https://github.com/fossas/team-analysis/issues/886

Checklist

  • I added tests for this PR's change (or confirmed tests are not viable).
  • If this PR introduced a user-visible change, I added documentation into docs/.
  • I updated Changelog.md if this change is externally facing. If this PR did not mark a release, I added my changes into an # Unreleased section at the top.
  • I updated *schema.json if I have made changes for .fossa.yml, fossa-deps.{json, yaml, yml}. You may also need to update these if you have added/removed new dependency (e.g. pip) or analysis target type (e.g. poetry).
  • I linked this PR to any referenced GitHub issues, if they exist.

Copy link
Contributor

@skilly-lily skilly-lily left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a short drive-by review, left some comments, nothing too significant.

src/App/Fossa/VSI/DynLinked.hs Outdated Show resolved Hide resolved
src/App/Fossa/VSI/DynLinked/Internal/Binary.hs Outdated Show resolved Hide resolved
Comment on lines 55 to 57
-- When adding new tactics in the future, ensure that they fail (through diagnostics) if they're not the last link in the chain.
-- <||> selects the first item to succeed without a diagnostic error.
resolved <- rpmTactic root file <||> debTactic root file <||> apkTactic table file
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think they should just always fail. That way, ordering is not critical for success.

Suggested change
-- When adding new tactics in the future, ensure that they fail (through diagnostics) if they're not the last link in the chain.
-- <||> selects the first item to succeed without a diagnostic error.
resolved <- rpmTactic root file <||> debTactic root file <||> apkTactic table file
-- When adding new tactics in the future, ensure that they fail through diagnostics.
-- <||> selects the first item to succeed without a diagnostic error.
resolved <- rpmTactic root file <||> debTactic root file <||> apkTactic table file

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idea for a utility:

dsum [a, b, c] = a <||> b <||> c

Naming comes from asum, but we should probably find a better name.

I think it could be implemented as

data DSumStart = DSumStart

instance ToDiagnostic DSumStart where
  renderDiagnostic _ = "dsum was called with an empty list"

dsum = foldr (<||>) (fatal DSumStart)

@jssblck jssblck requested a review from skilly-lily March 3, 2022 20:34
@jssblck jssblck marked this pull request as ready for review March 3, 2022 20:34
@jssblck
Copy link
Member Author

jssblck commented Mar 3, 2022

@scruffystuffs since you did a drive by I can't use the bot to select anyone else 😬 if you don't have time to review this let me know and I'll post in Slack.

@skilly-lily
Copy link
Contributor

Oops. Yeah I'm pretty swamped atm, you probably need someone else to review this.

@jssblck jssblck requested a review from a team March 3, 2022 20:40
@meghfossa meghfossa self-requested a review March 3, 2022 20:47
Copy link
Contributor

@meghfossa meghfossa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left few optional nits. There is minor open question re scan summary, please address it via comment/or/commit before merging.

additionalSourceUnits = mapMaybe (join . resultToMaybe) [manualSrcUnits, vsiResults, binarySearchResults]
traverse_ (Diag.flushLogs SevError SevDebug) [vsiResults, binarySearchResults, manualSrcUnits]
additionalSourceUnits = mapMaybe (join . resultToMaybe) [manualSrcUnits, vsiResults, binarySearchResults, dynamicLinkedResults]
traverse_ (Diag.flushLogs SevError SevDebug) [vsiResults, binarySearchResults, manualSrcUnits, dynamicLinkedResults]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want the dynamic deps results to show in scan summary, we would need to update this type, https://github.com/fossas/fossa-cli/blob/master/src/App/Fossa/Analyze/Types.hs#L40. And also few methods in App.Fossa.Analyze.ScanSummary.

If this is not in the AC of the ticket, ignore this.

src/App/Fossa/VSI/DynLinked.hs Outdated Show resolved Hide resolved
@jssblck
Copy link
Member Author

jssblck commented Mar 7, 2022

@meghfossa thanks for the comments, it now integrates cleanly with the scan summary:

[ INFO]
[ INFO] Scan Summary
[ INFO] ------------
[ INFO] fossa-cli branch vsi-dynamic-linking/tying-it-all-together (revision af19714b9803 (dirty) compiled with ghc-8.10)
[ INFO]
[ INFO] 1 projects scanned;  0 skipped,  1 succeeded,  0 failed,  1 analysis warning
[ INFO]
[ INFO] -
[ INFO] * dynamic linked dependency analysis: succeeded with 1 warning
[ INFO]   ** /lib/aarch64-linux-gnu/libpcre2-8.so.0
[ INFO]
[ INFO]
[ INFO] You can pass `--debug` option to eagerly show all warning and failure messages.
[ INFO] You can also view analysis summary with warning and error messages at: "/tmp/fossa-analyze-scan-summary.txt"
[ INFO] ------------
[ INFO]
[ INFO] Using project name: `binls`
[ INFO] Using revision: `1646690746`
[ INFO] Using branch: `vsi-dynamic-linking/tying-it-all-together`
[ INFO] ============================================================
[ INFO]
[ INFO]     View FOSSA Report:
[ INFO]     https://app.fossa.com/projects/custom%2b24357%2fbinls/refs/branch/vsi-dynamic-linking%2ftying-it-all-together/1646690746
[ INFO]
[ INFO] ============================================================

@jssblck jssblck merged commit f25758c into master Mar 7, 2022
@jssblck jssblck deleted the vsi-dynamic-linking/tying-it-all-together branch March 7, 2022 22:52
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants