Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(libscan): delete map that keeps all file contents detected by FindLock to save memory #1556

Merged
merged 5 commits into from
Nov 10, 2022

Conversation

kotakanbe
Copy link
Member

@kotakanbe kotakanbe commented Nov 2, 2022

What did you implement:

vulsdoc/vuls#219

Reduce memory usage of FindLock

When we run config.toml with FindLock specified, it found lock files, jars, etc., in all directories under root. Unfortunately, due to a rough implementation, the contents of the detected files were stored on a Map, which increased memory usage when the number of detected files was large. In some cases, this resulted in Kernel Out of memory.

  • before: Res: 707MB
       $ /usr/bin/time -v ./vuls scan localhost
        ....
        Maximum resident set size (kbytes): 707428
        ....

  • after: Res: 250MB
       $ /usr/bin/time -v ./vuls scan localhost
        ....
        Maximum resident set size (kbytes): 249820
        ....

FindLockDirs

Automatic lockfile detection

If findLock=true and findLockDirs are specified, libraries on the local file system can be automatically detected by the find command.

NOTE: When findLock = true, the target lockfile depends on the scan mode and scan user privilege.
When scan mode is fast, it depends on the privilege of the scan user. If the scan user does not have the root privilege, lockfiles that require the root privilege will not be detected.
When scan mode is fast-root, lockfiles are detected with root privileges.

[servers]
[servers.ubuntu]
host         = "xxx.xxx.xxx"
port        = "22"
user        = "tamachi"
keyPath     = "/Users/amachi/.ssh/id_dsa"
findLock = true # auto detect lockfile
findLockDirs = [
  "/path/to/prject/lib",
  "/path/to/prject2/lib",
]

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Checklist:

  • Write tests
  • Write documentation
  • Check that there aren't other open pull requests for the same issue/feature
  • Format your source code by make fmt
  • Pass the test by make test
  • Provide verification config / commands
  • Enable "Allow edits from maintainers" for this PR
  • Update the messages below

Is this ready for review?: YES

@kotakanbe kotakanbe changed the title fix(libscan): delete Map that keeps all files detected by FindLock to save memory fix(libscan): delete map that keeps all file contents detected by FindLock to save memory Nov 2, 2022
@kotakanbe kotakanbe marked this pull request as ready for review November 10, 2022 00:48
@kotakanbe kotakanbe merged commit 1d97e91 into master Nov 10, 2022
@kotakanbe kotakanbe deleted the fix-memory-use-libscan branch November 10, 2022 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant