does OSV works on a local db or using an API? #887
Replies: 8 comments
-
OSV-Scanner uses the osv.dev API, but we have an experimental local database option: https://google.github.io/osv-scanner/experimental/#local-database-option The local database option has a few limitations like commit level scanning is not supported, and is potentially slower as all the range matching logic is done locally, and has to download the entire database of an ecosystem before it can start scanning (around 10's of MBs in size, depending on the ecosystem).
There is no rate limiting currently on the API, and the speed depends on the query. You'll need to give more details on "large assets" before I can give a more specific answer.
No, the local-db option is an alternative limited option for specific use cases, and there's no plans to make it the default option. |
Beta Was this translation helpful? Give feedback.
-
Hi @another-rex ! thanks for the answer. If I want to find additional data about a CVE \ GHSA \ etc. for enrichment like the API, can I do so using the local DB? using API can cause other issues like API calls limitations (number of calls per minutes) |
Beta Was this translation helpful? Give feedback.
-
Yes, the localdb option uses the OSV.dev data dumps directly, which should container the full published advisory. See the documentation linked in the previous post for more information. The only difference in the data dump and what the API has access to should be an enumerated list of affected commits. However, the API should have no rate limits currently, if you do run into being rate limited, please make an issue reporting it, as it is probably a bug! |
Beta Was this translation helpful? Give feedback.
-
great @another-rex ! do you know if it contains all of the CVEs \ GHSA in NVD\Github? |
Beta Was this translation helpful? Give feedback.
-
It contains all GHSAs (specifically everything that is published to their advisory-database repository + all malicious packages advisories, which should cover all GHSAs, though we have found odd cases where a GHSA isn't published correctly, but that's generally addressed by the Github team pretty quickly). We do not contain all CVE's, only CVE's related to open source C/C++ projects (@andrewpollock might be able to give more information here). For OS packages, we generally only publish OS specific ones, not general CVEs, as we found general CVEs often contain false positives since OS distros commonly backport fixes to older versions. |
Beta Was this translation helpful? Give feedback.
-
Hi @tomerse-sg https://osv.dev/blog/posts/introducing-broad-c-c++-support/ discusses the CVEs from the NVD in a little more detail. In a nutshell, to be successfully converted, the CVE needs to:
or
If there are specific CVEs of interest to you that are not found to be in OSV, feel free to give me a list and I can take a look. If you have too many, I'll teach you how to determine it for yourself :-) |
Beta Was this translation helpful? Give feedback.
-
thanks for the answer! thanks! |
Beta Was this translation helpful? Give feedback.
-
Yes, all the source code for ingesting and serving OSV data is located in the OSV.dev repository: https://github.com/google/osv.dev Specifically the part for ingesting CVEs is located here: https://github.com/google/osv.dev/tree/master/vulnfeeds/cves |
Beta Was this translation helpful? Give feedback.
-
Hi,
Does OSV-Scanner works with a local db or with an API?
In case of an API, what are the limitations? how does it handles large assets?
Is it planned to use a local-DB and not an API?
Beta Was this translation helpful? Give feedback.
All reactions