-
-
Notifications
You must be signed in to change notification settings - Fork 11.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[📜] Spec for declaration of vulns to help scanners check find rate #1441
Comments
Hi @bkimminich , Adding https://owasp.slack.com/archives/CPMEWT342/p1597085979033700 and SasanLabs/VulnerableApp#164 for reference. thanks, |
|
Hi @bkimminich , Following are my queries:
such that anyone that is parsing knows what are the identifier types so that they need not to break values and then know what the identifier is.
Also This is quite different from my thinking where in i have written in a way that each URL has Vulnerabilities linked so that scanner Alert for that URL can be matched with the Vulnerabilities linked something like: url:
- "/vuln/xss/level1"
type:
- CWE:
- "CWE-79"
WASC:
- "WASC-8"
- CWE:
- "CWE-89"
WASC:
- "WASC-5" ZAP alerts: Url:
- "/vuln/xss/level1"
PluginId:
- "6"
VulnerabilityType:
CWE:
- "CWE-79" now URL is the key which can be used to map the findings. thanks, |
Flags are indicators that might show up in a scanner report. The URL that has an XSS problem can be a flag. The parameter name susceptible to the payload could be a second one. The responsible code snippet might be a third. Etc. The more flags the scanner report contains, the more likely it found the actual vulnerability. Even the CWE, WASC or OWASP ids could be just more flags, as at least one of those probably shows up in the scanner finding as a reference. I'd put those separate still to avoid confusion and make it a bit human readable too. But I don't think named properties per id issuer are really needed. |
Your ZAP alert example is perfect, it would match the URL flag and the CWE type in its finding, so it's very likely a true positive. If it would report the URL but some different CWE, it maybe missed or misclassified the actual vuln. |
Hi @bkimminich , Yes this makes sense and i think we can divide '/scanner' endpoint in SasanLabs/VulnerableApp into 2 parts one is the way to help scanners find the vulnerability (its requirements) and other is something similar to Also one doubt i have related to scanners is, are we thinking that scanner report/alerts will have all these flag information ? like say we have a flag and say scanners are not having that information in alerts then that can be little tricky or are we assuming that scanners will save the entire response in the report ? thanks, |
What I don't quite get is how telling a Scanner how a vulnerability can be discovered is helpful. It sounds to me as this gives the tools more information than it could ever have about a real app. I think the hints shouldn't go further than something like an OpenAPI definition of the API, to ensure that the scanner is able to assess all endpoints to its fullest potential. More than that feels like "cheating". Having hint to help the tools assess how many vulnerabilities it has discovered seems more helpful, thought this can normally be determined in a couple of minutes when comparing the findings against the vulnerabilities of the vuln app. Would be interesting how if this could help this process. |
No, that's why we might want to have rather smaller fragmented flags instead of long strings which might not be exact matches. Another option would be to support RegEx-flags optionalle or make flags RegExes by default. Here is a hopefully helpful example of how I imagine it to work: For the XSS vulnerability of the search box in Juice Shop the application:
name: OWASP Juice Shop
source: https://github.com/bkimminich/juice-shop
vulnerabilities:
-
type:
- "CWE-79"
- "CWE-749"
- "A7:2017"
- "WASC-8"
flags:
- "DomSanitizer"
- "bypassSecurityTrustHtml"
- "q="
- "#/search"
- "main-es[0-9]*.js" Here's a finding from a ZAP baseline scan report that actually found the usage of an insecure JavaScript function: My suggestion for the scanner provider would now be to run all the
The interpretation of this result is up to the vendor of the scanner in my opinion, but in this case it's pretty clear that ZAP found the root cause for some potential XSS, but - at least in this finding - not its execution. In the actual report I used the execution was not found, but we could imagine ZAP to report another finding from the same scan that matches some of the execution flags. As a scanner vendor I'd then be happy that I found some real issue from the execution and root cause angle. Well done. Repeating this for all specified vulnerabilities would give you a good picture of how well the scanner did on OWASP Juice Shop. Addendum: In the Juice Shop there are several XSS holes, and at least 3 or 4 of them come from using the |
Actually '/scanner' endpoint point is like an openAPI definition only, it helps scanners know about the endpoint requirement like
Now this information is telling that "JWT" query param is the one where scanners need to inject the payload. Scanners first find the endpoint (using spider) then they do the active scan (This is fully automated version where not manual intervention is there) but if they want to inject some payload they need to know where to provide the payload without that it is not possible for them to find the parameter which contains the payload. Now this scanner endpoint helps them to know the requirements of the endpoint. I was thinking to have URL (as a flag) and single end point to serve both the purposes but now __vuln.json makes more sense. Now the concern related to it is, in real applications this information is not there and how this information can be utiilsed ?
Going further may be scanner only understand this '/scanner' endpoint or openApi endpoint and incorporate it into the scan rule framework. Thanks, |
My own confusion with the But your proposed steps clearly separate the two concepts. That's probably the most important thing, to allow clear expectation management when choosing one or both the concepts for your own (vuln) app. |
I think we can have a call and discuss on this. I need to go through your previous comment too. Let me think more and then if possible we can have a call to discuss. |
Hi @bkimminich , Thanks for the amazing explanation. Following are my thoughts: vulnerabilities:
-
type:
- CWE :
- 79
- A7:
- 2017
- WASC:
- 8
flags:
- Mandatory:
- "main-es[0-9]*.js"
- Additional:
- "#/search"
- "DomSanitizer"
- "bypassSecurityTrustHtml"
- "q=" I also think at the same point that mandatory is something like URL (may be always) and everything else can go the same way you have explained. Also we can divide flags further into Regex one's and non-Regex one's, this separation can be either as: flags:
- Mandatory:
- Regex:
- "main-es[0-9]*.js"
- Non-Regex:
- "#/search"
- Additional:
- Regex:
- Non-Regex:
- "DomSanitizer"
- "bypassSecurityTrustHtml"
- "q=" or we can have some other way to differentiate it, However i also think that we can even go with regex always. thanks, |
Having mandatory flags would be too limiting or at least put too much anticipation into the whole thing. A scanner might explore the search field and get an XSS executed, another one - like ZAP - might find the insecure function call in the minified JS, yet another might find something in the source code (I didn't add flags for that yet) etc. |
But URL might remain the same and that can be mandatory and in case of minified versions we need not to qualify the parameter as mandatory because it changes at runtime. |
I get your point, but if you are a SAST scanner you won't know about any URLs. You'd love some class names, function or variable names or a line of code reference etc. |
So the usecase which i was thinking is, say an endpoint is vulnerable to sqlInjection like This can suit to our usecase where ZAP can configure the scan job and that job is capable to rate them self. I think same can be applied to real world applications where it can help those applications to do some kind of basic security sanity using this approach. Please correct me if i am thinking wrong here. |
I would prefer real-world-app-developers to focus on not having Injection issues in their webapp instead of writing any declarations to help security tools find potential Injection issues. That's what you pay the security tool vendors for, to find the broken stuff. If they don't, they're not good at their job. Giving them training wheels won't make them better...😁 But giving them something to compare their unassisted (!) success rate against as a vulnapp-developer, that helps them work on their deficits... 😇 |
Ahh you are talking about SAST tools like Fortify or SonarQube etc ? I haven't thought about them. Thanks for mentioning them. vulnerabilities:
-
scannerType:
- "Dynamic scanning"
type:
- CWE :
- 79
- A7:
- 2017
- WASC:
- 8
flags:
- Mandatory:
- "main-es[0-9]*.js"
- Additional:
- "#/search"
- "DomSanitizer"
- "bypassSecurityTrustHtml"
- "q="
-
scannerType:
- "Static scanning"
type:
- CWE :
- 79
- A7:
- 2017
- WASC:
- 8
flags:
- Additional:
- "#/search"
- "DomSanitizer"
- "bypassSecurityTrustHtml"
- "q=" |
As a vulnapp developer, I don't really want to do the exercise of assuming what a DAST and what a SAST scanner will find. I can't imagine many other will... I also don't want to prescribe how many |
Hmm makes sense. 👍 |
Haha, actually developers are not writing it, if we look at Swagger etc they are doing similar job only (giving a schema to clients to know how to call) similarly my attempt was to give the same information from vulnerableApp. Ok i will rethink on this. 👍 |
cc @sj @greysteil who may well have some SAST related comments |
Hi @bkimminich , I was thinking more about it and was thinking that if we are not going to categorise these flag then it might make Scanner's job much more tough reason being the one who is writing the vulnerability knows more about the flags like "#search" is an identifier to search box than the one who is understanding vuln.json. And coming to the point of DAST and SAST, i think we just need to categorise between one who is accessing the Source Code and one who is doing a black box scanning like ZAP (Where JS/Html can be an exception to this) thanks, |
I agree that putting more time into a more explicit vuln specification makes the life of the scanner vendor a lot easier. But it makes the like of the vuln app developer a lot harder, because now they have to come up with the mapping between flags more likely to be found by SAST vs. DAST tools etc. With the simple "here's a set of flags, good luck!" approach the interpretation becomes entirely the scanner vendor's job. Which to me seems like fair play, as they're improving their often commercial products with it, while most vuln apps are open source maintained by volunteers. |
Yes agreed however I was thinking that say we have just given flags(current approach) and now say ZAP wants to use them then they need to create something similar to the vuln.json with a boolean which states that this flag is useful for them and this flag is not useful for them (say file they named as vuln_zap.json) and they will start using vuln_zap.json file instead, which can reduce the usage and also vuln_zap.json might miss newer updates. so i was thinking that we can just add a category say Unknown/Not_Categorised and once the users of vulnerableApp use our specification they can tell us the correct category in that case it can help us as well as the scanners and one scanner's work can help others too. The newer vulnerabilities can have or we can try to have these categories and the older code can be maintained by community. thanks, |
Okay, actually we could throw the flags into different arrays, as already done with
|
yes yes, i was thinking something similar: vulnerabilities:
-
type:
- CWE :
- 79
- A7:
- 2017
- WASC:
- 8
flags:
- source:
- "main-es[0-9]*.js"
- "#/search"
- runtime:
- "/search"
- not_categorised:
- "DomSanitizer"
- "bypassSecurityTrustHtml"
- "q=" thanks, |
I'm on board with the general idea, except for the unnecessary type split into sub categories. I'll write up a conclusive example in the next days using such a split into different flag types. |
|
(see #1441 for background information and to leave feedback)
|
Draft 3 looks like has some issues as http_method is not there in first object and as it is a list (thinking from Java prespective) it should be present but with null value. Also how can some scanner know that this location is useful for them. I think it would be great if we can have Code and Runtime added in it. |
I did an attempt with And the |
Agreed, js/html/static resources are some kind of exceptions however i think that can reside at both the places with say type. This will make spec more cleaner as then whatever information for SAST is there in the configuration is relevant for them else they need to remove it or assume it that it is not for them and at runtime it might be tough for them. I agree this will increase duplicates but i think these cases are only for static resources. thanks, |
I've moved the most current drafts and the overall proposal into this GitHub Gist to decouple any discussions from Juice Shop. Please feel free to make suggestions and discuss the drafts over there! Thanks! |
This thread has been automatically locked because it has not had recent activity after it was closed. 🔒 Please open a new issue for regressions or related bugs. |
Back in 2016 an idea of having a
__vulns.json
file in vulnerable applications came up and was prepared by members of the OWASP ZAP, VWAD and Juice Shop teams. It was supposed to allow scanners/tools to assess their success rate byvulnerabilities
arrayflags
array in their own reported finding descriptions/root causes etc.The Juice Shop version of this file looked like this back then:
The JSON spec was the following (see https://github.com/OWASP/OWASP-VWAD/blob/6a8b54004db5b48278e82f2cc856fdc229c80ca3/src/owasp-wiki/deprecated/vulns-json/schema.json)
Because the idea was not pursued further and it saw no adoption, it was dropped in 2017.
In this issue ideas can be collected and the original schema can be taken apart. Personally, I still think a light-weight approach is a good idea, but some meta data could be added (like URL of the application repository as a unique identifier) or existing one improves (like the - in hindsight - seemingly arbitrary enumeration of
type
entries) - Any constructive criticism is welcome.The text was updated successfully, but these errors were encountered: