Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: global matchers #5701

Merged
merged 11 commits into from
Oct 14, 2024
Merged

feat: global matchers #5701

merged 11 commits into from
Oct 14, 2024

Conversation

dwisiswant0
Copy link
Member

@dwisiswant0 dwisiswant0 commented Oct 7, 2024

Proposed changes

Close #4549

How has this been tested?

Templates:

  1. http-templ-1.yaml
id: http-templ-1

info:
  name: "http-templ-1"
  author: "dwisiswant0"
  description: "HTTP Template 1"
  severity: "info"
  tags: "test"

http:
  - method: "GET"
    path:
      - "{{BaseURL}}/nonexistent"
  1. matchers-static-templ-1.yaml
id: matchers-static-templ-1

info:
  name: "matchers-static-templ-1"
  author: "dwisiswant0"
  description: "Matchers-Static Template 1"
  severity: "info"
  tags: "test"

http:
  - global-matchers: true # enables global matching and applies them globally to all events from other templates

    matchers-condition: and
    matchers:
      - type: word
        words:
          - The requested URL

      - type: word
        words:
          - was not found on this server

    extractors:
      # - type: regex
      #   group: 1
      #   regex:
      #     - '<title>(.*)<\/title>'

      - type: dsl
        dsl:
          - template\-id # Output: matchers-static-templ-1 (current)
          - origin\-template\-id # Output: http-templ-1

Proof:

$ go run cmd/nuclei/main.go -u https://www.google.com -c 1 -t http-templ-1.yaml -t matchers-static-templ-1.yaml -silent
[matchers-static-templ-1] [static] [http] [info] https://www.google.com/nonexistent ["matchers-static-templ-1","http-templ-1"]

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Ice3man543 <ice3man543@users.noreply.github.com>
Signed-off-by: Dwi Siswanto <git@dw1.io>
Signed-off-by: Dwi Siswanto <git@dw1.io>
also use `Set` method instead of `maps.Clone`

Signed-off-by: Dwi Siswanto <git@dw1.io>
@dwisiswant0 dwisiswant0 marked this pull request as ready for review October 8, 2024 21:07
@dwisiswant0 dwisiswant0 requested review from Ice3man543, dogancanbakir and ehsandeep and removed request for dogancanbakir October 8, 2024 21:07
@dwisiswant0
Copy link
Member Author

After giving it some serious thought about the convention - me and @Ice3man543.

(I'll leave this here for tracking purposes)

Initially, the idea was to use the field name passive, but I realized that this could lead to confusion since it also appears as a flag and serves a different purpose (according to types.Options, it's for offline HTTP functionality).

Then I suggested changing the name to matchers-global, but after thinking it over, I found that this doesn't quite make sense because the requests defined in the template are not actually processed.

So, I decided to go with matchers-static. This name clearly indicates that these matchers are marked as static and will apply globally to all (result) events (specifically response) originating from other templates.

@dogancanbakir
Copy link
Member

How about something at the template level? The feature name would be something like passive template or global template. Once enabled, we won't require the requests section, only the matchers / extractors.

@dwisiswant0
Copy link
Member Author

How about something at the template level? The feature name would be something like passive template or global template. Once enabled, we won't require the requests section, only the matchers / extractors.

I definitely prefer the term static over passive (since its already used for something else). I feel like if we move the field to the top level, it could cause breaking changes.

The thing is, the current setup actually works; for global matching, users still need to define which protocol they want to use. The problem right now is that the abstraction is not on the operators side, so it only works for the specific http-protocol-based templates instead of being applied to all protocols that we support (for continuity). I'm curious if other protocols need this too. Maybe we should wait for @Ice3man543 to explain, or I can throw this into Notion as a proposal for discussion.

Copy link
Member

@ehsandeep ehsandeep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Use of global-matchers instead of matcher-static in template and json(l) and global instead of static in CLI output line.
# example template
id: basic-example

info:
  name: Test HTTP Template
  author: pdteam
  severity: info

http:
  - global-matchers: true # enables global matching and applies them globally to all events from other templates

    matchers:
      - type: word
        words:
          - The requested URL
        negative: true
  1. missing failed event with -ms option.

Current:

$ go run cmd/nuclei/main.go -u https://www.google.com -c 1 -t http-templ-1.yaml -t matchers-static-templ-1.yaml -ms -silent

[matchers-static-templ-1] [matched] [global] [http] [info] https://www.google.com/nonexistent ["matchers-static-templ-1","http-templ-1"]

Expected:

$ go run cmd/nuclei/main.go -u https://www.google.com -c 1 -t http-templ-1.yaml -t matchers-static-templ-1.yaml -ms -silent

[http-templ-1] [failed] [http] [info] https://www.google.com/nonexistent
[matchers-static-templ-1] [matched] [global] [http] [info] https://www.google.com/nonexistent ["matchers-static-templ-1","http-templ-1"]

Signed-off-by: Dwi Siswanto <git@dw1.io>
Signed-off-by: Dwi Siswanto <git@dw1.io>
This will end up generating 2 events from the same
`scan.ScanContext` if one of the templates has
`global-matchers` enabled. This way, non-
`global-matchers` templates can enter the
`writeFailureCallback` func to log failure output.

Signed-off-by: Dwi Siswanto <git@dw1.io>
Signed-off-by: Dwi Siswanto <git@dw1.io>
Signed-off-by: Dwi Siswanto <git@dw1.io>
@dwisiswant0
Copy link
Member Author

dwisiswant0 commented Oct 11, 2024

On hold.

nuclei -validate -t http/global-matchers/

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.3.4

		projectdiscovery.io

[VER] Started metrics server at localhost:9092
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2098aec]

goroutine 1 [running]:
github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader.(*Store).areWorkflowOrTemplatesValid(0xc000824be0, 0xc001076738?, 0x0, 0x35d2e48)
	/home/dw1/Development/PD/nuclei/pkg/catalog/loader/loader.go:357 +0x20c
github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader.(*Store).areTemplatesValid(...)
	/home/dw1/Development/PD/nuclei/pkg/catalog/loader/loader.go:334
github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader.(*Store).ValidateTemplates(0xc000824be0)
	/home/dw1/Development/PD/nuclei/pkg/catalog/loader/loader.go:320 +0x345
github.com/projectdiscovery/nuclei/v3/internal/runner.(*Runner).RunEnumeration(0xc000c66000)
	/home/dw1/Development/PD/nuclei/internal/runner/runner.go:559 +0x15b4
main.main()
	/home/dw1/Development/PD/nuclei/cmd/nuclei/main.go:184 +0x6d5

UPDATE: Fixed in 12f6cb5

@dwisiswant0 dwisiswant0 marked this pull request as draft October 11, 2024 18:22
Signed-off-by: Dwi Siswanto <git@dw1.io>
Signed-off-by: Dwi Siswanto <git@dw1.io>
@dwisiswant0 dwisiswant0 marked this pull request as ready for review October 11, 2024 19:27
@ehsandeep ehsandeep merged commit cc5c550 into dev Oct 14, 2024
10 of 12 checks passed
@ehsandeep ehsandeep deleted the dwisiswant0/feat/global-matchers branch October 14, 2024 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Global passive matchers / extractors for nuclei scan
4 participants