feat: global matchers

[dwisiswant0]

Proposed changes

Close #4549

How has this been tested?

Templates:

1. http-templ-1.yaml

id: http-templ-1

info:
  name: "http-templ-1"
  author: "dwisiswant0"
  description: "HTTP Template 1"
  severity: "info"
  tags: "test"

http:
  - method: "GET"
    path:
      - "{{BaseURL}}/nonexistent"

2. matchers-static-templ-1.yaml

id: matchers-static-templ-1

info:
  name: "matchers-static-templ-1"
  author: "dwisiswant0"
  description: "Matchers-Static Template 1"
  severity: "info"
  tags: "test"

http:
  - global-matchers: true # enables global matching and applies them globally to all events from other templates

    matchers-condition: and
    matchers:
      - type: word
        words:
          - The requested URL

      - type: word
        words:
          - was not found on this server

    extractors:
      # - type: regex
      #   group: 1
      #   regex:
      #     - '<title>(.*)<\/title>'

      - type: dsl
        dsl:
          - template\-id # Output: matchers-static-templ-1 (current)
          - origin\-template\-id # Output: http-templ-1

Proof:

$ go run cmd/nuclei/main.go -u https://www.google.com -c 1 -t http-templ-1.yaml -t matchers-static-templ-1.yaml -silent
[matchers-static-templ-1] [static] [http] [info] https://www.google.com/nonexistent ["matchers-static-templ-1","http-templ-1"]

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[dwisiswant0]

After giving it some serious thought about the convention - me and @Ice3man543.

(I'll leave this here for tracking purposes)

Initially, the idea was to use the field name passive, but I realized that this could lead to confusion since it also appears as a flag and serves a different purpose (according to types.Options, it's for offline HTTP functionality).

Then I suggested changing the name to matchers-global, but after thinking it over, I found that this doesn't quite make sense because the requests defined in the template are not actually processed.

So, I decided to go with matchers-static. This name clearly indicates that these matchers are marked as static and will apply globally to all (result) events (specifically response) originating from other templates.

[dogancanbakir]

How about something at the template level? The feature name would be something like passive template or global template. Once enabled, we won't require the requests section, only the matchers / extractors.

[dwisiswant0]

How about something at the template level? The feature name would be something like passive template or global template. Once enabled, we won't require the requests section, only the matchers / extractors.

I definitely prefer the term static over passive (since its already used for something else). I feel like if we move the field to the top level, it could cause breaking changes.

The thing is, the current setup actually works; for global matching, users still need to define which protocol they want to use. The problem right now is that the abstraction is not on the operators side, so it only works for the specific http-protocol-based templates instead of being applied to all protocols that we support (for continuity). I'm curious if other protocols need this too. Maybe we should wait for @Ice3man543 to explain, or I can throw this into Notion as a proposal for discussion.

[ehsandeep]

1. Use of global-matchers instead of matcher-static in template and json(l) and global instead of static in CLI output line.

# example template
id: basic-example

info:
  name: Test HTTP Template
  author: pdteam
  severity: info

http:
  - global-matchers: true # enables global matching and applies them globally to all events from other templates

    matchers:
      - type: word
        words:
          - The requested URL
        negative: true

2. missing failed event with -ms option.

Current:

$ go run cmd/nuclei/main.go -u https://www.google.com -c 1 -t http-templ-1.yaml -t matchers-static-templ-1.yaml -ms -silent

[matchers-static-templ-1] [matched] [global] [http] [info] https://www.google.com/nonexistent ["matchers-static-templ-1","http-templ-1"]

Expected:

$ go run cmd/nuclei/main.go -u https://www.google.com -c 1 -t http-templ-1.yaml -t matchers-static-templ-1.yaml -ms -silent

[http-templ-1] [failed] [http] [info] https://www.google.com/nonexistent
[matchers-static-templ-1] [matched] [global] [http] [info] https://www.google.com/nonexistent ["matchers-static-templ-1","http-templ-1"]

[dwisiswant0]

On hold.

nuclei -validate -t http/global-matchers/

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.3.4

		projectdiscovery.io

[VER] Started metrics server at localhost:9092
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2098aec]

goroutine 1 [running]:
github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader.(*Store).areWorkflowOrTemplatesValid(0xc000824be0, 0xc001076738?, 0x0, 0x35d2e48)
	/home/dw1/Development/PD/nuclei/pkg/catalog/loader/loader.go:357 +0x20c
github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader.(*Store).areTemplatesValid(...)
	/home/dw1/Development/PD/nuclei/pkg/catalog/loader/loader.go:334
github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader.(*Store).ValidateTemplates(0xc000824be0)
	/home/dw1/Development/PD/nuclei/pkg/catalog/loader/loader.go:320 +0x345
github.com/projectdiscovery/nuclei/v3/internal/runner.(*Runner).RunEnumeration(0xc000c66000)
	/home/dw1/Development/PD/nuclei/internal/runner/runner.go:559 +0x15b4
main.main()
	/home/dw1/Development/PD/nuclei/cmd/nuclei/main.go:184 +0x6d5

UPDATE: Fixed in 12f6cb5