Skip to content

Commit

Permalink
[Filebeat] Fix threatintel.indicator.url.full field not populating (#…
Browse files Browse the repository at this point in the history
…26508)

* #26351: Fix Threat Intel Full URL field

* update changelog

* remove commented items

* updated pipelines per comments
  • Loading branch information
legoguy1000 authored Jun 29, 2021
1 parent 9e670f7 commit c45aba5
Show file tree
Hide file tree
Showing 12 changed files with 312 additions and 41 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -388,6 +388,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add improvements to the azure activitylogs and platformlogs ingest pipelines. {pull}26148[26148]
- Fix `kibana.log` pipeline when `event.duration` calculation becomes a Long. {issue}24556[24556] {pull}25675[25675]
- Removed incorrect `http.request.referrer` field from `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
- Fix `threatintel.indicator.url.full` not being populated. {issue}26351[26351] {pull}26508[26508]

*Heartbeat*

Expand Down
18 changes: 4 additions & 14 deletions x-pack/filebeat/module/threatintel/abuseurl/ingest/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,6 @@ processors:
- set:
field: threatintel.indicator.type
value: url
- set:
field: threatintel.indicator.url.scheme
value: https
if: ctx?.threatintel?.abuseurl?.url.startsWith('https:')
- set:
field: threatintel.indicator.url.scheme
value: http
if: ctx?.threatintel?.abuseurl?.url.startsWith('http:')
- date:
field: threatintel.abuseurl.date_added
target_field: threatintel.indicator.first_seen
Expand All @@ -51,11 +43,10 @@ processors:
target_field: threatintel.indicator.url
keep_original: true
remove_if_successful: true
- rename:
field: threatintel.abuseurl.url
target_field: threatintel.indicator.url.full
ignore_missing: true
if: ctx?.threatintel?.indicator?.url?.original == null && ctx?.threatintel?.abuseurl?.url != null
- set:
field: threatintel.indicator.url.full
copy_from: threatintel.indicator.url.original
ignore_empty_value: true
- rename:
field: threatintel.abuseurl.host
target_field: threatintel.indicator.domain
Expand All @@ -65,7 +56,6 @@ processors:
target_field: event.reference
ignore_missing: true


# Host can be both IP addresses and domain names
- grok:
field: threatintel.abuseurl.host
Expand Down

Large diffs are not rendered by default.

11 changes: 5 additions & 6 deletions x-pack/filebeat/module/threatintel/anomali/ingest/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ processors:
if: "ctx?.threatintel?.anomali?.valid_from != null"
- grok:
field: threatintel.anomali.pattern
patterns:
patterns:
- "^\\[%{DATA:_tmp.threattype}:value%{SPACE}=%{SPACE}'%{DATA:_tmp.threatvalue}'\\]"
- rename:
field: _tmp.threattype
Expand All @@ -82,11 +82,10 @@ processors:
keep_original: true
remove_if_successful: true
if: ctx?.threatintel?.indicator?.type == 'url'
- rename:
field: _tmp.threatvalue
target_field: threatintel.indicator.url.full
ignore_missing: true
if: ctx?.threatintel?.indicator?.type == 'url' && ctx?.threatintel?.indicator?.url?.original == null
- set:
field: threatintel.indicator.url.full
copy_from: threatintel.indicator.url.original
ignore_empty_value: true
- rename:
field: _tmp.threatvalue
target_field: threatintel.indicator.email.address
Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,11 @@ processors:
field: error.message
value: 'Cannot parse url field `{{{ json.url }}}`: {{{ _ingest.on_failure_message }}}'

- set:
field: threatintel.indicator.url.full
copy_from: threatintel.indicator.url.original
ignore_empty_value: true

- rename:
field: json.country
target_field: threatintel.indicator.geo.country_iso_code
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "ax1a6o38z.example.org",
"threatintel.indicator.url.full": "https://ax1a6o38z.example.org/enec3i/f1n8fv?4shpqq9=fbo9osx8p",
"threatintel.indicator.url.original": "https://ax1a6o38z.example.org/enec3i/f1n8fv?4shpqq9=fbo9osx8p",
"threatintel.indicator.url.path": "/enec3i/f1n8fv",
"threatintel.indicator.url.query": "4shpqq9=fbo9osx8p",
Expand Down Expand Up @@ -426,6 +427,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "beko3.example.com",
"threatintel.indicator.url.full": "https://beko3.example.com/vkelnz/jdz6zf-ga?g39fu=88309ge",
"threatintel.indicator.url.original": "https://beko3.example.com/vkelnz/jdz6zf-ga?g39fu=88309ge",
"threatintel.indicator.url.path": "/vkelnz/jdz6zf-ga",
"threatintel.indicator.url.query": "g39fu=88309ge",
Expand Down Expand Up @@ -550,6 +552,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "sevs82.example.com",
"threatintel.indicator.url.full": "http://sevs82.example.com/c5-d/hdajog?4rs78hl=wvwi",
"threatintel.indicator.url.original": "http://sevs82.example.com/c5-d/hdajog?4rs78hl=wvwi",
"threatintel.indicator.url.path": "/c5-d/hdajog",
"threatintel.indicator.url.query": "4rs78hl=wvwi",
Expand Down Expand Up @@ -989,6 +992,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "faahk3drf.example.net",
"threatintel.indicator.url.full": "http://faahk3drf.example.net/julf98x5/0g1t8f?cbffxs2qv=vwgz",
"threatintel.indicator.url.original": "http://faahk3drf.example.net/julf98x5/0g1t8f?cbffxs2qv=vwgz",
"threatintel.indicator.url.path": "/julf98x5/0g1t8f",
"threatintel.indicator.url.query": "cbffxs2qv=vwgz",
Expand Down Expand Up @@ -1191,6 +1195,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "p9okf0.example.org",
"threatintel.indicator.url.full": "http://p9okf0.example.org/jyb3n8f/f55vfyt48?s2n=0t2d",
"threatintel.indicator.url.original": "http://p9okf0.example.org/jyb3n8f/f55vfyt48?s2n=0t2d",
"threatintel.indicator.url.path": "/jyb3n8f/f55vfyt48",
"threatintel.indicator.url.query": "s2n=0t2d",
Expand Down Expand Up @@ -1236,6 +1241,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "fxkeo24m.example.com",
"threatintel.indicator.url.full": "https://fxkeo24m.example.com/y75tg7sw/jnnu9xmc?apus=ob1hnba4",
"threatintel.indicator.url.original": "https://fxkeo24m.example.com/y75tg7sw/jnnu9xmc?apus=ob1hnba4",
"threatintel.indicator.url.path": "/y75tg7sw/jnnu9xmc",
"threatintel.indicator.url.query": "apus=ob1hnba4",
Expand Down Expand Up @@ -1596,6 +1602,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "ke4ffyj5.example.com",
"threatintel.indicator.url.full": "http://ke4ffyj5.example.com/t-9ikyrtt/ai91?s6u=3y1",
"threatintel.indicator.url.original": "http://ke4ffyj5.example.com/t-9ikyrtt/ai91?s6u=3y1",
"threatintel.indicator.url.path": "/t-9ikyrtt/ai91",
"threatintel.indicator.url.query": "s6u=3y1",
Expand Down Expand Up @@ -1757,6 +1764,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "rl27d.example.net",
"threatintel.indicator.url.full": "https://rl27d.example.net/ko6/4rtt?b12=o4mgzz2kk",
"threatintel.indicator.url.original": "https://rl27d.example.net/ko6/4rtt?b12=o4mgzz2kk",
"threatintel.indicator.url.path": "/ko6/4rtt",
"threatintel.indicator.url.query": "b12=o4mgzz2kk",
Expand Down Expand Up @@ -1841,6 +1849,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "6ygk0y.example.com",
"threatintel.indicator.url.full": "http://6ygk0y.example.com/t520/4twe?ql4bhkpop=yfpkef",
"threatintel.indicator.url.original": "http://6ygk0y.example.com/t520/4twe?ql4bhkpop=yfpkef",
"threatintel.indicator.url.path": "/t520/4twe",
"threatintel.indicator.url.query": "ql4bhkpop=yfpkef",
Expand Down Expand Up @@ -1885,6 +1894,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "rcsr9o.example.net",
"threatintel.indicator.url.full": "http://rcsr9o.example.net/e6f/08b?8d2y=d-42fr-",
"threatintel.indicator.url.original": "http://rcsr9o.example.net/e6f/08b?8d2y=d-42fr-",
"threatintel.indicator.url.path": "/e6f/08b",
"threatintel.indicator.url.query": "8d2y=d-42fr-",
Expand Down Expand Up @@ -2089,6 +2099,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "cc7d.example.com",
"threatintel.indicator.url.full": "https://cc7d.example.com/kxxwobg/hd6omn?tr8=essb",
"threatintel.indicator.url.original": "https://cc7d.example.com/kxxwobg/hd6omn?tr8=essb",
"threatintel.indicator.url.path": "/kxxwobg/hd6omn",
"threatintel.indicator.url.query": "tr8=essb",
Expand Down Expand Up @@ -2252,6 +2263,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "v9aqrp81q.example.net",
"threatintel.indicator.url.full": "http://v9aqrp81q.example.net/psuj4bs/rvp?qufy=ymryh",
"threatintel.indicator.url.original": "http://v9aqrp81q.example.net/psuj4bs/rvp?qufy=ymryh",
"threatintel.indicator.url.path": "/psuj4bs/rvp",
"threatintel.indicator.url.query": "qufy=ymryh",
Expand Down Expand Up @@ -2491,6 +2503,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "o4kqv8b8.example.net",
"threatintel.indicator.url.full": "https://o4kqv8b8.example.net/gm4d-9gt/v2iqt?x65ry67ao=skta9rp",
"threatintel.indicator.url.original": "https://o4kqv8b8.example.net/gm4d-9gt/v2iqt?x65ry67ao=skta9rp",
"threatintel.indicator.url.path": "/gm4d-9gt/v2iqt",
"threatintel.indicator.url.query": "x65ry67ao=skta9rp",
Expand Down Expand Up @@ -2811,6 +2824,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "91p0p.example.com",
"threatintel.indicator.url.full": "https://91p0p.example.com/easx3j6iy/xvnchuoa?dvkljl=h21",
"threatintel.indicator.url.original": "https://91p0p.example.com/easx3j6iy/xvnchuoa?dvkljl=h21",
"threatintel.indicator.url.path": "/easx3j6iy/xvnchuoa",
"threatintel.indicator.url.query": "dvkljl=h21",
Expand Down Expand Up @@ -2970,6 +2984,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "lzr6.example.org",
"threatintel.indicator.url.full": "https://lzr6.example.org/a7og/4vpv?e7k5=wun",
"threatintel.indicator.url.original": "https://lzr6.example.org/a7og/4vpv?e7k5=wun",
"threatintel.indicator.url.path": "/a7og/4vpv",
"threatintel.indicator.url.query": "e7k5=wun",
Expand Down Expand Up @@ -3130,6 +3145,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "932.example.com",
"threatintel.indicator.url.full": "http://932.example.com/1xmdjyom/tf3inx1?s6zgr=ajgw",
"threatintel.indicator.url.original": "http://932.example.com/1xmdjyom/tf3inx1?s6zgr=ajgw",
"threatintel.indicator.url.path": "/1xmdjyom/tf3inx1",
"threatintel.indicator.url.query": "s6zgr=ajgw",
Expand Down Expand Up @@ -3258,6 +3274,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "0te9x75e.example.net",
"threatintel.indicator.url.full": "https://0te9x75e.example.net/y2cbl5ov5/u-s9?vhppw120=bt0ze0du3",
"threatintel.indicator.url.original": "https://0te9x75e.example.net/y2cbl5ov5/u-s9?vhppw120=bt0ze0du3",
"threatintel.indicator.url.path": "/y2cbl5ov5/u-s9",
"threatintel.indicator.url.query": "vhppw120=bt0ze0du3",
Expand Down Expand Up @@ -3304,6 +3321,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "b7qdtnl8f.example.org",
"threatintel.indicator.url.full": "http://b7qdtnl8f.example.org/z2a-tx3ip/7cv?9a67ct3mb=ijse",
"threatintel.indicator.url.original": "http://b7qdtnl8f.example.org/z2a-tx3ip/7cv?9a67ct3mb=ijse",
"threatintel.indicator.url.path": "/z2a-tx3ip/7cv",
"threatintel.indicator.url.query": "9a67ct3mb=ijse",
Expand Down Expand Up @@ -3434,6 +3452,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "tfva.example.org",
"threatintel.indicator.url.full": "https://tfva.example.org/iih3qkj/b04g7?dwosh0qmt=wi9ao",
"threatintel.indicator.url.original": "https://tfva.example.org/iih3qkj/b04g7?dwosh0qmt=wi9ao",
"threatintel.indicator.url.path": "/iih3qkj/b04g7",
"threatintel.indicator.url.query": "dwosh0qmt=wi9ao",
Expand Down Expand Up @@ -3480,6 +3499,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "erg2.example.com",
"threatintel.indicator.url.full": "https://erg2.example.com/4ys/vywa93c?7oru=evpi",
"threatintel.indicator.url.original": "https://erg2.example.com/4ys/vywa93c?7oru=evpi",
"threatintel.indicator.url.path": "/4ys/vywa93c",
"threatintel.indicator.url.query": "7oru=evpi",
Expand Down Expand Up @@ -3531,6 +3551,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "0elz6c.example.com",
"threatintel.indicator.url.full": "https://0elz6c.example.com/3nhx/cadsn6?kfcj94=gnl",
"threatintel.indicator.url.original": "https://0elz6c.example.com/3nhx/cadsn6?kfcj94=gnl",
"threatintel.indicator.url.path": "/3nhx/cadsn6",
"threatintel.indicator.url.query": "kfcj94=gnl",
Expand Down Expand Up @@ -3577,6 +3598,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "6i0-utr.example.com",
"threatintel.indicator.url.full": "https://6i0-utr.example.com/hsv/50qcugwt?xcl=ofr",
"threatintel.indicator.url.original": "https://6i0-utr.example.com/hsv/50qcugwt?xcl=ofr",
"threatintel.indicator.url.path": "/hsv/50qcugwt",
"threatintel.indicator.url.query": "xcl=ofr",
Expand Down Expand Up @@ -3714,6 +3736,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "e5el.example.net",
"threatintel.indicator.url.full": "http://e5el.example.net/rncer/fky?8tc53bbz=1pd-6w5",
"threatintel.indicator.url.original": "http://e5el.example.net/rncer/fky?8tc53bbz=1pd-6w5",
"threatintel.indicator.url.path": "/rncer/fky",
"threatintel.indicator.url.query": "8tc53bbz=1pd-6w5",
Expand Down Expand Up @@ -3758,6 +3781,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "eryz36i.example.net",
"threatintel.indicator.url.full": "http://eryz36i.example.net/9a86hdj/zti5r9fx?ahz=l7dsg01qo",
"threatintel.indicator.url.original": "http://eryz36i.example.net/9a86hdj/zti5r9fx?ahz=l7dsg01qo",
"threatintel.indicator.url.path": "/9a86hdj/zti5r9fx",
"threatintel.indicator.url.query": "ahz=l7dsg01qo",
Expand Down Expand Up @@ -3804,6 +3828,7 @@
"threatintel.indicator.provider": "Default Organization",
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "i-pb.example.com",
"threatintel.indicator.url.full": "http://i-pb.example.com/pjmy3/w0tgzb?noe1pr9=eiwcfihd",
"threatintel.indicator.url.original": "http://i-pb.example.com/pjmy3/w0tgzb?noe1pr9=eiwcfihd",
"threatintel.indicator.url.path": "/pjmy3/w0tgzb",
"threatintel.indicator.url.query": "noe1pr9=eiwcfihd",
Expand Down
23 changes: 12 additions & 11 deletions x-pack/filebeat/module/threatintel/misp/ingest/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -110,13 +110,13 @@ processors:
if: "ctx?.threatintel?.indicator?.type == 'file' && ctx?.threatintel?.misp?.attribute?.type == 'filename'"
- grok:
field: threatintel.misp.attribute.type
patterns:
patterns:
- "%{WORD}\\|%{WORD:_tmp.hashtype}"
ignore_missing: true
if: ctx?.threatintel?.misp?.attribute?.type.startsWith('filename|')
- grok:
field: threatintel.misp.attribute.value
patterns:
patterns:
- "%{DATA:threatintel.indicator.file.name}\\|%{GREEDYDATA:_tmp.hashvalue}"
ignore_missing: true
if: ctx?.threatintel?.misp?.attribute?.type.startsWith('filename|')
Expand All @@ -136,11 +136,12 @@ processors:
keep_original: true
remove_if_successful: true
if: ctx?.threatintel?.indicator?.type == 'url' && ctx?.threatintel?.misp?.attribute?.type != 'uri'
- rename:
field: threatintel.misp.attribute.value
target_field: threatintel.indicator.url.full
ignore_missing: true
if: "ctx?.threatintel?.indicator?.type == 'url' && ctx?.threatintel?.indicator?.url?.original == null && ctx?.threatintel?.misp?.attribute?.type != 'uri'"

- set:
field: threatintel.indicator.url.full
copy_from: threatintel.indicator.url.original
ignore_empty_value: true
if: "ctx?.threatintel?.indicator?.type == 'url' && ctx?.threatintel?.misp?.attribute?.type != 'uri'"

## Regkey indicator operations
- set:
Expand All @@ -154,7 +155,7 @@ processors:
if: "ctx?.threatintel?.indicator?.type == 'windows-registry-key' && ctx?.threatintel?.misp?.attribute?.type == 'regkey'"
- grok:
field: threatintel.misp.attribute.value
patterns:
patterns:
- "%{DATA:threatintel.indicator.registry.key}\\|%{DATA:threatintel.indicator.registry.value}"
ignore_missing: true
if: "ctx?.threatintel?.misp?.attribute?.type == 'regkey|value'"
Expand Down Expand Up @@ -192,13 +193,13 @@ processors:
if: "ctx?.threatintel?.indicator?.type == 'ipv4-addr' && ctx?.threatintel?.misp?.attribute?.type != 'domain|ip' && !['ip-src|port', 'ip-dst|port'].contains(ctx?.threatintel?.misp?.attribute?.type)"
- grok:
field: threatintel.misp.attribute.value
patterns:
patterns:
- "%{DATA:threatintel.indicator.domain}\\|%{IP:threatintel.indicator.ip}"
ignore_missing: true
if: ctx.threatintel?.misp?.attribute?.type == 'domain|ip'
- grok:
field: threatintel.misp.attribute.value
patterns:
patterns:
- "%{IP:threatintel.indicator.ip}\\|%{NUMBER:threatintel.indicator.port}"
ignore_missing: true
if: "['ip-src|port', 'ip-dst|port'].contains(ctx.threatintel?.misp?.attribute?.type)"
Expand Down Expand Up @@ -245,7 +246,7 @@ processors:
.filter(t -> t.startsWith('tlp:'))
.map(t -> t.replace('tlp:', ''))
.collect(Collectors.toList());
ctx.tags = tags;
ctx.threatintel.indicator.marking = [ 'tlp': tlpTags ];
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@
"threatintel.indicator.scanner_stats": 2,
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "www.virustotal.com",
"threatintel.indicator.url.full": "https://www.virustotal.com/file/7fa4482bfbca550ce296d8e791b1091d60d733ea8042167fd0eb853530584452/analysis/1486030116/",
"threatintel.indicator.url.original": "https://www.virustotal.com/file/7fa4482bfbca550ce296d8e791b1091d60d733ea8042167fd0eb853530584452/analysis/1486030116/",
"threatintel.indicator.url.path": "/file/7fa4482bfbca550ce296d8e791b1091d60d733ea8042167fd0eb853530584452/analysis/1486030116/",
"threatintel.indicator.url.scheme": "https",
Expand Down Expand Up @@ -527,6 +528,7 @@
"threatintel.indicator.scanner_stats": 0,
"threatintel.indicator.type": "url",
"threatintel.indicator.url.domain": "get.adobe.com",
"threatintel.indicator.url.full": "http://get.adobe.com/stats/AbfFcBebD/?q=",
"threatintel.indicator.url.original": "http://get.adobe.com/stats/AbfFcBebD/?q=",
"threatintel.indicator.url.path": "/stats/AbfFcBebD/",
"threatintel.indicator.url.query": "q=",
Expand Down
Loading

0 comments on commit c45aba5

Please sign in to comment.