Skip to content

Commit

Permalink
Add pipeline tests to Cisco ASA (elastic#414)
Browse files Browse the repository at this point in the history 
* Add pipeline test to Cisco ASA

Add missing field definitions.

Correct JSON data types for fields.
- event.code
- network.iana_number

TODO:
Fix warnings:
[0] parsing field value failed: field "destination.port"''s Go type, string, does not match the expected field type: long
[1] parsing field value failed: field "source.port"''s Go type, string, does not match the expected field type: long
cisco/asa test-not-ip.log:
[0] parsing field value failed: field "destination.nat.port"''s Go type, string, does not match the expected field type: long
[1] parsing field value failed: field "source.nat.port"''s Go type, string, does not match the expected field type: long
[2] parsing field value failed: field "syslog.facility"''s Go type, float64, does not match the expected field type: keyword
cisco/asa test-sample.log:
[0] parsing field value failed: field "source.nat.port"''s Go type, string, does not match the expected field type: long

Fix error.message in generated pipeline.

* Check for unset raw_date temp field.

* Convert NAT ports to integer

* Upgrade ASA pipeline from Beats

* Fix internal prefix

* Fixes to the pipeline and remove bad test lines

* Missing fields

* Regenerate golden files

* config to ignore event.ingested

* Update Cisco package to 0.8.0

* Mage fmt

* Update docs fields

* Use test case configuration to set numeric keywords

* Run `elastic-package format`

* Update dependencies

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
  • Loading branch information
@andrewkroh @adriansr
andrewkroh and adriansr authored Dec 14, 2020
1 parent f6feb13 commit 6ee910d
Show file tree
Hide file tree
Showing 33 changed files with 31,134 additions and 31 deletions.
2 changes: 1 addition & 1 deletion go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ go 1.12

require (
github.com/blang/semver v3.5.1+incompatible
github.com/elastic/elastic-package v0.0.0-20201202151510-fe69843289af
github.com/elastic/elastic-package v0.0.0-20201211101121-88f40b569b78
github.com/elastic/package-registry v0.13.0
github.com/magefile/mage v1.10.0
github.com/pkg/errors v0.9.1
Expand Down
8 changes: 4 additions & 4 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -84,17 +84,17 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/elastic/elastic-package v0.0.0-20201202151510-fe69843289af h1:KgL6xzIw8JfEAv1yguO3nFigtlGVvVSVWzAUQGnDsYY=
github.com/elastic/elastic-package v0.0.0-20201202151510-fe69843289af/go.mod h1:2dVvy0o3MA3QcUIIbIx3fI3+0iZj+GWx+sWamlUPknk=
github.com/elastic/elastic-package v0.0.0-20201211101121-88f40b569b78 h1:O88BTmG/DtGywk6Y965Ay7d51qPMl1cpRa55Ytlnz80=
github.com/elastic/elastic-package v0.0.0-20201211101121-88f40b569b78/go.mod h1:Fvhveu6PsnX82/yGGMoTExmh3B2ecBZ4EIjiR0TSm18=
github.com/elastic/go-elasticsearch/v7 v7.9.0 h1:UEau+a1MiiE/F+UrDj60kqIHFWdzU1M2y/YtBU2NC2M=
github.com/elastic/go-elasticsearch/v7 v7.9.0/go.mod h1:OJ4wdbtDNk5g503kvlHLyErCgQwwzmDtaFC4XyOxXA4=
github.com/elastic/go-ucfg v0.8.3/go.mod h1:iaiY0NBIYeasNgycLyTvhJftQlQEUO2hpF+FX0JKxzo=
github.com/elastic/go-ucfg v0.8.4-0.20200415140258-1232bd4774a6 h1:Ehbr7du4rSSEypR8zePr0XRbMhO4PJgcHC9f8fDbgAg=
github.com/elastic/go-ucfg v0.8.4-0.20200415140258-1232bd4774a6/go.mod h1:iaiY0NBIYeasNgycLyTvhJftQlQEUO2hpF+FX0JKxzo=
github.com/elastic/package-registry v0.13.0 h1:RdIzD111v1chu0L3MrbUGqgt6RZ9a9DVN3XMgjgOAu0=
github.com/elastic/package-registry v0.13.0/go.mod h1:oQx3Tg9ynuC6APd0o0OHud9kyPX6S6IzdJp/R4Hj1HY=
github.com/elastic/package-spec/code/go v0.0.0-20201202075901-3b4d8207554d h1:9GXCPO5Ahy12Qo+JKvdujuihSyqXyxeBQsxg+c13OaY=
github.com/elastic/package-spec/code/go v0.0.0-20201202075901-3b4d8207554d/go.mod h1:3W6uyBFCE4/NPcVPb+ZuoLJTMLu8BCTc+PRFDutSvfE=
github.com/elastic/package-spec/code/go v0.0.0-20201210164239-22bc835bcf04 h1:DfZR0hCpA1yE/bG9q1FJoeajvbrjWn945bXyavC8knE=
github.com/elastic/package-spec/code/go v0.0.0-20201210164239-22bc835bcf04/go.mod h1:3W6uyBFCE4/NPcVPb+ZuoLJTMLu8BCTc+PRFDutSvfE=
github.com/emirpasic/gods v1.12.0 h1:QAUIPSaCu4G+POclxeqb3F+WPpdKqFGlw36+yOzGlrg=
github.com/emirpasic/gods v1.12.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
Expand Down
69 changes: 69 additions & 0 deletions packages/cisco/data_stream/asa/_dev/test/pipeline/test-additional-messages.log
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
May 5 17:51:17 dev01: %FTD-6-302013: Built inbound TCP connection 111111111 for net:10.10.10.10/53500 (8.8.8.8/53500) to fw111:192.168.2.2/53500 (8.8.5.4/53500)
May 5 17:51:17 dev01: %FTD-6-302015: Built inbound UDP connection 111111111 for net:10.10.10.10/53500 (8.8.8.8/53500) to fw111:192.168.2.2/53500 (8.8.5.4/53500)
May 5 17:51:17 dev01: %FTD-6-302020: Built inbound ICMP connection for faddr 10.10.10.10/0 gaddr 8.8.8.8/0 laddr 192.168.2.2/0 type 3 code 3
May 5 17:51:17 dev01: %FTD-7-609002: Teardown local-host net:192.168.2.2 duration 0:00:00
May 5 17:51:17 dev01: %FTD-7-609001: Built local-host net:192.168.2.2
May 5 17:51:17 dev01: %FTD-6-302020: Built inbound ICMP connection for faddr 10.10.10.10/0 gaddr 8.8.8.8/0 laddr 192.168.2.2/0 type 3 code 1
May 5 17:51:17 dev01: %FTD-6-805001: Offloaded TCP Flow for connection 111111111 from fw111:10.10.10.10/111 (8.8.8.8/111) to fw111:192.168.2.2/111 (8.8.5.4/111)
May 5 17:51:17 dev01: %FTD-6-805002: TCP Flow is no longer offloaded for connection 941243214 from net:10.192.18.4/51261 (10.192.18.4/51261) to fw109:10.192.70.66/443 (10.192.70.66/443)
May 5 17:51:17 dev01: %FTD-7-710005: UDP request discarded from 192.168.2.2/68 to fw111:10.10.10.10/67
May 5 17:51:17 dev01: %FTD-6-303002: FTP connection from net:192.168.2.2/63656 to fw111:10.192.18.4/21, user testuser Stored file /export/home/sysm/ftproot/sdsdsds/tmp.log
May 5 17:51:17 dev01: %FTD-7-710006: VRRP request discarded from 192.168.2.2 to fw111:192.18.4
May 5 17:51:17 dev01: %FTD-4-313005: No matching connection for ICMP error message: icmp src fw111:10.192.33.100 dst fw111:192.18.4 (type 3, code 3) on fw111 interface. Original IP payload: udp src 192.18.4/53 dst 8.8.8.8/10872.
May 5 18:16:21 dev01: %ASA-6-302021: Teardown ICMP connection for faddr 192.168.2.2/0 gaddr 8.8.8.8/2 laddr 10.10.10.10/2 type 8 code 0
May 5 18:22:35 dev01: %ASA-7-609001: Built local-host net:10.10.10.10
May 5 18:24:31 dev01: %ASA-7-609002: Teardown local-host identity:10.10.10.10 duration 0:00:00
May 5 18:29:32 dev01: %ASA-6-302020: Built inbound ICMP connection for faddr 10.10.10.10/0 gaddr 8.8.8.8/0 laddr 10.192.46.90/0
May 5 18:29:32 dev01: %ASA-6-302020: Built outbound ICMP connection for faddr 10.10.10.10/0 gaddr 8.8.8.8/0 laddr 192.168.2.2/0 type 3 code 3
May 5 18:29:32 dev01: %ASA-6-302014: Teardown TCP connection 2960892904 for out111:10.10.10.10/443 to fw111:192.168.2.2/55225 duration 0:00:00 bytes 0 TCP Reset-I
May 5 18:29:32 dev01: %ASA-6-302013: Built outbound TCP connection 1588662 for intfacename:192.168.2.2/80 (8.8.8.8/80) to net:10.10.10.10/54839 (8.8.8.8/54839)
May 5 18:29:32 dev01: %ASA-6-302012: Teardown dynamic UDP translation from fw111:10.10.10.10/54230 to out111:192.168.2.2/54230 duration 0:00:00
May 5 18:40:50 dev01: %ASA-4-313004: Denied ICMP type=0, from laddr 10.10.10.10 on interface fw502 to 192.168.2.2: no matching session
May 5 18:40:50 dev01: %ASA-6-305011: Built dynamic TCP translation from fw111:10.10.10.10/57006 to out111:192.168.2.2/57006
May 5 18:40:50 dev01: %ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/43803 to 10.10.10.10/14322 flags SYN on interface out111
May 5 18:40:50 dev01: %ASA-2-302016: Teardown UDP connection 1671727 for intfacename:10.10.10.10/161 to net:192.186.2.2/53356 duration 0:02:04 bytes 64585
May 5 18:40:50 dev01: %ASA-2-302015: Built outbound UDP connection 1743372 for intfacename:10.10.10.10/161 (8.8.8.4/161) to net:192.168.2.2/22638 (8.8.8.8/22638)
May 5 18:40:50 dev01: %ASA-2-302015: Built outbound UDP connection 1743372 for intfacename:10.10.10.10/161 (8.8.8.4/161) to net:192.168.2.2/22638 (8.8.8.8/22638)
May 5 18:40:50 dev01: %ASA-4-106023: Deny tcp src fw111:10.10.10.10/64388 dst out111:192.168.2.2/443 by access-group "out1111_access_out" [0x47e21ef4, 0x47e21ef4]
May 5 18:40:50 dev01: %ASA-4-106021: Deny TCP reverse path check from 192.168.2.2 to 10.10.10.10 on interface fw111
May 5 19:02:58 dev01: %ASA-2-106006: Deny inbound UDP from 192.168.2.2/65020 to 10.10.10.10/65020 on interface fw111
May 5 19:02:58 dev01: %ASA-6-106015: Deny TCP (no connection) from 192.168.2.2/53089 to 10.10.10.10/443 flags FIN PSH ACK on interface out111
May 5 19:02:58 dev01: %ASA-6-106015: Deny TCP (no connection) from 192.168.2.2/17127 to 10.10.10.10/443 flags PSH ACK on interface out111
May 5 19:02:58 dev01: %ASA-6-106015: Deny TCP (no connection) from 192.168.2.2/24223 to 10.10.10.10/443 flags RST on interface fw111
May 5 19:02:58 dev01: %ASA-6-302022: Built director stub TCP connection for fw1111:10.10.10.10/38540 (8.8.8.5/38540) to net:192.168.2.2/10051 (8.8.8.8/10051)
May 5 19:02:58 dev01: %ASA-6-302022: Built forwarder stub TCP connection for fw111:10.10.10.10/38540 (8.8.8.5/38540) to net:192.168.2.2/10051 (8.8.8.8/10051)
May 5 19:02:58 dev01: %ASA-6-302022: Built backup stub TCP connection for fw111:10.10.10.10/38540 (8.8.8.5/38540) to net:192.1682.2.2/10051 (8.8.8.8/10051)
May 5 19:02:58 dev01: %ASA-6-302023: Teardown stub TCP connection for fw111:10.10.10.10/39210 to net:192.168.2.2/10051 duration 0:00:00 forwarded bytes 0 Cluster flow with CLU closed on owner
May 5 19:02:58 dev01: %ASA-6-302023: Teardown stub TCP connection for net:10.10.10.10/10051 to unknown:192.168.2.2/39222 duration 0:00:00 forwarded bytes 0 Forwarding or redirect flow removed to create director or backup flow
May 5 19:03:27 dev01: %ASA-7-111009: User 'aaaa' executed cmd: show access-list fw211111_access_out brief
May 5 19:02:26 dev01: %ASA-7-111009: User 'aaaa' executed cmd: show access-list aaa_out brief
May 5 19:02:26 dev01: %ASA-6-106100: access-list fw111_out permitted tcp ptaaac/192.168.2.2(62157) -> fw111/10.10.10.10(3452) hit-cnt 1 first hit [0x38ff326b, 0x00000000]
May 5 19:02:26 dev01: %ASA-6-106100: access-list fw111_out permitted tcp net/192.168.2.2(49033) -> fw111/10.10.10.10(6007) hit-cnt 2 300-second interval [0x38ff326b, 0x00000000]
May 5 19:02:26 dev01: %ASA-6-302027: Teardown stub ICMP connection for fw1111:10.10.10.10/6426 to net:192.168.2.2/0 duration 1:00:04 forwarded bytes 56 Cluster flow with CLU closed on owner
May 5 19:02:26 dev01: %ASA-6-302026: Built director stub ICMP connection for fw111:10.10.10.10/32004 (8.8.8.5) to net:192.168.2.2/0 (8.8.8.8)
May 5 19:02:26 dev01: %ASA-7-710005: UDP request discarded from 10.10.10.10/1985 to net:192.168.2.2/1985
May 5 19:02:26 dev01: %ASA-6-302025: Teardown stub UDP connection for net:192.168.2.2/123 to unknown:10.10.10.10/123 duration 0:01:00 forwarded bytes 48 Cluster flow with CLU removed from due to idle timeout
May 5 19:02:26 dev01: %ASA-6-302024: Built backup stub UDP connection for net:192.168.2.2/9051 (8.8.8.5(19051) to fw111:10.10.10.10/123 (8.8.8.8/123)
May 5 19:02:26 dev01: %ASA-3-106014: Deny inbound icmp src fw111:10.10.10.10 dst fw111:10.10.10.10(type 8, code 0)
May 5 19:02:25 dev01: %ASA-4-733100: [192.168.2.2] drop rate-1 exceeded. Current burst rate is 0 per second, max configured rate is -4; Current average rate is 7 per second, max configured rate is -4; Cumulative total count is 9063
May 5 19:02:25 dev01: %ASA-3-106010: Deny inbound sctp src fw111:10.10.10.10/5114 dst fw111:10.10.10.10/2
May 5 19:02:25 dev01: %ASA-4-507003: tcp flow from fw111:10.10.10.10/49574 to out111:192.168.2.2/80 terminated by inspection engine, reason - disconnected, dropped packet.
Apr 27 04:18:49 dev01: %ASA-5-304001: 10.20.30.40 Accessed URL 10.20.30.40:http://10.20.30.40/
Apr 27 04:18:49 dev01: %ASA-5-304001: 10.20.30.40 Accessed URL someuser@10.20.30.40:http://10.20.30.40/IOFUHSIU98[0]
Apr 27 17:54:52 dev01: %ASA-5-304001: 10.20.30.40 Accessed JAVA URL 10.20.30.40:http://10.20.30.40/some/longer/url-asd-er9789870[0]_=23
Apr 27 04:18:49 dev01: %ASA-5-304001: 10.20.30.40 Accessed JAVA URL someuser@10.20.30.40:http://10.20.30.40/
Apr 27 04:12:23 dev01: %ASA-6-302304: Teardown TCP state-bypass connection 2751765169 from server.deflan:1.2.3.4/54242 to server.deflan:2.3.4.5/9101 duration 1:00:02 bytes 245 Connection timeout
Apr 27 02:02:02 dev01: %ASA-4-106023: Deny tcp src outside:10.10.10.2/56444 dst srv:192.168.2.2/51635(testhostname.domain) by access-group "global_access_1"
Oct 20 2019 15:15:15 dev01: %ASA-5-106100: access-list testrulename denied tcp insideintf/somedomainname.local(27218) -> OUTSIDE/195.122.12.242(53) hit-cnt 1 first hit [0x16847359, 0x00000000]
Apr 27 02:03:03 dev01: %ASA-5-111004: console end configuration: OK
Apr 27 02:03:03 dev01: %ASA-5-111010: User 'enable_15', running 'CLI' from IP 10.10.0.87, executed 'clear'
Apr 27 02:03:03 dev01: %ASA-5-502103: User priv level changed: Uname: enable_15 From: 1 To: 15
Apr 27 02:03:03 dev01: %ASA-6-605004: Login denied from 10.10.1.212/51923 to FCD-FS-LAN:10.10.1.254/https for user "*****"
Apr 27 02:03:03 dev01: %ASA-6-611102: User authentication failed: IP address: 10.10.0.87, Uname: admin
Apr 27 02:03:03 dev01: %ASA-6-605005: Login permitted from 10.10.0.87/6651 to FCD-FS-LAN:10.10.1.254/ssh for user "admin"
Apr 27 02:03:03 dev01: %ASA-6-611101: User authentication succeeded: IP address: 10.10.0.87, Uname: admin
Apr 27 02:03:03 dev01: %ASA-5-713049: Group = 91.240.17.178, IP = 91.240.17.178, Security negotiation complete for LAN-to-LAN Group (91.240.17.178) Responder, Inbound SPI = 0x276b1da2, Outbound SPI = 0x0e1a581d
Apr 27 02:03:03 dev01: %ASA-4-113019: Group = 91.240.17.178, Username = 91.240.17.178, IP = 91.240.17.178, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:32m:16s, Bytes xmt: 297103, Bytes rcv: 1216163, Reason: User Requested
Apr 27 02:03:03 dev01: %ASA-4-722051: Group some-policy User testuser IP 8.8.8.8 IPv4 Address 8.8.4.4 IPv6 address 2001:4860:4860::8888 assigned to session
Apr 27 02:03:03 dev01: %ASA-6-716002: Group another-policy User testuser IP 8.8.8.8 WebVPN session terminated: User Requested.
Apr 27 02:03:03 dev01: %ASA-3-710003: TCP access denied by ACL from 104.46.88.19/6370 to outside:195.74.114.34/23
10 changes: 10 additions & 0 deletions packages/cisco/data_stream/asa/_dev/test/pipeline/test-additional-messages.log-config.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"dynamic_fields": {
"event.ingested": ".*"
},
"numeric_keyword_fields": [
"network.iana_number",
"event.code",
"syslog.facility"
]
}
Loading

0 comments on commit 6ee910d

Please sign in to comment.