Possible regression in field values validation

[ebeahan]

Summary

In elastic-package v0.84.0+, elastic-package test pipeline -v no longer produces errors due if ECS field value fails validation.

The same package and version (in the below example, cisco_asa version 2.21.0) tested using v0.83.2 does produce validation errors.

Comparison

v0.83.2

---

$ go install github.com/elastic/elastic-package@v0.83.2

$ elastic-package version
2023/09/11 12:14:48  INFO New version is available - v0.86.1. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.86.1
elastic-package v0.83.2 version-hash undefined (build time: unknown)

$ elastic-package test pipeline -v
2023/09/11 12:14:58  WARN CommitHash is undefined, in both /Users/ericbeahan/.elastic-package/version and the compiled binary, config may be out of date.
2023/09/11 12:14:58 DEBUG Enable verbose logging
2023/09/11 12:14:58  INFO New version is available - v0.86.1. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.86.1
Run pipeline tests for the package
--- Test results for package: cisco_asa - START ---
FAILURE DETAILS:
cisco_asa/log test-additional-messages.log:
[0] parsing field value failed: field "event.type" value "change" is not one of the expected values (access, allowed, connection, denied, end, info, protocol, start) for any of the values of "event.category" (network)
[1] parsing field value failed: field "event.type" value "deletion" is not one of the expected values (access, allowed, connection, denied, end, info, protocol, start) for any of the values of "event.category" (network)
[2] parsing field value failed: field "event.type" value "error" is not one of the expected values (access, allowed, connection, denied, end, info, protocol, start) for any of the values of "event.category" (network)
cisco_asa/log test-anyconnect-messages.log:
[0] parsing field value failed: field "event.type" value "error" is not one of the expected values (access, allowed, connection, denied, end, info, protocol, start) for any of the values of "event.category" (network)
cisco_asa/log test-sample.log:
[0] parsing field value failed: field "event.type" value "deletion" is not one of the expected values (access, allowed, connection, denied, end, info, protocol, start) for any of the values of "event.category" (network)


╭───────────┬─────────────┬───────────┬──────────────────────────────┬─────────────────────────────────────────────────────────────────────────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                    │ RESULT                                                                      │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼──────────────────────────────┼─────────────────────────────────────────────────────────────────────────────┼──────────────┤
│ cisco_asa │ log         │ pipeline  │ test-additional-messages.log │ FAIL: test case failed: one or more problems with fields found in documents │     22.607ms │
│ cisco_asa │ log         │ pipeline  │ test-anyconnect-messages.log │ FAIL: test case failed: one or more problems with fields found in documents │   5.322584ms │
│ cisco_asa │ log         │ pipeline  │ test-asa-fix.log             │ PASS                                                                        │  17.565792ms │
│ cisco_asa │ log         │ pipeline  │ test-asa-missing-groups.log  │ PASS                                                                        │  10.212708ms │
│ cisco_asa │ log         │ pipeline  │ test-asa.log                 │ PASS                                                                        │  89.401125ms │
│ cisco_asa │ log         │ pipeline  │ test-dap-records.log         │ PASS                                                                        │   6.171625ms │
│ cisco_asa │ log         │ pipeline  │ test-filtered.log            │ PASS                                                                        │   4.049042ms │
│ cisco_asa │ log         │ pipeline  │ test-hostnames.log           │ PASS                                                                        │     7.7675ms │
│ cisco_asa │ log         │ pipeline  │ test-non-canonical.log       │ PASS                                                                        │  19.610458ms │
│ cisco_asa │ log         │ pipeline  │ test-not-ip.log              │ PASS                                                                        │   7.265417ms │
│ cisco_asa │ log         │ pipeline  │ test-sample.log              │ FAIL: test case failed: one or more problems with fields found in documents │     43.255ms │
│ cisco_asa │ log         │ pipeline  │ test-sip.log                 │ PASS                                                                        │  10.267375ms │
╰───────────┴─────────────┴───────────┴──────────────────────────────┴─────────────────────────────────────────────────────────────────────────────┴──────────────╯
--- Test results for package: cisco_asa - END   ---
Done
Error: one or more test cases failed

v0.84.0+

---

$ go install github.com/elastic/elastic-package@v0.84.0

$ elastic-package version
2023/09/11 12:15:25  WARN CommitHash is undefined, in both /Users/ericbeahan/.elastic-package/version and the compiled binary, config may be out of date.
2023/09/11 12:15:25  INFO New version is available - v0.86.1. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.86.1
elastic-package v0.84.0 version-hash undefined (build time: unknown)

$ elastic-package test pipeline -v
2023/09/11 12:15:30  WARN CommitHash is undefined, in both /Users/ericbeahan/.elastic-package/version and the compiled binary, config may be out of date.
2023/09/11 12:15:30 DEBUG Enable verbose logging
2023/09/11 12:15:30  INFO New version is available - v0.86.1. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.86.1
Run pipeline tests for the package
--- Test results for package: cisco_asa - START ---
╭───────────┬─────────────┬───────────┬──────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                    │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼──────────────────────────────┼────────┼──────────────┤
│ cisco_asa │ log         │ pipeline  │ test-additional-messages.log │ PASS   │   26.69075ms │
│ cisco_asa │ log         │ pipeline  │ test-anyconnect-messages.log │ PASS   │  15.653833ms │
│ cisco_asa │ log         │ pipeline  │ test-asa-fix.log             │ PASS   │  13.698042ms │
│ cisco_asa │ log         │ pipeline  │ test-asa-missing-groups.log  │ PASS   │  12.466291ms │
│ cisco_asa │ log         │ pipeline  │ test-asa.log                 │ PASS   │  99.582625ms │
│ cisco_asa │ log         │ pipeline  │ test-dap-records.log         │ PASS   │   4.954625ms │
│ cisco_asa │ log         │ pipeline  │ test-filtered.log            │ PASS   │    4.13625ms │
│ cisco_asa │ log         │ pipeline  │ test-hostnames.log           │ PASS   │    4.68525ms │
│ cisco_asa │ log         │ pipeline  │ test-non-canonical.log       │ PASS   │  17.809375ms │
│ cisco_asa │ log         │ pipeline  │ test-not-ip.log              │ PASS   │   5.496333ms │
│ cisco_asa │ log         │ pipeline  │ test-sample.log              │ PASS   │     60.208ms │
│ cisco_asa │ log         │ pipeline  │ test-sip.log                 │ PASS   │   3.503375ms │
╰───────────┴─────────────┴───────────┴──────────────────────────────┴────────┴──────────────╯
--- Test results for package: cisco_asa - END   ---
Done

[jsoriano]

Hey @ebeahan, thanks for reporting this issue! This was indeed a regression, sorry for the problems caused!

We have a fix in #1452, but we have seen that applying this change would make several security packages to fail, see https://fleet-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations%2FPR-7817/detail/PR-7817/1/tests.

We have two possible approaches to solve this:

• We merge the fix, release an elastic-package version, and replace the wrong values in the same pull request where we update elastic-package in the integrations repo.
• Your team fix the packages, then we merge the fix, and follow the usual procedure.

If your team has availability to fix the packages, we could wait to release the fix till then, but this is risky, as nothing would prevent to introduce the issue in more packages.
If we merge the fix now, we would need someone from your team to review the changes in your packages when we update the integrations repo.

We could also merge the fix, and force-merge the update in integrations. This would break CI for the affected packages, but would avoid introducing the issue in more cases.

Let me know what you prefer.

cc @mrodm

[ebeahan]

We could also merge the fix, and force-merge the update in integrations. This would break CI for the affected packages, but would avoid introducing the issue in more cases.

@jsoriano After chatting with @andrewkroh and some others from Security Integrations, we feel this is the way to proceed. Decouples our timeline for fixing from other teams benefiting from the validation.

[jsoriano]

Thanks for confirming!

[jsoriano]

Fix merged in integrations.