[Filebeat][Cisco ASA] log enhancement and performance #20831

pcosic · 2020-08-27T15:34:11Z

What does this PR do?

This PR resolve some reported issues with ECS, and is adding new message pattern for ASA logs

Overview of the Changes:

Adding 15 new message pattern with dissect processor

434004
434002
713905
750002
750003
110002
419002
602304
602303
713120
713202
713901
713904
713906
713905

fix parsing error not extracting event.outcome and network.transport from 106015
all other processors associated with the new message id's have been updated and extended
further additional fields were derived from the logs
changed event.outcome in script processor to ECS
adding anchors to grok patterns with no conditional and to grok processors that using more than one pattern
adding new event.action for user creation/deletion or bypass events
fix 106014

Why is it important?

We think that these are one of the most used message types in Cisco ASA logs.
adding the anchors increases the throughput/performance. It is described in more detail in this blog article (https://www.elastic.co/blog/do-you-grok-grok)
Wee need more event.actions for specific logs/events

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.
I have made corresponding changes to the documentation

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

- Fixed some ECS issues - added anchors on grok patterns for performance - added messages: ------------------------- 434004 434002 713905 750002 750003 110002 419002 602304 602303 713120 713202 713901 713904 713906 713905 ------------------------- - with the messages pattern added also this commit add four new event action types in the script that mapped event actions to the event.kind/category/type - added set processor for adding outcome, action and protocol if necessary for the new messages

elasticmachine · 2020-08-27T15:34:13Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-08-27T15:34:14Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-08-27T15:39:11Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: marc-gr commented: jenkins run tests
Start Time: 2020-12-21T14:29:27.229+0000
Duration: 50 min 35 sec

Test stats 🧪

Test	Results
Failed	1
Passed	2444
Skipped	263
Total	2708

Test errors

Expand to view the tests failures

`Build&Test / x-pack/filebeat-build / test_fileset_file_203_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'values_changed': {"root['rsa.time.event_time']": {'new_value': '2020-12-23T02:09:07.000Z', 'old_value': '2019-12-23T02:09:07.000Z'}, "root['@timestamp']": {'new_value': '2020-12-23T02:09:07.000Z', 'old_value': '2019-12-23T02:09:07.000Z'}}}, full object:    {'rsa.internal.event_desc': 'TFTPD NAK ERROR', 'rsa.internal.messageid': 'TFTPD_NAK_ERR', 'rsa.counters.dclass_c1': 357, 'rsa.time.month': 'Dec', 'rsa.time.day': '23', 'rsa.time.event_time': '2020-12-23T02:09:07.000Z', 'rsa.misc.event_type': 'TFTPD_NAK_ERR', 'rsa.misc.pid': '1471', 'rsa.misc.result_code': 'ptatems', 'process.name': 'niamquis.exe', 'process.pid': 1471, 'log.offset': 2274, 'fileset.name': 'junos', 'tags': ['juniper.junos', 'forwarded'], 'input.type': 'log', 'observer.product': 'Junos', 'observer.vendor': 'Juniper', 'observer.type': 'Routers', '@timestamp': '2020-12-23T02:09:07.000Z', 'service.type': 'juniper', 'event.original': 'Dec 23 00:09:07 niamquis.exe[1471]: TFTPD_NAK_ERR : nak error ptatems, 357', 'event.code': 'TFTPD_NAK_ERR', 'event.module': 'juniper', 'event.action': 'TFTPD_NAK_ERR', 'event.dataset': 'juniper.junos'} assert 1 == 0  +  where 1 = len({'values_changed': {"root['rsa.time.event_time']": {'new_value': '2020-12-23T02:09:07.000Z', 'old_value': '2019-12-23T...9:07.000Z'}, "root['@timestamp']": {'new_value': '2020-12-23T02:09:07.000Z', 'old_value': '2019-12-23T02:09:07.000Z'}}})

Expand to view the stacktrace

 a = (<test_xpack_modules.XPackTest testMethod=test_fileset_file_203_juniper>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../filebeat/tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
../../filebeat/tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_xpack_modules.XPackTest testMethod=test_fileset_file_203_juniper>
test_file = '/go/src/github.com/elastic/beats/x-pack/filebeat/module/juniper/junos/test/generated.log'
objects = [{'@timestamp': '2020-01-29T08:09:59.000Z', 'agent': {'ephemeral_id': '03cc9c39-858b-45c9-8751-aea5800acb91', 'id': '0...ELL_RELAY_MODE_UNSPECIFIED', 'dataset': 'juniper.junos', 'ingested': '2020-12-21T15:08:47.396054298Z', ...}, ...}, ...]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'values_changed': {"root['rsa.time.event_time']": {'new_value': '2020-12-23T02:09:07.000Z', 'old_value': '2019-12-23T02:09:07.000Z'}, "root['@timestamp']": {'new_value': '2020-12-23T02:09:07.000Z', 'old_value': '2019-12-23T02:09:07.000Z'}}}, full object: 
E             {'rsa.internal.event_desc': 'TFTPD NAK ERROR', 'rsa.internal.messageid': 'TFTPD_NAK_ERR', 'rsa.counters.dclass_c1': 357, 'rsa.time.month': 'Dec', 'rsa.time.day': '23', 'rsa.time.event_time': '2020-12-23T02:09:07.000Z', 'rsa.misc.event_type': 'TFTPD_NAK_ERR', 'rsa.misc.pid': '1471', 'rsa.misc.result_code': 'ptatems', 'process.name': 'niamquis.exe', 'process.pid': 1471, 'log.offset': 2274, 'fileset.name': 'junos', 'tags': ['juniper.junos', 'forwarded'], 'input.type': 'log', 'observer.product': 'Junos', 'observer.vendor': 'Juniper', 'observer.type': 'Routers', '@timestamp': '2020-12-23T02:09:07.000Z', 'service.type': 'juniper', 'event.original': 'Dec 23 00:09:07 niamquis.exe[1471]: TFTPD_NAK_ERR : nak error ptatems, 357', 'event.code': 'TFTPD_NAK_ERR', 'event.module': 'juniper', 'event.action': 'TFTPD_NAK_ERR', 'event.dataset': 'juniper.junos'}
E           assert 1 == 0
E            +  where 1 = len({'values_changed': {"root['rsa.time.event_time']": {'new_value': '2020-12-23T02:09:07.000Z', 'old_value': '2019-12-23T...9:07.000Z'}, "root['@timestamp']": {'new_value': '2020-12-23T02:09:07.000Z', 'old_value': '2019-12-23T02:09:07.000Z'}}})

../../filebeat/tests/system/test_modules.py:217: AssertionError

Steps errors

Expand to view the steps failures

`x-pack/filebeat-build - mage build test`

Took 31 min 39 sec . View more details on here
Description: mage build test

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2020-12-21T15:14:57.167Z] FAILED tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_203_juniper
[2020-12-21T15:14:57.167Z] ================== 1 failed, 304 passed in 1376.20s (0:22:56) ==================
[2020-12-21T15:14:57.167Z] >> python test: Integration Testing Complete
[2020-12-21T15:14:57.167Z] Error: running "/go/src/github.com/elastic/beats/build/ve/docker/bin/pytest --timeout=90 --durations=20 --junit-xml=build/TEST-python-integration.xml tests/system/test_filebeat_xpack.py tests/system/test_http_endpoint.py tests/system/test_xpack_modules.py" failed with exit code 1
[2020-12-21T15:14:59.692Z] Error: running "docker-compose -p filebeat_8_0_0_c4122fd5d0-snapshot run -e DOCKER_COMPOSE_PROJECT_NAME=filebeat_8_0_0_c4122fd5d0-snapshot -e BEAT_STRICT_PERMS=false -e STACK_ENVIRONMENT=snapshot -e TESTING_ENVIRONMENT=snapshot -e GOCACHE=/go/src/github.com/elastic/beats/build/docker-gocache -v /var/lib/jenkins/workspace/Beats_beats_PR-20831/pkg/mod/cache/download:/gocache:ro -e GOPROXY=file:///gocache,direct -e EXEC_UID=1155 -e EXEC_GID=1156 -e TEST_COVERAGE=true -e RACE_DETECTOR=true -e TEST_TAGS=null,oracle -e MODULE=cisco -e BEATS_INSIDE_INTEGRATION_TEST_ENV=true -e GOFLAGS=-mod=readonly beat /go/src/github.com/elastic/beats/x-pack/filebeat/build/mage-linux-amd64 pythonIntegTest" failed with exit code 1
[2020-12-21T15:15:00.034Z] Client: Docker Engine - Community
[2020-12-21T15:15:00.034Z]  Version:           20.10.1
[2020-12-21T15:15:00.034Z]  API version:       1.41
[2020-12-21T15:15:00.034Z]  Go version:        go1.13.15
[2020-12-21T15:15:00.034Z]  Git commit:        831ebea
[2020-12-21T15:15:00.034Z]  Built:             Tue Dec 15 04:34:59 2020
[2020-12-21T15:15:00.034Z]  OS/Arch:           linux/amd64
[2020-12-21T15:15:00.034Z]  Context:           default
[2020-12-21T15:15:00.034Z]  Experimental:      true
[2020-12-21T15:15:00.034Z] 
[2020-12-21T15:15:00.034Z] Server: Docker Engine - Community
[2020-12-21T15:15:00.034Z]  Engine:
[2020-12-21T15:15:00.034Z]   Version:          20.10.1
[2020-12-21T15:15:00.034Z]   API version:      1.41 (minimum version 1.12)
[2020-12-21T15:15:00.034Z]   Go version:       go1.13.15
[2020-12-21T15:15:00.034Z]   Git commit:       f001486
[2020-12-21T15:15:00.034Z]   Built:            Tue Dec 15 04:32:40 2020
[2020-12-21T15:15:00.034Z]   OS/Arch:          linux/amd64
[2020-12-21T15:15:00.034Z]   Experimental:     false
[2020-12-21T15:15:00.034Z]  containerd:
[2020-12-21T15:15:00.034Z]   Version:          1.4.3
[2020-12-21T15:15:00.034Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2020-12-21T15:15:00.034Z]  runc:
[2020-12-21T15:15:00.034Z]   Version:          1.0.0-rc92
[2020-12-21T15:15:00.034Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2020-12-21T15:15:00.034Z]  docker-init:
[2020-12-21T15:15:00.034Z]   Version:          0.19.0
[2020-12-21T15:15:00.034Z]   GitCommit:        de40ad0
[2020-12-21T15:15:00.034Z] Unable to find image 'alpine:3.4' locally
[2020-12-21T15:15:00.966Z] 3.4: Pulling from library/alpine
[2020-12-21T15:15:01.223Z] c1e54eec4b57: Pulling fs layer
[2020-12-21T15:15:01.481Z] c1e54eec4b57: Verifying Checksum
[2020-12-21T15:15:01.481Z] c1e54eec4b57: Download complete
[2020-12-21T15:15:01.739Z] c1e54eec4b57: Pull complete
[2020-12-21T15:15:01.739Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2020-12-21T15:15:01.739Z] Status: Downloaded newer image for alpine:3.4
[2020-12-21T15:15:03.923Z] + python .ci/scripts/pre_archive_test.py
[2020-12-21T15:15:05.820Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2020-12-21T15:15:05.828Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20831/src/github.com/elastic/beats/build
[2020-12-21T15:15:06.128Z] + rm -rf ve
[2020-12-21T15:15:06.128Z] + find . -type d -name vendor -exec rm -r {} ;
[2020-12-21T15:15:06.139Z] Recording test results
[2020-12-21T15:15:07.186Z] [Checks API] No suitable checks publisher found.
[2020-12-21T15:15:07.536Z] + tar --version
[2020-12-21T15:15:07.877Z] + tar --exclude=test-build-artifacts-x-pack/filebeat-build.tgz -czf test-build-artifacts-x-pack/filebeat-build.tgz .
[2020-12-21T15:15:54.718Z] [INFO] Override default googleStorageUpload with some sleep
[2020-12-21T15:15:54.729Z] Sleeping for 55 sec
[2020-12-21T15:16:49.743Z] [Google Cloud Storage Plugin] Found 1 files to upload from pattern: test-build-artifacts-x-pack/filebeat-build.tgz
[2020-12-21T15:16:50.100Z] [Google Cloud Storage Plugin] Uploading: test-build-artifacts-x-pack/filebeat-build.tgz
[2020-12-21T15:17:00.556Z] + python .ci/scripts/search_system_tests.py
[2020-12-21T15:17:00.568Z] [INFO] system-tests='build/x-pack/filebeat/build/system-tests'. If no empty then let's create a tarball
[2020-12-21T15:17:00.882Z] + tar --version
[2020-12-21T15:17:01.184Z] + tar --exclude=x-pack-filebeat--system-tests-linux.tgz -czf x-pack-filebeat--system-tests-linux.tgz build/x-pack/filebeat/build/system-tests
[2020-12-21T15:17:27.735Z] [INFO] Override default googleStorageUpload with some sleep
[2020-12-21T15:17:27.745Z] Sleeping for 1 min 22 sec
[2020-12-21T15:18:49.759Z] [Google Cloud Storage Plugin] Found 1 files to upload from pattern: x-pack-filebeat--system-tests-linux.tgz
[2020-12-21T15:18:49.813Z] [Google Cloud Storage Plugin] Uploading: x-pack-filebeat--system-tests-linux.tgz
[2020-12-21T15:18:55.907Z] Client: Docker Engine - Community
[2020-12-21T15:18:55.907Z]  Version:           20.10.1
[2020-12-21T15:18:55.907Z]  API version:       1.41
[2020-12-21T15:18:55.907Z]  Go version:        go1.13.15
[2020-12-21T15:18:55.907Z]  Git commit:        831ebea
[2020-12-21T15:18:55.907Z]  Built:             Tue Dec 15 04:34:59 2020
[2020-12-21T15:18:55.907Z]  OS/Arch:           linux/amd64
[2020-12-21T15:18:55.907Z]  Context:           default
[2020-12-21T15:18:55.907Z]  Experimental:      true
[2020-12-21T15:18:55.907Z] 
[2020-12-21T15:18:55.907Z] Server: Docker Engine - Community
[2020-12-21T15:18:55.907Z]  Engine:
[2020-12-21T15:18:55.907Z]   Version:          20.10.1
[2020-12-21T15:18:55.907Z]   API version:      1.41 (minimum version 1.12)
[2020-12-21T15:18:55.907Z]   Go version:       go1.13.15
[2020-12-21T15:18:55.907Z]   Git commit:       f001486
[2020-12-21T15:18:55.907Z]   Built:            Tue Dec 15 04:32:40 2020
[2020-12-21T15:18:55.907Z]   OS/Arch:          linux/amd64
[2020-12-21T15:18:55.907Z]   Experimental:     false
[2020-12-21T15:18:55.907Z]  containerd:
[2020-12-21T15:18:55.907Z]   Version:          1.4.3
[2020-12-21T15:18:55.907Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2020-12-21T15:18:55.907Z]  runc:
[2020-12-21T15:18:55.907Z]   Version:          1.0.0-rc92
[2020-12-21T15:18:55.907Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2020-12-21T15:18:55.907Z]  docker-init:
[2020-12-21T15:18:55.907Z]   Version:          0.19.0
[2020-12-21T15:18:55.907Z]   GitCommit:        de40ad0
[2020-12-21T15:19:01.404Z] Failed in branch x-pack/filebeat-build
[2020-12-21T15:19:01.470Z] Stage "Packaging" skipped due to earlier failure(s)
[2020-12-21T15:19:01.512Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20831/src/github.com/elastic/beats
[2020-12-21T15:19:01.697Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats_PR-20831
[2020-12-21T15:19:01.770Z] [INFO] getVaultSecret: Getting secrets
[2020-12-21T15:19:01.816Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-12-21T15:19:03.755Z] + chmod 755 generate-build-data.sh
[2020-12-21T15:19:03.755Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20831/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20831/runs/3 FAILURE 2975132
[2020-12-21T15:19:03.755Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20831/runs/3/steps/?limit=10000 -o steps-info.json
[2020-12-21T15:19:06.015Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20831/runs/3/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	1
Passed	2444
Skipped	263
Total	2708

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / x-pack/filebeat-build / test_fileset_file_203_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest

elasticmachine · 2020-08-27T15:39:32Z

Pinging @elastic/siem (Team:SIEM)

fix parsing error and add enhancements

fix 602303

P1llus · 2020-09-06T22:11:51Z

@pcosic Would you be able to generate some new test files to see how it would impact the test files?

Inside the x-pack filebeat folder run:
GENERATE=1 INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=cisco make system-tests

pcosic · 2020-09-08T12:50:28Z

@P1llus Sorry that I didn't do this earlier. I committed all the changes and here is the Output of the test:

------------------------------------- generated xml file: /go/src/github.com/elastic/beats/x-pack/filebeat/build/TEST-python-integration.xml -------------------------------------
============================================================================== slowest 20 durations ==============================================================================
16.32s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_00_cisco
9.48s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_15_cisco
9.03s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_03_cisco
7.28s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_20_cisco
6.92s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_06_cisco
6.34s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_18_cisco
5.22s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_05_cisco
4.50s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_11_cisco
4.25s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_12_cisco
4.13s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_07_cisco
3.62s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_01_cisco
3.48s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_08_cisco
3.19s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_14_cisco
3.06s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_02_cisco
2.97s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_10_cisco
2.88s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_04_cisco
2.83s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_16_cisco
2.78s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_19_cisco
2.77s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_17_cisco
2.75s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_09_cisco
========================================================================= 35 passed in 131.67s (0:02:11) =========================================================================
>> python test: Integration Testing Complete

Is there anything else needed?

x-pack/filebeat/module/cisco/asa/_meta/fields.yml

x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml

botelastic · 2020-10-18T09:36:16Z

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

felix-lessoer · 2020-10-18T09:55:30Z

+1

P1llus · 2020-10-20T08:53:43Z

@pcosic are you able to merge with master? There is a conflict with x-pack/filebeat/module/cisco/fields.go

P1llus · 2020-10-20T13:35:43Z

run tests

commit after running tests.

P1llus · 2020-10-20T13:53:46Z

I just had to write a comment to trigger the CI, no worries :)

pcosic · 2020-10-21T11:26:03Z

Can someone run a test again?

P1llus · 2020-10-21T11:30:03Z

jenkins run tests

pcosic · 2020-10-21T12:51:04Z

It looks like I failed because of 106014, originally I did not want to change this message id with my PR.
Locally my tests succeed and passed on all tests.

P1llus · 2020-10-30T10:29:24Z

Hey @pcosic ,

Sorry for the delay, you are indeed correct that not all issues with the test logs are related to your PR. There might be some delays with the answers, but let me also tag in @jamiehynds to see if he can keep an overview on this PR and maybe share some details.

This fixing finally 106014. We have, afaik, two options. Use IPORHOST to not match '(type' or using '(?<destination.address>[^ (]*)' so we only dispense on space or '(' for the case destination.address is weird. NOTSPACE is not work in this case.

pcosic · 2020-10-30T11:58:38Z

sorry this was my mistake. I think 106014 shouldn't be a problem anymore, and the PR would pass the tests with the last commit. Maybe run test again 👍 ?

andrewstucki · 2020-10-30T15:05:50Z

jenkins run tests

elasticmachine · 2020-10-30T16:09:28Z

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	1947
Skipped	259
Total	2206

…ePattern

P1llus · 2020-10-30T17:42:29Z

Jenkins run tests

pcosic · 2020-11-06T09:24:35Z

I have changed what you @adriansr requested at the Cr and it would be nice to get your approval

botelastic · 2020-12-16T13:15:42Z

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

marc-gr · 2020-12-17T08:45:42Z

hello @pcosic ! Can you please update the branch with master so I can run the tests again? Thanks!

…ePattern

pcosic · 2020-12-21T12:00:44Z

you can start the test @marc-gr if you want.
but I still need an approval for the CR right?

marc-gr · 2020-12-21T14:29:02Z

jenkins run tests

elasticmachine · 2021-01-13T02:07:01Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh · 2021-03-24T18:55:06Z

I pulled these commits into a new PR, resolved the merge conflicts, and added a changelog entry. #24744

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 27, 2020

jsoriano added the Team:SIEM label Aug 27, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 27, 2020

jsoriano added enhancement Filebeat Filebeat review labels Aug 27, 2020

pcosic added 3 commits August 31, 2020 16:55

Update asa-ftd-pipeline.yml

8e0491e

Update asa-ftd-pipeline.yml

b13af6e

fix parsing error and add enhancements

Update asa-ftd-pipeline.yml

eafaae2

fix 602303

testing for PR and some minor fixes

40814cc

adriansr suggested changes Sep 16, 2020

View reviewed changes

x-pack/filebeat/module/cisco/asa/_meta/fields.yml Outdated Show resolved Hide resolved

x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml Outdated Show resolved Hide resolved

pcosic added 2 commits September 18, 2020 11:16

commit for requested changes

b992fcd

newline

5386064

botelastic bot added the Stalled label Oct 18, 2020

botelastic bot removed the Stalled label Oct 18, 2020

pcosic added 2 commits October 20, 2020 11:37

test

1775792

Merge branch 'master' into ingestCiscoMessagePattern

14aebc0

make test commit

029083f

commit after running tests.

pcosic closed this Oct 30, 2020

pcosic reopened this Oct 30, 2020

fixed 106014 finally

1e9da38

This fixing finally 106014. We have, afaik, two options. Use IPORHOST to not match '(type' or using '(?<destination.address>[^ (]*)' so we only dispense on space or '(' for the case destination.address is weird. NOTSPACE is not work in this case.

pcosic added 2 commits October 30, 2020 17:45

Merge remote-tracking branch 'upstream/master' into ingestCiscoMessag…

e664cd6

…ePattern

after test commit

babe7b5

pcosic requested a review from adriansr October 30, 2020 19:47

botelastic bot added the Stalled label Dec 16, 2020

botelastic bot removed the Stalled label Dec 17, 2020

pcosic added 2 commits December 17, 2020 11:12

Merge remote-tracking branch 'upstream/master' into ingestCiscoMessag…

faf2659

…ePattern

Test after merge

763132e

andrewkroh added Team:Security-External Integrations and removed Team:SIEM labels Jan 13, 2021

adriansr self-assigned this Jan 19, 2021

andrewkroh mentioned this pull request Mar 24, 2021

[Filebeat][Cisco ASA] log enhancement and performance #24744

Merged

6 tasks

andrewkroh closed this Mar 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat][Cisco ASA] log enhancement and performance #20831

[Filebeat][Cisco ASA] log enhancement and performance #20831

pcosic commented Aug 27, 2020 •

edited

Loading

elasticmachine commented Aug 27, 2020

elasticmachine commented Aug 27, 2020

elasticmachine commented Aug 27, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

`Build&Test / x-pack/filebeat-build / test_fileset_file_203_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest`

`x-pack/filebeat-build - mage build test`

`Error signal`

Test stats 🧪

Genuine test errors

elasticmachine commented Aug 27, 2020

P1llus commented Sep 6, 2020

pcosic commented Sep 8, 2020 •

edited

Loading

botelastic bot commented Oct 18, 2020

felix-lessoer commented Oct 18, 2020

P1llus commented Oct 20, 2020

P1llus commented Oct 20, 2020

P1llus commented Oct 20, 2020

pcosic commented Oct 21, 2020

P1llus commented Oct 21, 2020

pcosic commented Oct 21, 2020

P1llus commented Oct 30, 2020

pcosic commented Oct 30, 2020

andrewstucki commented Oct 30, 2020

elasticmachine commented Oct 30, 2020 •

edited by jenkins-beats-ci bot

Loading

Test stats 🧪

P1llus commented Oct 30, 2020

pcosic commented Nov 6, 2020 •

edited

Loading

botelastic bot commented Dec 16, 2020

marc-gr commented Dec 17, 2020

pcosic commented Dec 21, 2020

marc-gr commented Dec 21, 2020

elasticmachine commented Jan 13, 2021

andrewkroh commented Mar 24, 2021

[Filebeat][Cisco ASA] log enhancement and performance #20831

[Filebeat][Cisco ASA] log enhancement and performance #20831

Conversation

pcosic commented Aug 27, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Aug 27, 2020

elasticmachine commented Aug 27, 2020

elasticmachine commented Aug 27, 2020 • edited by jenkins-beats-ci bot Loading

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Build&Test / x-pack/filebeat-build / test_fileset_file_203_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest

Steps errors

x-pack/filebeat-build - mage build test

Error signal

Log output

🐛 Flaky test report

Test stats 🧪

Genuine test errors

elasticmachine commented Aug 27, 2020

P1llus commented Sep 6, 2020

pcosic commented Sep 8, 2020 • edited Loading

botelastic bot commented Oct 18, 2020

felix-lessoer commented Oct 18, 2020

P1llus commented Oct 20, 2020

P1llus commented Oct 20, 2020

P1llus commented Oct 20, 2020

pcosic commented Oct 21, 2020

P1llus commented Oct 21, 2020

pcosic commented Oct 21, 2020

P1llus commented Oct 30, 2020

pcosic commented Oct 30, 2020

andrewstucki commented Oct 30, 2020

elasticmachine commented Oct 30, 2020 • edited by jenkins-beats-ci bot Loading

💚 Flaky test report

Test stats 🧪

P1llus commented Oct 30, 2020

pcosic commented Nov 6, 2020 • edited Loading

botelastic bot commented Dec 16, 2020

marc-gr commented Dec 17, 2020

pcosic commented Dec 21, 2020

marc-gr commented Dec 21, 2020

elasticmachine commented Jan 13, 2021

andrewkroh commented Mar 24, 2021

pcosic commented Aug 27, 2020 •

edited

Loading

elasticmachine commented Aug 27, 2020 •

edited by jenkins-beats-ci bot

Loading

`Build&Test / x-pack/filebeat-build / test_fileset_file_203_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest`

`x-pack/filebeat-build - mage build test`

`Error signal`

pcosic commented Sep 8, 2020 •

edited

Loading

elasticmachine commented Oct 30, 2020 •

edited by jenkins-beats-ci bot

Loading

pcosic commented Nov 6, 2020 •

edited

Loading