New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[libbeat] Decode xml enhance wineventlog #25109

Closed

marc-gr wants to merge 4 commits into elastic:master from marc-gr:decode_xml_enhance_wineventlog

Contributor

marc-gr commented Apr 15, 2021

What does this PR do?

Adds ECS mappings
Adds RenderInfo when possible

Why is it important?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
~~- [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

marc-gr added enhancement Team:Security-External Integrations backport-v7.13.0 labels

marc-gr requested a review from leehinman

April 15, 2021 10:38

Collaborator

elasticmachine commented Apr 15, 2021

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

botelastic bot added needs_team and removed needs_team labels

marc-gr added 2 commits

April 15, 2021 12:48


          Move enrich raw functionality to common package

9b36790


          Enrich Raw fields when possible in decode_xml

a266251

marc-gr force-pushed the decode_xml_enhance_wineventlog branch from f2356cc to 84d1ef2 Compare

April 15, 2021 10:48


          Add ECS mappings when decoding wineventlog xml

b9c0498

marc-gr force-pushed the decode_xml_enhance_wineventlog branch from 84d1ef2 to b9c0498 Compare

April 15, 2021 11:05

Collaborator

elasticmachine commented Apr 15, 2021 •

edited by jenkins-beats-ci bot

Loading

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25109 updated
Start Time: 2021-04-19T09:57:54.550+0000
Duration: 89 min 33 sec
Commit: 4090afe

Test stats 🧪

Test	Results
Failed	174
Passed	46645
Skipped	5129
Total	51948

Trends 🧪

Test errors

Expand to view the tests failures

> Show only the first 10 test failures

`Build&Test / filebeat-build / test_fileset_file_000_traefik – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'traefik.access.user_identifier': '-', 'traefik.access.frontend_name': 'Host-host-1', 'traefik.access.backend_url': 'http://172.19.0.3:5601', 'traefik.access.request_count': 262, 'source.address': '192.168.33.1', 'source.ip': '192.168.33.1', 'fileset.name': 'access', 'url.original': '/ui/favicons/favicon-16x16.png', 'input.type': 'log', '@timestamp': '2017-10-02T20:22:07.000Z', 'related.ip': ['192.168.33.1'], 'service.type': 'traefik', 'http.request.referrer': 'http://example.com/login', 'http.request.method': 'GET', 'http.response.status_code': 304, 'http.response.body.bytes': 0, 'http.version': '1.1', 'event.duration': 2000000, 'event.kind': 'event', 'event.module': 'traefik', 'event.category': ['web'], 'event.type': ['access'], 'event.dataset': 'traefik.access', 'event.outcome': 'success', 'user.name': '-', 'user_agent.original': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36', 'user_agent.os.name': 'Linux', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '61.0.3163.100'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_000_traefik>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_000_traefik>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/traefik/access/test/test.log'
objects = [{'@timestamp': '2017-10-02T20:22:07.000Z', 'agent': {'ephemeral_id': '7686169d-5328-47f0-9398-021e1938e0cf', 'id': '7...ry': ['web'], 'created': '2021-04-19T10:34:14.672Z', 'dataset': 'traefik.access', 'duration': 8000000, ...}, ...}, ...]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'traefik.access.user_identifier': '-', 'traefik.access.frontend_name': 'Host-host-1', 'traefik.access.backend_url': 'http://172.19.0.3:5601', 'traefik.access.request_count': 262, 'source.address': '192.168.33.1', 'source.ip': '192.168.33.1', 'fileset.name': 'access', 'url.original': '/ui/favicons/favicon-16x16.png', 'input.type': 'log', '@timestamp': '2017-10-02T20:22:07.000Z', 'related.ip': ['192.168.33.1'], 'service.type': 'traefik', 'http.request.referrer': 'http://example.com/login', 'http.request.method': 'GET', 'http.response.status_code': 304, 'http.response.body.bytes': 0, 'http.version': '1.1', 'event.duration': 2000000, 'event.kind': 'event', 'event.module': 'traefik', 'event.category': ['web'], 'event.type': ['access'], 'event.dataset': 'traefik.access', 'event.outcome': 'success', 'user.name': '-', 'user_agent.original': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36', 'user_agent.os.name': 'Linux', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '61.0.3163.100'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_002_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 244, 'destination.address': '10.100.220.70', 'destination.port': 80, 'destination.ip': '10.100.220.70', 'source.address': '10.100.118.31', 'source.ip': '10.100.118.31', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.sub_status': 4, 'iis.access.win32_status': 2, '@timestamp': '2018-08-28T18:24:25.000Z', 'related.ip': ['10.100.118.31', '10.100.220.70'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 792000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR[ 2.0.50727](tel: 2050727); .NET CLR 3.0.30729)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '8.1', 'user_agent.os.full': 'Windows 8.1', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '11.0'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_002_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_002_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-iis-7.5.log'
objects = [{'@timestamp': '2018-08-28T18:24:25.000Z', 'agent': {'ephemeral_id': '3adceb3e-7ac9-4c0f-9adb-1c9cc4873969', 'id': '7...:0000:0000:3257:9652', 'ip': '2001:cdba:0000:0000:0000:0000:3257:9652', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 244, 'destination.address': '10.100.220.70', 'destination.port': 80, 'destination.ip': '10.100.220.70', 'source.address': '10.100.118.31', 'source.ip': '10.100.118.31', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.sub_status': 4, 'iis.access.win32_status': 2, '@timestamp': '2018-08-28T18:24:25.000Z', 'related.ip': ['10.100.118.31', '10.100.220.70'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 792000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR[ 2.0.50727](tel: 2050727); .NET CLR 3.0.30729)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '8.1', 'user_agent.os.full': 'Windows 8.1', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '11.0'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_003_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'destination.address': '10.24.129.162', 'destination.port': 80, 'destination.domain': 'images.hogeschoolrotterdam.nl', 'destination.ip': '10.24.129.162', 'source.address': '10.24.136.240', 'source.ip': '10.24.136.240', 'fileset.name': 'access', 'url.path': '/favicon.ico', 'network.forwarded_ip': '116.189.86.89', 'input.type': 'log', 'iis.access.site_name': 'W3SVC2', 'iis.access.server_name': 'freca1', 'iis.access.sub_status': 0, 'iis.access.win32_status': 2, '@timestamp': '2020-10-04T22:00:34.000Z', 'related.ip': ['10.24.136.240', '10.24.129.162'], 'service.type': 'iis', 'http.request.referrer': 'https://images.hogeschoolrotterdam.nl/Blob/adeec119008c48758c1a6be53aeeb2ac/34ff475072d54117bcb46ea7f023bd87.jpg?width=1200&height=630&mode=crop', 'http.request.method': 'GET', 'http.request.body.bytes': 534, 'http.response.status_code': 404, 'http.response.body.bytes': 1437, 'http.version': '1.1', 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '81.0.'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_003_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_003_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-x-forward-for-extended.log'
objects = [{'@timestamp': '2020-10-04T22:00:34.000Z', 'agent': {'ephemeral_id': '5f4638e0-9d30-4110-9aab-6f97df5b2806', 'id': '7...162', 'domain': 'images.hogeschoolrotterdam.nl', 'ip': '10.24.129.162', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'destination.address': '10.24.129.162', 'destination.port': 80, 'destination.domain': 'images.hogeschoolrotterdam.nl', 'destination.ip': '10.24.129.162', 'source.address': '10.24.136.240', 'source.ip': '10.24.136.240', 'fileset.name': 'access', 'url.path': '/favicon.ico', 'network.forwarded_ip': '116.189.86.89', 'input.type': 'log', 'iis.access.site_name': 'W3SVC2', 'iis.access.server_name': 'freca1', 'iis.access.sub_status': 0, 'iis.access.win32_status': 2, '@timestamp': '2020-10-04T22:00:34.000Z', 'related.ip': ['10.24.136.240', '10.24.129.162'], 'service.type': 'iis', 'http.request.referrer': 'https://images.hogeschoolrotterdam.nl/Blob/adeec119008c48758c1a6be53aeeb2ac/34ff475072d54117bcb46ea7f023bd87.jpg?width=1200&height=630&mode=crop', 'http.request.method': 'GET', 'http.request.body.bytes': 534, 'http.response.status_code': 404, 'http.response.body.bytes': 1437, 'http.version': '1.1', 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '81.0.'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_004_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 331, 'destination.address': '::1%0', 'destination.port': 80, 'destination.domain': 'example.com', 'destination.ip': '::1', 'source.address': '::1%0', 'source.ip': '::1', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.site_name': 'W3SVC1', 'iis.access.server_name': 'MACHINE-NAME', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T10:11:12.000Z', 'related.ip': ['::1', '::1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.request.body.bytes': 456, 'http.response.status_code': 200, 'http.response.body.bytes': 123, 'http.version': '1.1', 'event.duration': 789000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36', 'user_agent.os.name': 'Mac OS X', 'user_agent.os.version': '10.14.0', 'user_agent.os.full': 'Mac OS X 10.14.0', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Mac', 'user_agent.version': '70.0.3538.102'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_004_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_004_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-ipv6zone.log'
objects = [{'@timestamp': '2018-01-01T10:11:12.000Z', 'agent': {'ephemeral_id': '58b7e61f-0b30-4f43-be8d-63304c99868c', 'id': '7...estination': {'address': '::1%0', 'domain': 'example.com', 'ip': '::1', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 331, 'destination.address': '::1%0', 'destination.port': 80, 'destination.domain': 'example.com', 'destination.ip': '::1', 'source.address': '::1%0', 'source.ip': '::1', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.site_name': 'W3SVC1', 'iis.access.server_name': 'MACHINE-NAME', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T10:11:12.000Z', 'related.ip': ['::1', '::1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.request.body.bytes': 456, 'http.response.status_code': 200, 'http.response.body.bytes': 123, 'http.version': '1.1', 'event.duration': 789000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36', 'user_agent.os.name': 'Mac OS X', 'user_agent.os.version': '10.14.0', 'user_agent.os.full': 'Mac OS X 10.14.0', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Mac', 'user_agent.version': '70.0.3538.102'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_005_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'destination.address': '192.168.16.11', 'destination.port': 443, 'destination.ip': '192.168.16.11', 'source.address': '192.168.7.63', 'source.ip': '192.168.7.63', 'fileset.name': 'access', 'url.path': '/Production-UI/api/finance/legacy/GeneralLedger/LoadBatchTotals', 'network.forwarded_ip': '192.168.198.23', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2020-10-07T23:00:17.000Z', 'related.ip': ['192.168.7.63', '192.168.16.11'], 'service.type': 'iis', 'http.request.referrer': 'https://onesolfarm.ggcity.org/Production-UI/ui/uiscreens/generalledger/GLJEUB', 'http.request.method': 'POST', 'http.response.status_code': 200, 'event.duration': 26000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '85.0.4183.121'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_005_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_005_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-x-forward-for.log'
objects = [{'@timestamp': '2020-10-07T23:00:17.000Z', 'agent': {'ephemeral_id': 'eb6e6222-9bcb-4bc7-96b3-16ba66efeb05', 'id': '7...destination': {'address': '192.168.16.11', 'ip': '192.168.16.11', 'port': 443}, 'ecs': {'version': '1.9.0'}, ...}, ...]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'destination.address': '192.168.16.11', 'destination.port': 443, 'destination.ip': '192.168.16.11', 'source.address': '192.168.7.63', 'source.ip': '192.168.7.63', 'fileset.name': 'access', 'url.path': '/Production-UI/api/finance/legacy/GeneralLedger/LoadBatchTotals', 'network.forwarded_ip': '192.168.198.23', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2020-10-07T23:00:17.000Z', 'related.ip': ['192.168.7.63', '192.168.16.11'], 'service.type': 'iis', 'http.request.referrer': 'https://onesolfarm.ggcity.org/Production-UI/ui/uiscreens/generalledger/GLJEUB', 'http.request.method': 'POST', 'http.response.status_code': 200, 'event.duration': 26000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '85.0.4183.121'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_006_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'destination.address': '10.44.0.136', 'destination.port': 8080, 'destination.ip': '10.44.0.136', 'source.address': '10.50.6.188', 'source.ip': '10.50.6.188', 'fileset.name': 'access', 'url.path': '/pbserver/..À¯..À¯..À¯..À¯..À¯../winnt/system32/cmd.exe', 'url.query': '/c+dir+c:\\+/OG', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 64, '@timestamp': '2018-12-31T12:02:53.000Z', 'related.ip': ['10.50.6.188', '10.44.0.136'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': 'XP', 'user_agent.os.full': 'Windows XP', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '8.0'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_006_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_006_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-iis-7.2.log'
objects = [{'@timestamp': '2018-12-31T12:02:53.000Z', 'agent': {'ephemeral_id': '864addb9-55c7-4faf-b760-1e67df91f127', 'id': '7... ...}, 'destination': {'address': '10.44.0.136', 'ip': '10.44.0.136', 'port': 8080}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'destination.address': '10.44.0.136', 'destination.port': 8080, 'destination.ip': '10.44.0.136', 'source.address': '10.50.6.188', 'source.ip': '10.50.6.188', 'fileset.name': 'access', 'url.path': '/pbserver/..À¯..À¯..À¯..À¯..À¯../winnt/system32/cmd.exe', 'url.query': '/c+dir+c:\\+/OG', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 64, '@timestamp': '2018-12-31T12:02:53.000Z', 'related.ip': ['10.50.6.188', '10.44.0.136'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': 'XP', 'user_agent.os.full': 'Windows XP', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '8.0'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_007_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['user_agent.device.type'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object:    {'log.offset': 257, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.ip': '85.181.35.98', 'fileset.name': 'access', 'url.path': '/', 'url.query': 'q=100', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T08:09:10.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 200, 'event.duration': 123000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '7', 'user_agent.os.full': 'Windows 7', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '57.0.'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['user_agent.device.type'], root['source.geo.region_iso...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_007_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_007_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test.log'
objects = [{'@timestamp': '2018-01-01T08:09:10.000Z', 'agent': {'ephemeral_id': '2720ac1e-3671-4051-83f9-379e25aeb57e', 'id': '7..., ...}, 'destination': {'address': '10.44.0.136', 'ip': '10.44.0.136', 'port': 443}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['user_agent.device.type'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object: 
E             {'log.offset': 257, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.ip': '85.181.35.98', 'fileset.name': 'access', 'url.path': '/', 'url.query': 'q=100', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T08:09:10.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 200, 'event.duration': 123000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '7', 'user_agent.os.full': 'Windows 7', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '57.0.'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['user_agent.device.type'], root['source.geo.region_iso...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_010_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object:    {'log.offset': 286, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.port': 2780, 'source.ip': '85.181.35.98', 'fileset.name': 'error', 'url.original': '/ThisIsMyUrl.htm', 'input.type': 'log', 'iis.error.reason_phrase': 'Hostname', '@timestamp': '2018-01-01T09:10:11.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 400, 'http.version': '1.1', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.error', 'event.outcome': 'failure'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_010_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_010_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/error/test/test.log'
objects = [{'@timestamp': '2018-01-01T08:09:10.000Z', 'agent': {'ephemeral_id': '9953b6bf-7bdf-4160-8ed2-1ea0f79e2c17', 'id': '7...beat', ...}, 'destination': {'address': '127.0.0.1', 'ip': '127.0.0.1', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object: 
E             {'log.offset': 286, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.port': 2780, 'source.ip': '85.181.35.98', 'fileset.name': 'error', 'url.original': '/ThisIsMyUrl.htm', 'input.type': 'log', 'iis.error.reason_phrase': 'Hostname', '@timestamp': '2018-01-01T09:10:11.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 400, 'http.version': '1.1', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.error', 'event.outcome': 'failure'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_013_haproxy – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}}}, full object:    {'process.name': 'haproxy', 'process.pid': 32450, 'log.offset': 0, 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 38862, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4'], 'service.type': 'haproxy', 'haproxy.server_name': 'docs', 'haproxy.backend_queue': 0, 'haproxy.total_waiting_time_ms': 0, 'haproxy.termination_state': '----', 'haproxy.connection_wait_time_ms': 1, 'haproxy.backend_name': 'docs_microservice', 'haproxy.http.request.raw_request_line': 'GET /component---src-pages-index-js-4b15624544f97cf0bb8f.js HTTP/1.1', 'haproxy.http.request.captured_cookie': '-', 'haproxy.http.request.captured_headers': ['docs.example.internal'], 'haproxy.http.request.time_wait_ms': 0, 'haproxy.http.request.time_wait_without_data_ms': 0, 'haproxy.http.response.captured_cookie': '-', 'haproxy.http.response.captured_headers': [], 'haproxy.frontend_name': 'incoming~', 'haproxy.server_queue': 0, 'haproxy.bytes_read': 168, 'haproxy.connections.server': 0, 'haproxy.connections.retries': 0, 'haproxy.connections.active': 6, 'haproxy.connections.backend': 0, 'haproxy.connections.frontend': 6, 'http.response.status_code': 304, 'http.response.bytes': 168, 'event.duration': 2000000, 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web'], 'event.dataset': 'haproxy.log', 'event.outcome': 'success'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...143.2104, 'old_value': 37.6172}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}}})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_013_haproxy>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_013_haproxy>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/haproxy/log/test/haproxy.log'
objects = [{'@timestamp': '2018-07-30T09:03:52.726-02:00', 'agent': {'ephemeral_id': 'fb8939a1-ac92-4445-b8ec-f88bbb55e309', 'id...ory': ['web'], 'dataset': 'haproxy.log', 'duration': 2000000, 'ingested': '2021-04-19T10:34:41.119988201Z', ...}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}}}, full object: 
E             {'process.name': 'haproxy', 'process.pid': 32450, 'log.offset': 0, 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 38862, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4'], 'service.type': 'haproxy', 'haproxy.server_name': 'docs', 'haproxy.backend_queue': 0, 'haproxy.total_waiting_time_ms': 0, 'haproxy.termination_state': '----', 'haproxy.connection_wait_time_ms': 1, 'haproxy.backend_name': 'docs_microservice', 'haproxy.http.request.raw_request_line': 'GET /component---src-pages-index-js-4b15624544f97cf0bb8f.js HTTP/1.1', 'haproxy.http.request.captured_cookie': '-', 'haproxy.http.request.captured_headers': ['docs.example.internal'], 'haproxy.http.request.time_wait_ms': 0, 'haproxy.http.request.time_wait_without_data_ms': 0, 'haproxy.http.response.captured_cookie': '-', 'haproxy.http.response.captured_headers': [], 'haproxy.frontend_name': 'incoming~', 'haproxy.server_queue': 0, 'haproxy.bytes_read': 168, 'haproxy.connections.server': 0, 'haproxy.connections.retries': 0, 'haproxy.connections.active': 6, 'haproxy.connections.backend': 0, 'haproxy.connections.frontend': 6, 'http.response.status_code': 304, 'http.response.bytes': 168, 'event.duration': 2000000, 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web'], 'event.dataset': 'haproxy.log', 'event.outcome': 'success'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...143.2104, 'old_value': 37.6172}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}}})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_014_haproxy – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}}}, full object:    {'process.name': 'haproxy', 'process.pid': 24551, 'log.offset': 0, 'destination.port': 5000, 'destination.ip': '1.2.3.4', 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 40780, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4', '1.2.3.4'], 'service.type': 'haproxy', 'haproxy.mode': 'HTTP', 'haproxy.frontend_name': 'main', 'haproxy.source': '1.2.3.4', 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'haproxy.log'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...143.2104, 'old_value': 37.6172}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}}})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_014_haproxy>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_014_haproxy>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/haproxy/log/test/default.log'
objects = [{'@timestamp': '2021-09-20T15:42:59.000-02:00', 'agent': {'ephemeral_id': '9d82d9cf-12c3-46fd-b646-bc9596cdb7f0', 'id...2e46f285', 'type': 'filebeat', ...}, 'destination': {'ip': '1.2.3.4', 'port': 5000}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}}}, full object: 
E             {'process.name': 'haproxy', 'process.pid': 24551, 'log.offset': 0, 'destination.port': 5000, 'destination.ip': '1.2.3.4', 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 40780, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4', '1.2.3.4'], 'service.type': 'haproxy', 'haproxy.mode': 'HTTP', 'haproxy.frontend_name': 'main', 'haproxy.source': '1.2.3.4', 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'haproxy.log'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...143.2104, 'old_value': 37.6172}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}}})

tests/system/test_modules.py:217: AssertionError

Steps errors

Expand to view the steps failures

`filebeat-build - mage build test`

Took 19 min 30 sec . View more details on here
Description: mage build test

`packetbeat-windows-7-32-windows-7-32-bit - mage build unitTest`

Took 2 min 2 sec . View more details on here
Description: mage build unitTest

`gsutil -m -q cp -a public-read test-build-artifacts-packetbeat-windows-7-32-windows-7-32-bit-tgz gs:`

Took 0 min 3 sec . View more details on here
Description: @echo off gsutil -m -q cp -a public-read test-build-artifacts-packetbeat-windows-7-32-windows-7-32-bit-tgz gs://beats-ci-temp/Beats/beats/PR-25109-4

`x-pack/filebeat-build - mage build test`

Took 33 min 57 sec . View more details on here
Description: mage build test

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-04-19T11:24:30.316Z] 0.20s call     x-pack/functionbeat/tests/system/test_base.py::Test::test_export_ilm_policy
[2021-04-19T11:24:30.316Z] 0.12s call     x-pack/functionbeat/tests/system/test_base.py::Test::test_export_function_invalid_conf
[2021-04-19T11:24:30.316Z] 
[2021-04-19T11:24:30.316Z] (13 durations < 0.005s hidden.  Use -vv to show these durations.)
[2021-04-19T11:24:30.316Z] ======================== 7 passed, 1 skipped in 2.88s =========================
[2021-04-19T11:24:30.316Z] >> python test: Unit Testing Complete
[2021-04-19T11:24:30.680Z] 
[2021-04-19T11:24:30.681Z] C:\Users\jenkins\workspace\PR-25109-4-66f57bc7-6325-47fb-bf60-b1b87df8b5e0\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-04-19T11:24:32.365Z] 
[2021-04-19T11:24:32.365Z] C:\Users\jenkins\workspace\PR-25109-4-66f57bc7-6325-47fb-bf60-b1b87df8b5e0\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-04-19T11:24:32.626Z] Copy .\x-pack\functionbeat\build into build\x-pack\functionbeat\build
[2021-04-19T11:24:32.627Z] Copy .\x-pack\functionbeat\null\build into build\x-pack\functionbeat\null\build
[2021-04-19T11:24:33.373Z] [Checks API] No suitable checks publisher found.
[2021-04-19T11:24:33.397Z] Running in C:\Users\jenkins\workspace\PR-25109-4-66f57bc7-6325-47fb-bf60-b1b87df8b5e0\src\github.com\elastic\beats\build
[2021-04-19T11:24:33.429Z] Recording test results
[2021-04-19T11:24:33.813Z] 
[2021-04-19T11:24:33.813Z] C:\Users\jenkins\workspace\PR-25109-4-75a53c05-75ac-46b5-8386-e54fe29c0b6c\src\github.com\elastic\beats>go clean -modcache 
[2021-04-19T11:24:35.720Z] [Checks API] No suitable checks publisher found.
[2021-04-19T11:24:36.100Z] 
[2021-04-19T11:24:36.100Z] C:\Users\jenkins\workspace\PR-25109-4-66f57bc7-6325-47fb-bf60-b1b87df8b5e0\src\github.com\elastic\beats>go clean -modcache 
[2021-04-19T11:24:45.193Z] + gsutil --version
[2021-04-19T11:24:46.652Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-19T11:24:47.065Z] + gcloud auth activate-service-account --key-file ****
[2021-04-19T11:24:47.660Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-19T11:24:47.988Z] + gsutil -m -q cp -a public-read eC1wYWNrL21ldHJpY2JlYXQtd2luZG93cy0yMDA4LXdpbmRvd3MtMjAwOC1yMjQwOTBhZmU2ZmU1MDk5OGZmMTg1MGM0YmE2OTdhMmY0NTdiZDMzZWE gs://beats-ci-temp/ci/cache/
[2021-04-19T11:24:58.655Z] warn: failed to upgrade pip (ignoring): running "null\build\ve\windows\Scripts\pip install -U pip" failed with exit code 1============================= test session starts =============================
[2021-04-19T11:24:58.655Z] platform win32 -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
[2021-04-19T11:24:58.655Z] rootdir: C:\Users\jenkins\workspace\PR-25109-4-784c6924-9e49-4c35-aa50-1f3541acfa27\src\github.com\elastic\beats, configfile: pytest.ini
[2021-04-19T11:24:58.655Z] plugins: rerunfailures-9.0, timeout-1.3.4
[2021-04-19T11:24:58.655Z] timeout: 90.0s
[2021-04-19T11:24:58.655Z] timeout method: thread
[2021-04-19T11:24:58.655Z] timeout func_only: True
[2021-04-19T11:24:58.655Z] collected 155 items
[2021-04-19T11:24:58.655Z] 
[2021-04-19T11:25:07.853Z] + gsutil --version
[2021-04-19T11:25:09.304Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-19T11:25:09.625Z] + gcloud auth activate-service-account --key-file ****
[2021-04-19T11:25:10.197Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-19T11:25:10.526Z] + gsutil -m -q cp -a public-read eC1wYWNrL2F1ZGl0YmVhdC13aW5kb3dzLTIwMDgtd2luZG93cy0yMDA4LXIyNDA5MGFmZTZmZTUwOTk4ZmYxODUwYzRiYTY5N2EyZjQ1N2JkMzNlYQ gs://beats-ci-temp/ci/cache/
[2021-04-19T11:25:12.504Z] tests\system\test_filebeat_xpack.py .....                                [  3%]
[2021-04-19T11:25:22.198Z] + gsutil --version
[2021-04-19T11:25:23.653Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-19T11:25:23.974Z] + gcloud auth activate-service-account --key-file ****
[2021-04-19T11:25:24.318Z] + gsutil --version
[2021-04-19T11:25:24.547Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-19T11:25:24.873Z] + gsutil -m -q cp -a public-read cGFja2V0YmVhdC13aW5kb3dzLTIwMDgtd2luZG93cy0yMDA4LXIyNDA5MGFmZTZmZTUwOTk4ZmYxODUwYzRiYTY5N2EyZjQ1N2JkMzNlYQ gs://beats-ci-temp/ci/cache/
[2021-04-19T11:25:25.767Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-19T11:25:26.091Z] + gcloud auth activate-service-account --key-file ****
[2021-04-19T11:25:26.472Z] tests\system\test_http_endpoint.py .........                             [  9%]
[2021-04-19T11:25:26.472Z] tests\system\test_xpack_modules.py sssssssssssssssssssssssssssssssssssss [ 32%]
[2021-04-19T11:25:26.664Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-19T11:25:26.733Z] ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 79%]
[2021-04-19T11:25:26.993Z] ssssssssssssssssssssssssssssssss                                         [100%]
[2021-04-19T11:25:26.993Z] 
[2021-04-19T11:25:26.993Z] - generated xml file: C:\Users\jenkins\workspace\PR-25109-4-784c6924-9e49-4c35-aa50-1f3541acfa27\src\github.com\elastic\beats\x-pack\filebeat\build\TEST-python-unit.xml -
[2021-04-19T11:25:26.993Z] ============================ slowest 20 durations =============================
[2021-04-19T11:25:26.993Z] 3.55s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_index_pattern_migration
[2021-04-19T11:25:26.993Z] 3.52s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_index_pattern
[2021-04-19T11:25:26.993Z] 3.03s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_template
[2021-04-19T11:25:26.993Z] 2.97s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_config
[2021-04-19T11:25:26.993Z] 1.92s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_request
[2021-04-19T11:25:26.993Z] 1.75s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_empty_body
[2021-04-19T11:25:26.993Z] 1.73s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_wrong_auth_header
[2021-04-19T11:25:26.993Z] 1.71s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_get_request
[2021-04-19T11:25:26.993Z] 1.71s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_ilm_policy
[2021-04-19T11:25:26.993Z] 1.71s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_wrong_content_header
[2021-04-19T11:25:26.993Z] 1.70s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_wrong_auth_value
[2021-04-19T11:25:26.993Z] 1.70s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_correct_auth_header
[2021-04-19T11:25:26.993Z] 1.69s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_malformed_json
[2021-04-19T11:25:26.993Z] 0.27s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_missing_auth_value
[2021-04-19T11:25:26.993Z] 
[2021-04-19T11:25:26.993Z] (6 durations < 0.005s hidden.  Use -vv to show these durations.)
[2021-04-19T11:25:26.993Z] ====================== 14 passed, 141 skipped in 30.04s =======================
[2021-04-19T11:25:26.993Z] >> python test: Unit Testing Complete
[2021-04-19T11:25:27.317Z] + gsutil -m -q cp -a public-read eC1wYWNrL2Z1bmN0aW9uYmVhdC13aW5kb3dzLTIwMDgtd2luZG93cy0yMDA4LXIyNDA5MGFmZTZmZTUwOTk4ZmYxODUwYzRiYTY5N2EyZjQ1N2JkMzNlYQ gs://beats-ci-temp/ci/cache/
[2021-04-19T11:25:27.439Z] 
[2021-04-19T11:25:27.439Z] C:\Users\jenkins\workspace\PR-25109-4-784c6924-9e49-4c35-aa50-1f3541acfa27\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-04-19T11:25:29.131Z] 
[2021-04-19T11:25:29.131Z] C:\Users\jenkins\workspace\PR-25109-4-784c6924-9e49-4c35-aa50-1f3541acfa27\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-04-19T11:25:29.393Z] Copy .\x-pack\filebeat\build into build\x-pack\filebeat\build
[2021-04-19T11:25:29.393Z] Copy .\x-pack\filebeat\null\build into build\x-pack\filebeat\null\build
[2021-04-19T11:25:29.415Z] Running in C:\Users\jenkins\workspace\PR-25109-4-784c6924-9e49-4c35-aa50-1f3541acfa27\src\github.com\elastic\beats\build
[2021-04-19T11:25:29.435Z] Recording test results
[2021-04-19T11:25:32.719Z] [Checks API] No suitable checks publisher found.
[2021-04-19T11:25:33.118Z] 
[2021-04-19T11:25:33.118Z] C:\Users\jenkins\workspace\PR-25109-4-784c6924-9e49-4c35-aa50-1f3541acfa27\src\github.com\elastic\beats>go clean -modcache 
[2021-04-19T11:26:21.341Z] + gsutil --version
[2021-04-19T11:26:22.794Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-19T11:26:23.120Z] + gcloud auth activate-service-account --key-file ****
[2021-04-19T11:26:23.695Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-19T11:26:24.020Z] + gsutil -m -q cp -a public-read eC1wYWNrL2ZpbGViZWF0LXdpbmRvd3MtMjAwOC13aW5kb3dzLTIwMDgtcjI0MDkwYWZlNmZlNTA5OThmZjE4NTBjNGJhNjk3YTJmNDU3YmQzM2Vh gs://beats-ci-temp/ci/cache/
[2021-04-19T11:26:25.611Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-19T11:26:25.691Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25109/src/github.com/elastic/beats
[2021-04-19T11:26:26.388Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25109
[2021-04-19T11:26:26.446Z] [INFO] getVaultSecret: Getting secrets
[2021-04-19T11:26:26.563Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-19T11:26:27.405Z] + chmod 755 generate-build-data.sh
[2021-04-19T11:26:27.405Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25109/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25109/runs/4 FAILURE 5312592
[2021-04-19T11:26:27.956Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25109/runs/4/steps/?limit=10000 -o steps-info.json
[2021-04-19T11:26:36.001Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25109/runs/4/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	174
Passed	46645
Skipped	5129
Total	51948

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / filebeat-build / test_fileset_file_000_traefik – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_002_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_003_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_004_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_005_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_006_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_007_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_010_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_013_haproxy – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_014_haproxy – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_065_auditd – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_068_auditd – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_071_nginx – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_072_nginx – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_073_nginx – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_074_nginx – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_102_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_103_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_104_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_105_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_106_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_109_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_133_system – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_136_system – filebeat.tests.system.test_modules.Test
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_001_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_002_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_003_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_004_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_005_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_006_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_007_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_008_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_009_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_010_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_011_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_012_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_013_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_014_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_015_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_016_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_017_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_018_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_019_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_020_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_021_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_022_zscaler – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_023_zscaler – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_024_azure – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_027_azure – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_032_fortinet – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_036_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_037_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_038_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_039_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_040_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_046_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_053_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_059_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_060_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_061_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_064_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_065_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_068_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_069_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_077_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_079_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_082_checkpoint – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_083_sonicwall – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_093_gcp – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_095_gcp – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_101_f5 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_113_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_114_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_115_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_116_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_117_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_118_squid – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_119_squid – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_122_tomcat – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_123_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_124_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_128_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_131_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_132_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_135_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_136_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_138_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_140_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_141_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_142_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_143_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_144_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_145_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_146_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_147_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_148_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_149_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_150_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_151_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_152_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest

Contributor

leehinman commented Apr 15, 2021

I'm thinking adding fields outside of target_field is unexpected, and we should avoid that if possible.

One other option would be to populate target_field with the missing values, that means that consumers of decode_xml and winlog have to deal with different input. We can do it, but that seems unexpected as well.

A second option would be a new processor that does the XML decode, the decode of the Raw fields, & ECS mappings. The output of this new processor should be as close to the output of winlog as possible.

I'm leaning towards the new processor, but that probably puts us past feature freeze.

@andrewkroh thoughts?

Contributor Author

marc-gr commented Apr 15, 2021

I'm thinking adding fields outside of target_field is unexpected, and we should avoid that if possible.

One other option would be to populate target_field with the missing values, that means that consumers of decode_xml and winlog have to deal with different input. We can do it, but that seems unexpected as well.

A second option would be a new processor that does the XML decode, the decode of the Raw fields, & ECS mappings. The output of this new processor should be as close to the output of winlog as possible.

I'm leaning towards the new processor, but that probably puts us past feature freeze.

@andrewkroh thoughts?

I agree that if as part of the decoding we want to map ECS fields probably going with a new processor is the cleanest approach.

Member

andrewkroh commented Apr 15, 2021

Setting those ECS fields does kind of murky up behavior when we have a target_field so I would avoid adding them directly to the processor. Maybe in the decode_xml examples shown in the documentation we could demonstrate how to use the convert processor to populate more ECS fields? Or is it more complicated than that?

I didn't look at what the RenderInfo changes were.

marc-gr mentioned this pull request

[libbeat] New decode xml wineventlog processor #25115

Merged

6 tasks


          Remove ECS mapping

4090afe

marc-gr force-pushed the decode_xml_enhance_wineventlog branch from 4a3bffe to 4090afe Compare

April 19, 2021 09:57

Contributor Author

marc-gr commented Apr 19, 2021

Closing in favor of #25115 approach

marc-gr closed this

marc-gr deleted the decode_xml_enhance_wineventlog branch

April 19, 2021 15:57

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-v7.13.0 enhancement