Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initialize support for ignore_inactive in filestream input #25036

Merged
merged 7 commits into from
Apr 19, 2021
Merged

Initialize support for ignore_inactive in filestream input #25036

merged 7 commits into from
Apr 19, 2021

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Apr 13, 2021 •
edited
Loading

What does this PR do?

This PR adds support for a more flexible file ignoring in filestream input. A new setting is introduced named ignore_inactive. At the moment it only supports two values: since_last_start and since_first_start.

If since_last_start is selected, the input ignores every file that has not been updated since Filebeat has been started. If since_first_start is chosen files that haven't been written since Filebeat has been started the first time on a given host are ignored.

I am submitting tests in a follow-up PR.

Why is it important?

It extends the existing ingore_older support of filestream.

Open question

Should we save the size of the files to the registry, so if they get updated the input can start with the new lines?

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@kvch kvch added needs_backport PR is waiting to be backported to other branches. needs_docs labels Apr 13, 2021
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 13, 2021
@kvch kvch requested a review from urso April 13, 2021 12:47
@kvch kvch added the Team:Elastic-Agent Label for the Agent team label Apr 13, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 13, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Apr 13, 2021 •
edited by jenkins-beats-ci bot
Loading

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #25036 updated

  • Start Time: 2021-04-19T12:06:41.072+0000

  • Duration: 62 min 0 sec

  • Commit: cbf9923

Test stats 🧪

Test Results
Failed 175
Passed 46645
Skipped 5129
Total 51949

Trends 🧪

Image of Build Times

Image of Tests

Test errors 175

Expand to view the tests failures

> Show only the first 10 test failures

Build&Test / filebeat-build / test_fileset_file_000_traefik – filebeat.tests.system.test_modules.Test
    Expand to view the error details

     AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'traefik.access.user_identifier': '-', 'traefik.access.frontend_name': 'Host-host-1', 'traefik.access.backend_url': 'http://172.19.0.3:5601', 'traefik.access.request_count': 262, 'source.address': '192.168.33.1', 'source.ip': '192.168.33.1', 'fileset.name': 'access', 'url.original': '/ui/favicons/favicon-16x16.png', 'input.type': 'log', '@timestamp': '2017-10-02T20:22:07.000Z', 'related.ip': ['192.168.33.1'], 'service.type': 'traefik', 'http.request.referrer': 'http://example.com/login', 'http.request.method': 'GET', 'http.response.status_code': 304, 'http.response.body.bytes': 0, 'http.version': '1.1', 'event.duration': 2000000, 'event.kind': 'event', 'event.module': 'traefik', 'event.category': ['web'], 'event.type': ['access'], 'event.dataset': 'traefik.access', 'event.outcome': 'success', 'user.name': '-', 'user_agent.original': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36', 'user_agent.os.name': 'Linux', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '61.0.3163.100'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

    Expand to view the stacktrace

     a = (<test_modules.Test testMethod=test_fileset_file_000_traefik>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_000_traefik>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/traefik/access/test/test.log'
objects = [{'@timestamp': '2017-10-02T20:22:07.000Z', 'agent': {'ephemeral_id': 'a1e3b87a-7c42-457a-a0a1-31e7ff186ad2', 'id': 'd...ry': ['web'], 'created': '2021-04-19T12:38:56.578Z', 'dataset': 'traefik.access', 'duration': 8000000, ...}, ...}, ...]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'traefik.access.user_identifier': '-', 'traefik.access.frontend_name': 'Host-host-1', 'traefik.access.backend_url': 'http://172.19.0.3:5601', 'traefik.access.request_count': 262, 'source.address': '192.168.33.1', 'source.ip': '192.168.33.1', 'fileset.name': 'access', 'url.original': '/ui/favicons/favicon-16x16.png', 'input.type': 'log', '@timestamp': '2017-10-02T20:22:07.000Z', 'related.ip': ['192.168.33.1'], 'service.type': 'traefik', 'http.request.referrer': 'http://example.com/login', 'http.request.method': 'GET', 'http.response.status_code': 304, 'http.response.body.bytes': 0, 'http.version': '1.1', 'event.duration': 2000000, 'event.kind': 'event', 'event.module': 'traefik', 'event.category': ['web'], 'event.type': ['access'], 'event.dataset': 'traefik.access', 'event.outcome': 'success', 'user.name': '-', 'user_agent.original': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36', 'user_agent.os.name': 'Linux', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '61.0.3163.100'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

Build&Test / filebeat-build / test_fileset_file_002_iis – filebeat.tests.system.test_modules.Test
    Expand to view the error details

     AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 244, 'destination.address': '10.100.220.70', 'destination.port': 80, 'destination.ip': '10.100.220.70', 'source.address': '10.100.118.31', 'source.ip': '10.100.118.31', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.sub_status': 4, 'iis.access.win32_status': 2, '@timestamp': '2018-08-28T18:24:25.000Z', 'related.ip': ['10.100.118.31', '10.100.220.70'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 792000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR[ 2.0.50727](tel: 2050727); .NET CLR 3.0.30729)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '8.1', 'user_agent.os.full': 'Windows 8.1', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '11.0'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

    Expand to view the stacktrace

     a = (<test_modules.Test testMethod=test_fileset_file_002_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_002_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-iis-7.5.log'
objects = [{'@timestamp': '2018-08-28T18:24:25.000Z', 'agent': {'ephemeral_id': '2b271b6e-e928-4b00-8c37-999946a369b4', 'id': 'd...:0000:0000:3257:9652', 'ip': '2001:cdba:0000:0000:0000:0000:3257:9652', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 244, 'destination.address': '10.100.220.70', 'destination.port': 80, 'destination.ip': '10.100.220.70', 'source.address': '10.100.118.31', 'source.ip': '10.100.118.31', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.sub_status': 4, 'iis.access.win32_status': 2, '@timestamp': '2018-08-28T18:24:25.000Z', 'related.ip': ['10.100.118.31', '10.100.220.70'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 792000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR[ 2.0.50727](tel: 2050727); .NET CLR 3.0.30729)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '8.1', 'user_agent.os.full': 'Windows 8.1', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '11.0'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

Build&Test / filebeat-build / test_fileset_file_003_iis – filebeat.tests.system.test_modules.Test
    Expand to view the error details

     AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'destination.address': '10.24.129.162', 'destination.port': 80, 'destination.domain': 'images.hogeschoolrotterdam.nl', 'destination.ip': '10.24.129.162', 'source.address': '10.24.136.240', 'source.ip': '10.24.136.240', 'fileset.name': 'access', 'url.path': '/favicon.ico', 'network.forwarded_ip': '116.189.86.89', 'input.type': 'log', 'iis.access.site_name': 'W3SVC2', 'iis.access.server_name': 'freca1', 'iis.access.sub_status': 0, 'iis.access.win32_status': 2, '@timestamp': '2020-10-04T22:00:34.000Z', 'related.ip': ['10.24.136.240', '10.24.129.162'], 'service.type': 'iis', 'http.request.referrer': 'https://images.hogeschoolrotterdam.nl/Blob/adeec119008c48758c1a6be53aeeb2ac/34ff475072d54117bcb46ea7f023bd87.jpg?width=1200&height=630&mode=crop', 'http.request.method': 'GET', 'http.request.body.bytes': 534, 'http.response.status_code': 404, 'http.response.body.bytes': 1437, 'http.version': '1.1', 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '81.0.'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

    Expand to view the stacktrace

     a = (<test_modules.Test testMethod=test_fileset_file_003_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_003_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-x-forward-for-extended.log'
objects = [{'@timestamp': '2020-10-04T22:00:34.000Z', 'agent': {'ephemeral_id': '00bf41bb-9479-4dcf-9641-df51d61cfa9a', 'id': 'd...162', 'domain': 'images.hogeschoolrotterdam.nl', 'ip': '10.24.129.162', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'destination.address': '10.24.129.162', 'destination.port': 80, 'destination.domain': 'images.hogeschoolrotterdam.nl', 'destination.ip': '10.24.129.162', 'source.address': '10.24.136.240', 'source.ip': '10.24.136.240', 'fileset.name': 'access', 'url.path': '/favicon.ico', 'network.forwarded_ip': '116.189.86.89', 'input.type': 'log', 'iis.access.site_name': 'W3SVC2', 'iis.access.server_name': 'freca1', 'iis.access.sub_status': 0, 'iis.access.win32_status': 2, '@timestamp': '2020-10-04T22:00:34.000Z', 'related.ip': ['10.24.136.240', '10.24.129.162'], 'service.type': 'iis', 'http.request.referrer': 'https://images.hogeschoolrotterdam.nl/Blob/adeec119008c48758c1a6be53aeeb2ac/34ff475072d54117bcb46ea7f023bd87.jpg?width=1200&height=630&mode=crop', 'http.request.method': 'GET', 'http.request.body.bytes': 534, 'http.response.status_code': 404, 'http.response.body.bytes': 1437, 'http.version': '1.1', 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '81.0.'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

Build&Test / filebeat-build / test_fileset_file_004_iis – filebeat.tests.system.test_modules.Test
    Expand to view the error details

     AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 331, 'destination.address': '::1%0', 'destination.port': 80, 'destination.domain': 'example.com', 'destination.ip': '::1', 'source.address': '::1%0', 'source.ip': '::1', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.site_name': 'W3SVC1', 'iis.access.server_name': 'MACHINE-NAME', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T10:11:12.000Z', 'related.ip': ['::1', '::1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.request.body.bytes': 456, 'http.response.status_code': 200, 'http.response.body.bytes': 123, 'http.version': '1.1', 'event.duration': 789000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36', 'user_agent.os.name': 'Mac OS X', 'user_agent.os.version': '10.14.0', 'user_agent.os.full': 'Mac OS X 10.14.0', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Mac', 'user_agent.version': '70.0.3538.102'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

    Expand to view the stacktrace

     a = (<test_modules.Test testMethod=test_fileset_file_004_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_004_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-ipv6zone.log'
objects = [{'@timestamp': '2018-01-01T10:11:12.000Z', 'agent': {'ephemeral_id': 'd9c8c56c-304d-4f4f-873d-225abe6313c7', 'id': 'd...estination': {'address': '::1%0', 'domain': 'example.com', 'ip': '::1', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 331, 'destination.address': '::1%0', 'destination.port': 80, 'destination.domain': 'example.com', 'destination.ip': '::1', 'source.address': '::1%0', 'source.ip': '::1', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.site_name': 'W3SVC1', 'iis.access.server_name': 'MACHINE-NAME', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T10:11:12.000Z', 'related.ip': ['::1', '::1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.request.body.bytes': 456, 'http.response.status_code': 200, 'http.response.body.bytes': 123, 'http.version': '1.1', 'event.duration': 789000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36', 'user_agent.os.name': 'Mac OS X', 'user_agent.os.version': '10.14.0', 'user_agent.os.full': 'Mac OS X 10.14.0', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Mac', 'user_agent.version': '70.0.3538.102'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

Build&Test / filebeat-build / test_fileset_file_005_iis – filebeat.tests.system.test_modules.Test
    Expand to view the error details

     AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'destination.address': '192.168.16.11', 'destination.port': 443, 'destination.ip': '192.168.16.11', 'source.address': '192.168.7.63', 'source.ip': '192.168.7.63', 'fileset.name': 'access', 'url.path': '/Production-UI/api/finance/legacy/GeneralLedger/LoadBatchTotals', 'network.forwarded_ip': '192.168.198.23', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2020-10-07T23:00:17.000Z', 'related.ip': ['192.168.7.63', '192.168.16.11'], 'service.type': 'iis', 'http.request.referrer': 'https://onesolfarm.ggcity.org/Production-UI/ui/uiscreens/generalledger/GLJEUB', 'http.request.method': 'POST', 'http.response.status_code': 200, 'event.duration': 26000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '85.0.4183.121'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

    Expand to view the stacktrace

     a = (<test_modules.Test testMethod=test_fileset_file_005_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_005_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-x-forward-for.log'
objects = [{'@timestamp': '2020-10-07T23:00:17.000Z', 'agent': {'ephemeral_id': '532482a2-8f49-4c9b-aafd-e6c4789eb125', 'id': 'd...destination': {'address': '192.168.16.11', 'ip': '192.168.16.11', 'port': 443}, 'ecs': {'version': '1.9.0'}, ...}, ...]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'destination.address': '192.168.16.11', 'destination.port': 443, 'destination.ip': '192.168.16.11', 'source.address': '192.168.7.63', 'source.ip': '192.168.7.63', 'fileset.name': 'access', 'url.path': '/Production-UI/api/finance/legacy/GeneralLedger/LoadBatchTotals', 'network.forwarded_ip': '192.168.198.23', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2020-10-07T23:00:17.000Z', 'related.ip': ['192.168.7.63', '192.168.16.11'], 'service.type': 'iis', 'http.request.referrer': 'https://onesolfarm.ggcity.org/Production-UI/ui/uiscreens/generalledger/GLJEUB', 'http.request.method': 'POST', 'http.response.status_code': 200, 'event.duration': 26000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '85.0.4183.121'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

Build&Test / filebeat-build / test_fileset_file_006_iis – filebeat.tests.system.test_modules.Test
    Expand to view the error details

     AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'destination.address': '10.44.0.136', 'destination.port': 8080, 'destination.ip': '10.44.0.136', 'source.address': '10.50.6.188', 'source.ip': '10.50.6.188', 'fileset.name': 'access', 'url.path': '/pbserver/..À¯..À¯..À¯..À¯..À¯../winnt/system32/cmd.exe', 'url.query': '/c+dir+c:\\+/OG', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 64, '@timestamp': '2018-12-31T12:02:53.000Z', 'related.ip': ['10.50.6.188', '10.44.0.136'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': 'XP', 'user_agent.os.full': 'Windows XP', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '8.0'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

    Expand to view the stacktrace

     a = (<test_modules.Test testMethod=test_fileset_file_006_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_006_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-iis-7.2.log'
objects = [{'@timestamp': '2018-12-31T12:02:53.000Z', 'agent': {'ephemeral_id': '1cb64ac5-23f9-4afe-b91f-c6e9852a0ab6', 'id': 'd... ...}, 'destination': {'address': '10.44.0.136', 'ip': '10.44.0.136', 'port': 8080}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'destination.address': '10.44.0.136', 'destination.port': 8080, 'destination.ip': '10.44.0.136', 'source.address': '10.50.6.188', 'source.ip': '10.50.6.188', 'fileset.name': 'access', 'url.path': '/pbserver/..À¯..À¯..À¯..À¯..À¯../winnt/system32/cmd.exe', 'url.query': '/c+dir+c:\\+/OG', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 64, '@timestamp': '2018-12-31T12:02:53.000Z', 'related.ip': ['10.50.6.188', '10.44.0.136'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': 'XP', 'user_agent.os.full': 'Windows XP', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '8.0'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

Build&Test / filebeat-build / test_fileset_file_007_iis – filebeat.tests.system.test_modules.Test
    Expand to view the error details

     AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['user_agent.device.type'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object:    {'log.offset': 257, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.ip': '85.181.35.98', 'fileset.name': 'access', 'url.path': '/', 'url.query': 'q=100', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T08:09:10.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 200, 'event.duration': 123000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '7', 'user_agent.os.full': 'Windows 7', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '57.0.'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['user_agent.device...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

    Expand to view the stacktrace

     a = (<test_modules.Test testMethod=test_fileset_file_007_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_007_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test.log'
objects = [{'@timestamp': '2018-01-01T08:09:10.000Z', 'agent': {'ephemeral_id': 'ae89fbfc-21d6-49e2-96d8-5f7e9c93d7b1', 'id': 'd..., ...}, 'destination': {'address': '10.44.0.136', 'ip': '10.44.0.136', 'port': 443}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['user_agent.device.type'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object: 
E             {'log.offset': 257, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.ip': '85.181.35.98', 'fileset.name': 'access', 'url.path': '/', 'url.query': 'q=100', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T08:09:10.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 200, 'event.duration': 123000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '7', 'user_agent.os.full': 'Windows 7', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '57.0.'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['user_agent.device...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

tests/system/test_modules.py:217: AssertionError

Build&Test / filebeat-build / test_fileset_file_010_iis – filebeat.tests.system.test_modules.Test
    Expand to view the error details

     AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object:    {'log.offset': 286, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.port': 2780, 'source.ip': '85.181.35.98', 'fileset.name': 'error', 'url.original': '/ThisIsMyUrl.htm', 'input.type': 'log', 'iis.error.reason_phrase': 'Hostname', '@timestamp': '2018-01-01T09:10:11.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 400, 'http.version': '1.1', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.error', 'event.outcome': 'failure'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

    Expand to view the stacktrace

     a = (<test_modules.Test testMethod=test_fileset_file_010_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_010_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/error/test/test.log'
objects = [{'@timestamp': '2018-01-01T08:09:10.000Z', 'agent': {'ephemeral_id': '9cd4ca40-7771-4dd3-bef9-7c7c444d96d2', 'id': 'd...beat', ...}, 'destination': {'address': '127.0.0.1', 'ip': '127.0.0.1', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object: 
E             {'log.offset': 286, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.port': 2780, 'source.ip': '85.181.35.98', 'fileset.name': 'error', 'url.original': '/ThisIsMyUrl.htm', 'input.type': 'log', 'iis.error.reason_phrase': 'Hostname', '@timestamp': '2018-01-01T09:10:11.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 400, 'http.version': '1.1', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.error', 'event.outcome': 'failure'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

tests/system/test_modules.py:217: AssertionError

Build&Test / filebeat-build / test_fileset_file_013_haproxy – filebeat.tests.system.test_modules.Test
    Expand to view the error details

     AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}, "root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}}}, full object:    {'process.name': 'haproxy', 'process.pid': 32450, 'log.offset': 0, 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 38862, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4'], 'service.type': 'haproxy', 'haproxy.server_name': 'docs', 'haproxy.backend_queue': 0, 'haproxy.total_waiting_time_ms': 0, 'haproxy.termination_state': '----', 'haproxy.connection_wait_time_ms': 1, 'haproxy.backend_name': 'docs_microservice', 'haproxy.http.request.raw_request_line': 'GET /component---src-pages-index-js-4b15624544f97cf0bb8f.js HTTP/1.1', 'haproxy.http.request.captured_cookie': '-', 'haproxy.http.request.captured_headers': ['docs.example.internal'], 'haproxy.http.request.time_wait_ms': 0, 'haproxy.http.request.time_wait_without_data_ms': 0, 'haproxy.http.response.captured_cookie': '-', 'haproxy.http.response.captured_headers': [], 'haproxy.frontend_name': 'incoming~', 'haproxy.server_queue': 0, 'haproxy.bytes_read': 168, 'haproxy.connections.server': 0, 'haproxy.connections.retries': 0, 'haproxy.connections.active': 6, 'haproxy.connections.backend': 0, 'haproxy.connections.frontend': 6, 'http.response.status_code': 304, 'http.response.bytes': 168, 'event.duration': 2000000, 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web'], 'event.dataset': 'haproxy.log', 'event.outcome': 'success'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...143.2104, 'old_value': 37.6172}, "root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}}})

    Expand to view the stacktrace

     a = (<test_modules.Test testMethod=test_fileset_file_013_haproxy>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_013_haproxy>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/haproxy/log/test/haproxy.log'
objects = [{'@timestamp': '2018-07-30T09:03:52.726-02:00', 'agent': {'ephemeral_id': 'be106082-2755-4944-9da0-a5f5c2bf7afa', 'id...ory': ['web'], 'dataset': 'haproxy.log', 'duration': 2000000, 'ingested': '2021-04-19T12:39:21.171507534Z', ...}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}, "root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}}}, full object: 
E             {'process.name': 'haproxy', 'process.pid': 32450, 'log.offset': 0, 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 38862, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4'], 'service.type': 'haproxy', 'haproxy.server_name': 'docs', 'haproxy.backend_queue': 0, 'haproxy.total_waiting_time_ms': 0, 'haproxy.termination_state': '----', 'haproxy.connection_wait_time_ms': 1, 'haproxy.backend_name': 'docs_microservice', 'haproxy.http.request.raw_request_line': 'GET /component---src-pages-index-js-4b15624544f97cf0bb8f.js HTTP/1.1', 'haproxy.http.request.captured_cookie': '-', 'haproxy.http.request.captured_headers': ['docs.example.internal'], 'haproxy.http.request.time_wait_ms': 0, 'haproxy.http.request.time_wait_without_data_ms': 0, 'haproxy.http.response.captured_cookie': '-', 'haproxy.http.response.captured_headers': [], 'haproxy.frontend_name': 'incoming~', 'haproxy.server_queue': 0, 'haproxy.bytes_read': 168, 'haproxy.connections.server': 0, 'haproxy.connections.retries': 0, 'haproxy.connections.active': 6, 'haproxy.connections.backend': 0, 'haproxy.connections.frontend': 6, 'http.response.status_code': 304, 'http.response.bytes': 168, 'event.duration': 2000000, 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web'], 'event.dataset': 'haproxy.log', 'event.outcome': 'success'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...143.2104, 'old_value': 37.6172}, "root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}}})

tests/system/test_modules.py:217: AssertionError

Build&Test / filebeat-build / test_fileset_file_014_haproxy – filebeat.tests.system.test_modules.Test
    Expand to view the error details

     AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}, "root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}}}, full object:    {'process.name': 'haproxy', 'process.pid': 24551, 'log.offset': 0, 'destination.port': 5000, 'destination.ip': '1.2.3.4', 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 40780, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4', '1.2.3.4'], 'service.type': 'haproxy', 'haproxy.mode': 'HTTP', 'haproxy.frontend_name': 'main', 'haproxy.source': '1.2.3.4', 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'haproxy.log'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...143.2104, 'old_value': 37.6172}, "root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}}})

    Expand to view the stacktrace

     a = (<test_modules.Test testMethod=test_fileset_file_014_haproxy>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_014_haproxy>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/haproxy/log/test/default.log'
objects = [{'@timestamp': '2021-09-20T15:42:59.000-02:00', 'agent': {'ephemeral_id': '7922070b-0aff-4325-aba2-37480bed91e3', 'id...4af56220', 'type': 'filebeat', ...}, 'destination': {'ip': '1.2.3.4', 'port': 5000}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}, "root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}}}, full object: 
E             {'process.name': 'haproxy', 'process.pid': 24551, 'log.offset': 0, 'destination.port': 5000, 'destination.ip': '1.2.3.4', 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 40780, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4', '1.2.3.4'], 'service.type': 'haproxy', 'haproxy.mode': 'HTTP', 'haproxy.frontend_name': 'main', 'haproxy.source': '1.2.3.4', 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'haproxy.log'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...143.2104, 'old_value': 37.6172}, "root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}}})

tests/system/test_modules.py:217: AssertionError

Steps errors 6

Expand to view the steps failures

filebeat-build - mage build test
  • Took 18 min 25 sec . View more details on here
  • Description: mage build test
metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 33 min 33 sec . View more details on here
  • Description: mage pythonIntegTest
packetbeat-windows-7-32-windows-7-32-bit - mage build unitTest
  • Took 2 min 15 sec . View more details on here
  • Description: mage build unitTest
gsutil -m -q cp -a public-read test-build-artifacts-packetbeat-windows-7-32-windows-7-32-bit-tgz gs:
  • Took 0 min 4 sec . View more details on here
  • Description: @echo off gsutil -m -q cp -a public-read test-build-artifacts-packetbeat-windows-7-32-windows-7-32-bit-tgz gs://beats-ci-temp/Beats/beats/PR-25036-10
x-pack/filebeat-build - mage build test
  • Took 35 min 3 sec . View more details on here
  • Description: mage build test
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output 

[2021-04-19T13:06:13.864Z]   Git commit:       46229ca
[2021-04-19T13:06:13.864Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-19T13:06:13.864Z]   OS/Arch:          linux/amd64
[2021-04-19T13:06:13.864Z]   Experimental:     false
[2021-04-19T13:06:13.864Z]  containerd:
[2021-04-19T13:06:13.864Z]   Version:          1.4.4
[2021-04-19T13:06:13.864Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-19T13:06:13.864Z]  runc:
[2021-04-19T13:06:13.865Z]   Version:          1.0.0-rc93
[2021-04-19T13:06:13.865Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-19T13:06:13.865Z]  docker-init:
[2021-04-19T13:06:13.865Z]   Version:          0.19.0
[2021-04-19T13:06:13.865Z]   GitCommit:        de40ad0
[2021-04-19T13:06:13.865Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-19T13:06:13.865Z] Unable to find image 'alpine:3.4' locally
[2021-04-19T13:06:14.436Z] 3.4: Pulling from library/alpine
[2021-04-19T13:06:14.739Z] c1e54eec4b57: Pulling fs layer
[2021-04-19T13:06:15.001Z] c1e54eec4b57: Verifying Checksum
[2021-04-19T13:06:15.001Z] c1e54eec4b57: Download complete
[2021-04-19T13:06:15.263Z] c1e54eec4b57: Pull complete
[2021-04-19T13:06:15.263Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-04-19T13:06:15.263Z] Status: Downloaded newer image for alpine:3.4
[2021-04-19T13:06:17.177Z] Change permissions with write access of all files inside the specific folder
[2021-04-19T13:06:18.629Z] Running in /var/lib/jenkins/workspace/PR-25036-10-ef54408d-bfe7-4eab-91ce-aec328427526/src/github.com/elastic/beats/build
[2021-04-19T13:06:18.931Z] + rm -rf ve
[2021-04-19T13:06:18.931Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-04-19T13:06:19.313Z] + python .ci/scripts/pre_archive_test.py
[2021-04-19T13:06:21.229Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-04-19T13:06:21.253Z] Running in /var/lib/jenkins/workspace/PR-25036-10-ef54408d-bfe7-4eab-91ce-aec328427526/src/github.com/elastic/beats/build
[2021-04-19T13:06:21.279Z] Recording test results
[2021-04-19T13:06:26.452Z] [Checks API] No suitable checks publisher found.
[2021-04-19T13:06:26.865Z] + tar --version
[2021-04-19T13:06:27.289Z] + tar --exclude=test-build-artifacts-x-pack-filebeat-build-tgz -czf test-build-artifacts-x-pack-filebeat-build-tgz .
[2021-04-19T13:07:24.819Z] + gsutil --version
[2021-04-19T13:07:24.819Z] /var/lib/jenkins/workspace/PR-25036-10-ef54408d-bfe7-4eab-91ce-aec328427526/src/github.com/elastic/beats/build@tmp/durable-f015827f/script.sh: 1: /var/lib/jenkins/workspace/PR-25036-10-ef54408d-bfe7-4eab-91ce-aec328427526/src/github.com/elastic/beats/build@tmp/durable-f015827f/script.sh: gsutil: not found
[2021-04-19T13:07:24.885Z] Running in /var/lib/jenkins/workspace/PR-25036-10-ef54408d-bfe7-4eab-91ce-aec328427526/src/github.com/elastic/beats/build@tmp
[2021-04-19T13:07:25.222Z] + wget --version
[2021-04-19T13:07:25.574Z] + wget -q -O gsutil.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-319.0.0-linux-x86_64.tar.gz
[2021-04-19T13:07:29.294Z] + tar -xpf gsutil.tar.gz --strip-components=1
[2021-04-19T13:07:31.984Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-19T13:07:32.426Z] + gcloud auth activate-service-account --key-file ****
[2021-04-19T13:07:33.820Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-19T13:07:34.199Z] + gsutil -m -q cp -a public-read test-build-artifacts-x-pack-filebeat-build-tgz gs://beats-ci-temp/Beats/beats/PR-25036-10
[2021-04-19T13:07:38.930Z] + python .ci/scripts/search_system_tests.py
[2021-04-19T13:07:38.957Z] [INFO] system-tests='build/x-pack/filebeat/build/system-tests'. If no empty then let's create a tarball
[2021-04-19T13:07:39.341Z] + tar --version
[2021-04-19T13:07:40.365Z] + tar --exclude=x-pack-filebeat--system-tests-linux-tgz -czf x-pack-filebeat--system-tests-linux-tgz build/x-pack/filebeat/build/system-tests
[2021-04-19T13:08:12.897Z] + gsutil --version
[2021-04-19T13:08:12.897Z] /var/lib/jenkins/workspace/PR-25036-10-ef54408d-bfe7-4eab-91ce-aec328427526/src/github.com/elastic/beats@tmp/durable-3dccc5a8/script.sh: 1: /var/lib/jenkins/workspace/PR-25036-10-ef54408d-bfe7-4eab-91ce-aec328427526/src/github.com/elastic/beats@tmp/durable-3dccc5a8/script.sh: gsutil: not found
[2021-04-19T13:08:12.949Z] Running in /var/lib/jenkins/workspace/PR-25036-10-ef54408d-bfe7-4eab-91ce-aec328427526/src/github.com/elastic/beats@tmp
[2021-04-19T13:08:13.282Z] + wget --version
[2021-04-19T13:08:13.620Z] + wget -q -O gsutil.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-319.0.0-linux-x86_64.tar.gz
[2021-04-19T13:08:15.387Z] + tar -xpf gsutil.tar.gz --strip-components=1
[2021-04-19T13:08:18.033Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-19T13:08:18.366Z] + gcloud auth activate-service-account --key-file ****
[2021-04-19T13:08:19.751Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-19T13:08:20.087Z] + gsutil -m -q cp -a public-read x-pack-filebeat--system-tests-linux-tgz gs://beats-ci-temp/Beats/beats/PR-25036-10
[2021-04-19T13:08:23.835Z] + go clean -modcache
[2021-04-19T13:08:27.477Z] Cleaning up /var/lib/jenkins/workspace/PR-25036-10-ef54408d-bfe7-4eab-91ce-aec328427526
[2021-04-19T13:08:27.477Z] Client: Docker Engine - Community
[2021-04-19T13:08:27.477Z]  Version:           20.10.3
[2021-04-19T13:08:27.477Z]  API version:       1.41
[2021-04-19T13:08:27.477Z]  Go version:        go1.13.15
[2021-04-19T13:08:27.477Z]  Git commit:        48d30b5
[2021-04-19T13:08:27.477Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-04-19T13:08:27.478Z]  OS/Arch:           linux/amd64
[2021-04-19T13:08:27.478Z]  Context:           default
[2021-04-19T13:08:27.478Z]  Experimental:      true
[2021-04-19T13:08:27.478Z] 
[2021-04-19T13:08:27.478Z] Server: Docker Engine - Community
[2021-04-19T13:08:27.478Z]  Engine:
[2021-04-19T13:08:27.478Z]   Version:          20.10.3
[2021-04-19T13:08:27.478Z]   API version:      1.41 (minimum version 1.12)
[2021-04-19T13:08:27.478Z]   Go version:       go1.13.15
[2021-04-19T13:08:27.478Z]   Git commit:       46229ca
[2021-04-19T13:08:27.478Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-19T13:08:27.478Z]   OS/Arch:          linux/amd64
[2021-04-19T13:08:27.478Z]   Experimental:     false
[2021-04-19T13:08:27.478Z]  containerd:
[2021-04-19T13:08:27.478Z]   Version:          1.4.4
[2021-04-19T13:08:27.478Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-19T13:08:27.478Z]  runc:
[2021-04-19T13:08:27.478Z]   Version:          1.0.0-rc93
[2021-04-19T13:08:27.478Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-19T13:08:27.478Z]  docker-init:
[2021-04-19T13:08:27.478Z]   Version:          0.19.0
[2021-04-19T13:08:27.478Z]   GitCommit:        de40ad0
[2021-04-19T13:08:27.478Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-19T13:08:31.755Z] Change permissions with write access of all files inside the specific folder
[2021-04-19T13:08:32.065Z] Running in /var/lib/jenkins/workspace/PR-25036-10-ef54408d-bfe7-4eab-91ce-aec328427526
[2021-04-19T13:08:38.539Z] Failed in branch x-pack/filebeat-build
[2021-04-19T13:08:38.931Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-19T13:08:39.106Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25036/src/github.com/elastic/beats
[2021-04-19T13:08:40.210Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25036
[2021-04-19T13:08:40.288Z] [INFO] getVaultSecret: Getting secrets
[2021-04-19T13:08:40.450Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-19T13:08:41.239Z] + chmod 755 generate-build-data.sh
[2021-04-19T13:08:41.239Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25036/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25036/runs/10 FAILURE 3719901
[2021-04-19T13:08:41.490Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25036/runs/10/steps/?limit=10000 -o steps-info.json
[2021-04-19T13:08:51.367Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25036/runs/10/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test Results
Failed 175
Passed 46645
Skipped 5129
Total 51949

Genuine test errors 100

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: Build&Test / filebeat-build / test_fileset_file_000_traefik – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_002_iis – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_003_iis – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_004_iis – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_005_iis – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_006_iis – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_007_iis – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_010_iis – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_013_haproxy – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_014_haproxy – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_065_auditd – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_068_auditd – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_071_nginx – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_072_nginx – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_073_nginx – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_074_nginx – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_102_apache – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_103_apache – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_104_apache – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_105_apache – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_106_apache – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_109_apache – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_133_system – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / filebeat-build / test_fileset_file_136_system – filebeat.tests.system.test_modules.Test
  • Name: Build&Test / metricbeat-pythonIntegTest / test_node_stats – metricbeat.module.logstash.test_logstash.Test
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_001_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_002_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_003_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_004_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_005_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_006_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_007_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_008_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_009_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_010_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_011_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_012_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_013_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_014_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_015_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_016_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_017_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_018_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_019_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_020_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_021_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_022_zscaler – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_023_zscaler – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_024_azure – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_027_azure – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_032_fortinet – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_036_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_037_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_038_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_039_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_040_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_046_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_053_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_059_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_060_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_061_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_064_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_065_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_068_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_069_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_077_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_079_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_082_checkpoint – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_083_sonicwall – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_093_gcp – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_095_gcp – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_101_f5 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_113_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_114_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_115_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_116_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_117_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_118_squid – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_119_squid – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_122_tomcat – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_123_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_124_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_128_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_131_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_132_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_135_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_136_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_138_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_140_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_141_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_142_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_143_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_144_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_145_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_146_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_147_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_148_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_149_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_150_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
  • Name: Build&Test / x-pack/filebeat-build / test_fileset_file_151_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest

@kvch
Copy link
Contributor Author

kvch commented Apr 15, 2021

jenkins run tests

@kvch
Copy link
Contributor Author

kvch commented Apr 15, 2021

jenkins run tests

@kvch
Copy link
Contributor Author

kvch commented Apr 15, 2021

Failing test is unrelated.

urso
urso reviewed Apr 15, 2021
View reviewed changes
libbeat/cmd/instance/beat.go
@@ -695,7 +697,8 @@ func (b *Beat) configure(settings Settings) error {

func (b *Beat) loadMeta(metaPath string) error {
type meta struct {
UUID uuid.UUID `json:"uuid"`
UUID uuid.UUID `json:"uuid"`
FirstStart time.Time `json:"first_start"`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will need some testing. the stdlib JSON encoder/decoder might not play well with that type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my manual tests, it worked like a charm. What might be the problem?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern would be that timestamp serialization and parsing is not supported. If it works +1. We have some tests for loadMeta I think. Still would be nice to enhance the existing tests.

@kvch kvch force-pushed the feature-filebeat-filestream-ignore-older branch from e1f211b to 3c6caf8 Compare April 16, 2021 14:45
@urso
Copy link

urso commented Apr 16, 2021

Change LGTM. Can you please update the tests for loadMeta?

@kvch kvch merged commit 3f39dd8 into elastic:master Apr 19, 2021
kvch added a commit to kvch/beats that referenced this pull request Apr 19, 2021
@kvch
Add support for ignore_inactive in filestream input (elastic#25036)
1cdbd60 
This PR adds support for a more flexible file ignoring in `filestream` input. A new setting is introduced named `ignore_inactive`. At the moment it only supports two values: `since_last_start` and `since_first_start`.

If `since_last_start` is selected, the input ignores every file that has not been updated since Filebeat has been started. If `since_first_start` is chosen files that haven't been written since Filebeat has been started the first time on a given host are ignored.

(cherry picked from commit 3f39dd8)
@kvch kvch mentioned this pull request Apr 19, 2021
Merged
3 tasks
@kvch kvch added v7.13.0 and removed needs_backport PR is waiting to be backported to other branches. labels Apr 19, 2021
kvch added a commit that referenced this pull request Apr 19, 2021
@kvch
Add support for ignore_inactive in filestream input (#25036) (#25140)
927d53f 
This PR adds support for a more flexible file ignoring in `filestream` input. A new setting is introduced named `ignore_inactive`. At the moment it only supports two values: `since_last_start` and `since_first_start`.

If `since_last_start` is selected, the input ignores every file that has not been updated since Filebeat has been started. If `since_first_start` is chosen files that haven't been written since Filebeat has been started the first time on a given host are ignored.

(cherry picked from commit 3f39dd8)
v1v added a commit to v1v/beats that referenced this pull request Apr 20, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into feature/customise…
2d9fb4d 
…-github-pr-comment-template

* upstream/master:
  [Ingest Manager] Keep http and logging config during enroll (elastic#25132)
  Refactor kubernetes autodiscover to avoid skipping short-living pods (elastic#24742)
  [libbeat] New decode xml wineventlog processor (elastic#25115)
  Add svc to agent k8s clusterRole (elastic#25146)
  Add awsfargate module to collect container logs from Amazon ECS on Fargate (elastic#25041)
  [Filebeat][Cisco ASA] log enhancement and performance (elastic#24744)
  Watch kubernetes namespaces for autodiscover metadata for pods (elastic#25117)
  Cyberark Privileged Access Security module (elastic#24803)
  [Elastic Agent] Log the container command output with LOGS_PATH (elastic#25150)
  Fix for tests after `device...` field has been removed (elastic#25141)
  [Ingest Manager] Restart process on output change (elastic#24907)
  Set --insecure in container when FLEET_SERVER_ENABLE and FLEET_INSECURE set. (elastic#25137)
  [filebeat] Update documentation / changelog / beta warnings for the syslog input (elastic#25047)
  Add support for ignore_inactive in filestream input (elastic#25036)
  Fix bug with annotations dedot config on k8s not used (elastic#25111)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@urso urso urso approved these changes

Assignees
No one assigned
Labels
needs_docs Team:Elastic-Agent Label for the Agent team v7.13.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kvch @elasticmachine @urso