[7.13](backport #25674) [Filebeat] Fix bad append for abusechmalware #25965

mergify · 2021-05-27T12:34:19Z

This is an automatic backport of pull request #25674 done by Mergify.
Cherry-pick of fafe6fb has failed:

On branch mergify/bp/7.13/pr-25674
Your branch is up to date with 'origin/7.13'.

You are currently cherry-picking commit fafe6fb27.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   x-pack/filebeat/module/threatintel/abusemalware/ingest/pipeline.yml

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   x-pack/filebeat/module/threatintel/abusemalware/test/abusechmalware.ndjson.log-expected.json

To fix up this pull request, you can check it out locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

@Mergifyio refresh will re-evaluate the rules
@Mergifyio rebase will rebase this PR on its base branch
@Mergifyio update will merge the base branch into this PR
@Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

look at your merge queues
generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.io/

* 25151: Fix bad append for abusechmalware * update changelog (cherry picked from commit fafe6fb) # Conflicts: # x-pack/filebeat/module/threatintel/abusemalware/test/abusechmalware.ndjson.log-expected.json

elasticmachine · 2021-05-27T13:33:15Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25965 opened
Start Time: 2021-05-27T12:34:33.791+0000
Duration: 58 min 25 sec
Commit: 3bdf553

Test stats 🧪

Test	Results
Failed	1
Passed	7034
Skipped	1193
Total	8228

Trends 🧪

Test errors

Expand to view the tests failures

`Build&Test / x-pack/filebeat-build / test_fileset_file_116_threatintel – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest`

Expand to view the error details

 json.decoder.JSONDecodeError: Expecting value: line 13 column 1 (char 464)

Expand to view the stacktrace

 a = (<test_xpack_modules.XPackTest testMethod=test_fileset_file_116_threatintel>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../filebeat/tests/system/test_modules.py:107: in test_fileset_file
    cfgfile=cfgfile)
../../filebeat/tests/system/test_modules.py:193: in run_on_file
    self._test_expected_events(test_file, objects)
../../filebeat/tests/system/test_modules.py:209: in _test_expected_events
    expected = json.load(f)
/usr/lib/python3.7/json/__init__.py:296: in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
/usr/lib/python3.7/json/__init__.py:348: in loads
    return _default_decoder.decode(s)
/usr/lib/python3.7/json/decoder.py:337: in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <json.decoder.JSONDecoder object at 0x7ff1847ef710>
s = '[\n    {\n        "event.category": "threat",\n        "event.dataset": "threatintel.abusemalware",\n        "event.k...reatintel.indicator.first_seen": "2021-01-14T06:04:20.000Z",\n        "threatintel.indicator.type": "file"\n    }\n]\n'
idx = 0

    def raw_decode(self, s, idx=0):
        """Decode a JSON document from ``s`` (a ``str`` beginning with
        a JSON document) and return a 2-tuple of the Python
        representation and the index in ``s`` where the document ended.
    
        This can be used to decode a JSON document from a string that may
        have extraneous data at the end.
    
        """
        try:
            obj, end = self.scan_once(s, idx)
        except StopIteration as err:
>           raise JSONDecodeError("Expecting value", s, err.value) from None
E           json.decoder.JSONDecodeError: Expecting value: line 13 column 1 (char 464)

/usr/lib/python3.7/json/decoder.py:355: JSONDecodeError

Steps errors

Expand to view the steps failures

`x-pack/filebeat-build - mage build test`

Took 37 min 4 sec . View more details on here
Description: mage build test

`x-pack/filebeat-packaging-linux - mage package`

Took 6 min 46 sec . View more details on here
Description: mage package

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-05-27T13:30:40.027Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-05-27T13:30:40.028Z]  OS/Arch:           linux/amd64
[2021-05-27T13:30:40.028Z]  Context:           default
[2021-05-27T13:30:40.028Z]  Experimental:      true
[2021-05-27T13:30:40.028Z] 
[2021-05-27T13:30:40.028Z] Server: Docker Engine - Community
[2021-05-27T13:30:40.028Z]  Engine:
[2021-05-27T13:30:40.028Z]   Version:          20.10.3
[2021-05-27T13:30:40.028Z]   API version:      1.41 (minimum version 1.12)
[2021-05-27T13:30:40.028Z]   Go version:       go1.13.15
[2021-05-27T13:30:40.028Z]   Git commit:       46229ca
[2021-05-27T13:30:40.028Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-05-27T13:30:40.028Z]   OS/Arch:          linux/amd64
[2021-05-27T13:30:40.028Z]   Experimental:     false
[2021-05-27T13:30:40.028Z]  containerd:
[2021-05-27T13:30:40.028Z]   Version:          1.4.4
[2021-05-27T13:30:40.028Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-05-27T13:30:40.028Z]  runc:
[2021-05-27T13:30:40.028Z]   Version:          1.0.0-rc93
[2021-05-27T13:30:40.028Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-05-27T13:30:40.028Z]  docker-init:
[2021-05-27T13:30:40.028Z]   Version:          0.19.0
[2021-05-27T13:30:40.028Z]   GitCommit:        de40ad0
[2021-05-27T13:30:40.028Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-05-27T13:30:40.028Z] Unable to find image 'alpine:3.4' locally
[2021-05-27T13:30:40.600Z] 3.4: Pulling from library/alpine
[2021-05-27T13:30:40.861Z] c1e54eec4b57: Pulling fs layer
[2021-05-27T13:30:41.121Z] c1e54eec4b57: Verifying Checksum
[2021-05-27T13:30:41.121Z] c1e54eec4b57: Download complete
[2021-05-27T13:30:41.383Z] c1e54eec4b57: Pull complete
[2021-05-27T13:30:41.383Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-05-27T13:30:41.383Z] Status: Downloaded newer image for alpine:3.4
[2021-05-27T13:30:43.297Z] Change permissions with write access of all files inside the specific folder
[2021-05-27T13:30:44.722Z] Running in /var/lib/jenkins/workspace/PR-25965-1-1dc030de-b5d0-4c24-9064-15ff3ea0377b/src/github.com/elastic/beats/build
[2021-05-27T13:30:45.054Z] + rm -rf ve
[2021-05-27T13:30:45.054Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-05-27T13:30:45.445Z] + python .ci/scripts/pre_archive_test.py
[2021-05-27T13:30:47.992Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-05-27T13:30:48.025Z] Running in /var/lib/jenkins/workspace/PR-25965-1-1dc030de-b5d0-4c24-9064-15ff3ea0377b/src/github.com/elastic/beats/build
[2021-05-27T13:30:48.072Z] Recording test results
[2021-05-27T13:30:49.998Z] [Checks API] No suitable checks publisher found.
[2021-05-27T13:30:50.335Z] + tar --version
[2021-05-27T13:30:50.700Z] + tar --exclude=test-build-artifacts-x-pack-filebeat-build-tgz -czf test-build-artifacts-x-pack-filebeat-build-tgz .
[2021-05-27T13:31:47.397Z] + gsutil --version
[2021-05-27T13:31:50.753Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-05-27T13:31:51.266Z] + gcloud auth activate-service-account --key-file ****
[2021-05-27T13:31:52.206Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-05-27T13:31:52.602Z] + gsutil -m -q cp -a public-read test-build-artifacts-x-pack-filebeat-build-tgz gs://beats-ci-temp/Beats/beats/PR-25965-1
[2021-05-27T13:31:57.372Z] + python .ci/scripts/search_system_tests.py
[2021-05-27T13:31:57.403Z] [INFO] system-tests='build/x-pack/filebeat/build/system-tests'. If no empty then let's create a tarball
[2021-05-27T13:31:57.763Z] + tar --version
[2021-05-27T13:31:58.135Z] + tar --exclude=x-pack-filebeat--system-tests-linux-tgz -czf x-pack-filebeat--system-tests-linux-tgz build/x-pack/filebeat/build/system-tests
[2021-05-27T13:32:37.316Z] + gsutil --version
[2021-05-27T13:32:38.757Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-05-27T13:32:39.156Z] + gcloud auth activate-service-account --key-file ****
[2021-05-27T13:32:39.730Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-05-27T13:32:40.121Z] + gsutil -m -q cp -a public-read x-pack-filebeat--system-tests-linux-tgz gs://beats-ci-temp/Beats/beats/PR-25965-1
[2021-05-27T13:32:45.966Z] + go clean -modcache
[2021-05-27T13:32:49.607Z] Cleaning up /var/lib/jenkins/workspace/PR-25965-1-1dc030de-b5d0-4c24-9064-15ff3ea0377b
[2021-05-27T13:32:49.607Z] Client: Docker Engine - Community
[2021-05-27T13:32:49.608Z]  Version:           20.10.3
[2021-05-27T13:32:49.608Z]  API version:       1.41
[2021-05-27T13:32:49.608Z]  Go version:        go1.13.15
[2021-05-27T13:32:49.608Z]  Git commit:        48d30b5
[2021-05-27T13:32:49.608Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-05-27T13:32:49.608Z]  OS/Arch:           linux/amd64
[2021-05-27T13:32:49.608Z]  Context:           default
[2021-05-27T13:32:49.608Z]  Experimental:      true
[2021-05-27T13:32:49.608Z] 
[2021-05-27T13:32:49.608Z] Server: Docker Engine - Community
[2021-05-27T13:32:49.608Z]  Engine:
[2021-05-27T13:32:49.608Z]   Version:          20.10.3
[2021-05-27T13:32:49.608Z]   API version:      1.41 (minimum version 1.12)
[2021-05-27T13:32:49.608Z]   Go version:       go1.13.15
[2021-05-27T13:32:49.608Z]   Git commit:       46229ca
[2021-05-27T13:32:49.608Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-05-27T13:32:49.608Z]   OS/Arch:          linux/amd64
[2021-05-27T13:32:49.608Z]   Experimental:     false
[2021-05-27T13:32:49.608Z]  containerd:
[2021-05-27T13:32:49.608Z]   Version:          1.4.4
[2021-05-27T13:32:49.608Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-05-27T13:32:49.608Z]  runc:
[2021-05-27T13:32:49.608Z]   Version:          1.0.0-rc93
[2021-05-27T13:32:49.608Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-05-27T13:32:49.608Z]  docker-init:
[2021-05-27T13:32:49.608Z]   Version:          0.19.0
[2021-05-27T13:32:49.608Z]   GitCommit:        de40ad0
[2021-05-27T13:32:49.608Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-05-27T13:32:50.999Z] Change permissions with write access of all files inside the specific folder
[2021-05-27T13:32:51.309Z] Running in /var/lib/jenkins/workspace/PR-25965-1-1dc030de-b5d0-4c24-9064-15ff3ea0377b
[2021-05-27T13:32:56.821Z] Failed in branch x-pack/filebeat-build
[2021-05-27T13:32:57.451Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-05-27T13:32:57.586Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25965/src/github.com/elastic/beats
[2021-05-27T13:32:58.134Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25965
[2021-05-27T13:32:58.243Z] [INFO] getVaultSecret: Getting secrets
[2021-05-27T13:32:58.303Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-05-27T13:32:59.319Z] + chmod 755 generate-build-data.sh
[2021-05-27T13:32:59.319Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25965/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25965/runs/1 FAILURE 3505253
[2021-05-27T13:32:59.319Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25965/runs/1/steps/?limit=10000 -o steps-info.json
[2021-05-27T13:33:00.231Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25965/runs/1/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	1
Passed	7034
Skipped	1193
Total	8228

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / x-pack/filebeat-build / test_fileset_file_116_threatintel – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest

mergify · 2021-05-28T15:28:00Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/7.13/pr-25674 upstream/mergify/bp/7.13/pr-25674
git merge upstream/7.13
git push upstream mergify/bp/7.13/pr-25674

mergify · 2021-06-09T13:28:06Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/7.13/pr-25674 upstream/mergify/bp/7.13/pr-25674
git merge upstream/7.13
git push upstream mergify/bp/7.13/pr-25674

mergify · 2021-06-15T12:25:24Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/7.13/pr-25674 upstream/mergify/bp/7.13/pr-25674
git merge upstream/7.13
git push upstream mergify/bp/7.13/pr-25674

elasticmachine · 2021-06-24T11:34:10Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

jsoriano · 2021-06-24T11:34:59Z

Hey @P1llus @legoguy1000, is this backport still needed in 7.13? It has some conflicts that would need to be addressed. If not needed, please close it. Thanks!

[Filebeat] Fix bad append for abusechmalware (#25674)

3bdf553

* 25151: Fix bad append for abusechmalware * update changelog (cherry picked from commit fafe6fb) # Conflicts: # x-pack/filebeat/module/threatintel/abusemalware/test/abusechmalware.ndjson.log-expected.json

mergify bot added backport conflicts There is a conflict in the backported pull request labels May 27, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 27, 2021

jsoriano assigned P1llus Jun 24, 2021

jsoriano added the Team:Security-External Integrations label Jun 24, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 24, 2021

P1llus closed this Jun 24, 2021

mergify bot deleted the mergify/bp/7.13/pr-25674 branch June 24, 2021 11:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[7.13](backport #25674) [Filebeat] Fix bad append for abusechmalware #25965

[7.13](backport #25674) [Filebeat] Fix bad append for abusechmalware #25965

mergify bot commented May 27, 2021 •

edited by zube bot

Loading

elasticmachine commented May 27, 2021

Build stats

Test stats 🧪

Trends 🧪

`Build&Test / x-pack/filebeat-build / test_fileset_file_116_threatintel – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest`

`x-pack/filebeat-build - mage build test`

`x-pack/filebeat-packaging-linux - mage package`

`Error signal`

Test stats 🧪

Genuine test errors

mergify bot commented May 28, 2021

mergify bot commented Jun 9, 2021

mergify bot commented Jun 15, 2021

elasticmachine commented Jun 24, 2021

jsoriano commented Jun 24, 2021

[7.13](backport #25674) [Filebeat] Fix bad append for abusechmalware #25965

[7.13](backport #25674) [Filebeat] Fix bad append for abusechmalware #25965

Conversation

mergify bot commented May 27, 2021 • edited by zube bot Loading

elasticmachine commented May 27, 2021

💔 Tests Failed

Build stats

Test stats 🧪

Trends 🧪

Test errors

Build&Test / x-pack/filebeat-build / test_fileset_file_116_threatintel – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest

Steps errors

x-pack/filebeat-build - mage build test

x-pack/filebeat-packaging-linux - mage package

Error signal

Log output

🐛 Flaky test report

Test stats 🧪

Genuine test errors

mergify bot commented May 28, 2021

mergify bot commented Jun 9, 2021

mergify bot commented Jun 15, 2021

elasticmachine commented Jun 24, 2021

jsoriano commented Jun 24, 2021

mergify bot commented May 27, 2021 •

edited by zube bot

Loading

`Build&Test / x-pack/filebeat-build / test_fileset_file_116_threatintel – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest`

`x-pack/filebeat-build - mage build test`

`x-pack/filebeat-packaging-linux - mage package`

`Error signal`