Cherry-pick #24425 to 7.x: Add tests for truncated and symlinked files #25118

kvch · 2021-04-15T17:33:03Z

Cherry-pick of PR #24425 to 7.x branch. Original message:

What does this PR do?

Add support for better support truncated files with symlinks and adds migrates related tests from test_harvester.py.

Why is it important?

Increase coverage.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~- [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

…tic#24425) (cherry picked from commit a9279cd)

elasticmachine · 2021-04-15T17:33:06Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-04-15T18:17:21Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25118 opened
Start Time: 2021-04-15T17:33:52.968+0000
Duration: 43 min 13 sec
Commit: 92b81be

Test stats 🧪

Test	Results
Failed	0
Passed	9546
Skipped	1972
Total	11518

Trends 🧪

Steps errors

Expand to view the steps failures

Show only the first 10 steps failures

`Docker login`

Took 0 min 1 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 1 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 1 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 1 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 1 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 9 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`untar gsutil`

Took 0 min 0 sec . View more details on here
Description: tar -xpf gsutil.tar.gz --strip-components=1

`Docker login`

Took 0 min 2 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 2 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 9 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

Log output

Expand to view the last 100 lines of log output

[2021-04-15T18:11:09.724Z] C:\Users\jenkins\workspace\PR-25118-1-c043c572-ce7f-40c7-8ae2-5d46e04137b9\src\github.com\elastic\beats>go clean -modcache 
[2021-04-15T18:11:44.974Z] warn: failed to upgrade pip (ignoring): running "null\build\ve\windows\Scripts\pip install -U pip" failed with exit code 1============================= test session starts =============================
[2021-04-15T18:11:44.974Z] platform win32 -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
[2021-04-15T18:11:44.974Z] rootdir: C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats, configfile: pytest.ini
[2021-04-15T18:11:44.974Z] plugins: rerunfailures-9.0, timeout-1.3.4
[2021-04-15T18:11:44.974Z] timeout: 90.0s
[2021-04-15T18:11:44.974Z] timeout method: thread
[2021-04-15T18:11:44.974Z] timeout func_only: True
[2021-04-15T18:11:44.974Z] collected 348 items
[2021-04-15T18:11:44.974Z] 
[2021-04-15T18:11:44.974Z] tests\system\test_autodiscover.py ss                                     [  0%]
[2021-04-15T18:11:44.974Z] tests\system\test_base.py ......ss                                       [  2%]
[2021-04-15T18:11:44.974Z] tests\system\test_cmd.py ...                                             [  3%]
[2021-04-15T18:11:46.786Z] tests\system\test_container.py ...                                       [  4%]
[2021-04-15T18:12:08.834Z] tests\system\test_crawler.py ......s..........                           [  9%]
[2021-04-15T18:12:09.714Z] tests\system\test_deprecated.py ..                                       [ 10%]
[2021-04-15T18:12:11.485Z] tests\system\test_fields.py ...                                          [ 10%]
[2021-04-15T18:12:13.330Z] tests\system\test_generate.py ...                                        [ 11%]
[2021-04-15T18:12:33.195Z] + gsutil --version
[2021-04-15T18:12:34.635Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-15T18:12:35.163Z] + gcloud auth activate-service-account --key-file ****
[2021-04-15T18:12:35.755Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-15T18:12:36.143Z] + gsutil -m -q cp -a public-read eC1wYWNrL2ZpbGViZWF0LXdpbmRvd3MtMTAtd2luZG93cy0xMDkyYjgxYmVlOTFjNTQ3Nzg3Y2E3Nzc0NmEzY2VjODNmNjhiNGY0Mzc gs://beats-ci-temp/ci/cache/
[2021-04-15T18:12:43.402Z] tests\system\test_harvester.py ......s..............                     [ 17%]
[2021-04-15T18:12:44.581Z] tests\system\test_index_pattern.py ..                                    [ 18%]
[2021-04-15T18:13:11.374Z] tests\system\test_input.py .........ss...........                        [ 24%]
[2021-04-15T18:13:19.030Z] tests\system\test_json.py ..............                                 [ 28%]
[2021-04-15T18:13:20.251Z] tests\system\test_keystore.py ..                                         [ 29%]
[2021-04-15T18:13:20.251Z] tests\system\test_load.py sss                                            [ 30%]
[2021-04-15T18:13:20.251Z] tests\system\test_ml.py ss                                               [ 30%]
[2021-04-15T18:13:20.251Z] tests\system\test_modules.py sssssssssssssssssssssssssssssssssssssssssss [ 43%]
[2021-04-15T18:13:20.524Z] ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 63%]
[2021-04-15T18:13:20.784Z] ssssssssssssssssssssssssss                                               [ 71%]
[2021-04-15T18:13:37.837Z] tests\system\test_multiline.py .........                                 [ 73%]
[2021-04-15T18:13:37.837Z] tests\system\test_pipeline.py s                                          [ 74%]
[2021-04-15T18:13:44.999Z] tests\system\test_processors.py .............                            [ 77%]
[2021-04-15T18:13:45.949Z] tests\system\test_publisher.py .                                         [ 78%]
[2021-04-15T18:13:45.949Z] tests\system\test_redis.py s                                             [ 78%]
[2021-04-15T18:14:15.110Z] tests\system\test_registrar.py sss........ssss......ss..                 [ 85%]
[2021-04-15T18:14:15.110Z] tests\system\test_registrar_upgrade.py sssss                             [ 87%]
[2021-04-15T18:14:37.333Z] tests\system\test_reload_inputs.py .......                               [ 89%]
[2021-04-15T18:14:46.327Z] tests\system\test_reload_modules.py ...s..                               [ 90%]
[2021-04-15T18:14:46.327Z] tests\system\test_setup.py s                                             [ 91%]
[2021-04-15T18:14:57.614Z] tests\system\test_shutdown.py ..s..                                      [ 92%]
[2021-04-15T18:14:58.838Z] tests\system\test_stdin.py ...                                           [ 93%]
[2021-04-15T18:15:00.588Z] tests\system\test_syslog.py ...ssss                                      [ 95%]
[2021-04-15T18:15:03.544Z] tests\system\test_tcp.py .....                                           [ 96%]
[2021-04-15T18:15:06.484Z] tests\system\test_tcp_tls.py ......                                      [ 98%]
[2021-04-15T18:15:06.484Z] tests\system\test_udp.py s                                               [ 98%]
[2021-04-15T18:15:06.484Z] tests\system\test_unix.py ssss                                           [100%]
[2021-04-15T18:15:06.484Z] 
[2021-04-15T18:15:06.484Z] - generated xml file: C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats\filebeat\build\TEST-python-unit.xml -
[2021-04-15T18:15:06.484Z] ============================ slowest 20 durations =============================
[2021-04-15T18:15:06.484Z] 15.85s call     filebeat/tests/system/test_harvester.py::Test::test_close_timeout
[2021-04-15T18:15:06.484Z] 6.08s call     filebeat/tests/system/test_registrar.py::Test::test_restart_state
[2021-04-15T18:15:06.484Z] 5.27s call     filebeat/tests/system/test_shutdown.py::Test::test_shutdown_wait_timeout
[2021-04-15T18:15:06.484Z] 4.60s call     filebeat/tests/system/test_crawler.py::Test::test_encodings
[2021-04-15T18:15:06.484Z] 4.56s call     filebeat/tests/system/test_reload_inputs.py::Test::test_reload_add
[2021-04-15T18:15:06.484Z] 4.37s call     filebeat/tests/system/test_crawler.py::Test::test_multiple_appends
[2021-04-15T18:15:06.484Z] 3.85s call     filebeat/tests/system/test_input.py::Test::test_rotating_close_inactive_low_write_rate
[2021-04-15T18:15:06.484Z] 3.78s call     filebeat/tests/system/test_input.py::Test::test_close_inactive_file_rotation_and_removal2
[2021-04-15T18:15:06.484Z] 3.75s call     filebeat/tests/system/test_multiline.py::Test::test_close_timeout_with_multiline
[2021-04-15T18:15:06.484Z] 3.68s call     filebeat/tests/system/test_input.py::Test::test_harvester_limit
[2021-04-15T18:15:06.484Z] 3.63s call     filebeat/tests/system/test_reload_inputs.py::Test::test_reload_same_config
[2021-04-15T18:15:06.484Z] 3.58s call     filebeat/tests/system/test_reload_inputs.py::Test::test_start_stop_replace
[2021-04-15T18:15:06.484Z] 3.57s call     filebeat/tests/system/test_reload_inputs.py::Test::test_reload_same_input
[2021-04-15T18:15:06.484Z] 3.54s call     filebeat/tests/system/test_registrar.py::Test::test_state_after_rotation_ignore_older
[2021-04-15T18:15:06.484Z] 3.52s call     filebeat/tests/system/test_registrar.py::Test::test_state_after_rotation
[2021-04-15T18:15:06.484Z] 3.49s call     filebeat/tests/system/test_reload_modules.py::Test::test_start_stop
[2021-04-15T18:15:06.484Z] 3.48s call     filebeat/tests/system/test_reload_inputs.py::Test::test_start_stop
[2021-04-15T18:15:06.484Z] 3.28s call     filebeat/tests/system/test_shutdown.py::Test::test_shutdown
[2021-04-15T18:15:06.484Z] 2.88s call     filebeat/tests/system/test_multiline.py::Test::test_timeout
[2021-04-15T18:15:06.484Z] 2.50s call     filebeat/tests/system/test_registrar.py::Test::test_ignore_older_state_clean_inactive
[2021-04-15T18:15:06.484Z] ================ 166 passed, 182 skipped in 209.57s (0:03:29) =================
[2021-04-15T18:15:06.745Z] >> python test: Unit Testing Complete
[2021-04-15T18:15:07.277Z] 
[2021-04-15T18:15:07.277Z] C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-04-15T18:15:10.183Z] 
[2021-04-15T18:15:10.183Z] C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-04-15T18:15:15.469Z] Copy .\filebeat\build into build\filebeat\build
[2021-04-15T18:15:15.469Z] Copy .\filebeat\null\build into build\filebeat\null\build
[2021-04-15T18:15:15.511Z] Running in C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats\build
[2021-04-15T18:15:15.581Z] Recording test results
[2021-04-15T18:15:17.234Z] [Checks API] No suitable checks publisher found.
[2021-04-15T18:15:17.787Z] 
[2021-04-15T18:15:17.787Z] C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats>go clean -modcache 
[2021-04-15T18:16:56.602Z] + gsutil --version
[2021-04-15T18:16:58.126Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-15T18:16:58.498Z] + gcloud auth activate-service-account --key-file ****
[2021-04-15T18:16:59.078Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-15T18:16:59.682Z] + gsutil -m -q cp -a public-read ZmlsZWJlYXQtd2luZG93cy0xMC13aW5kb3dzLTEwOTJiODFiZWU5MWM1NDc3ODdjYTc3NzQ2YTNjZWM4M2Y2OGI0ZjQzNw gs://beats-ci-temp/ci/cache/
[2021-04-15T18:17:02.738Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-15T18:17:02.899Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25118/src/github.com/elastic/beats
[2021-04-15T18:17:04.442Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25118
[2021-04-15T18:17:04.550Z] [INFO] getVaultSecret: Getting secrets
[2021-04-15T18:17:04.696Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-15T18:17:05.860Z] + chmod 755 generate-build-data.sh
[2021-04-15T18:17:05.861Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25118/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25118/runs/1 FAILURE 2592603
[2021-04-15T18:17:06.411Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25118/runs/1/steps/?limit=10000 -o steps-info.json
[2021-04-15T18:17:07.755Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25118/runs/1/tests/?status=FAILED -o tests-errors.json

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	9546
Skipped	1972
Total	11518

Add tests for truncated and symlinked files in filestream input (elas…

92b81be

…tic#24425) (cherry picked from commit a9279cd)

kvch added [zube]: In Review backport Team:Elastic-Agent Label for the Agent team labels Apr 15, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 15, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 15, 2021

kvch merged commit e748e45 into elastic:7.x Apr 16, 2021

zube bot added [zube]: Done and removed [zube]: In Review labels Apr 16, 2021

zube bot removed the [zube]: Done label Jul 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick #24425 to 7.x: Add tests for truncated and symlinked files #25118

Cherry-pick #24425 to 7.x: Add tests for truncated and symlinked files #25118

kvch commented Apr 15, 2021 •

edited by zube bot

Loading

elasticmachine commented Apr 15, 2021

elasticmachine commented Apr 15, 2021

Build stats

Test stats 🧪

Trends 🧪

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`untar gsutil`

`Docker login`

`Docker login`

`Docker login`

Test stats 🧪

Cherry-pick #24425 to 7.x: Add tests for truncated and symlinked files #25118

Cherry-pick #24425 to 7.x: Add tests for truncated and symlinked files #25118

Conversation

kvch commented Apr 15, 2021 • edited by zube bot Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Apr 15, 2021

elasticmachine commented Apr 15, 2021

💔 Build Failed

Build stats

Test stats 🧪

Trends 🧪

Steps errors

Docker login

Docker login

Docker login

Docker login

Docker login

Docker login

untar gsutil

Docker login

Docker login

Docker login

Log output

💚 Flaky test report

Test stats 🧪

kvch commented Apr 15, 2021 •

edited by zube bot

Loading

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`untar gsutil`

`Docker login`

`Docker login`

`Docker login`