Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7.x] Introduce approvaltest module (#4112) #4162

Merged
merged 2 commits into from
Sep 8, 2020

Conversation

axw
Copy link
Member

@axw axw commented Sep 7, 2020

Backports the following commits to 7.x:

* Introduce new approvaltest module

We move tests/approvals and tests/scripts/approvals.go intoa new module,
github.com/elastic/apm-server/approvaltest. This new module will be used
by both apm-server (integration tests) and the systemtest module.

We also reduce the API surface area to two functions: ApproveJSON, and
ApproveEventDocs. The ApproveJSON function can be used for approving
arbitrary JSON-encoded values, while ApproveEventDocs is specifically
intended for approving JSON-encoded Elasticsearch event documents.

To enable the API reduction, we introduce beatertest.EncodeEventDoc(s),
which encode events using the libbeat Elasticsearch output. In some
tests we previously used the libbeat JSON codec directly, which led to
@metadata being included in diffs. The output removes @metadata, hence
it no longer features in our approval diffs.

* Update approved files

- Since we now use the Elasticsearch output, rather than directly
  using the libbeat JSON codec, `@metadata` is no longer part of the
  diff, and `@timestamp` is encoded slightly differently.

- For dynamic fields like `profile.id`, we now replace the value
  with a known string ("dynamic") rather than ignoring the field.

* .ci/docker/golang-mage: support multiple modules
@axw axw added the backport label Sep 7, 2020
@apmmachine
Copy link
Contributor

apmmachine commented Sep 7, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #4162 updated]

  • Start Time: 2020-09-07T22:57:35.374+0000

  • Duration: 34 min 33 sec

Test stats 🧪

Test Results
Failed 1
Passed 3032
Skipped 139
Total 3172

Test errors

Expand to view the tests failures

  • Name: Build and Test / OSX build-test / TestAggregatorRun – spanmetrics

    • Age: 1
    • Duration: 0.02
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: Compress

    • Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage

    • Duration: 0 min 0 sec

    • Start Time: 2020-09-07T23:15:01.348+0000

    • log

  • Name: Download Codecov

    • Description: #!/bin/bash set -x curl -sSLo codecov.sh https://codecov.io/bash

    • Duration: 2 min 22 sec

    • Start Time: 2020-09-07T23:15:03.551+0000

    • log

  • Name: Compress

    • Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests

    • Duration: 0 min 0 sec

    • Start Time: 2020-09-07T23:31:06.356+0000

    • log

  • Name: Test Sync

    • Description: ./script/jenkins/sync.sh

    • Duration: 3 min 54 sec

    • Start Time: 2020-09-07T23:11:43.043+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-09-07T23:27:41.264Z] go: downloading github.com/opencontainers/go-digest v1.0.0
[2020-09-07T23:27:41.264Z] go: downloading github.com/containerd/containerd v1.3.6
[2020-09-07T23:27:41.264Z] go: downloading golang.org/x/sys v0.0.0-20200217220822-9197077df867
[2020-09-07T23:27:41.264Z] go: downloading github.com/mitchellh/mapstructure v1.1.2
[2020-09-07T23:27:41.264Z] go: downloading golang.org/x/net v0.0.0-20200202094626-16171245cfb2
[2020-09-07T23:27:41.264Z] go: downloading github.com/stretchr/testify v1.6.1
[2020-09-07T23:27:41.264Z] go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
[2020-09-07T23:27:41.264Z] go: downloading golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e
[2020-09-07T23:27:41.264Z] go: downloading github.com/pmezard/go-difflib v1.0.0
[2020-09-07T23:27:41.264Z] go: downloading github.com/elastic/go-sysinfo v1.1.1
[2020-09-07T23:27:41.264Z] go: downloading github.com/jaegertracing/jaeger v1.18.1
[2020-09-07T23:27:41.264Z] go: downloading github.com/gogo/protobuf v1.2.1
[2020-09-07T23:27:41.264Z] go: downloading github.com/tidwall/match v1.0.1
[2020-09-07T23:27:41.529Z] go: downloading github.com/prometheus/procfs v0.0.10
[2020-09-07T23:27:41.529Z] go: downloading howett.net/plist v0.0.0-20181124034731-591f970eefbb
[2020-09-07T23:27:41.529Z] go: downloading github.com/tidwall/pretty v1.0.1
[2020-09-07T23:27:41.529Z] go: downloading github.com/opencontainers/image-spec v1.0.1
[2020-09-07T23:27:41.793Z] go: downloading google.golang.org/genproto v0.0.0-20200218151345-dad8c97a84f5
[2020-09-07T23:27:41.793Z] go: downloading github.com/golang/protobuf v1.3.4
[2020-09-07T23:27:41.793Z] go: downloading golang.org/x/text v0.3.2
[2020-09-07T23:27:42.055Z] go: downloading github.com/gogo/googleapis v1.0.1-0.20180501115203-b23578765ee5
[2020-09-07T23:27:42.055Z] go: downloading github.com/opentracing/opentracing-go v1.1.0
[2020-09-07T23:31:03.833Z] apm-server_elasticsearch_1 is up-to-date
[2020-09-07T23:31:03.833Z] apm-server_kibana_1 is up-to-date
[2020-09-07T23:31:03.833Z] === RUN   TestAPIKeyCreate
[2020-09-07T23:31:03.833Z] 2020/09/07 23:28:15 Building apm-server...
[2020-09-07T23:31:03.833Z] 2020/09/07 23:30:35 Built /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-4162/src/github.com/elastic/apm-server/apm-server
[2020-09-07T23:31:03.833Z] --- PASS: TestAPIKeyCreate (142.72s)
[2020-09-07T23:31:03.833Z] === RUN   TestAPIKeyCreateExpiration
[2020-09-07T23:31:03.833Z] --- PASS: TestAPIKeyCreateExpiration (2.02s)
[2020-09-07T23:31:03.833Z] === RUN   TestAPIKeyInvalidateName
[2020-09-07T23:31:03.833Z] --- PASS: TestAPIKeyInvalidateName (3.06s)
[2020-09-07T23:31:03.833Z] === RUN   TestAPIKeyInvalidateID
[2020-09-07T23:31:03.833Z] --- PASS: TestAPIKeyInvalidateID (2.02s)
[2020-09-07T23:31:03.833Z] === RUN   TestAPMServerEnvironment
[2020-09-07T23:31:03.833Z] === RUN   TestAPMServerEnvironment/container
[2020-09-07T23:31:03.833Z] === PAUSE TestAPMServerEnvironment/container
[2020-09-07T23:31:03.833Z] === RUN   TestAPMServerEnvironment/systemd
[2020-09-07T23:31:03.833Z] === PAUSE TestAPMServerEnvironment/systemd
[2020-09-07T23:31:03.833Z] === RUN   TestAPMServerEnvironment/macos_service
[2020-09-07T23:31:03.833Z] === PAUSE TestAPMServerEnvironment/macos_service
[2020-09-07T23:31:03.833Z] === RUN   TestAPMServerEnvironment/windows_service
[2020-09-07T23:31:03.833Z] === PAUSE TestAPMServerEnvironment/windows_service
[2020-09-07T23:31:03.833Z] === CONT  TestAPMServerEnvironment/container
[2020-09-07T23:31:03.833Z] === CONT  TestAPMServerEnvironment/macos_service
[2020-09-07T23:31:03.833Z] === CONT  TestAPMServerEnvironment/windows_service
[2020-09-07T23:31:03.833Z] === CONT  TestAPMServerEnvironment/systemd
[2020-09-07T23:31:03.833Z] --- PASS: TestAPMServerEnvironment (0.00s)
[2020-09-07T23:31:03.833Z]     --- PASS: TestAPMServerEnvironment/systemd (0.46s)
[2020-09-07T23:31:03.833Z]     --- PASS: TestAPMServerEnvironment/macos_service (0.47s)
[2020-09-07T23:31:03.833Z]     --- PASS: TestAPMServerEnvironment/windows_service (0.49s)
[2020-09-07T23:31:03.833Z]     --- PASS: TestAPMServerEnvironment/container (0.50s)
[2020-09-07T23:31:03.833Z] === RUN   TestAPMServerInstrumentation
[2020-09-07T23:31:03.833Z] --- PASS: TestAPMServerInstrumentation (2.96s)
[2020-09-07T23:31:03.833Z] === RUN   TestJaegerGRPC
[2020-09-07T23:31:03.833Z] --- PASS: TestJaegerGRPC (2.75s)
[2020-09-07T23:31:03.833Z] === RUN   TestJaegerGRPCSampling
[2020-09-07T23:31:03.833Z] --- PASS: TestJaegerGRPCSampling (2.27s)
[2020-09-07T23:31:03.833Z] === RUN   TestAPMServerRequestLoggingValid
[2020-09-07T23:31:03.833Z] --- PASS: TestAPMServerRequestLoggingValid (0.31s)
[2020-09-07T23:31:03.833Z] === RUN   TestAPMServerMonitoring
[2020-09-07T23:31:03.833Z] --- PASS: TestAPMServerMonitoring (2.30s)
[2020-09-07T23:31:03.833Z] === RUN   TestAPMServerMonitoringBuiltinUser
[2020-09-07T23:31:03.833Z] --- PASS: TestAPMServerMonitoringBuiltinUser (1.92s)
[2020-09-07T23:31:03.833Z] === RUN   TestAPMServerOnboarding
[2020-09-07T23:31:03.833Z] --- PASS: TestAPMServerOnboarding (2.79s)
[2020-09-07T23:31:03.833Z] PASS
[2020-09-07T23:31:03.833Z] ok  	github.com/elastic/apm-server/systemtest	166.968s
[2020-09-07T23:31:03.833Z] === RUN   TestAPMServer
[2020-09-07T23:31:03.833Z] 2020/09/07 23:28:12 Building apm-server...
[2020-09-07T23:31:03.833Z] 2020/09/07 23:30:35 Built /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-4162/src/github.com/elastic/apm-server/apm-server
[2020-09-07T23:31:03.833Z] --- PASS: TestAPMServer (145.22s)
[2020-09-07T23:31:03.833Z] === RUN   TestUnstartedAPMServer
[2020-09-07T23:31:03.833Z] --- PASS: TestUnstartedAPMServer (0.00s)
[2020-09-07T23:31:03.833Z] === RUN   TestExpvar
[2020-09-07T23:31:03.833Z] --- PASS: TestExpvar (0.31s)
[2020-09-07T23:31:03.833Z] PASS
[2020-09-07T23:31:03.833Z] ok  	github.com/elastic/apm-server/systemtest/apmservertest	145.543s
[2020-09-07T23:31:03.833Z] ?   	github.com/elastic/apm-server/systemtest/estest	[no test files]
[2020-09-07T23:31:03.833Z] + cleanup
[2020-09-07T23:31:03.833Z] + rm -rf /tmp/tmp.4f9YmR2MZA
[2020-09-07T23:31:03.833Z] + .ci/scripts/docker-get-logs.sh
[2020-09-07T23:31:04.925Z] Post stage
[2020-09-07T23:31:04.936Z] Running in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-4162/src/github.com/elastic/apm-server
[2020-09-07T23:31:04.959Z] Archiving artifacts
[2020-09-07T23:31:05.393Z] Recording test results
[2020-09-07T23:31:05.997Z] [WARN] tar: pathPrefix parameter is deprecated.
[2020-09-07T23:31:06.325Z] + tar --version
[2020-09-07T23:31:06.645Z] + tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests
[2020-09-07T23:31:06.645Z] tar: system-tests: Cannot stat: No such file or directory
[2020-09-07T23:31:06.645Z] tar: Exiting with failure status due to previous errors
[2020-09-07T23:31:06.659Z] [INFO] system-tests-linux-files.tgz was not compressed or archived : script returned exit code 2
[2020-09-07T23:31:06.862Z] Stage "Release" skipped due to when conditional
[2020-09-07T23:31:07.721Z] Running on Jenkins in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-4162
[2020-09-07T23:31:07.777Z] [INFO] getVaultSecret: Getting secrets
[2020-09-07T23:31:07.854Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-07T23:31:08.693Z] + chmod 755 generate-build-data.sh
[2020-09-07T23:31:08.693Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4162/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4162/runs/2 UNSTABLE 2013059
[2020-09-07T23:31:11.176Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4162/runs/2/steps/?limit=10000 -o steps-info.json
[2020-09-07T23:31:11.427Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4162/runs/2/tests/?status=FAILED -o tests-errors.json

Copy in approvaltest/go.mod (and systemtest/go.mod for good measure)
to fix `go mod download` at the top level. Also run `go mod download`
for the sub-modules, to speed up repeat runs of docker-system-tests.
@axw axw merged commit 5e2b7e8 into elastic:7.x Sep 8, 2020
@axw axw deleted the backport/7.x/pr-4112 branch September 8, 2020 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants