Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: fix TestNewReceiver race conditions #42084

Merged
merged 1 commit into from
Dec 17, 2024
Merged

test: fix TestNewReceiver race conditions #42084

merged 1 commit into from
Dec 17, 2024

Conversation

kruskall
Copy link
Member

Proposed commit message

update zapcore to use observer zaptest package to avoid race condition on zapLogs bytes.Buffer

use atomic for countLogs to avoid race condition on primitive type

bytes.Buffer race condition:

=== RUN   TestNewReceiver
==================
WARNING: DATA RACE
Write at 0x00c0007bdf70 by goroutine 58:
  bytes.(*Buffer).Write()
      bytes/buffer.go:176 +0x3d
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Debug()
      go.uber.org/zap@v1.27.0/sugar.go:149 +0xc4
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:108 +0x1d
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*queueReader).run()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/queue_reader.go:49 +0x3a
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.gowrap1()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:102 +0x44

Previous write at 0x00c0007bdf70 by goroutine 57:
  bytes.(*Buffer).Write()
      bytes/buffer.go:176 +0x3d
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Debug()
      go.uber.org/zap@v1.27.0/sugar.go:149 +0xe4
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:108 +0x8c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).run()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:110 +0x58
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.func1()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:91 +0x84

Goroutine 58 (running) created at:
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:102 +0x3e4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newOutputController()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:98 +0x9c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.New()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/pipeline.go:149 +0x5a4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:100 +0x396
  github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver()
      github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:504 +0x356f
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.createReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/factory.go:36 +0x1e4
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:65 +0xc44
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44

Goroutine 57 (running) created at:
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:89 +0x31c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newOutputController()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:98 +0x9c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.New()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/pipeline.go:149 +0x5a4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:100 +0x396
  github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver()
      github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:504 +0x356f
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.createReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/factory.go:36 +0x1e4
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:65 +0xc44
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44
==================
==================
WARNING: DATA RACE
Read at 0x00c0007bdf50 by goroutine 58:
  bytes.(*Buffer).tryGrowByReslice()
      bytes/buffer.go:116 +0x4b
  bytes.(*Buffer).Write()
      bytes/buffer.go:177 +0xbc
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Debug()
      go.uber.org/zap@v1.27.0/sugar.go:149 +0xc4
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:108 +0x1d
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*queueReader).run()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/queue_reader.go:49 +0x3a
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.gowrap1()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:102 +0x44

Previous write at 0x00c0007bdf50 by goroutine 57:
  bytes.(*Buffer).grow()
      bytes/buffer.go:151 +0x2d3
  bytes.(*Buffer).Write()
      bytes/buffer.go:179 +0xc4
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Debug()
      go.uber.org/zap@v1.27.0/sugar.go:149 +0xe4
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:108 +0x8c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).run()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:110 +0x58
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.func1()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:91 +0x84

Goroutine 58 (running) created at:
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:102 +0x3e4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newOutputController()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:98 +0x9c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.New()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/pipeline.go:149 +0x5a4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:100 +0x396
  github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver()
      github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:504 +0x356f
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.createReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/factory.go:36 +0x1e4
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:65 +0xc44
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44

Goroutine 57 (running) created at:
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:89 +0x31c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newOutputController()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:98 +0x9c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.New()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/pipeline.go:149 +0x5a4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:100 +0x396
  github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver()
      github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:504 +0x356f
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.createReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/factory.go:36 +0x1e4
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:65 +0xc44
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44
==================
==================
WARNING: DATA RACE
Write at 0x00c00018b200 by goroutine 58:
  runtime.slicecopy()
      runtime/slice.go:355 +0x0
  bytes.(*Buffer).Write()
      bytes/buffer.go:181 +0x118
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Debug()
      go.uber.org/zap@v1.27.0/sugar.go:149 +0xc4
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:108 +0x1d
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*queueReader).run()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/queue_reader.go:49 +0x3a
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.gowrap1()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:102 +0x44

Previous write at 0x00c00018b200 by goroutine 57:
  runtime.slicecopy()
      runtime/slice.go:355 +0x0
  bytes.growSlice()
      bytes/buffer.go:249 +0x119
  bytes.(*Buffer).grow()
      bytes/buffer.go:151 +0x2b7
  bytes.(*Buffer).Write()
      bytes/buffer.go:179 +0xc4
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Debug()
      go.uber.org/zap@v1.27.0/sugar.go:149 +0xe4
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:108 +0x8c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).run()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:110 +0x58
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.func1()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:91 +0x84

Goroutine 58 (running) created at:
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:102 +0x3e4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newOutputController()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:98 +0x9c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.New()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/pipeline.go:149 +0x5a4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:100 +0x396
  github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver()
      github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:504 +0x356f
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.createReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/factory.go:36 +0x1e4
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:65 +0xc44
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44

Goroutine 57 (running) created at:
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:89 +0x31c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newOutputController()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:98 +0x9c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.New()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/pipeline.go:149 +0x5a4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:100 +0x396
  github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver()
      github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:504 +0x356f
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.createReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/factory.go:36 +0x1e4
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:65 +0xc44
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44
==================
==================
WARNING: DATA RACE
Read at 0x00c0007bdf50 by goroutine 28:
  bytes.(*Buffer).tryGrowByReslice()
      bytes/buffer.go:116 +0x4b
  bytes.(*Buffer).Write()
      bytes/buffer.go:177 +0xbc
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Infof()
      go.uber.org/zap@v1.27.0/sugar.go:204 +0x479
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Infof()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:156 +0x426
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:105 +0xc7
  github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver()
      github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:504 +0x356f
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.createReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/factory.go:36 +0x1e4
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:65 +0xc44
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44

Previous write at 0x00c0007bdf50 by goroutine 58:
  bytes.(*Buffer).tryGrowByReslice()
      bytes/buffer.go:117 +0xa8
  bytes.(*Buffer).Write()
      bytes/buffer.go:177 +0xbc
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Debug()
      go.uber.org/zap@v1.27.0/sugar.go:149 +0xc4
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:108 +0x1d
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*queueReader).run()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/queue_reader.go:49 +0x3a
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.gowrap1()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:102 +0x44

Goroutine 28 (running) created at:
  testing.(*T).Run()
      testing/testing.go:1743 +0x825
  testing.runTests.func1()
      testing/testing.go:2168 +0x85
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.runTests()
      testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      testing/testing.go:2034 +0xf17
  main.main()
      _testmain.go:47 +0x164

Goroutine 58 (running) created at:
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:102 +0x3e4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newOutputController()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:98 +0x9c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.New()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/pipeline.go:149 +0x5a4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:100 +0x396
  github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver()
      github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:504 +0x356f
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.createReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/factory.go:36 +0x1e4
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:65 +0xc44
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44
==================
==================
WARNING: DATA RACE
Write at 0x00c0007bdf70 by goroutine 28:
  bytes.(*Buffer).Write()
      bytes/buffer.go:176 +0x3d
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*Logger).Info()
      go.uber.org/zap@v1.27.0/logger.go:247 +0x64
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.(*metricbeatReceiver).Shutdown()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver.go:34 +0x58
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:91 +0x1308
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44

Previous write at 0x00c0007bdf70 by goroutine 60:
  bytes.(*Buffer).Write()
      bytes/buffer.go:176 +0x3d
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Debug()
      go.uber.org/zap@v1.27.0/sugar.go:149 +0x3f3
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:108 +0x3ab
  github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).handleBatchSig()
      github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/ackloop.go:82 +0x333
  github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).run()
      github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/ackloop.go:54 +0x3d6
  github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func2()
      github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/broker.go:172 +0x9c

Goroutine 28 (running) created at:
  testing.(*T).Run()
      testing/testing.go:1743 +0x825
  testing.runTests.func1()
      testing/testing.go:2168 +0x85
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.runTests()
      testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      testing/testing.go:2034 +0xf17
  main.main()
      _testmain.go:47 +0x164

Goroutine 60 (running) created at:
  github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue()
      github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/broker.go:170 +0x17c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.queueFactoryForUserConfig.FactoryForSettings.func1()
      github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/broker.go:148 +0x8e
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).createQueueIfNeeded()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:275 +0x292
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).Set()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:125 +0xee
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.New()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/pipeline.go:154 +0x6b5
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:100 +0x396
  github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver()
      github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:504 +0x356f
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.createReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/factory.go:36 +0x1e4
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:65 +0xc44
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44
==================
==================
WARNING: DATA RACE
Read at 0x00c0007bdf50 by goroutine 28:
  bytes.(*Buffer).tryGrowByReslice()
      bytes/buffer.go:116 +0x4b
  bytes.(*Buffer).Write()
      bytes/buffer.go:177 +0xbc
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*Logger).Info()
      go.uber.org/zap@v1.27.0/logger.go:247 +0x64
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.(*metricbeatReceiver).Shutdown()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver.go:34 +0x58
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:91 +0x1308
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44

Previous write at 0x00c0007bdf50 by goroutine 60:
  bytes.(*Buffer).tryGrowByReslice()
      bytes/buffer.go:117 +0xa8
  bytes.(*Buffer).Write()
      bytes/buffer.go:177 +0xbc
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Debug()
      go.uber.org/zap@v1.27.0/sugar.go:149 +0x3f3
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:108 +0x3ab
  github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).handleBatchSig()
      github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/ackloop.go:82 +0x333
  github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).run()
      github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/ackloop.go:54 +0x3d6
  github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func2()
      github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/broker.go:172 +0x9c

Goroutine 28 (running) created at:
  testing.(*T).Run()
      testing/testing.go:1743 +0x825
  testing.runTests.func1()
      testing/testing.go:2168 +0x85
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.runTests()
      testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      testing/testing.go:2034 +0xf17
  main.main()
      _testmain.go:47 +0x164

Goroutine 60 (running) created at:
  github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue()
      github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/broker.go:170 +0x17c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.queueFactoryForUserConfig.FactoryForSettings.func1()
      github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/broker.go:148 +0x8e
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).createQueueIfNeeded()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:275 +0x292
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).Set()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:125 +0xee
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.New()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/pipeline.go:154 +0x6b5
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:100 +0x396
  github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver()
      github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:504 +0x356f
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.createReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/factory.go:36 +0x1e4
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:65 +0xc44
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44
==================
==================
WARNING: DATA RACE
Write at 0x00c0007bdf70 by goroutine 63:
  bytes.(*Buffer).Write()
      bytes/buffer.go:176 +0x3d
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*Logger).Debug()
      go.uber.org/zap@v1.27.0/logger.go:239 +0x67
  github.com/elastic/beats/v7/metricbeat/mb/module.init.MakeDebug.func1()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/global.go:32 +0x94
  github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).run.deferwrap2()
      github.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:218 +0xb5
  runtime.deferreturn()
      runtime/panic.go:605 +0x5d
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1()
      github.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:175 +0x535
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.gowrap1()
      github.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:176 +0x41

Previous write at 0x00c0007bdf70 by goroutine 66:
  bytes.(*Buffer).Write()
      bytes/buffer.go:176 +0x3d
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Debug()
      go.uber.org/zap@v1.27.0/sugar.go:149 +0x41d
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:108 +0x3cc
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Close()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/client.go:147 +0x378
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Stop.func1()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner.go:95 +0x61
  sync.(*Once).doSlow()
      sync/once.go:76 +0xe1
  sync.(*Once).Do()
      sync/once.go:67 +0x44
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Stop()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner.go:93 +0x5e
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Stop.func1()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner_group.go:63 +0x78
  sync.(*Once).doSlow()
      sync/once.go:76 +0xe1
  sync.(*Once).Do()
      sync/once.go:67 +0x44
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Stop()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner_group.go:61 +0x5e
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run.func1()
      github.com/elastic/beats/v7/metricbeat/beater/metricbeat.go:243 +0xb7

Goroutine 63 (running) created at:
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start()
      github.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:162 +0x16b
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start.func1()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner.go:82 +0x64
  sync.(*Once).doSlow()
      sync/once.go:76 +0xe1
  sync.(*Once).Do()
      sync/once.go:67 +0x44
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner.go:81 +0x5e
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start.func1()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner_group.go:55 +0x78
  sync.(*Once).doSlow()
      sync/once.go:76 +0xe1
  sync.(*Once).Do()
      sync/once.go:67 +0x44
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner_group.go:53 +0x5e
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run()
      github.com/elastic/beats/v7/metricbeat/beater/metricbeat.go:236 +0x121
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.(*metricbeatReceiver).Start.func1()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver.go:25 +0xc2

Goroutine 66 (running) created at:
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run()
      github.com/elastic/beats/v7/metricbeat/beater/metricbeat.go:240 +0xab
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.(*metricbeatReceiver).Start.func1()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver.go:25 +0xc2
==================
==================
WARNING: DATA RACE
Read at 0x00c0007bdf50 by goroutine 63:
  bytes.(*Buffer).tryGrowByReslice()
      bytes/buffer.go:116 +0x4b
  bytes.(*Buffer).Write()
      bytes/buffer.go:177 +0xbc
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
    testing.go:1399: race detected during execution of test
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*Logger).Debug()
      go.uber.org/zap@v1.27.0/logger.go:239 +0x67
  github.com/elastic/beats/v7/metricbeat/mb/module.init.MakeDebug.func1()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/global.go:32 +0x94
  github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).run.deferwrap2()
      github.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:218 +0xb5
  runtime.deferreturn()
      runtime/panic.go:605 +0x5d
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1()
      github.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:175 +0x535
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.gowrap1()
      github.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:176 +0x41

Previous write at 0x00c0007bdf50 by goroutine 66:
  bytes.(*Buffer).tryGrowByReslice()
      bytes/buffer.go:117 +0xa8
  bytes.(*Buffer).Write()
      bytes/buffer.go:177 +0xbc
  go.uber.org/zap/zapcore.(*writerWrapper).Write()
      <autogenerated>:1 +0x6c
  go.uber.org/zap/zapcore.(*ioCore).Write()
      go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0x192
  go.elastic.co/ecszap.core.Write()
      go.elastic.co/ecszap@v1.0.2/core.go:75 +0x2ed
  go.elastic.co/ecszap.(*core).Write()
      <autogenerated>:1 +0xcf
  github.com/elastic/elastic-agent-libs/logp.(*selectiveCore).Write()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/selective.go:85 +0xef
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x1ef
  go.uber.org/zap.(*SugaredLogger).log()
      go.uber.org/zap@v1.27.0/sugar.go:355 +0x12f
  go.uber.org/zap.(*SugaredLogger).Debug()
      go.uber.org/zap@v1.27.0/sugar.go:149 +0x41d
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      github.com/elastic/elastic-agent-libs@v0.17.4/logp/logger.go:108 +0x3cc
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Close()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/client.go:147 +0x378
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Stop.func1()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner.go:95 +0x61
  sync.(*Once).doSlow()
      sync/once.go:76 +0xe1
  sync.(*Once).Do()
      sync/once.go:67 +0x44
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Stop()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner.go:93 +0x5e
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Stop.func1()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner_group.go:63 +0x78
  sync.(*Once).doSlow()
      sync/once.go:76 +0xe1
  sync.(*Once).Do()
      sync/once.go:67 +0x44
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Stop()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner_group.go:61 +0x5e
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run.func1()
      github.com/elastic/beats/v7/metricbeat/beater/metricbeat.go:243 +0xb7

Goroutine 63 (running) created at:
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start()
      github.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:162 +0x16b
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start.func1()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner.go:82 +0x64
  sync.(*Once).doSlow()
      sync/once.go:76 +0xe1
  sync.(*Once).Do()
      sync/once.go:67 +0x44
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner.go:81 +0x5e
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start.func1()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner_group.go:55 +0x78
  sync.(*Once).doSlow()
      sync/once.go:76 +0xe1
  sync.(*Once).Do()
      sync/once.go:67 +0x44
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start()
      github.com/elastic/beats/v7/metricbeat/mb/module/runner_group.go:53 +0x5e
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run()
      github.com/elastic/beats/v7/metricbeat/beater/metricbeat.go:236 +0x121
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.(*metricbeatReceiver).Start.func1()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver.go:25 +0xc2

Goroutine 66 (running) created at:
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run()
      github.com/elastic/beats/v7/metricbeat/beater/metricbeat.go:240 +0xab
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.(*metricbeatReceiver).Start.func1()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver.go:25 +0xc2
==================
--- FAIL: TestNewReceiver (13.06s)

countLogs race condition:

==================
WARNING: DATA RACE
Write at 0x00c000731918 by goroutine 62:
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver.func1()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:55 +0x6d
  go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs()
      go.opentelemetry.io/collector/consumer@v0.109.0/logs.go:26 +0x83
  go.opentelemetry.io/collector/consumer.(*baseLogs).ConsumeLogs()
      <autogenerated>:1 +0x29
  github.com/elastic/beats/v7/libbeat/outputs/otelconsumer.(*otelConsumer).logsPublish()
      github.com/elastic/beats/v7/libbeat/outputs/otelconsumer/otelconsumer.go:103 +0xe41
  github.com/elastic/beats/v7/libbeat/outputs/otelconsumer.(*otelConsumer).Publish()
      github.com/elastic/beats/v7/libbeat/outputs/otelconsumer/otelconsumer.go:74 +0x6c
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*clientWorker).run()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/client_worker.go:101 +0x15e
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker.gowrap1()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/client_worker.go:75 +0x61

Previous read at 0x00c000731918 by goroutine 85:
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver.func2()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:77 +0x33

Goroutine 62 (running) created at:
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/client_worker.go:75 +0x364
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).Set()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:141 +0x48f
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.New()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/pipeline.go:154 +0x6b5
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:100 +0x396
  github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver()
      github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:504 +0x356f
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.createReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/factory.go:36 +0x1e4
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:60 +0xaab
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44

Goroutine 85 (finished) created at:
  github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver.TestNewReceiver()
      github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver/receiver_test.go:77 +0xf2a
  testing.tRunner()
      testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      testing/testing.go:1743 +0x44
==================
    testing.go:1399: race detected during execution of test
--- FAIL: TestNewReceiver (16.06s)

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

  • [ ]

How to test this PR locally

 go test ./x-pack/metricbeat/mbreceiver/ -run=TestNewReceiver -race -v

Related issues

Use cases

Screenshots

Logs

@kruskall
test: fix TestNewReceiver race conditions
578a432 
update zapcore to use observer zaptest package to avoid
race condition on zapLogs bytes.Buffer

use atomic for countLogs to avoid race condition on
primitive type
@kruskall kruskall requested a review from a team as a code owner December 17, 2024 15:29
@kruskall kruskall requested review from mauri870 and faec December 17, 2024 15:29
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 17, 2024
@botelastic
Copy link

botelastic bot commented Dec 17, 2024

This pull request doesn't have a Team:<team> label.

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 17, 2024

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kruskall? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 17, 2024

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Dec 17, 2024
leehinman
leehinman approved these changes Dec 17, 2024
View reviewed changes
Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank You.

If you have the time can you add the same changes to:
https://github.com/elastic/beats/blob/main/x-pack/filebeat/fbreceiver/receiver_test.go

or I can after this is merged.

@kruskall
Copy link
Member Author

If you have the time can you add the same changes to:
https://github.com/elastic/beats/blob/main/x-pack/filebeat/fbreceiver/receiver_test.go

or I can after this is merged.

Thank you! 🙇

I plan to fix all the race conditions spotted by the race detector so it should be a matter of time

@kruskall kruskall merged commit d115014 into elastic:main Dec 17, 2024
21 checks passed
@kruskall kruskall deleted the test/fix-receiver-race branch December 17, 2024 16:45
mergify bot pushed a commit that referenced this pull request Dec 17, 2024
@kruskall @mergify
test: fix TestNewReceiver race conditions (#42084)
aea490a 
update zapcore to use observer zaptest package to avoid
race condition on zapLogs bytes.Buffer

use atomic for countLogs to avoid race condition on
primitive type

(cherry picked from commit d115014)
@mergify mergify bot mentioned this pull request Dec 17, 2024
Merged
6 tasks
kruskall added a commit that referenced this pull request Dec 17, 2024
@mergify @kruskall
test: fix TestNewReceiver race conditions (#42084) (#42090)
6e72e29 
update zapcore to use observer zaptest package to avoid
race condition on zapLogs bytes.Buffer

use atomic for countLogs to avoid race condition on
primitive type

(cherry picked from commit d115014)

Co-authored-by: kruskall <99559985+kruskall@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leehinman leehinman leehinman approved these changes

@mauri870 mauri870 Awaiting requested review from mauri870 mauri870 is a code owner automatically assigned from elastic/elastic-agent-data-plane

@faec faec Awaiting requested review from faec faec is a code owner automatically assigned from elastic/elastic-agent-data-plane

Assignees

@kruskall kruskall

Labels
backport-8.x Automated backport to the 8.x branch with mergify enhancement needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kruskall @leehinman