Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use read mutex when iterating over informers map #160

Merged
merged 1 commit into from
Sep 25, 2018

Conversation

JoelSpeed
Copy link
Contributor

@JoelSpeed JoelSpeed commented Sep 25, 2018

I'm building a controller using the package and have been running my tests using ginkgo -race, I came across this race condition which led me to a small fix.

WARNING: DATA RACE
Write at 0x00c000e65440 by goroutine 23:
  runtime.mapassign()
      /usr/local/Cellar/go/1.11/libexec/src/runtime/map.go:549 +0x0
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).Get.func2()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:180 +0x456
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).Get()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:190 +0x161
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal.(*InformersMap).Get()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/deleg_map.go:84 +0x1f0
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache.(*informerCache).Get()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache/informer_cache.go:53 +0x1c1
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/client.(*DelegatingReader).Get()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/client/split.go:49 +0x1a4
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/client.(*DelegatingClient).Get()
      <autogenerated>:1 +0xd1
  github.com/pusher/faros/pkg/controller/gittrack.(*ReconcileGitTrack).fetchDeployKey()
      /Users/joel/go/src/github.com/pusher/faros/pkg/controller/gittrack/gittrack_controller.go:157 +0x28b
  github.com/pusher/faros/pkg/controller/gittrack.glob..func3.9.1.4()
      /Users/joel/go/src/github.com/pusher/faros/pkg/controller/gittrack/gittrack_controller_test.go:836 +0xfd
  github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync()
      /Users/joel/go/src/github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:113 +0xbd
  github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/leafnodes.(*runner).run()
      /Users/joel/go/src/github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:64 +0x16a
  github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run()
      /Users/joel/go/src/github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/leafnodes/it_node.go:26 +0xa2
  github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/spec.(*Spec).runSample()
      /Users/joel/go/src/github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/spec/spec.go:203 +0x766
  github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/spec.(*Spec).Run()
      /Users/joel/go/src/github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/spec/spec.go:138 +0x145
  github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec()
      /Users/joel/go/src/github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/specrunner/spec_runner.go:200 +0x172
  github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs()
      /Users/joel/go/src/github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/specrunner/spec_runner.go:170 +0x4ad
  github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run()
      /Users/joel/go/src/github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/specrunner/spec_runner.go:66 +0x149
  github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/suite.(*Suite).Run()
      /Users/joel/go/src/github.com/pusher/faros/vendor/github.com/onsi/ginkgo/internal/suite/suite.go:62 +0x3e1
  github.com/pusher/faros/vendor/github.com/onsi/ginkgo.RunSpecsWithCustomReporters()
      /Users/joel/go/src/github.com/pusher/faros/vendor/github.com/onsi/ginkgo/ginkgo_dsl.go:221 +0x367
  github.com/pusher/faros/vendor/github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters()
      /Users/joel/go/src/github.com/pusher/faros/vendor/github.com/onsi/ginkgo/ginkgo_dsl.go:209 +0x129
  github.com/pusher/faros/pkg/controller/gittrack.TestBee()
      /Users/joel/go/src/github.com/pusher/faros/pkg/controller/gittrack/gittrack_controller_suite_test.go:65 +0xc4
  testing.tRunner()
      /usr/local/Cellar/go/1.11/libexec/src/testing/testing.go:827 +0x162

Previous read at 0x00c000e65440 by goroutine 242:
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).HasSyncedFuncs()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:130 +0x87
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal.(*InformersMap).WaitForCacheSync()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/deleg_map.go:67 +0x66
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache.(*informerCache).WaitForCacheSync()
      <autogenerated>:1 +0x63
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/cache.Cache.WaitForCacheSync-fm()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:139 +0x50
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:141 +0x28f
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).start.func1.2()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:270 +0x4c

Goroutine 23 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.11/libexec/src/testing/testing.go:878 +0x650
  testing.runTests.func1()
      /usr/local/Cellar/go/1.11/libexec/src/testing/testing.go:1119 +0xa8
  testing.tRunner()
      /usr/local/Cellar/go/1.11/libexec/src/testing/testing.go:827 +0x162
  testing.runTests()
      /usr/local/Cellar/go/1.11/libexec/src/testing/testing.go:1117 +0x4ee
  testing.(*M).Run()
      /usr/local/Cellar/go/1.11/libexec/src/testing/testing.go:1034 +0x2ee
  main.main()
      _testmain.go:42 +0x221

Goroutine 242 (running) created at:
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).start.func1()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:269 +0x215
  github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).start()
      /Users/joel/go/src/github.com/pusher/faros/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:275 +0x42

The Get and the HasSynced methods are getting called concurrently in my tests and I noticed there is a read lock used in one but not the other, this PR fixes that

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Sep 25, 2018
@DirectXMan12
Copy link
Contributor

/approve
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 25, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DirectXMan12, JoelSpeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 25, 2018
@k8s-ci-robot k8s-ci-robot merged commit b2824f6 into kubernetes-sigs:master Sep 25, 2018
@JoelSpeed JoelSpeed deleted the fix-race branch September 25, 2018 14:44
justinsb pushed a commit to justinsb/controller-runtime that referenced this pull request Dec 7, 2018
Use read mutex when iterating over informers map
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants