-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Skipped failing test and added race condition check to CI #1217
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #1217 +/- ##
=========================================
Coverage ? 86.94%
=========================================
Files ? 43
Lines ? 11258
Branches ? 0
=========================================
Hits ? 9788
Misses ? 1005
Partials ? 465 ☔ View full report in Codecov by Sentry. |
@@ -76,7 +76,9 @@ jobs: | |||
terraform_version: '1.7.*' | |||
terraform_wrapper: false | |||
- name: Terraform Acceptance Test (APIC ${{ matrix.apic_host.name }}) | |||
run: go test github.com/CiscoDevNet/terraform-provider-aci/v2/internal/provider -v -timeout 300m -coverprofile=coverage.out -covermode=atomic | |||
# TODO: Fix TestAccAciRestManaged_importWithIpv6 to resolve data race failures occurring in the CI |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be an interesting fix.
Authenticate() has c.skipLoggingPayload whose value is being changed and reset in the end.
Now Authenticate() is called from multiple go-routines, one being MakeRestRequestRaw() via DoRestRequestEscapeHtml and the other go-routine handling MakeRestRequestRaw() within the Authenticate() function itself. This might intermittently be creating a race condition due to c.skipLoggingPayload state being read by one and written by the other at the same time.
I think this is what's causing it.
It would be great if this PR is updated with the fix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @shrsr, yeah this sounds like what could be happening based on the stack trace error of the data race warnings. Its odd that its only happening consistently in this one test. It appears to be quite a complex fix that will effect every other module, hence I created another issue for it. #1218
For now this PR will turn on the race condition check in the CI and skip this one test to stop PR's failing randomly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's only happening for this test because of the way the config. is specified in the test. As seen below, two BDs are being created in parallel. This is only showing up during the CI because more than one cpu is utilized by the go test environment in the CI workflow enabling parallelism. The utilization of more than one CPU also causes parallel refreshes in TF acceptance tests further adding to race conditions.
resource "aci_rest_managed" "tenant_%[2]s" {
dn = "uni/tn-%[1]s"
class_name = "fvTenant"
content = {
name = "%[1]s"
}
child {
rn = "ctx-VRF1"
class_name = "fvCtx"
content = {
name = "VRF1"
}
}
child {
rn = "ctx-VRF2"
class_name = "fvCtx"
content = {
name = "VRF2"
}
}
}
resource "aci_rest_managed" "bd_%[2]s" {
dn = "${aci_rest_managed.tenant_%[2]s.id}/BD-%[1]s"
class_name = "fvBD"
content = {
name = "%[1]s"
}
child {
rn = "rsctx"
class_name = "fvRsCtx"
content = {
tnFvCtxName = "VRF1"
}
}
}
resource "aci_rest_managed" "subnet_%[2]s" {
dn = "${aci_rest_managed.bd_%[2]s.id}/subnet-[%[3]s]"
class_name = "fvSubnet"
content = {
ip = "%[3]s"
scope = "private"
ipDPLearning = "enabled"
ctrl = "nd"
}
}
resource "aci_rest_managed" "bd_%[2]s_2" {
dn = "${aci_rest_managed.tenant_%[2]s.id}/BD-%[1]s_2"
class_name = "fvBD"
content = {
name = "%[1]s_2"
}
child {
rn = "rsctx"
class_name = "fvRsCtx"
content = {
tnFvCtxName = "VRF2"
}
}
child {
rn = "subnet-[%[4]s]"
class_name = "fvSubnet"
content = {
ip = "%[4]s"
scope = "private"
ipDPLearning = "enabled"
ctrl = "nd"
}
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of skipping the test, we could enforce the CI to use 1 cpu for now while the issue is being fixed.
What do you think?
go test github.com/CiscoDevNet/terraform-provider-aci/v2/internal/provider -v -race -cpu 1 -timeout 300m -coverprofile=coverage.out -covermode=atomic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like it! I'll try it out now. 😸
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@shrsr Unfortunately with -cpu 1
the data race is still detected. I'll revert that commit and skip the test for now. 😞
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah looks like there are other factors that come into play in containers and simply setting the core to 1 is not enough to mask the race.
@@ -76,7 +76,9 @@ jobs: | |||
terraform_version: '1.7.*' | |||
terraform_wrapper: false | |||
- name: Terraform Acceptance Test (APIC ${{ matrix.apic_host.name }}) | |||
run: go test github.com/CiscoDevNet/terraform-provider-aci/v2/internal/provider -v -timeout 300m -coverprofile=coverage.out -covermode=atomic | |||
# TODO: Fix TestAccAciRestManaged_importWithIpv6 to resolve data race failures occurring in the CI |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's only happening for this test because of the way the config. is specified in the test. As seen below, two BDs are being created in parallel. This is only showing up during the CI because more than one cpu is utilized by the go test environment in the CI workflow enabling parallelism. The utilization of more than one CPU also causes parallel refreshes in TF acceptance tests further adding to race conditions.
resource "aci_rest_managed" "tenant_%[2]s" {
dn = "uni/tn-%[1]s"
class_name = "fvTenant"
content = {
name = "%[1]s"
}
child {
rn = "ctx-VRF1"
class_name = "fvCtx"
content = {
name = "VRF1"
}
}
child {
rn = "ctx-VRF2"
class_name = "fvCtx"
content = {
name = "VRF2"
}
}
}
resource "aci_rest_managed" "bd_%[2]s" {
dn = "${aci_rest_managed.tenant_%[2]s.id}/BD-%[1]s"
class_name = "fvBD"
content = {
name = "%[1]s"
}
child {
rn = "rsctx"
class_name = "fvRsCtx"
content = {
tnFvCtxName = "VRF1"
}
}
}
resource "aci_rest_managed" "subnet_%[2]s" {
dn = "${aci_rest_managed.bd_%[2]s.id}/subnet-[%[3]s]"
class_name = "fvSubnet"
content = {
ip = "%[3]s"
scope = "private"
ipDPLearning = "enabled"
ctrl = "nd"
}
}
resource "aci_rest_managed" "bd_%[2]s_2" {
dn = "${aci_rest_managed.tenant_%[2]s.id}/BD-%[1]s_2"
class_name = "fvBD"
content = {
name = "%[1]s_2"
}
child {
rn = "rsctx"
class_name = "fvRsCtx"
content = {
tnFvCtxName = "VRF2"
}
}
child {
rn = "subnet-[%[4]s]"
class_name = "fvSubnet"
content = {
ip = "%[4]s"
scope = "private"
ipDPLearning = "enabled"
ctrl = "nd"
}
}
}
@@ -76,7 +76,9 @@ jobs: | |||
terraform_version: '1.7.*' | |||
terraform_wrapper: false | |||
- name: Terraform Acceptance Test (APIC ${{ matrix.apic_host.name }}) | |||
run: go test github.com/CiscoDevNet/terraform-provider-aci/v2/internal/provider -v -timeout 300m -coverprofile=coverage.out -covermode=atomic | |||
# TODO: Fix TestAccAciRestManaged_importWithIpv6 to resolve data race failures occurring in the CI |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of skipping the test, we could enforce the CI to use 1 cpu for now while the issue is being fixed.
What do you think?
go test github.com/CiscoDevNet/terraform-provider-aci/v2/internal/provider -v -race -cpu 1 -timeout 300m -coverprofile=coverage.out -covermode=atomic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Long term fix is ready here #1236. Cancelling this PR. |
Skipped the
aci_rest_managed importWithIpv6
test from CI to stop random failures in the acceptance test.Issue #1218 is created to fix this single test case.