Seeing client timeouts when creating new feature flags

[lunnydan]

Hello,

I am seeing occasional client timeouts while creating new feature flags. The behaviour seems to be sporadic and sometimes can be solved with a reapply. The variations don't seem to be the source as it has successfully applied over 300 flags with all different variation types. Below is an example of a flag which is "stuck" or unable to be applied via terraform.

Plan:

  # module.feature_flags["enable-claim-loyalty-points"].launchdarkly_feature_flag.feature_flag[0] will be created
  + resource "launchdarkly_feature_flag" "feature_flag" {
      + archived           = false
      + description        = "Enables loyalty points to be claimed"
      + id                 = (known after apply)
      + include_in_snippet = (known after apply)
      + key                = "enable-claim-loyalty-points"
      + maintainer_id      = (known after apply)
      + name               = "enable-claim-loyalty-points"
      + project_key        = "TH"
      + tags               = [
          + "terraform-managed",
        ]
      + temporary          = true
      + variation_type     = "boolean"
      + variations {
          + value = "true"
        }
      + variations {
          + value = "false"
        }
    }

Error:

│ Error: failed to create flag "enable-claim-loyalty-points" in project "TH": Post "https://app.launchdarkly.com/api/v2/flags/TH": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
│ 
│   with module.feature_flags["enable-claim-loyalty-points"].launchdarkly_feature_flag.feature_flag[0],
│   on .terraform/modules/feature_flags/launchdarkly/feature-flag/main.tf line 34, in resource "launchdarkly_feature_flag" "feature_flag":
│   34: resource "launchdarkly_feature_flag" "feature_flag" {

[sloloris]

Hi @lunnydan!

Thanks for bringing this to our attention. As you identified, this may not be a provider issue but an issue with our client, but in any case I plan to dig into this in the next few days to see if I can resolve it and/or redirect it to a team who can.

Have a great week!

[sloloris]

Hi @lunnydan!

Follow-up inquiry: does the project you're creating these flags in have a lot of environments? We batch-create environment-specific flag configurations, and it seems that occasionally see timeouts when creating flags in a project with many environments.

However, I also noticed that it was timing out while awaiting headers. I looked into this a bit and it seems this is commonly a DNS / firewall issue - is there any possibility that your local firewall settings may be affecting this?

[lunnydan]

Hi @sloloris,

We currently have 3 environments (dev, staging, production). Currently our terraform applies all run on github actions so the underlying infrastructure is managed by github. It's possible that there are some DNS/Firewall settings on their side that are impacting this. I will try to replicate an apply on another resource as well as get some better debug logs to determine what specifically is failing.

[lunnydan]

Hey @sloloris,

I have done some more investigation and was able to reproduce the issue on a local Mac OS X operating system on a completely separate network. This does lead me to believe this issue is related to either the launchdarkly api return calls or the way the launchdarkly provider is calling the api. I have added some additional debug logs to help investigate but the root cause seems to be a set of calls returning 409 error codes when trying to create feature flags.

Terraform version

Terraform v1.7.3
on darwin_arm64

Providers version

    launchdarkly = {
      source  = "launchdarkly/launchdarkly"
      version = "~> 2.17.0"
    }

2024-03-06T09:06:14.069-0500 [DEBUG] provider.terraform-provider-launchdarkly_v2.17.0: 2024/03/06 09:06:14 [DEBUG] POST https://app.launchdarkly.com/api/v2/flags/TH (status: 409): retrying in 2s (6 left)
2024-03-06T09:06:14.312-0500 [DEBUG] provider.terraform-provider-launchdarkly_v2.17.0: 2024/03/06 09:06:14 [DEBUG] POST https://app.launchdarkly.com/api/v2/flags/TH (status: 409): retrying in 2s (7 left)
2024-03-06T09:06:14.504-0500 [DEBUG] provider.terraform-provider-launchdarkly_v2.17.0: 2024/03/06 09:06:14 [DEBUG] POST https://app.launchdarkly.com/api/v2/flags/TH (status: 409): retrying in 2s (5 left)
module.feature_flags["enableOrderTimedFireSICOM"].launchdarkly_feature_flag.feature_flag[0]: Still creating... [20s elapsed]
module.feature_flags["enable-aep-quick-actions"].launchdarkly_feature_flag.feature_flag[0]: Still creating... [20s elapsed]
module.feature_flags["darkmode-feature-enabled"].launchdarkly_feature_flag.feature_flag[0]: Still creating... [20s elapsed]
2024-03-06T09:06:15.417-0500 [ERROR] provider.terraform-provider-launchdarkly_v2.17.0: Response contains error diagnostic: @module=sdk.proto tf_provider_addr=registry.terraform.io/launchdarkly/launchdarkly tf_resource_type=launchdarkly_feature_flag @caller=github.com/hashicorp/terraform-plugin-go@v0.18.0/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_severity=ERROR diagnostic_summary="failed to create flag \"enable-aep-quick-actions\" in project \"TH\": Post \"https://app.launchdarkly.com/api/v2/flags/TH\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" tf_proto_version=5.3 tf_rpc=ApplyResourceChange diagnostic_detail="" tf_req_id=a0e56831-17eb-1e6f-7137-cbc580760dbd timestamp=2024-03-06T09:06:15.417-0500
2024-03-06T09:06:15.426-0500 [ERROR] provider.terraform-provider-launchdarkly_v2.17.0: Response contains error diagnostic: @caller=github.com/hashicorp/terraform-plugin-go@v0.18.0/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail="" tf_resource_type=launchdarkly_feature_flag tf_rpc=ApplyResourceChange diagnostic_severity=ERROR diagnostic_summary="failed to create flag \"darkmode-feature-enabled\" in project \"TH\": Post \"https://app.launchdarkly.com/api/v2/flags/TH\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" tf_provider_addr=registry.terraform.io/launchdarkly/launchdarkly @module=sdk.proto tf_proto_version=5.3 tf_req_id=b27e71ed-81b3-c128-0045-23e9c05553a3 timestamp=2024-03-06T09:06:15.426-0500
2024-03-06T09:06:15.427-0500 [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-03-06T09:06:15.427-0500 [ERROR] vertex "module.feature_flags[\"enable-aep-quick-actions\"].launchdarkly_feature_flag.feature_flag[0]" error: failed to create flag "enable-aep-quick-actions" in project "TH": Post "https://app.launchdarkly.com/api/v2/flags/TH": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-03-06T09:06:15.432-0500 [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-03-06T09:06:15.432-0500 [ERROR] vertex "module.feature_flags[\"darkmode-feature-enabled\"].launchdarkly_feature_flag.feature_flag[0]" error: failed to create flag "darkmode-feature-enabled" in project "TH": Post "https://app.launchdarkly.com/api/v2/flags/TH": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-03-06T09:06:15.469-0500 [ERROR] provider.terraform-provider-launchdarkly_v2.17.0: Response contains error diagnostic: tf_proto_version=5.3 tf_req_id=4c49cef9-4926-9f3b-d38a-1d97fdbf2d32 tf_resource_type=launchdarkly_feature_flag tf_rpc=ApplyResourceChange @module=sdk.proto @caller=github.com/hashicorp/terraform-plugin-go@v0.18.0/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_severity=ERROR tf_provider_addr=registry.terraform.io/launchdarkly/launchdarkly diagnostic_summary="failed to create flag \"enableOrderTimedFireSICOM\" in project \"TH\": Post \"https://app.launchdarkly.com/api/v2/flags/TH\": context deadline exceeded" diagnostic_detail="" timestamp=2024-03-06T09:06:15.469-0500
2024-03-06T09:06:15.474-0500 [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-03-06T09:06:15.474-0500 [ERROR] vertex "module.feature_flags[\"enableOrderTimedFireSICOM\"].launchdarkly_feature_flag.feature_flag[0]" error: failed to create flag "enableOrderTimedFireSICOM" in project "TH": Post "https://app.launchdarkly.com/api/v2/flags/TH": context deadline exceeded

[sloloris]

Hi @lunnydan,

I've also spoken to our support engineer and they will likely follow up with you as well, but it looks to me like this tends to happen when a 409 is returned from our API and we retry. This is a bug from our end; we should not be retrying on 409s, but rather returning an error directly, so I will investigate that.

If you have examples of when this is not the case, please do pass the logs along, as that would be an additional issue.

However, you also should not be trying to create Terraform resources the normal why (with a config) if they are resources that already exist on our platform. If you want to manage existing resources in Terraform, you first need to import them before attempting to apply any configs. You can do so using this code:

# Import a feature flag using the feature flag's ID in the format `project_key/flag_key`.
terraform import launchdarkly_feature_flag.enable_loyalty_claim_points th/enable-claim-loyalty-points

Thank you for bringing this to our attention and please do let me know if you continue to see this issue after importing your existing resources. I will also be attempting to roll out a fix to our retry logic in the coming days.

[lunnydan]

Thanks for getting back to me @sloloris.

I did some more investigation around this issue and found that the issue was not related to having the flag created twice by the core terraform code (or creating a resource that already exists as implied). The issue I observed was the launchdarkly provider creating the resource but not updating the state file to reflect it's creation.

I did get around this by importing the resource, however I still observed the behaviour when creating additional new feature flags.

[sloloris]

Hi @lunnydan,

I see - if I've understood correctly, it sounds like there are two pieces to this problem:

1. When retrying while attempting to create an existing feature flag, the connection sometimes hangs and then returns this error. This is solved by importing existing flags into Terraform before attempting to apply their configurations.
2. The situation described above is being artificially created because, after the first apply for flags that do not yet exist, the configurations are not saved to state. Subsequent applies then find no record of the resource in state and attempt to create where they should be updating a resource that already exists in the state.

Does this sound right to you?

I also saw that in the logs you provided to our support team, there is frequently a debug log like this one:

2024-03-06T09:06:15.474-0500 [DEBUG] State storage *remote.State declined to persist a state snapshot

which could be the source of the problem and would likely be due to some sort of remote state configuration or concurrency setting issue. However, I don't understand why this would happen only for our provider's resources, and it is also possible that this only happens in the 409 case as I've not seen your other logs. I am also confused as to why importing a resource seems to update your remote state as expected (if I have understood correctly) but only creating one does not.

If you are consistently seeing this issue, may I ask how many flags you are creating at a time so I can try to reproduce in more realistic conditions? Also, do you use the provider to manage any other LD resources or do you only manage feature flags?

In the meantime, would you mind setting this http_timeout field to 30 or 60 seconds and seeing if you can continue to reproduce the problem? Our default is 20 seconds.

So sorry this is taking so long to get sorted. Unfortunately, I've not been able to reproduce this issue on our end, so it's proving difficult to provide a concrete solution.

[lunnydan]

Hey @sloloris,

The problem you described sounds correct.

I did also notice that the remote state declines to persist a snapshot as you mentioned. I will look into this to see if we have something misconfigured but we don't see this issue with any other provider or even any other launchdarkly resource. I do agree though that it's strange that this would be unique to your provider.

We are currently creating around 350 flags, however the error does not happen consistently on all flags (which have all now been created/configured via manual imports). We are also configuring feature flag environments and segments via the launchdarkly provider without issues minus an exception when the state is not aware that a flag exists currently (we have worked around this with imports as mentioned).

I will try increasing the http_timeout for any new flags being added and see if the issue comes up again, I did not see this configuration option previously.

Also, no problem about this taking a while. It does seem like a niche issue and I appreciate your team looking into it and we are able to work around it for now.

[sloloris]

Hi @lunnydan,

Thank you for the information and for your patience. We have a team member who used to work on Terraform at Hashicorp so I will speak to them and see if they have any insights. I'm glad we were able to find workarounds that worked but I'm sorry you had to import so many flags, can't have been fun!

[lunnydan]

Hey @sloloris,

I am happy to report that since increasing the http_timeout setting on the launchdarkly provider the issue has not been observed again. This may have just been a result of a client timeout causing the state file to not get updated properly.

Feel free to close this issue, thanks again for all your help!