[BUG] CF org not found after creating CF environment via BTP terraform

[SeanKilleen]

Is there an existing issue for this?

• I have searched the existing issues

What version of Terraform are you using?

0.6.0-beta2

What type of issue are you facing

bug report

Describe the bug

I am hitting a strange issue that I think may be related to my local settings but I am unsure how to verify, and I believe this team understands the inner workings enough to help me understand if it's a TF issue or how to resolve.

My goal: create a CF environment using SAP BTP Terraform; then use the CloudFoundry provider to operate on that org to do things like create spaces, create instances, etc.

This is code that was previously working (with the latest CF provider, double-checked) but that I had shelved and am now revisiting. The difference is, we are now operating in a different global account than last time and I don't seem to be able to get the CF CLI to "see" orgs in that new global account despite also being a GA admin there.

I:

• Have a subaccount with a region of us10
• Create a CloudFoundry org, also within us10, via BTP TF
• With the CF provider, attempt to use a data block to find the CF org (exact same name as created org; using a local to be sure.)

I receive an error on the data block when attempting to apply from the CF Provider, saying that the element is not found.

So I figured I would try with the CF CLI. When I sign into the CF CLI with my username and password and the SAP us10 endpoint, I see three CloudFoundry orgs...from my original global account. And no CF orgs from the new global account (which exists alongside that old account).

I'm trying to understand: how can I retrieve the values for that new account? I can see it in BTP, the name matches exactly what I expect, and yet I cannot see it in the CF CLI.

I'm wondering if perhaps I need to regenerate my credentials somehow in order for it to have both global accounts available to it?

Expected Behavior

No response

Steps To Reproduce

No response

Add screenshots to help explain your problem

No response

Additional context

No response

[SeanKilleen]

Thinking that this seems more like an issue at the BTP <-> CF level where CF should be seeing all of the orgs I Have access to, but instead only sees orgs from the first global account I was associated with instead of all of them.

Created a forum post here to explore it: https://answers.sap.com/questions/14009654/viewing-cloudfoundry-orgs-across-multiple-global-a.html. Excerpt:

Summary:

    Given I am previously an administrator on Global Account A
    ...And Global account A contains a CF org 123
    ...And I am newly also an administrator on Global Account B
    ...And Global account B contains a CF org 456
    When I log into the CF cli with my credentials via "cf login"
    ...And I run "cf orgs"
    How can I see both CF org 123 and CF org 456? (currently only org 123 is displayed)

If you feel confident that this is outside of the TF provider's realm, feel free to close this.

[SeanKilleen]

Actually this may be a BTP provider issue after all.

From the comments on my SAP forum post, I discovered that the CF API url for the newly created CF Org for some reason is https://api.cf.us10-001.hana.ondemand.com instead of https://api.cf.us10.hana.ondemand.com.

If the BTP provider is creating this with the incorrect CF API URL, that could have caused this issue, because I don't have the option to set that it seems, and the region is set to us10 for the subaccount so I'm assuming I should be able to anticipate the same region for the CF API (us10, not us10-001)

[lechnerc77]

@SeanKilleen This is not an isue with the Terraform provider, but the way the CF orgs are created on BTP. The CF orgs are depending on their regional assignment created in the so called extension landscapes (see https://help.sap.com/docs/custom-domain/custom-domain-manager/extension-landscapes). This impacts the CF API endpoint.

After creation of the CF enviroment via the Terraform provider you can fetch the right API endpoint URL from the labels that are created for the environment. We have an example for this here: https://github.com/SAP-samples/btp-terraform-samples/tree/main/released/modules/environment/cloudfoundry/envinstance_cf

The envinstance_cf_outputs.tf file contains the code snippet you are looking for:

output "cf_api_endpoint" {
  value       = lookup(jsondecode(btp_subaccount_environment_instance.cf.labels), "API Endpoint", "not found")
  description = "API endpoint of the Cloud Foundry environment."
}

There is no way to influence where your CF environment gets placed if you are in a region that is working with extension landscapes.

[SeanKilleen]

@lechnerc77 being able to fetch the CF endpoint via the label is a perfectly fine solution; I didn't know it was populated there. Thanks for the explanation!

[SeanKilleen]

@lechnerc77 I know this is a closed issue but I have what I hope will be a quick follow-up question just to confirm my understanding.

My thinking:

• Terraform provider information must be statically known
• The CloudFoundry API URL is a part of the CloudFoundry provider
• But the CF API URL isn't known anymore in the SAP BTP environment, and has to be retrieved dynamically.

Therefore, I think my best option (if it's possible, I have to research) is:

• Create multiple providers with aliases, for each possible CF API URL that could come back.
• Retrieve the API URL dynamically as you describe
• Attempt to programmatically select a provider based on the API URL matching.

Is that the recommended approach here?

[lechnerc77]

@SeanKilleen : I would recommend a different approach. I would split the setup into two parts namely (assumption is that the setup is executed via CI/CD pipeline):

• first step would be the setup of the BTP specific parts including the creation of the CF environment via the Terraform Provider for SAP BTP. One output parameter of this part is the URL of the CF API endpoint
• second step would be the setup of the CF specific parts. Here I would define the API endpoint needed for the configuration of the CF provider as a variable and feed the output of step one as input of step 2.

The price of this approach is that the Terraform state is split even for the same environment.

[SeanKilleen]

@lechnerc77 this isn't currently in CI but will be eventually. I rather liked the idea of having a single Terraform apply command, but given the current constraints, I guess it's worth the effort to split it up at this point and it doesn't hurt. Thanks for the insight!

[SeanKilleen]

PS it's mainly not in CI because I can't figure out the best path forward to get it to get the Kyma pieces to run headless (currently pops up a window for OAuth login). I'll get that working eventually, it's just a little lower on the priority list.

[lechnerc77]

@SeanKilleen : Yes I thought so, definitely not the solution you would like. However, I still hope that Hashicorp might come up with a solution as the requests for such a dynamic setup of providers are out there for quite some time now (see hashicorp/terraform#25244 and hashicorp/terraform#19932).

Concerning Kyma: the current flow does not allow a "headless" configuration. There is something in the backlog (see kyma-project/kyma#18198) but the original request also dates back now approx. two years to enable something like that for automation purposes.

[SeanKilleen]

Then I suppose our deployments won't wind up in CI anytime soon. 😞 I'll see if I can hack together something with a service account.

But regardless, this gives me a tangible path forward and I appreciate it, so thank you!

[SeanKilleen]

For those who might be curious, I created a different path forward that is still a little hacky but that seems to work OK and doesn't involve me having to split our terraform.

My thinking/context:

• This is a one-time pain, so is it really worth splitting up our TF for?
• I also have the advantage of knowing that most of my CF stuff is keyed off of the lookup of the CF org, which won't exist if the wrong endpoint is selected.

So, my approach:

• Set up provider aliases (I have two scenarios that might have different regions)
• Set cf_api_url in a variable
• Add an output for the URL within BTP.

So, worst case scenario -- let's say I choose us10 when it should be us10-001. In this case, My TF will fail the first time with a "not found". In the output, I'll see the CF URL that comes from the BTP environment, and then in the TF resource definition I have a comment that they should match. I really only need to fix that once per environment creation it seems, which is up-front pain I can deal with.

This worked in practice with my second region that I needed. Since the first was in us10-001 I assumed the second would be in us20-001 but it was still in us20. I got the error, saw the output, checked things against it, made the change, and it worked.

So the end result: still not desired, still quite messy, but working fine for our purposes.