Consumer account after subscribing the new SaaS Sample application- do not create 2 HDI containers(for data,for side-car)

[Mihirjha13]

[Mihirjha13]

In the kyma runtime -
a.)I can see that new api rules are created for new subscriptions in consumer sub account and
b.) instances in Kyma are running (saas-registry and xsuaa)

Seems something in broker pod, I found below in the log:

2021-06-04T15:48:55.706513884Z | {"log":"{"timestamp":"1622821735.706346273","source":"faq-broker","message":"faq-broker.k8s-client-creation","log_level":3,"data":{"error":"Get \"https://api.b341387.kyma.internal.live.k8s.ondemand.com:443/api?timeout=32s\\\": dial tcp 20.73.106.44:443: connect: connection refused","message":"error creating k8s client","trace":"goroutine 1 [running]:\ code.cloudfoundry.org/lager.(*logger).Fatal(0xc00040a120, 0x140b07d, 0x13, 0x15b84a0, 0xc000104150, 0xc00000e028, 0x1, 0x1)\ \ /go/pkg/mod/code.cloudfoundry.org/lager@v2.0.0+incompatible/logger.go:138 +0xc6\ main.main()\ \ /workspace/main.go:33 +0x107e\ "}} ","time":"2021-06-04T15:48:55.706513884Z"}

On the namespace overview - everything looks fine.

[eikeb]

I have a similar issue with this sample.
All pods and deployments are green, but only one HDI container appears below that faq-saas-container service manager. Even if I subscribe new consumer accounts, no HDI containers are created.

The broker log contains this error message:
{"log":"CAP Response: "HTTP/1.1 403 Forbidden\r\ Content-Length: 74\r\ Content-Type: application/json; charset=utf-8\r\ Date: Fri, 04 Jun 2021 18:22:04 GMT\r\ Etag: W/\"4a-HB0FIz0zxq92WkMPKTrErwEf+5o\"\r\ Server: envoy\r\ X-Envoy-Upstream-Service-Time: 7\r\ X-Powered-By: Express\r\ \r\ {\"error\":{\"code\":\"403\",\"@Common.numericSeverity\":4,\"message\":\"Forbidden\"}}"{"timestamp":"1622830924.804577589","source":"faq-broker","message":"faq-broker.create-saas-instance","log_level":1,"data":{"message":"Creating APIRule for subdomain","subdomain":"faq-consumer3"}} ","time":"2021-06-04T18:22:04.804769587Z"}

[draschke]

And you don't have any trouble with your faq-backend Pod?
I'm asking because I don't get my issue solved even after recreating my trial account.
(error: Conflict, description: instance with same name exists for the current tenant ")

[eikeb]

@draschke I encountered that error too. Eventually, I deleted everything and started the deployment from scratch. After the deployment of the backend component I waited until it started up completely. Only then I deployed the other components. That did the trick for me.
Also make sure you have a SAP HANA Cloud instance started.

[draschke]

@eikeb. thanks, but it didn't solve my issue. (instance with same name exists for the current tenant)
Thought that you maybe struggle with the same issue but didn't notice that, because the Overview of the Pods doesn't show this issue.

[Mihirjha13]

Same for me - i had issue in backend but when I made the app unique in the manifest of user-interface\k8s and then deleted all pods,and deployments and did it fresh atleast the backend was fine. But This broker issue is still annoying and no clue what might be wrong. @IObert : perhaps you can help :)

[IObert]

Can you provide more details?

Are all pods running without any errors? Is HANA Cloud running as well? What error do you get during the subscription from the BTP cockpit? (Check out the Troubleshooting section here to find out how to see the detailed error message )

[Mihirjha13]

Hi Marius,
Except Broker pod all are fine. and there is no error while subscribing the SaaS service from consumer subaccount but rather I do not see any 2 new HDi created after new subscriptions.

The error in broker pod is:
2021-06-04T15:48:55.706513884Z | {"log":"{"timestamp":"1622821735.706346273","source":"faq-broker","message":"faq-broker.k8s-client-creation","log_level":3,"data":{"error":"Get "https://api.b341387.kyma.internal.live.k8s.ondemand.com:443/api?timeout=32s\\\": dial tcp 20.73.106.44:443: connect: connection refused","message":"error creating k8s client","trace":"goroutine 1 [running]:\ code.cloudfoundry.org/lager.(*logger).Fatal(0xc00040a120, 0x140b07d, 0x13, 0x15b84a0, 0xc000104150, 0xc00000e028, 0x1, 0x1)\ \ /go/pkg/mod/code.cloudfoundry.org/lager@v2.0.0+incompatible/logger.go:138 +0xc6\ main.main()\ \ /workspace/main.go:33 +0x107e\ "}} ","time":"2021-06-04T15:48:55.706513884Z"}

[IObert]

1. Did you do any modifications to the deployment.yaml file of the broker?

2. And can you confirm that HANA Cloud is accepting traffic from all IP addresses?

3. And can you see and decode the secret sm-credentials in the Kyma console?

[Mihirjha13]

1. there is no deployment.yaml but if you broker menifest , i dint change nothing
2. yes
3. yes

[Mihirjha13]

might this be issue ?

[Mihirjha13]

one more thing -
the kubctl tell me there is no secret named as "sm-credential"

but when i see the kyma UI it is there

may be i need to degrade kube client to 1.18 equal to server version?

[IObert]

My guess is that you didn't specify the namespace in the kubectl command

[eikeb]

Hi @IObert,

regarding your questions:

1. No, I did not change the manifest.yml file from the broker
2. Yes. The database allows connections from all IP addresses and is running
3. Yes, the sm-credentials secret is there and I can view the decoded data in the Kyma console.

All Pods and Deployments are green and running.

I tried again to create a new consumer and got the following error in the broker logs:
{"log":"CAP Response: "HTTP/1.1 403 Forbidden\r\ Content-Length: 74\r\ Content-Type: application/json; charset=utf-8\r\ Date: Mon, 07 Jun 2021 15:04:40 GMT\r\ Etag: W/\"4a-HB0FIz0zxq92WkMPKTrErwEf+5o\"\r\ Server: envoy\r\ X-Envoy-Upstream-Service-Time: 96\r\ X-Powered-By: Express\r\ \r\ {\"error\":{\"code\":\"403\",\"@Common.numericSeverity\":4,\"message\":\"Forbidden\"}}"{"timestamp":"1623078280.196769476","source":"faq-broker","message":"faq-broker.create-saas-instance","log_level":1,"data":{"message":"Creating APIRule for subdomain","subdomain":"faq-consumer"}} ","time":"2021-06-07T15:04:40.19695852Z"}

There seems to be a problem with the APIRule creation. Is there a way to debug that possibly?
EDIT: That not the problem I guess. I can see the APIRule in the Kyma console.

[eikeb]

@IObert: I just checked the requirements that you listed in the readme file and I seem to be missing these entitlements in my trial account:

• SaaS Provisioning (container and application plans)
• XSUAA: Authorization & Trust Management (broker plan)

Can that be the problem? I can't find a way to add them.

[IObert]

@IObert: I just checked the requirements that you listed in the readme file and I seem to be missing these entitlements in my trial account:

• SaaS Provisioning (container and application plans)
• XSUAA: Authorization & Trust Management (broker plan)

Can that be the problem? I can't find a way to add them.

No, that's just bad wording from my side. These two services are not based on entitlements any more and come with each CF-enabled subaccount. I will change the wording to avoid confusion :)

Regarding the original issue: I learned that @Mihirjha13 also works for SAP and we scheduled an internal meeting for Friday to try to find the root cause of this error. Once we find it, I'll update this issue. But I have a packed calendar this week, so I won't be able to provide much help until then.

[eikeb]

No worries, I'm looking forward to it :)

[eduardoarndt]

Hi @IObert,

regarding your questions:

1. No, I did not change the manifest.yml file from the broker
2. Yes. The database allows connections from all IP addresses and is running
3. Yes, the sm-credentials secret is there and I can view the decoded data in the Kyma console.

All Pods and Deployments are green and running.

I tried again to create a new consumer and got the following error in the broker logs: {"log":"CAP Response: "HTTP/1.1 403 Forbidden\r\ Content-Length: 74\r\ Content-Type: application/json; charset=utf-8\r\ Date: Mon, 07 Jun 2021 15:04:40 GMT\r\ Etag: W/"4a-HB0FIz0zxq92WkMPKTrErwEf+5o"\r\ Server: envoy\r\ X-Envoy-Upstream-Service-Time: 96\r\ X-Powered-By: Express\r\ \r\ {"error":{"code":"403","@Common.numericSeverity":4,"message":"Forbidden"}}"{"timestamp":"1623078280.196769476","source":"faq-broker","message":"faq-broker.create-saas-instance","log_level":1,"data":{"message":"Creating APIRule for subdomain","subdomain":"faq-consumer"}} ","time":"2021-06-07T15:04:40.19695852Z"}

There seems to be a problem with the APIRule creation. Is there a way to debug that possibly? EDIT: That not the problem I guess. I can see the APIRule in the Kyma console.

Hello, I've been following the guide here and came accross this same problem, here are the full logs of my faq-broker

$ kubectl logs deploy/faq-broker
Broker configured with:
Namespace in which brokered resources will be provisioned in: project-faq
Location of kubeconfig file if broker is run out of cluster:
Name of the Service that will be exposed via API Rule: faq-backend
Port of the Service that will be exposed via API Rule: 8080
Kyma gateway used for the ingress: kyma-gateway.kyma-system.svc.cluster.local
SaaS Provisioner Port: 8081
Domain of the kyma cluster: c-9569984.kyma.shoot.live.k8s-hana.ondemand.com
Name of the SaaS App Service that will be exposed via API Rule: kyma-faq-ui-shell
Port of the SaaS App Service that will be exposed via API Rule: 8081
Cap Provisioning URl http://faq-backend:8080
{"timestamp":"1634745967.033209324","source":"faq-broker","message":"faq-broker.start-saas-provisioner","log_level":1,"data":{"message":"Ready for APIRule creation on port 8081"}}
CAP Response: "HTTP/1.1 403 Forbidden\r\nContent-Length: 74\r\nContent-Type: application/json; charset=utf-8\r\nDate: Wed, 20 Oct 2021 16:11:06 GMT\r\nEtag: W/\"4a-HB0FIz0zxq92WkMPKTrErwEf+5o\"\r\nServer: envoy\r\nX-Envoy-Upstream-Service-Time: 104\r\nX-Powered-By: Express\r\n\r\n{\"error\":{\"code\":\"403\",\"@Common.numericSeverity\":4,\"message\":\"Forbidden\"}}"{"timestamp":"1634746266.957107544","source":"faq-broker","message":"faq-broker.create-saas-instance","log_level":1,"data":{"message":"Creating APIRule for subdomain","subdomain":"subscriber-bq3awjfr"}}
{"timestamp":"1634746266.981483936","source":"faq-broker","message":"faq-broker.create-saas-instance","log_level":1,"data":{"status":"Success"}}
2021/10/20 16:11:06 http: superfluous response.WriteHeader call from broker/pkg/kyma.(*KymaHandler).ServeHTTP (kyma.go:121)
CAP Response: "HTTP/1.1 403 Forbidden\r\nContent-Length: 74\r\nContent-Type: application/json; charset=utf-8\r\nDate: Wed, 20 Oct 2021 16:26:43 GMT\r\nEtag: W/\"4a-HB0FIz0zxq92WkMPKTrErwEf+5o\"\r\nServer: envoy\r\nX-Envoy-Upstream-Service-Time: 47\r\nX-Powered-By: Express\r\n\r\n{\"error\":{\"code\":\"403\",\"@Common.numericSeverity\":4,\"message\":\"Forbidden\"}}"{"timestamp":"1634747203.099887133","source":"faq-broker","message":"faq-broker.create-saas-instance","log_level":1,"data":{"message":"Creating APIRule for subdomain","subdomain":"subscriber-b-td18wvwi"}}
{"timestamp":"1634747203.126182318","source":"faq-broker","message":"faq-broker.create-saas-instance","log_level":1,"data":{"status":"Success"}}
2021/10/20 16:26:43 http: superfluous response.WriteHeader call from broker/pkg/kyma.(*KymaHandler).ServeHTTP (kyma.go:121)

The problem seems to come from the request being made at

kyma-mtx-faq-management/provider-components/broker/pkg/kyma/kyma.go

Line 73 in fa187fd

targetCapUrl := fmt.Sprintf("%s%s%s", strings.TrimSuffix(k.capProvisioningURL, "/"), "/mtx/v1/provisioning/tenant/", tenantId)

where capProvisioningURL is set to http://faq-backend:8080?

kyma-mtx-faq-management/provider-components/broker/k8s/manifest.yaml

Lines 39 to 40 in fa187fd

	- "--capProvisioningURL"
	- "http://faq-backend:8080"

Sadly I don't understand much about CAP and go to further debug this.

@IObert , did you find any solution for this?