Metal Install - No Tectonic-Console loads

[dmgaraway]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT or FEATURE REQUEST

BUG REPORT

Versions

tectonic_1.8.4-tectonic.3

Tectonic version (release or commit hash):

tectonic_1.8.4-tectonic.3

Terraform version (terraform version):

Terraform v0.10.7

Platform (aws|azure|openstack|metal|vmware):

metal

What happened?

Everything completes except the console will not load. Everything seems good except the tectonic-console will not load. I have checked the worker node and got these errors. 

tectonic-system   tectonic-console-6d4578d847-ff664                      0/1       CrashLoopBackOff   41         2h
tectonic-system   tectonic-console-6d4578d847-wlhzs                      0/1       CrashLoopBackOff   41         2h
tectonic-system   tectonic-identity-7484d469d7-sz85b                     1/1       Running            0          2h
tectonic-system   tectonic-identity-7484d469d7-wh8gv                     1/1       Running            0          2h
tectonic-system   tectonic-ingress-controller-q9jvr                      1/1       Running            0          2h
tectonic-system   tectonic-monitoring-auth-prometheus-5c7c499ff4-k895r   0/1       CrashLoopBackOff   49         2h

What you expected to happen?

everything work

How to reproduce it (as minimally and precisely as possible)?

reinstalled and happens consistently 

Anything else we need to know?

NO

References

No

Feature Request

Environment

Vmware ESXi

Desired Feature

To work

Other Information

Nothing

[bodgit]

This is looking suspiciously like the same problem I'm hitting. What does kubectl logs tectonic-console-6d4578d847-ff664 --namespace=tectonic-system show for you?

I'm getting the following for the same three crashing pods:

2018/03/2 16:12:42 http: Provider config sync still failing, retrying in 16s: Get https://mycluster.example.com/identity/.well-known/openid-configuration: dial tcp 172.16.142.69:443: getsockopt: connection refused

[dmgaraway]

This is what I'm getting.. Its like it cant get to the identity server but thats running fine. its like the router is not proxy for the services.

[root@matchbox install]# kubectl logs tectonic-console-6d4578d847-ff664 --namespace=tectonic-system
Error from server (NotFound): pods "tectonic-console-6d4578d847-ff664" not found
[root@matchbox install]# kubectl logs tectonic-console-6d4578d847-jcwh6 --namespace=tectonic-system
2018/03/3 23:32:17 http: Provider config sync failed, retrying in 1s: Get https://console.lab.com/identity/.well-known/openid-configuration: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2018/03/3 23:32:23 http: Provider config sync still failing, retrying in 2s: Get https://console.lab.com/identity/.well-known/openid-configuration: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2018/03/3 23:32:30 http: Provider config sync still failing, retrying in 4s: Get https://console.lab.com/identity/.well-known/openid-configuration: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2018/03/3 23:32:34 http: Updating provider config: config=oidc.ProviderConfig{Issuer:(*url.URL)(0xc42026c900), AuthEndpoint:(*url.URL)(0xc42026c980), TokenEndpoint:(*url.URL)(0xc42026ca00), UserInfoEndpoint:(*url.URL)(nil), KeysEndpoint:(*url.URL)(0xc42026ca80), RegistrationEndpoint:(*url.URL)(nil), ScopesSupported:[]string{"openid", "email", "groups", "profile", "offline_access"}, ResponseTypesSupported:[]string{"code"}, ResponseModesSupported:[]string(nil), GrantTypesSupported:[]string(nil), ACRValuesSupported:[]string(nil), SubjectTypesSupported:[]string{"public"}, IDTokenSigningAlgValues:[]string{"RS256"}, IDTokenEncryptionAlgValues:[]string(nil), IDTokenEncryptionEncValues:[]string(nil), UserInfoSigningAlgValues:[]string(nil), UserInfoEncryptionAlgValues:[]string(nil), UserInfoEncryptionEncValues:[]string(nil), ReqObjSigningAlgValues:[]string(nil), ReqObjEncryptionAlgValues:[]string(nil), ReqObjEncryptionEncValues:[]string(nil), TokenEndpointAuthMethodsSupported:[]string{"client_secret_basic"}, TokenEndpointAuthSigningAlgValuesSupported:[]string(nil), DisplayValuesSupported:[]string(nil), ClaimTypesSupported:[]string(nil), ClaimsSupported:[]string{"aud", "email", "email_verified", "exp", "iat", "iss", "locale", "name", "sub"}, ServiceDocs:(*url.URL)(nil), ClaimsLocalsSupported:[]string(nil), UILocalsSupported:[]string(nil), ClaimsParameterSupported:false, RequestParameterSupported:false, RequestURIParamaterSupported:false, RequireRequestURIRegistration:false, Policy:(*url.URL)(nil), TermsOfService:(*url.URL)(nil), ExpiresAt:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}
2018/03/3 23:32:34 http: Provider config sync no longer failing
2018/03/3 23:32:39 http: Provider config sync failed, retrying in 1s: Get https://console.lab.com/identity/.well-known/openid-configuration: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2018/03/3 23:32:40 http: Updating provider config: config=oidc.ProviderConfig{Issuer:(*url.URL)(0xc420175a80), AuthEndpoint:(*url.URL)(0xc420175b00), TokenEndpoint:(*url.URL)(0xc420175b80), UserInfoEndpoint:(*url.URL)(nil), KeysEndpoint:(*url.URL)(0xc420175c00), RegistrationEndpoint:(*url.URL)(nil), ScopesSupported:[]string{"openid", "email", "groups", "profile", "offline_access"}, ResponseTypesSupported:[]string{"code"}, ResponseModesSupported:[]string(nil), GrantTypesSupported:[]string(nil), ACRValuesSupported:[]string(nil), SubjectTypesSupported:[]string{"public"}, IDTokenSigningAlgValues:[]string{"RS256"}, IDTokenEncryptionAlgValues:[]string(nil), IDTokenEncryptionEncValues:[]string(nil), UserInfoSigningAlgValues:[]string(nil), UserInfoEncryptionAlgValues:[]string(nil), UserInfoEncryptionEncValues:[]string(nil), ReqObjSigningAlgValues:[]string(nil), ReqObjEncryptionAlgValues:[]string(nil), ReqObjEncryptionEncValues:[]string(nil), TokenEndpointAuthMethodsSupported:[]string{"client_secret_basic"}, TokenEndpointAuthSigningAlgValuesSupported:[]string(nil), DisplayValuesSupported:[]string(nil), ClaimTypesSupported:[]string(nil), ClaimsSupported:[]string{"aud", "email", "email_verified", "exp", "iat", "iss", "locale", "name", "sub"}, ServiceDocs:(*url.URL)(nil), ClaimsLocalsSupported:[]string(nil), UILocalsSupported:[]string(nil), ClaimsParameterSupported:false, RequestParameterSupported:false, RequestURIParamaterSupported:false, RequireRequestURIRegistration:false, Policy:(*url.URL)(nil), TermsOfService:(*url.URL)(nil), ExpiresAt:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}
2018/03/3 23:32:40 http: Provider config sync no longer failing
2018/03/3 23:32:40 cmd/main: Binding to 0.0.0.0:8080...
2018/03/3 23:32:40 cmd/main: not using TLS

[MartijnStraatman]

We are running on OpenStack and also experienced these issues. Is there any communication possible between workers and controllers? The time-out tells me network traffic on 443 is not possible.
Did you check the kubelet logs on the controllers (masters)

[bodgit]

As far as I can tell the Kubernetes cluster itself is working, I can use kubectl to inspect the cluster and I find all of the nodes and pods, etc. what seems to be the issue is that there's no iptables rule for forwarding traffic from port 443 to 32000 on the workers. I've reported this issue on the mailing list with no response other than @dmgaraway's "me too", a bit frustrating.

[squat]

@bodgit bare-metal installations of tectonic are not expected to run ingress on port 32000. Bare-metal installs should have a hostPort [0] based ingress controller listening directly on port 443 [1].

@dmgaraway can you manually curl https://console.lab.com/identity/.well-known/openid-configuration? If not, then the issue is that identity is not running correctly. If you can curl it then the issue is that console cannot reach identity for some reason, and without identity, console will not start correctly.

[0] https://github.com/coreos/tectonic-installer/blob/track-1/platforms/metal/tectonic.tf#L95
[1] https://github.com/coreos/tectonic-installer/blob/track-1/modules/tectonic/resources/manifests/ingress/hostport/daemonset.yaml

[bodgit]

@squat Sure, that's how I would expect it be running, however on my workers there's nothing listening on port 443 but I found it is listening on port 32000. For reference my thread on the google group is here: https://groups.google.com/d/topic/coreos-user/bsmWjYqdOCs/discussion

I found that https://console.lab.com:32000/identity/.well-known/openid-configuration worked fine, however the console pods are trying to use port 443 and failing. I've literally just ran the installer with the settings as noted in the above topic so I'm not sure how/why it's broken that way.

[bodgit]

Is there some debugging or commands I can run to help get you more info?

Edit: I found I still had this output in a terminal:

$ kubectl describe svc tectonic-lb --namespace=tectonic-system
Name:                   tectonic-lb
Namespace:              tectonic-system
Labels:                 component=ingress-controller
                        k8s-app=tectonic-lb
Annotations:            <none>
Selector:               component=ingress-controller,k8s-app=tectonic-lb
Type:                   NodePort
IP:                     10.3.47.177
Port:                   https   443/TCP
NodePort:               https   32000/TCP
Endpoints:              10.2.2.5:443
Port:                   http    80/TCP
NodePort:               http    32001/TCP
Endpoints:              10.2.2.5:80
Port:                   health  10254/TCP
NodePort:               health  32002/TCP
Endpoints:              10.2.2.5:10254
Session Affinity:       None
Events:                 <none>

which is what led me to try port 32000. So it looks like it's of type NodePort instead of HostPort?

[squat]

@bodgit I see. Your situation is a little different from @dmgaraway's. In your case, the tfvars posted in the google groups thread indicate that you deployed Tectonic onto vmware, which we treat as a separate platform from bare-metal. The configuration for the vmware ingress controller was changed very recently [0]. This PR changed the ingress controller to use a nodePort service on 32000, expecting you to have some load balancer proxying traffic from https://:443 to https://:32000. Without that proxy, console and identity will not work at port 443. For reference, AWS uses an ELB to perform this load balancing; bare-metal uses the hostPort strategy as we cannot provision a load balancer for the user.

One thing you can try to get your console working is to modify the issuer field in the tectonic-identity configMap so that it points to https://console.lab.com:32000/identity, then delete all tectonic-console pods and let them be spun up again. The console should now load correctly at https://console.lab.com:32000.

Alternatively, you can modify https://github.com/coreos/tectonic-installer/blob/track-1/platforms/vmware/tectonic.tf#L134 to be HostPort and create a new cluster. In any case, this is a legitimate regression for vmware, though seems to be a separate issue from @dmgaraway they say they are using bare-metal. @dmgaraway can you confirm the target platform?

cc @lander2k2

[0] #2911

[bodgit]

Thank you @squat for the explanation, I reverted #2911 and rebuilt the cluster and now the console works as expected. Is it worth me raising this as a separate issue?

[squat]

@bodgit yes please file as it’s own issue so we can track it. I’ll make sure the right people see it.

[bodgit]

I've raised #3080.

[squat]

@bodgit thanks.

@dmgaraway please let share your tfvars file. If you are also running on vmware, then we can close this issue in favor of #3080.

[dmgaraway]

First I want to thank you for taking time to respond (I've been stuck with no help for weeks). I've tried to install on 2 bare metal HP blades as well as virtualbox and vmware. Question, if i have virtual machines on Vmware ESX6.0 and use the tectonic bare metal installer, why would this problem occur? is there something about how the virtual machine interacts with the hypervisior that causes the issue?

The current build I have up was installed on Vmware (i was sick of dealing with the slowness of the bare metal instal). I followed the Tectonic installer but I dont have anything running on port 32000. is there a documented way i should be installing on vmware esx6.0?

[root@matchbox ~]# kubectl describe svc tectonic-lb --namespace=tectonic-system
Name: tectonic-lb
Namespace: tectonic-system
Labels: component=ingress-controller k8s-app=tectonic-lb
Annotations:
Selector: component=ingress-controller,k8s-app=tectonic-lb
Type: ClusterIP
IP: 10.3.117.41
Port: http 80/TCP
TargetPort: 80/TCP
Endpoints: 172.26.3.52:80
Port: https 443/TCP
TargetPort: 443/TCP
Endpoints: 172.26.3.52:443
Session Affinity: None
Events:

{
"tectonic_admin_email": "admin@lab.com",
"tectonic_admin_password": "xxxxxxx",
"tectonic_base_domain": "unused",
"tectonic_cluster_cidr": "10.2.0.0/16",
"tectonic_cluster_name": "123",
"tectonic_container_linux_version": "1576.4.0",
"tectonic_dns_name": "",
"tectonic_kube_apiserver_service_ip": "10.3.0.1",
"tectonic_kube_dns_service_ip": "10.3.0.10",
"tectonic_kube_etcd_service_ip": "10.3.0.15",
"tectonic_license_path": "./license.txt",
"tectonic_metal_controller_domain": "node1.lab.com",
"tectonic_metal_controller_domains": [
"node1.lab.com"
],
"tectonic_metal_controller_macs": [
"00:50:56:b4:75:94"
],
"tectonic_metal_controller_names": [
"node1"
],
"tectonic_metal_ingress_domain": "node2.lab.com",
"tectonic_metal_matchbox_ca": "-----BEGIN CERTIFICATE-----\nMIIFDTCCAvWgAwIBAgIJAPcNPHLtKcj5MA0GCSqGSIb3DQEBCwUAMBIxEDAOBgNV\nBAMMB2Zha2UtY2EwHhcNMTgwMjE4MDM1MjM1WhcNMjgwMjE2MDM1MjM1WjASMRAw\nDgYDVQQDDAdmYWtlLWNhMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA\np0dS0ZSsz8/Xo4EFWRgdQg2H2MBhWFaE/s30CzzamKf5DfLwJJtbLlgi2cv2Hs60\nU/zZ+2VEDI/ih2h0fc4JIejQzcfM0R1YswnEhRab9mJT4a7CS7wtdVKXyRrRD6c+\ns/KVg1d2FRTbvuVKtzhvT4Bi2RX5jvGCcl05sXTdDjsYYTI0AZUvxLMuVrrzbonV\nnv/u2yEGrFByv/C/ISN5VSGMylLwCdoQj7E586Yfjbg8w7+/dNVbrqtUWs3eI9/D\nem34NTn8Vl6uQT27v1t+wYOKR6staUFhU2DwFiwJf2wfYSnFaG47kFTFDlemYzeS\nLTjRPVkP9b/y2iX1XLiUGxaqTQnW9qDsoSz7Pil5Oo9K7QhNHimu1nHUz3AuXZr8\nBGa3Qss8vwdfuj/UNge0etbbT1GfgKn2KlvHB2FwR+Ig51xbjUzGzmKr8VTYFf4h\niEL7m3Fr3pfMYIsSRe5CgdCdsWXXPBgAuLWV3ZfITkJtJGVsYkdcC212W9IhMrLJ\nOnqzyaIU0j5amLS0sTTQL3UKoiujDCinm92ag5njL0SdEoBgfFdYInAf1CCv7jp0\n4joh11eqOhNzkleRr68rXTqu5U8cofaAabyM+iPJ3h4DyZ4NJPB/KoJDumAV/u69\nxqvVkNrrdbCXT9yBQoHQgANjtE42zRmzZSpF4ATXcK8CAwEAAaNmMGQwHQYDVR0O\nBBYEFBhv3TDIqtV1WmpRHu+5Z5Fm3Ga1MB8GA1UdIwQYMBaAFBhv3TDIqtV1WmpR\nHu+5Z5Fm3Ga1MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0G\nCSqGSIb3DQEBCwUAA4ICAQA8QbJj1k5TZcH00pSW+Gn5AVoG1zzmgldODzRDBg2N\n1crX3y6q/tlmoDhubfEzWwlu8dIp3Xcj12kD5fOW9L7TfdpCL+N1KQU73gXeUaNx\n03PXf+eMp9XHlYqFIZsGGH9OQ67Ug1BOxoXwV+1M37uPZpa0vC01TfACQH4bQdw1\n/xoNsf30pxDLYqoCxTnwt3JqOuwbJVB5yhSfd3BVL5/FEc/4RdtOgaAoGrEv0DN9\nABpiX0M9LK6Y1hiCRWcPmNlFbiz/vhyOZ1KCX9C6kMSQ1NMezX0AU9VmXBT01r0y\nVhcx1NxQjiHErl9030CGjwgbUzTnMxDkfXSkOqqdyw/UBZ+8TUB09ab4c5ixtRH8\ni1b7qEXOnrQ6hlMYQsfidnjxOHPoVpc1AwyxdHW5FQBcGZLiFXJtvwYSbRQarReh\n2Fri//XvtTamujQ5ATAzxXCEECICJXsDRmjHwkE667zWx0abGTnrSinrePf1iRnX\nwboIlV1PaEZ1jeNBJkEkF3t/TT7aqIhejoxu7E4Gd4zSubYeiln7Zsom22hGR70F\nRu9DklmdAnP01tVd1+Q69z74f4vlrgZ3yJO10rN9ReCdyoBbZ5i4F39Dj3KYGiE4\nThSUs3tySXDR8j4ontcEXH0Or3XgZwBYz8NQ2btOUzaP5CfAzKJ5fJFQdhbaYIwF\n6g==\n-----END CERTIFICATE-----\n",
"tectonic_metal_matchbox_client_cert": "-----BEGIN CERTIFICATE-----\nMIIEYDCCAkigAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwEjEQMA4GA1UEAwwHZmFr\nZS1jYTAeFw0xODAyMTgwMzUyMzVaFw0xOTAyMTgwMzUyMzVaMBYxFDASBgNVBAMM\nC2Zha2UtY2xpZW50MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArDYP\nf8lxVRQFIi6P/XbGZFp9NKvW8leu6bL8dDP/AcAk9CSHzehAFcUpU33ZB4jVzs1+\n4mRRvuBEzh9SopSzWj3eeVsthQWnDzxthSe0D8rBc5A8lrAGHE9zdcJLm2LzSz8i\nJqMbmdqR0W41A8QNwZINT/8/khDcNVIhCX+xKrFqEyNFi5JuXxi+NkvYOUY1mCzw\nzZmwppzOB0NK3BjAW5DBhLGqgLbw6B1n6O47Kfp3TRanown6c553a6q/ab5VRfFb\n2IDm/zRse38huDLmetRFZrCh42TKhsiJQpJvPDAOvgxpGaeb75hEk4Y4bSVXuk9R\nQLE7AYQh7L7063tCrwIDAQABo4G7MIG4MAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEB\nBAQDAgeAMDMGCWCGSAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQg\nQ2VydGlmaWNhdGUwHQYDVR0OBBYEFO7QbYTdIDIOezgU9+7bvdwXitVCMB8GA1Ud\nIwQYMBaAFBhv3TDIqtV1WmpRHu+5Z5Fm3Ga1MA4GA1UdDwEB/wQEAwIF4DATBgNV\nHSUEDDAKBggrBgEFBQcDAjANBgkqhkiG9w0BAQsFAAOCAgEADXVdxY32MVdra/tZ\nPshlCm6Xk6jBTnsXjlEJCu+GPWGYxUJIne/jI0qpohOIkyhBHuaLhRo5bnYcP8D/\nbwH7K4wvsmFWQRrNng0frqcCGASBK9cPPGCe1sb+Y7Wr84y5vBteIaOdApt/Tv30\n/syL536TrqBE+47mwsPDjODqnLCfVg3OQh7ScYyT1qE4yVVnnD2/nWcHfisWOz0/\nTlPnoIgwsyK16eiPeRqfjXYtJWrA7W+dDpITnTsxhzAEvWFMIHFrGuGm23Yfnovh\nk5QfrHZs7nElzO9Vqh/XFIalIq33ruYSy5jaZBYvaZ+MLVuGcKEAXjMCem/LLftv\nNn6f+X2GF7cwYesrUMSjJdoBNtDn827346W1j6AzrP75DCtbveoCSdq3Accom8wn\nlK0X1S8z3delHRdMnq0A+gOCm0FxbYdNDfuAUMbI+rOPEgwmfMSCqa6Vh5Em3aqh\n6phbi+NcGqkc+EySLkbTbqzruIETDqBT6+qVPwz07Z5kxRvcHTih1HTinuPJ2zl3\n7Xv7F6WInq1XvQKkNVj6PJfbLPOB63y6hD74Zp8hubsV9Ff7zxKA3SqVh5LFDhTq\nlN6jPLqXDEkduvDsSKLL39KH3R7duSO8sNid7KsU+ouwKNgjPTohYxxeLcTHvl+f\nLIva4Q14z8u0fXjd72pn8+HPrxs=\n-----END CERTIFICATE-----\n",
"tectonic_metal_matchbox_client_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEArDYPf8lxVRQFIi6P/XbGZFp9NKvW8leu6bL8dDP/AcAk9CSH\nzehAFcUpU33ZB4jVzs1+4mRRvuBEzh9SopSzWj3eeVsthQWnDzxthSe0D8rBc5A8\nlrAGHE9zdcJLm2LzSz8iJqMbmdqR0W41A8QNwZINT/8/khDcNVIhCX+xKrFqEyNF\ni5JuXxi+NkvYOUY1mCzwzZmwppzOB0NK3BjAW5DBhLGqgLbw6B1n6O47Kfp3TRan\nown6c553a6q/ab5VRfFb2IDm/zRse38huDLmetRFZrCh42TKhsiJQpJvPDAOvgxp\nGaeb75hEk4Y4bSVXuk9RQLE7AYQh7L7063tCrwIDAQABAoIBAHWhn8ir8wwoVrFY\nmOaLoUyfOvMlhfeXyVS8/BDsC35In3VdFp6hzFSSjn4Hcp3g7BsM52UBHY2CZ71c\nS/feFvzHUhYXH+rDR6/ymOThj+9Rkf68QYa0yhfAdZ+0jiyf1IxLnapCs/HOCNjD\nrNbbroHQn+NK+UNu3NxXM1XnsT71PZeJni5vixUEsOtm0/EG76wopimc8zBtMPVj\njLfv/R7IeGL5c961XBIozpxOhqWvj463eBeN5XLGNmz3xT4NthFbExcReOX6xntx\nnNIjpPS5LEr88Q+GYSdbkdAhD2RnAwQ7VjSzJT9ufXmNXb9G79vgC5DAtlDVgntc\nVDfQ0nECgYEA32riBObZnuSer+8otfyCawfxTNsmNim5rNRrs8FDJyPyg1kGeLHk\ngrbiw3D0/hmQ+oYu+FePL25mpnHdwxWxeepe/+rfLWX1pLonpPThl8P/R3pxAVqf\nIWk/ureKqKZMK2GXkb09Ruh/LLh0aow9Vj3iG672X94h8INNCeDXdsMCgYEAxVNt\n57fwJYpgOCkavhf3Vb+dnC3/nVFdxlxSeLIzzRSt/yeNX3ENaGtTRh3LuWigWzIT\n36g5ckCFqa6ccdvINbS1BdnGAQ/LUY+59ehiEXcxw9rhNlkgsn7Ts+JzpKd9ZAvG\nqS1a0tKgGtFX07sRZAZWaBVBt8y5eX70Lez//aUCgYBRMj0WXve0QY5Sjm6TRUP4\nthhCQFsw0BVE9KZ328MWFIBssAwkDTLpNqJPOVwwhQYlpmKnqtrE+DCiSTu5TMcF\nceI4zBl0HFwcE/OYhc7/IyJBzgQH4/F1aRTaPR5NkLKaCYvWUZpVjOW7UQYuOu2H\nzFHCkeHEkaxwBqgW8PodlQKBgQCQEOFlFD4Yqnalih0kPIsv575CkLLXQNieQjlU\nNYbsa/S9HTtyOy/21KTvxEFBFMo+yskHueJ3L/Rmdj1yQ2xhCZZAt0aA7+8ZlT3H\n8+nlpoiG3f8fluLwbvhRlTupMz+xE4fvbFWrJmwOKFlfFG0WAiqDw0E/2BByS+Yq\nwaYEoQKBgBrGwWPN+cQAGpZveumampBDyiGvuewjtQgWCArLmXlAROHyI+m6aBYy\nKzqWd8aoMQmdztoiaK66WVmkvqPONM1udkiqMgrIi+0pghy/6iM/xF0rOUPkFD3M\nb9eOiv++ML62IqwCs40yL4KI7TjCNuijojbWi7xoRmj9mgzEizqd\n-----END RSA PRIVATE KEY-----\n",
"tectonic_metal_matchbox_http_url": "http://172.26.3.50:8080",
"tectonic_metal_matchbox_rpc_endpoint": "172.26.3.50:8081",
"tectonic_metal_worker_domains": [
"node2.lab.com"
],
"tectonic_metal_worker_macs": [
"00:50:56:b4:15:f0"
],
"tectonic_metal_worker_names": [
"node2"
],
"tectonic_pull_secret_path": "./pull_secret.json",
"tectonic_service_cidr": "10.3.0.0/16",
"tectonic_ssh_authorized_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDh9aI2L1Zrt6q1HQAweiuMelBOq2fZGMuV89jzw2BVJUw5CFHSFatk2zzVQ8+LZwo87NAAE+6qtD1YyfSWqOJbE0VgEHiMPKgE4dq5izoc5PaZAh9OOamSuge18k0MgYHvHLVi4xBkM0av+lK3u+HBLqfYfL95otjIg2Rh/sECA9bCag1aJV1M5wIVPpUqrLf68nKSb1XJ6S4hGJCWyPWdMHbQLibj3KICevjtoxyFOvQoJISEZiWfgfm5gLYdQlw5tr0nc0m+yil5+Pq2eElpX7+pP5wBkUIg3JnvBJbPgSx0mH9t8/9KQeNMjxYaY2oFq+nhUnrXJ5y6+yPC2fs3 root@matchbox.lab.com"

[squat]

@dmgaraway Tectonic does have a VMware specific install, which is what @bodgit followed, though your approach should also work, however it is distinct.

The bare metal install flow never expects ingress on port 32000. On metal, ingress works by scheduling a daemonset to listen on all worker nodes on port 443.

Your tfvars above do not quite match the logs you posted. In the tfvars you specify the ingress domain as node2.lab.com, whereas in the logs I can see the console trying to contact identity at console.lab.com. The logs are from a cluster with a different configuration, no?

Assuming the ingress domain is node2.lab.com, the console should request the OIDC configuration at https://node2.lab.com/identity/.well-known/openid-configuration.

Please try manually curl-ing this URL. If this does not work, then there is likely a problem either with the ingress configuration or with the identity server. If it does work, but console cannot load the URL, then we need to dig a little more as the URL is accessible externally but not by console.

[squat]

@dmgaraway any update? Otherwise I'll close for now until we confirm you are still experiencing this issue.

[dmgaraway]

I have since tried to get an install going on virtualbox just like the video on the tectonic installer page. everything works when i select "bare metal" but i still run into the same issue stated in this thread. What am i doing wrong or is there another option to select when trying to simulate bare metal on virtual using virtualbox?

[squat]

Can you post the result of curling the OIDC configuration from a worker node? This URL should include the ingress domain i.e. it should be the same URL the console would use to access the configuration

[dmgaraway]

I found something interesting, when i get a list of nodes either one has an external IP. Isn't the ingress controller supposed to have an external IP?

NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node1.lab.com Ready master 10d v1.8.4+coreos.0 Container Linux by CoreOS 1632.3.0 (Ladybug) 4.14.19-coreos docker://Unknown
node2.lab.com Ready node 10d v1.8.4+coreos.0 Container Linux by CoreOS 1632.3.0 (Ladybug) 4.14.19-coreos docker://Unknown