network unreachable #528

Toerktumlare · 2017-01-16T10:02:25Z

I don't know if this is an issue or not

i pulled the Vagrant image and started it. Was able to login and start a project. But when trying to pull an image from docker hub i get network unrachable.

I have tried setting http_proxy in the vagrant and if i ssh into the machine i can pull and push images with docker inside the openshift-vagrant. But when using the openshift gui i cannot pull images.

i have tried setting the http_proxy in the docker-opts /etc/sysconfig/docker without any luck.

is this a known issue?

stevekuznetsov · 2017-01-16T12:59:47Z

I can't seem to reach that host from my machine -- can you ping it from inside the VM? How are you starting OpenShift? I suggest if possible doing a containerized start with oc cluster up.

Toerktumlare · 2017-01-16T14:33:42Z

i pulled the vagrant image and did a
vagrant up
then i accessed the gui at
localhost:8443
and logged in. and then i get that screenshot above

Then i sshd into the machine and did a oc cluster down and then tried to start the cluster with oc cluster up i then get an error saying that ports are still in use

-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.3.0 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... FAIL
   Error: a port needed by OpenShift is not available
   Caused By:
     Error: ports in use: [80 443 4001 7001 8443 10250 53]

the cluster doesn't seem to go down

stevekuznetsov · 2017-01-16T14:40:53Z

Ah, the pre-installed cluster is not created using oc cluster up -- you'll need to:

$ sudo systemctl stop openshift
$ oc cluster up

The systemctl unit name might be origin, not openshift -- I can't remember off the top of my head right now.

Toerktumlare · 2017-01-17T16:35:37Z

stopped the service with

$ sudo systemctl stop origin
$ oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.3.0 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... FAIL
Error: a port needed by OpenShift is not available
Error: ports in use: [80 443]

so apparently some http-service is not going down when i stop the service.
i can start the service again with:

$ sudo systemctl start openshift

and i can login to the gui but i still have the same problem, can't pull images from docker.hub with the gui

stevekuznetsov · 2017-01-17T18:19:31Z

Can you figure out if there is some rogue process still going?

sudo systemctl stop openshift
ps -ef | grep -e openshift -e origin
lsof -i :80
lsof -i :443

Toerktumlare · 2017-01-18T14:38:38Z

destroyed the vagrant, started it again. took down the service with:

sudo systemctl stop origin

started cluster with

[vagrant@localhost ~]$ oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.3.0 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ...
   Using nsenter mounter for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ...
   Using 10.0.2.15 as the server IP
-- Starting OpenShift container ...
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
FAIL
   Error: cannot access master readiness URL https://10.0.2.15:8443/healthz/ready
   Details:
     Last 10 lines of "origin" container log:
     I0118 13:36:34.697420   15833 ensure.go:209] Created default security context constraint restricted
     I0118 13:36:34.701958   15833 ensure.go:209] Created default security context constraint anyuid
     I0118 13:36:34.707975   15833 ensure.go:209] Created default security context constraint hostnetwork
     I0118 13:36:35.544447   15833 trace.go:61] Trace "Create /oapi/v1/namespaces/default/rolebindings" (started 2017-01-18 13:36:34.711919024 +0000 UTC):
     [11.305µs] [11.305µs] About to convert to expected version
     [35.204µs] [23.899µs] Conversion done
     [819.153245ms] [819.118041ms] About to store object in database
     [832.408942ms] [13.255697ms] Object stored in database
     [832.417261ms] [8.319µs] Self-link added
     [832.483773ms] [66.512µs] END

stevekuznetsov · 2017-01-18T15:05:19Z

@csrwng could you help a little with oc cluster up failures?

csrwng · 2017-01-18T15:25:20Z

Make sure that the firewall allows access to port 8443. Also after cluster up fails, can you try curl'ing that URL and see why it's not accessible ? Is the container (origin) still running?

naveenkumartangallapally · 2017-03-23T14:30:35Z

hi csrwng ,
i am also facing the same issue.not able to do oc cluster up.
[root@master ~]# oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for registry.access.redhat.com/openshift3/ose:v3.4.1.10 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... FAIL
Error: a port needed by OpenShift is not available
Caused By:
Error: ports in use: [10250]

and we have stopped origin and dns service also,but getting same error

csrwng · 2017-03-23T14:32:20Z

@naveenkumartangallapally try running sudo netstat -tnlp to see what process is using port 10250.

naveenkumartangallapally · 2017-03-23T14:33:18Z

i got below o/p

[root@master ~]# netstat -tnlp
Active Internet connections Proto Recv-Q Send-Q Local Address tcp 0 0 127.0.0.1:2380 tcp 0 0 0.0.0.0:111 tcp 0 0 0.0.0.0:20048 tcp 0 0 0.0.0.0:8053 tcp 0 0 0.0.0.0:22 tcp 0 0 127.0.0.1:25 tcp 0 0 0.0.0.0:8443 tcp 0 0 0.0.0.0:59261 tcp 0 0 0.0.0.0:2049 tcp 0 0 192.168.57.90:2379 tcp6 0 0 :::111 tcp6 0 0 :::20048 tcp6 0 0 :::22 tcp6 0 0 ::1:25 tcp6 0 0 :::2049 tcp6 0 0 :::9090 tcp6 0 0 :::10250 tcp6 0 0 :::48714 [root@master ~]# (only servers)
Foreign Address State PID/Program name
0.0.0.0:* LISTEN 1362/etcd
0.0.0.0:* LISTEN 1/systemd
0.0.0.0:* LISTEN 1386/rpc.mountd
0.0.0.0:* LISTEN 29970/openshift
0.0.0.0:* LISTEN 1869/sshd
0.0.0.0:* LISTEN 2557/master
0.0.0.0:* LISTEN 29970/openshift
0.0.0.0:* LISTEN 1373/rpc.statd
0.0.0.0:* LISTEN -
0.0.0.0:* LISTEN 1362/etcd
:::* LISTEN 1/systemd
:::* LISTEN 1386/rpc.mountd
:::* LISTEN 1869/sshd
:::* LISTEN 2557/master
:::* LISTEN -
:::* LISTEN 1/systemd
:::* LISTEN 29031/openshift
:::* LISTEN 1373/rpc.statd

csrwng · 2017-03-23T14:34:13Z

so it looks like you're running openshift (pid 29031) ... which is causing a conflict

naveenkumartangallapally · 2017-03-23T14:36:48Z

can u help me to fix the issue plz

csrwng · 2017-03-23T14:37:57Z

kill existing openshift related processes:
killall openshift or
kill 29031 29970

and then run oc cluster up

csrwng · 2017-03-23T14:38:29Z

or systemctl stop openshift

naveenkumartangallapally · 2017-03-23T14:40:16Z

getting same as error as below.
[root@master ~]# killall openshift
[root@master ~]# kill 29031 29970
-bash: kill: (29031) - No such process
-bash: kill: (29970) - No such process
[root@master ~]# oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for registry.access.redhat.com/openshift3/ose:v3.4.1.10 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... FAIL
Error: a port needed by OpenShift is not available
Caused By:
Error: ports in use: [8443 10250]
[root@master ~]# systemctl stop at^C
[root@master ~]# systemctl stop atomic-openshift-master
[root@master ~]# oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for registry.access.redhat.com/openshift3/ose:v3.4.1.10 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... FAIL
Error: a port needed by OpenShift is not available
Caused By:
Error: ports in use: [10250]
[root@master ~]#

csrwng · 2017-03-23T14:41:22Z

try systemctl stop atomic-openshift-node as well

naveenkumartangallapally · 2017-03-23T14:56:45Z

once the cluster is up dashboard is not accessible.might be because node is down?

dashboard error:
{"error":"invalid_request","error_description":"The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed.","state":"{"then":"/","nonce":"1490280762256-26489848221538182040344017095426601797094232054395413048189320121903021386031219"}"}

csrwng · 2017-03-23T15:09:47Z

Do you mean the web console? That doesn't require the node to be running. But the node is likely running anyway, since 'cluster up' runs the all-in-one which is master+node.

stevekuznetsov · 2017-03-23T15:10:19Z

I would also ask -- just to make sure before much debugging happens, can use you oc cluster up from your local environment? What do you need from the VM?

naveenkumartangallapally · 2017-03-23T15:21:24Z

My goal is to configure docker-registry, for which there were two containers created with name "docker-registry-1" & "router-1-deploy" when i used quick installation method. However, both the containers were in Error state... One of the blogs i browsed said the containers were in Error state because the oc cluster is not running, which is why i tried to make oc cluster.. Please correct me if I am wrong.

Just to add, when i rebooted master and node, i was able to login to web console, but the oc cluster was again in "not running" state. Are docker registry and router creation linked with openshift cluster status???

stevekuznetsov · 2017-03-23T15:40:37Z

Right, but what reasons are you using for running vagrant versus oc cluster up on the local host (not in the Vagrant VM)?

naveenkumartangallapally · 2017-03-23T15:52:01Z

if node is down?what is function of oc cluster?

stevekuznetsov · 2017-03-23T16:53:48Z

If you are trying to achieve a cluster with a router and a registry, one option is to install docker on your local system, install the oc tool on your local system, and use oc cluster up. The VM is a little more heavyweight approach and may be more error prone in some situations. If you need a RH system and the RH-supported Docker, you may look into using minishift.

naveenkumartangallapally · 2017-03-23T17:01:36Z

docker and oc are already installed and working fine. But after the oc cluster is up, we are unable to access the dashboard. I dont think we need minishift.

Deprecated output of "#oc cluster up" is as follows:

OpenShift server started.
The server is accessible via web console at:
https://10.0.2.15:8443

You are logged in as:
User: developer
Password: developer

To login as administrator:
oc login -u system:admin

[root@master ~]# oc get pods
No resources found.
[root@master ~]# oc get nodes
No resources found.
Error from server: User "developer" cannot list all nodes in the cluster
[root@master ~]#

However, we are unable to access the dashboard with "https://10.0.2.15:8443". Please let me know what are the nodes which are kept in openshift cluster. Is it the master and nodes OR containers created on the node??

stevekuznetsov · 2017-03-23T17:14:42Z

Ok. Again. please be advised that this codebase is deprecated and not supported. The currently supported method for starting a VM with an OpenShift cluster inside of it is minishift. I will not be able to spend too much time debugging this.

So the cluster is up and running fine inside the VM? If you use oc to make actions, they work? Can you reach the console from inside the VM? Maybe something is wrong with networking for the VM? Is it forwarding :8443?

naveenkumartangallapally · 2017-03-23T17:35:15Z

The cluster is up and running fine inside the VM?

Yes, it is...

[root@master ~]# oc cluster status
The OpenShift cluster was started 31 minutes ago

Web console URL: https://10.0.2.15:8443

Config is at host directory /var/lib/origin/openshift.local.config
Volumes are at host directory /var/lib/origin/openshift.local.volumes
Data will be discarded when cluster is destroyed
[root@master ~]#

If you use oc to make actions, they work?

Yes.. they do..

[root@master ~]# oc get pods -n default
NAME                       READY     STATUS              RESTARTS   AGE
docker-registry-1-deploy   0/1       ContainerCreating   0          35m
router-1-deploy            0/1       ContainerCreating   0          35m
[root@master ~]#

Is it forwarding :8443?

Yes..

[root@master ~]# telnet 10.0.2.15 8443
Trying 10.0.2.15...
Connected to 10.0.2.15.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
[root@master ~]#

stevekuznetsov · 2017-03-23T18:39:57Z

Can you hit the console from inside the VM?

@jwforres @spadgett have you seen this error from the console before? What does it mean:

dashboard error:
{
  "error": "invalid_request",
  "error_description": "The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed.",
  "state": "{
    "then": "/",
    "nonce":"1490280762256-26489848221538182040344017095426601797094232054395413048189320121903021386031219"
  }"
}

spadgett · 2017-03-23T19:48:26Z

@naveenkumartangallapally Are you connecting to the web console using a hostname that is not the public hostname? If so, you need add host to the allowed redirect URIs for the web console

$ oc login -u system:admin
$ oc patch oauthclient/openshift-web-console -p '{"redirectURIs":["https://<console-host>:<console-port>/"]}'

stevekuznetsov · 2017-03-23T20:50:07Z

I thought oc cluster up would patch that?

csrwng · 2017-03-23T20:53:09Z

It should do that. I suspect there's something else going in this case.

naveenkumartangallapally · 2017-03-24T11:46:49Z

i patched up Web console URL: https://10.0.2.15:8443

[root@master ~]# oc patch oauthclient/openshift-web-console -p '{"redirectURIs":["https://10.0.2.15:8443/"]}'
"openshift-web-console" patched
[root@master ~]#
but still i am not able to access web ui

naveenkumartangallapally · 2017-03-24T14:30:20Z

getting below error while creating docker registry volume:

[root@master ~]# oadm registry --service-account=registry --config=/etc/origin/master/admin.kubeconfig --images='registry.access.redhat.com/openshift3/ose-${component}:${version}' --mount-host=/root/registry

error: error getting client: Get https://master.example.com:8443/api: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, localhost, openshift, openshift.default, openshift.default.svc, openshift.default.svc.cluster.local, 10.0.2.15, 10.129.0.1, 127.0.0.1, 172.17.0.1, 172.30.0.1, 192.168.57.90, 192.168.58.90, not master.example.com

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

network unreachable #528

network unreachable #528

Toerktumlare commented Jan 16, 2017

stevekuznetsov commented Jan 16, 2017

Toerktumlare commented Jan 16, 2017 •

edited by stevekuznetsov

Loading

stevekuznetsov commented Jan 16, 2017

Toerktumlare commented Jan 17, 2017

stevekuznetsov commented Jan 17, 2017

Toerktumlare commented Jan 18, 2017

stevekuznetsov commented Jan 18, 2017

csrwng commented Jan 18, 2017

naveenkumartangallapally commented Mar 23, 2017

csrwng commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

csrwng commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

csrwng commented Mar 23, 2017

csrwng commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

csrwng commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

csrwng commented Mar 23, 2017

stevekuznetsov commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

stevekuznetsov commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

stevekuznetsov commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

stevekuznetsov commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017 •

edited by stevekuznetsov

Loading

stevekuznetsov commented Mar 23, 2017

spadgett commented Mar 23, 2017

stevekuznetsov commented Mar 23, 2017

csrwng commented Mar 23, 2017

naveenkumartangallapally commented Mar 24, 2017

naveenkumartangallapally commented Mar 24, 2017 •

edited by stevekuznetsov

Loading

network unreachable #528

network unreachable #528

Comments

Toerktumlare commented Jan 16, 2017

stevekuznetsov commented Jan 16, 2017

Toerktumlare commented Jan 16, 2017 • edited by stevekuznetsov Loading

stevekuznetsov commented Jan 16, 2017

Toerktumlare commented Jan 17, 2017

stevekuznetsov commented Jan 17, 2017

Toerktumlare commented Jan 18, 2017

stevekuznetsov commented Jan 18, 2017

csrwng commented Jan 18, 2017

naveenkumartangallapally commented Mar 23, 2017

csrwng commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

csrwng commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

csrwng commented Mar 23, 2017

csrwng commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

csrwng commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

csrwng commented Mar 23, 2017

stevekuznetsov commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

stevekuznetsov commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

stevekuznetsov commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017

stevekuznetsov commented Mar 23, 2017

naveenkumartangallapally commented Mar 23, 2017 • edited by stevekuznetsov Loading

stevekuznetsov commented Mar 23, 2017

spadgett commented Mar 23, 2017

stevekuznetsov commented Mar 23, 2017

csrwng commented Mar 23, 2017

naveenkumartangallapally commented Mar 24, 2017

naveenkumartangallapally commented Mar 24, 2017 • edited by stevekuznetsov Loading

Toerktumlare commented Jan 16, 2017 •

edited by stevekuznetsov

Loading

naveenkumartangallapally commented Mar 23, 2017 •

edited by stevekuznetsov

Loading

naveenkumartangallapally commented Mar 24, 2017 •

edited by stevekuznetsov

Loading