Restoring Unstable Cluster Setup on iCubParis002

[jeljaik]

Hello guys,

After solving many issues we had with the hardware of our two iCubs, I was trying today to setup the red ball demo, however I found the blades status quite unstable. I might need a bit of assistance on different issues that I will try to keep in this single issue. Let me start with the following problem:

When I do ssh icub-b1 this is what I get:

ssh icub-b1
Linux icub-netboot 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Exporting iCub variables for: iCubParis02

After which the terminal shows:
icub@no instead of the expected icub@icub-b1

If I try to do sudo hostname icub-b1 this is what I get:

icub@no:~$ sudo hostname icub-b1
sudo: unable to resolve host icub-b1

I have the same problem with all three blades. Can you help me diagnose this and the other issues I will mention after this one is solved?

As an additional information:

When I first ssh on any of the blades, this is what I get (notice the icub-netboot)

icub@icub-desktop:~$ ssh icub-b1
Linux icub-netboot 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
-bash: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8)
-bash: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8)
Exporting iCub variables for: iCubParis02
-bash: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8)

Which reveals also another issue we have with the locale settings, although I'm not sure it's related to the previous issue. One at a time I guess :)

[mbrunettini]

On one blade, please do the following commands:
cat /etc/issue
cat /etc/rc.local
cat /etc/hostname
And please post here the results

[jeljaik]

Right away!

cat /etc/issue
Debian GNU/Linux stretch/sid \n \l

cat /etc/rc.local
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser run level
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

/root/set-hostname-from-net.sh
mount -a -t nfs

/root/force_ntp_update.sh
exit 0

[mbrunettini]

Thanks, can you please issue also
cat /root/set-hostname-from-net.sh

[jeljaik]

#! /bin/sh
​_HOSTNAME_​=""
LEASES_FILE_PATH="/var/lib/dhcp"

get_hostname_from_dns() {
 if [ "$1" != "" ]
 then
    IP_ADDR=$( ip addr show $1 | grep inet | grep $1 | awk '{ print $2 }' | sed -e 's/\/24//')
    if [ "$IP_ADDR" != "" ]
    then
        ​_HOSTNAME_​=$( host $IP_ADDR |  awk '{ print $5 }' | cut -d'.' -f1 )
    fi
 fi
}

get_hostname_from_leases() {
 if [ "$1" != "" ]
 then
   if [ -f "${LEASES_FILE_PATH}/dhclient.${1}.leases" ]
   then
     ​_HOSTNAME_​=$( grep "host-name" ${LEASES_FILE_PATH}/dhclient.${1}.leases | awk '{ print $3 }' | sed -e 's/"//g' -e 's/;//g' )
   fi 
 fi
}

get_hostname_from_dns eth0
if [ "${​_HOSTNAME_​}" = "" ]
then
    get_hostname_from_leases eth0
fi
if [ "${​_HOSTNAME_​}" = "" ]
then
    get_hostname_from_dns eth1
fi
if [ "${​_HOSTNAME_​}" = "" ]
then
    get_hostname_from_leases eth1
fi
if [ "${​_HOSTNAME_​}" = "" ]
then
    exit 1
fi
echo $​_HOSTNAME_​
hostname "$​_HOSTNAME_​"
exit $?

[mbrunettini]

It looks like a strange mix..
A script created for the Wheezy release on Debian sid(older).. really odd..
can you try to
sudo /root/set-hostname-from-net.sh

[jeljaik]

For example on icub-b1 I get

icub@no:~$ sudo /root/set-hostname-from-net.sh
sudo: unable to resolve host no
[sudo] password for icub:
no

If I exit and ssh again I get the same icub@no

[mbrunettini]

ok, then:
su -
/root/set-hostname-from-net.sh

[jeljaik]

icub@no:~$ sudo su
sudo: unable to resolve host no
bash: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8)
root@no:/home/icub# /root/set-hostname-from-net.sh 
no

[mbrunettini]

NOT sudo, but
su -

[jeljaik]

Oops, sorry

icub@no su - /root/set-hostname-from-net.sh
No passwd entry for user '/root/set-hostname-from-net.sh

[mbrunettini]

Please read carefully, thanks:
su -
/root/set-hostname-from-net.sh
Usually, 2 LINES = 2 SEPARATE COMMANDS

[jeljaik]

Relax, no need for caps ;) I was having some issue using just su

Here you go,

icub@no:~$ su - 
Password: 
-su: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8)
-su: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8)
root@no:~# /root/set-hostname-from-net.sh 
no

[jeljaik]

ping! 🔔🙇

[mbrunettini]

Cut'n'paste the following command and execute on icub-b1
host 10.0.0.51
and copy here the result.
Thanks

[jeljaik]

Sure, thanks

icub@icub-desktop:~$ ssh icub-b1
Linux icub-netboot 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
-bash: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8)
-bash: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8)
Exporting iCub variables for: iCubParis02
-bash: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8)
icub@no:~$ host 10.0.0.51
;; connection timed out; no servers could be reached```

[jeljaik]

Thanks @lornat75 for the suggestion.
So far I have tried option 1 and rebooted icub-srv but then I can’t ssh icub-bx getting the following message:

icub@icub-srv:~$ ssh icub-b1
ssh_exchange_identification: Connection closed by remote host

So I went back to the original wheezy-x86_64 and it's just like before. For option 2 how can we proceed so that you provide me this version?

[lornat75]

Assuming the backup copy of wheezy-x86_64 (wheezy-x86_64.old) was taken before changing the system, I fear solution 2) would not be much better. Let's see if @mbrunettini has some suggestions on how to fix the problem you have with ssh.

[jeljaik]

@lornat75 True, I'll wait for his reply.

[mbrunettini]

ssh_exchange_identification: Connection closed by remote host
I must say that I've never seen this error before.
The netboot system (at least, we have to check the whole icub-srv later) seems to be really compromised: I suggest to delete and recreate it starting from our images.
We can provide a working version of
/srv/tftp/wheezy-x86_64
to replace the failing one. It is a compressed file of about 6GB, I can provide a download link
how do you think?

[jeljaik]

@mbrunettini I can try another version of wheezy-x86_64, yes. I'll be waiting for the link. Thank you so much.

[mbrunettini]

I'm uploading the image to the cloud. It's a .tar.gz of 3.1GB (I removed not used stuff).
I will post the link here when it's available
Once you have downloaded it, please:

1. switch off the 3 blades
2. check on icub-srv if there is enough space on /srv/tftp to keep the two old versions of
   /srv/tftp/wheezy-x86_64 (at least 10 GB will be necessary)
3. rename the two old version of
   /srv/tftp/wheezy-x86_64
4. copy the downloaded archive to icub-srv in the following path
   /srv/tftp
5. extract it

[jeljaik]

I will do @mbrunettini thanks!

[mbrunettini]

You can download it here:
https://istitutoitalianotecnologia-my.sharepoint.com/personal/matteo_brunettini_iit_it/_layouts/15/guestaccess.aspx?guestaccesstoken=iPdeNt3TVUszkR%2bhB18wve8vA6Htw7s%2fwV19eP9AEj4%3d&docid=0ae02626af7a34598b60509bc54adfa02&rev=1
Let us know

[jeljaik]

Alright, so a few things have changed! which I guess is something good.

I followed the steps you listed above and after rebooting icub-srv and the blades, when I ssh into any of them I notice the following things:

1. The password is changed to icubAdmin (ours was icub for all of them) which I assume is the default and a normal behaviour.
2. Regardless of where I ssh they all say icub-b2. Maybe this is normal too and an additional configuration step is needed. If so, please let me know.
   
3. As per comment Restoring Unstable Cluster Setup on iCubParis002 #126 (comment) I get the following from icub-b1 which is also good, since before it was always empty.

icub@icub-b2:~$ cat /etc/resolv.conf 
domain icub.local
search icub.local
nameserver 10.0.0.1

[mbrunettini]

Good! Well, not perfect, but a big step further.
Please do the following checks:

• ip addr
• nslookup icub-b1
• mount

and post here the results.
Thanks

P.S.
Later, please change the password to icub (so this is coherent with other machines in your setup)

[jeljaik]

Always on icub-b1 (hoping I assumed correctly 😅)

icub@icub-b2:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 84:34:97:11:b0:ec brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.51/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::8634:97ff:fe11:b0ec/64 scope link
       valid_lft forever preferred_lft forever
3: eth14: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 84:34:97:11:b0:ed brd ff:ff:ff:ff:ff:ff

icub@icub-b2:~$ nslookup icub-b1
Server:     10.0.0.1
Address:    10.0.0.1#53

Name:   icub-b1.icub.local
Address: 10.0.0.51

icub@icub-b2:~$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=4104691,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3284936k,mode=755)
10.0.0.1:/pxeroot on / type nfs (rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,nolock,proto=tcp,port=2049,timeo=7,retrans=3,sec=sys,local_lock=all,addr=10.0.0.1)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6569860k)
none on /tmp type tmpfs (rw,relatime)
none on /run type tmpfs (rw,relatime)
none on /var/tmp type tmpfs (rw,relatime)
none on /var/log type tmpfs (rw,relatime)
none on /var/mail type tmpfs (rw,relatime)
none on /var/spool/cron type tmpfs (rw,relatime)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6569860k)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
10.0.0.1:/exports/code on /usr/local/src/robot type nfs (rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.1,mountvers=3,mountport=60197,mountproto=udp,local_lock=none,addr=10.0.0.1,_netdev)
10.0.0.1:/exports/local_yarp on /home/icub/.local/share/yarp type nfs (rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.1,mountvers=3,mountport=60197,mountproto=udp,local_lock=none,addr=10.0.0.1,_netdev)

[mbrunettini]

Everithing seems ok.
To fix the hostname issue, on icub-b1 you have to modify the following file
/root/set-hostname-from-net.sh
as follows: on line 27 change from
get_hostname_from_dns
to
get_hostname_from_dns eth0
then reboot the 3 blades.

[jeljaik]

Perfetto!
I can now ssh into each one of them and see the proper host names. Overall, the cluster seems "stable" now. I will proceed upgrading source codes and so on. Should I have any additional problem I will open a different issue. Maybe this one can be renamed to the main problem that was tackled ... Restoring Unstable Cluster Setup (?)
Thanks for your patience.

[mbrunettini]

Issue renamed as requested.
@jeljaik ,please remember to change the icub password on the blades (changing on one will also replicate on the others) to the original one.
Can we close this issue?

[jeljaik]

Sure I will change it.
Thanks. Closing!