cli nightly does not work on different OS or storage drivers

[riuvshin]

after we merged this cd5db87

we have problems on mac and centOs

error is

INFO: Proxy: HTTP_PROXY=, HTTPS_PROXY=, NO_PROXY=*.local, 169.254/16
INFO: (codenvy cli): Checking network... (hint: '--fast' skips version, network, and nightly checks)
INFO: (codenvy cli): Loading cli...
date: can't stat '/var/lib/docker/image/aufs/imagedb/content/sha256/d4bbec79a4ac34b1ead51573f9c05b2e48c1eedc218b5783ed5339f3a9518fb4': No such file or directory

this happen because this path /var/lib/docker/image/aufs/ maybe different for different OS and storage drivers. in production systems we use special data pool devices for docker storage and afaik you can just go there and navigate through files "layers" they all compressed and not accessible.

@TylerJewell said that docker inspect -f '{{ .Created }}' codenvy/cli-saas:nightly this does not work for us because this date is not the date when image was written to disk, but why we need to know when it was written? As I understand it should be enough to know when image was built like if image was built today - ok this is accurate one if yesterday - no. We can assume here that nightly image we update each day. So I don't understand why that CREATED date does not work for us, I don;t know why we need to know when it was written to disk because it maybe written today but image can be outdated one, i.e we should know when locally pulled CLI image was built and if it was built more than 24h ago - we do pull it and ask to relaunch CLI

@benoitf @TylerJewell wdyt guys?

[TylerJewell]

@riuvshin - that check was the check we previously had. So we could revert some of the nightly changes that I made today and put that check back in. But that check is not very reliable as:

1. It reflects when the image was created - not written to disk
2. It doesn't account for time zones or different date settings on the computer where the image was saved vs. the computer where it was built.

So we could still depend upon that date, but we have to really add in some buffers on the assumption that you are comparing a date from another computer to your local date on your server.

[benoitf]

@riuvshin well as we can't start cli anymore on macOs yes I would say a revert should be best option while in parallel new checks can be tested widely

About timezone, increasing previous check from one hour to one day should be OK (as timezone is <24hour)

Maybe we could also have timestamps for each nightly deployed by the CI

like with the current date so checking previous versions is easy:
$ docker run eclipse/che:20170112

[riuvshin]

@benoitf @TylerJewell after this revert merged cli works for me on macOs but Im not able to run
codenvy saas on nightly it fails with this:

cli command:

docker run --rm -i  -v /var/run/docker.sock:/var/run/docker.sock \
                    -v /home/codenvy/codenvy:/data \
                    -v /home/codenvy/codenvy/backup:/backup \
                    codenvy/cli-saas:nightly start

output:

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
WARN: Did not detect TTY - interactive mode disabled
INFO: (codenvy cli): Checking network... (hint: '--fast' skips version, network, and nightly checks)
INFO: (codenvy cli): Loading cli...
INFO: Welcome to Codenvy SaaS!
INFO:
INFO: We did not auto-detect a valid HOST or IP address.
INFO: Pass CODENVY_HOST with your hostname or IP address.
INFO:
INFO: Rerun the CLI:
INFO:   docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock
INFO:                       -v <local-path>:/data
INFO:                       -e CODENVY_HOST=<your-ip-or-host>
INFO:                          codenvy/cli-saas:nightly start --pull

I don't understand what was changed but in /home/codenvy/codenvy/codenvy.env we already have CODENVY_HOST=nightly.codenvy-stg.com. Last time it was working Jan 11, 2017 5:31 PM (UA TIME)

I also tried to mount another folder to :/data like this is a new installation and it fails same way.

Im not able to reproduce this on MacOs, and it is hard for me to figure out how we detect CODENVY_HOST variable, can you guys help me on that?

[riuvshin]

So it seems that I've find how we set this we use docker_run --net host eclipse/che-ip:nightly but if I run docker run --net host eclipse/che-ip:nightly on nightly it gives nothing in output...

[riuvshin]

@benoitf @TylerJewell Ok I've found the issue, we've broke che-ip 10 days ago but it was working because we use che-ip:nightly and we don;t pull it. once i've pulled it yesterday it stop work.
here is the commit which break che ip on some OS 915f32f

in my case grep "scope global ${NETWORK_IF}"does not work because the output looks different

inet 172.30.21.70/24 brd 172.30.21.255 scope global dynamic eth0

here is PR that fix this issue #3690

this is very good example of a problem that we should not have hardcoded nightly tags for utility images in CLI...

[riuvshin]

reopen issue :(
we did a mistake our latest changes perfectly works but we;ve missed that if there is no changes to the CLI
during nightly build of cli image image does not pushing it says "already up to date" so image date still old one... now I;m not able update nightly QA env because CLI do exit.

@benoitf @TylerJewell we have to think again how to handle this....
as a first step I would remove exit. but we need to change the logic how we check image accuracy.
it seeems that date does not help us, we need some way to make sure that on docker hub image is newer than local one.

[TylerJewell]

I have proposed an updated branch - this time we are comparing digest IDs before and after a pull of the image. We'll use them for an equality check.