-
Notifications
You must be signed in to change notification settings - Fork 507
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
* fatal: client is offline #1114
Comments
Could you please give us the logs from the notary server container for this attempted push? The logs indicate that the server probably rejected the update. We'd have to see those to confirm why it did. As a note I have also previously seen this when there was a clock skew inside my docker VM such that containers inside thought that it was many hours or even days before the current time. I saw errors such as |
Thank you for prompt response. As suggested, I ran the command & found the vm itself has wrong date & getting error message while running docker run Please find below :[root@trustimage ~]# date
|
@shiva022 Thank you for the logs - they don't seem to contain any logs from the POST command, which should have been present from the push. I'm not sure about the timeout error during the Are you running docker 4 mac by any chance? If so, restarting docker 4 mac should fix the clock skew (see docker/for-mac#17 for the issue). If it is a general VM (such as via virtualbox, vmware, or qemu or something) could you try setting up NTP in that VM? Or if it's already set up and just not fixing itself, maybe restarting the VM would fix it? |
@cyli As NTP was not setup; so I did fresh installation with NTP setup.
|
@cyli Sorry for incomplete information. As require please find the output of date command [root@trustimage ~]# date -u |
Please find the push images snapshot with --debug mode: [root@dockerhost .docker]# docker --debug push trustimage.test.com:5000/registry:2 DEBU[0021] Making dir path: /root/.docker/trust/tuf/trustimage.test.com:5000/registry/changelist
|
@shiva022: I see this error in your notary server logs, it seems that there's a newer version already published on the server.
Have you manually edited the database by chance? |
@riyazdf By mistake my Virtual machine got shutdown & since running container got exited. I ran the below command to run the container again: [root@trustimage notary-master]# docker-compose down signer_1 | waiting for mysql://signer@tcp(mysql:3306)/notarysigner to come up.
|
@shiva022: looks like your notary services were waiting for the database to come up (after crash recovery), did it eventually connect? If not, can you try restarting the services? |
@riyazdf I restarted the service by the below command : |
@shiva022 Would it be ok to delete all your existing data and try again? If so, could you try |
@cyli The Issue is resolved. Since I neither configure DNS nor made host entry in host file hence, while pushing the image it was resolving the host by searching host trustimage.test.com globally. which causing the failure. |
Ah ok, thanks for letting us know! |
Hello Team,
I am running docker private registry on vm along with notary-server, notary-signer. To run notary server & signer I followed : https://github.com/docker/notary I have used docker-compose to run all components & everything is up and running:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7acbf8a4a527 registry:2 "/entrypoint.sh /e..." 20 hours ago Up 20 hours 0.0.0.0:5000->5000/tcp registry
68afe6a63cd5 notary_server "/usr/bin/env sh -..." 20 hours ago Up 20 hours 0.0.0.0:4443->4443/tcp, 0.0.0.0:32769->8080/tcp notary_server_1
ff6bfeb8b84a notary_signer "/usr/bin/env sh -..." 20 hours ago Up 20 hours notary_signer_1
9ac8a5e488f5 mariadb:10.1.10 "/docker-entrypoin..." 20 hours ago Up 20 hours 3306/tcp notary_mysql_1
go/src/github.com/docker/notary # notary-server version
{"level":"info","msg":"Version: 0.5.0, Git commit: c04e3e6","time":"2017-03-03T16:07:28Z"}
{"level":"fatal","msg":"Could not read config at :, viper error: Unsupported Config Type ""","time":"2017-03-03T16:07:28Z"}
/go/src/github.com/docker/notary # notary-signer version
{"level":"info","msg":"Version: 0.5.0, Git commit: c04e3e6","time":"2017-03-03T16:10:20Z"}
{"level":"fatal","msg":"Could not read config at :, viper error: Unsupported Config Type ""","time":"2017-03-03T16:10:20Z"}
/go/src/github.com/docker/notary #
docker info
Containers: 8
Running: 4
Paused: 0
Stopped: 4
Images: 29
Server Version: 1.13.1
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.el7.x86_64
Operating System: Red Hat Enterprise Linux
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.797 GiB
Name: trustimage.test.com
ID: DAMD:CG2G:3FR3:UR3R:CF75:HLWX:OES4:6S2I:TE4I:IR4H:CUTO:DUM5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
[root@trustimage fixtures]# docker version
Client:
Version: 1.13.1
API version: 1.26
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 06:38:28 2017
OS/Arch: linux/amd64
Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 06:38:28 2017
OS/Arch: linux/amd64
Experimental: false
[root@trustimage fixtures]#
I have setup docker host on another virtual machine & installed notary client & enable DOCKER_TRUST_CONTENT.
[root@dockerclient .docker]# echo $DOCKER_CONTENT_TRUST
1
[root@dockerclient .docker]# echo $DOCKER_CONTENT_TRUST_SERVER
https://notary-server:4443
[root@dockerclient .docker]#
[root@dockerclient .docker]# docker version
Client:
Version: 1.13.0
API version: 1.25
Go version: go1.7.3
Git commit: 49bf474
Built: Tue Jan 17 09:55:28 2017
OS/Arch: linux/amd64
Server:
Version: 1.13.0
API version: 1.25 (minimum version 1.12)
Go version: go1.7.3
Git commit: 49bf474
Built: Tue Jan 17 09:55:28 2017
OS/Arch: linux/amd64
Experimental: false
[root@dockerclient .docker]# notary version
notary
Version: 0.4.3
Git commit: 9211198
ISSUE:- I got stuck while pushing the image from docker host I am facing the below Issue: -
[root@dockerclient .docker]# docker --debug push trustimage.test.com:5000/regis_no5:v2.1
The push refers to a repository [trustimage.test.com:5000/regis_no5]
ad7b85eab471: Layer already exists
7436cad623e5: Layer already exists
61fd5549d17b: Layer already exists
5cc1e08c7c0e: Layer already exists
7a33e410db32: Layer already exists
0912b310db10: Layer already exists
6767cac03a2d: Layer already exists
f38182d44088: Layer already exists
b5aac7bb5fdf: Layer already exists
9f8566ee5135: Layer already exists
v2.1: digest: sha256:2369cdd47081bfbd26d8f169ed21dde9000cde98d9f7d4fa58411c8dabea131d size: 2395
Signing and pushing trust metadata
DEBU[0000] reading certificate directory: /root/.docker/tls/notary-server:4443
DEBU[0000] crt: /root/.docker/tls/notary-server:4443/ca.crt
DEBU[0000] No yubikey found, using alternative key storage: no library found
DEBU[0000] received HTTP status 404 when requesting root.
DEBU[0000] No yubikey found, using alternative key storage: no library found
DEBU[0000] generated ECDSA key with keyID: 0643b68e02f1b5df36c30dc0402e3476ae1b6182c5d41cc9a01d2f165c70558f
DEBU[0000] generated new ecdsa key for role: root and keyID: 0643b68e02f1b5df36c30dc0402e3476ae1b6182c5d41cc9a01d2f165c70558f
DEBU[0000] No yubikey found, using alternative key storage: no library found
You are about to create a new root signing key passphrase. This passphrase
will be used to protect the most sensitive key in your signing system. Please
choose a long, complex passphrase and be careful to keep the password and the
key file itself secure and backed up. It is highly recommended that you use a
password manager to generate the passphrase and keep it safe. There will be no
way to recover this key. You can find the key in your config directory.
Enter passphrase for new root key with ID 0643b68:
Repeat passphrase for new root key with ID 0643b68:
DEBU[0009] No yubikey found, using alternative key storage: no library found
DEBU[0009] generated ECDSA key with keyID: f50e718989e4f8a10e37f0973b4887d2c50fa13c1b1ec19f35477d25f0b84405
DEBU[0009] generated new ecdsa key for role: targets and keyID: f50e718989e4f8a10e37f0973b4887d2c50fa13c1b1ec19f35477d25f0b84405
Enter passphrase for new repository key with ID f50e718 (trustimage.test.com:5000/regis_no5):
Repeat passphrase for new repository key with ID f50e718 (trustimage.test.com:5000/regis_no5):
DEBU[0015] got remote timestamp ecdsa key with keyID: 7ea8415278180d587d73a0a0b0e050d8dfb26bc10aee33450e75e6bcd353ef8e
DEBU[0015] got remote snapshot ecdsa key with keyID: 3914a421932ca217cf2368925b455a2146ea2382aeacad0da098d26cbc4d5e6d
DEBU[0015] generating new snapshot...
DEBU[0015] Saving changes to Trusted Collection.
DEBU[0015] signing root...
DEBU[0015] sign called with 1/1 required keys
DEBU[0015] No yubikey found, using alternative key storage: no library found
DEBU[0015] sign called with 0/0 required keys
DEBU[0015] sign targets called for role targets
DEBU[0015] sign called with 1/1 required keys
DEBU[0015] No yubikey found, using alternative key storage: no library found
DEBU[0015] sign called with 0/0 required keys
Finished initializing "trustimage.test.com:5000/regis_no5"
DEBU[0015] Making dir path: /root/.docker/trust/tuf/trustimage.test.com:5000/regis_no5/changelist
DEBU[0015] Adding target "v2.1" with sha256 "2369cdd47081bfbd26d8f169ed21dde9000cde98d9f7d4fa58411c8dabea131d" and size 2395 bytes.
DEBU[0015] Making dir path: /root/.docker/trust/tuf/trustimage.test.com:5000/regis_no5/changelist
DEBU[0015] entered ValidateRoot with dns: trustimage.test.com:5000/regis_no5
DEBU[0015] found the following root keys: [5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da]
DEBU[0015] found 1 valid leaf certificates for trustimage.test.com:5000/regis_no5: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] found 1 leaf certs, of which 1 are valid leaf certs for trustimage.test.com:5000/regis_no5
DEBU[0015] checking root against trust_pinning config%!(EXTRA string=trustimage.test.com:5000/regis_no5)
DEBU[0015] checking trust-pinning for cert: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] role has key IDs: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] verifying signature for key ID: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] root validation succeeded for trustimage.test.com:5000/regis_no5
DEBU[0015] entered ValidateRoot with dns: trustimage.test.com:5000/regis_no5
DEBU[0015] found the following root keys: [5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da]
DEBU[0015] found 1 valid leaf certificates for trustimage.test.com:5000/regis_no5: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] found 1 leaf certs, of which 1 are valid leaf certs for trustimage.test.com:5000/regis_no5
DEBU[0015] checking root against trust_pinning config%!(EXTRA string=trustimage.test.com:5000/regis_no5)
DEBU[0015] checking trust-pinning for cert: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] role has key IDs: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] verifying signature for key ID: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] root validation succeeded for trustimage.test.com:5000/regis_no5
DEBU[0015] received HTTP status 404 when requesting root.
DEBU[0015] Loading trusted collection.
DEBU[0015] entered ValidateRoot with dns: trustimage.test.com:5000/regis_no5
DEBU[0015] found the following root keys: [5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da]
DEBU[0015] found 1 valid leaf certificates for trustimage.test.com:5000/regis_no5: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] found 1 leaf certs, of which 1 are valid leaf certs for trustimage.test.com:5000/regis_no5
DEBU[0015] checking root against trust_pinning config%!(EXTRA string=trustimage.test.com:5000/regis_no5)
DEBU[0015] checking trust-pinning for cert: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] role has key IDs: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] verifying signature for key ID: 5e64dc63bfa9678304a877719085ca8672bd7b41ab94d37e8c390a9755a610da
DEBU[0015] root validation succeeded for trustimage.test.com:5000/regis_no5
DEBU[0015] targets role has key IDs: f50e718989e4f8a10e37f0973b4887d2c50fa13c1b1ec19f35477d25f0b84405
DEBU[0015] verifying signature for key ID: f50e718989e4f8a10e37f0973b4887d2c50fa13c1b1ec19f35477d25f0b84405
DEBU[0015] changelist add: v2.1
DEBU[0015] No yubikey found, using alternative key storage: no library found
DEBU[0015] applied 1 change(s)
DEBU[0015] sign targets called for role targets
DEBU[0015] sign called with 1/1 required keys
DEBU[0015] No yubikey found, using alternative key storage: no library found
DEBU[0015] sign called with 0/0 required keys
DEBU[0015] generating new snapshot...
DEBU[0015] signing snapshot...
DEBU[0015] sign called with 1/1 required keys
DEBU[0015] No yubikey found, using alternative key storage: no library found
DEBU[0015] Client does not have the key to sign snapshot. Assuming that server should sign the snapshot.
Failed to sign "trustimage.test.com:5000/regis_no5":v2.1 - The root metadata is invalid: could not validate the path to a trusted root: unable to retrieve valid leaf certificates
The root metadata is invalid: could not validate the path to a trusted root: unable to retrieve valid leaf certificates
Kindly look into it. Your earliest help would be appreciated.
The text was updated successfully, but these errors were encountered: