harbor-registry with F5 BIG-IP ingress does not work when docker push images (unauthorized to access repository)

[teramucho]

i set up harbor-registry with F5 BIG-IP ingress controller.

using harbor-helm branch: master, latest version

and results... clean installed

and harbor portal is working well

login is good and creating project is good

but...docker client have trouble with docker login and push images

when i try to docker login harbor registory, docker client show TLS handshake time out error

so i edit my docker daemon.json files to add insecure registries

"insecure-registries": ["core.skt.ai"]

and next try, docker client show http://core.skt.ai to https://core.skt.ai/service?token ... error

so i edit realm http://core.skt.ai in harbor registry configmap (https --> http)

then i finally login to harbor registry with admin & password

the real problem is here...

when i tried to push my local images, it send me such errors...

[svcapp_su@ddp-deveco-gpu01 ~]$ sudo docker push core.skt.ai/deveco/nginx:1.17
The push refers to repository [core.skt.ai/deveco/nginx]
22439467ad99: Preparing
b4a29beac87c: Preparing
488dfecc21b1: Preparing
unauthorized: unauthorized to access repository: deveco/nginx, action: push

i wonder how to solve this problem...
harbor core give me such errors

2020-02-19T09:01:30Z [ERROR] [/server/registry/error/error.go:25]: failed to handle the request POST /v2/deveco/nginx/blobs/uploads/: unauthorized to access repository: deveco/nginx, action: push, UNAUTHORIZED, unauthorized to access repository: deveco/nginx, action: push
2020/02/19 09:01:30.447 [D] [transaction.go:70] | 172.27.15.196| 401 | 2.550614ms| match| POST /v2/deveco/nginx/blobs/uploads/ r:/*
2020-02-19T09:01:30Z [ERROR] [/server/registry/error/error.go:25]: failed to handle the request POST /v2/deveco/nginx/blobs/uploads/: unauthorized to access repository: deveco/nginx, action: push, UNAUTHORIZED, unauthorized to access repository: deveco/nginx, action: push
2020/02/19 09:01:30.447 [D] [transaction.go:70] | 172.27.15.196| 401 | 2.852277ms| match| POST /v2/deveco/nginx/blobs/uploads/ r:/*
2020-02-19T09:01:30Z [ERROR] [/server/registry/error/error.go:25]: failed to handle the request POST /v2/deveco/nginx/blobs/uploads/: unauthorized to access repository: deveco/nginx, action: push, UNAUTHORIZED, unauthorized to access repository: deveco/nginx, action: push
2020/02/19 09:01:30.448 [D] [transaction.go:70] | 172.27.15.196| 401 | 2.902966ms| match| POST /v2/deveco/nginx/blobs/uploads/ r:/*

this is my harbor-helm value.yaml files

[svcapp_su@ddp-deveco-master01 logging]$ kubectl get pod -n harbor-helm
NAME READY STATUS RESTARTS AGE
harbor-harbor-chartmuseum-7cd84fdfc9-8shkv 1/1 Running 3 31h
harbor-harbor-clair-69cfc88854-brjfp 2/2 Running 7 31h
harbor-harbor-core-86c547987-7vf24 1/1 Running 5 31h
harbor-harbor-database-0 1/1 Running 2 31h
harbor-harbor-jobservice-79fc59d6c-jvcrz 1/1 Running 4 31h
harbor-harbor-notary-server-8d55f5847-5rbnd 1/1 Running 3 31h
harbor-harbor-notary-signer-6648c6ddbc-8wj22 1/1 Running 2 31h
harbor-harbor-portal-67c6b747f-b6kjm 1/1 Running 3 31h
harbor-harbor-redis-0 1/1 Running 3 31h
harbor-harbor-registry-84bbd9869f-55snb 2/2 Running 4 30h

[svcapp_su@ddp-deveco-master01 logging]$ kubectl get svc -n harbor-helm
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
harbor-harbor-chartmuseum ClusterIP 10.233.33.237 80/TCP 31h
harbor-harbor-clair ClusterIP 10.233.58.2 8080/TCP 31h
harbor-harbor-core ClusterIP 10.233.55.246 80/TCP 31h
harbor-harbor-database ClusterIP 10.233.13.10 5432/TCP 31h
harbor-harbor-jobservice ClusterIP 10.233.43.45 80/TCP 31h
harbor-harbor-notary-server ClusterIP 10.233.24.244 4443/TCP 31h
harbor-harbor-notary-signer ClusterIP 10.233.14.74 7899/TCP 31h
harbor-harbor-portal ClusterIP 10.233.33.136 80/TCP 31h
harbor-harbor-redis ClusterIP 10.233.33.101 6379/TCP 31h
harbor-harbor-registry ClusterIP 10.233.57.155 5000/TCP,8080/TCP 31h

[svcapp_su@ddp-deveco-master01 harbor-helm]$ kubectl get ing -n harbor-helm
NAME HOSTS ADDRESS PORTS AGE
harbor-harbor-ingress core.skt.ai,notary.skt.ai 172.27.15.242 80, 443 31h

[teramucho]

value.yaml

[svcapp_su@ddp-deveco-master01 harbor-helm]$ cat values.yaml
expose:

Set the way how to expose the service. Set the type as "ingress",

"clusterIP", "nodePort" or "loadBalancer" and fill the information

in the corresponding section

type: ingress
tls:
# Enable the tls or not. Note: if the type is "ingress" and the tls
# is disabled, the port must be included in the command when pull/push
# images. Refer to goharbor/harbor#5291
# for the detail.
enabled: true
# Fill the name of secret if you want to use your own TLS certificate.
# The secret contains keys named:
# "tls.crt" - the certificate (required)
# "tls.key" - the private key (required)
# "ca.crt" - the certificate of CA (optional), this enables the download
# link on portal to download the certificate of CA
# These files will be generated automatically if the "secretName" is not set
secretName: ""
# By default, the Notary service will use the same cert and key as
# described above. Fill the name of secret if you want to use a
# separated one. Only needed when the type is "ingress".
notarySecretName: ""
# The common name used to generate the certificate, it's necessary
# when the type isn't "ingress" and "secretName" is null
commonName: ""
ingress:
hosts:
core: core.skt.ai
notary: notary.skt.ai
# set to the type of ingress controller if it has specific requirements.
# leave as default for most ingress controllers.
# set to gce if using the GCE ingress controller
# set to ncp if using the NCP (NSX-T Container Plugin) ingress controller
controller: default
annotations:
ingress.kubernetes.io/allow-http: "true"
ingress.kubernetes.io/ssl-redirect: "false"
ingress.kubernetes.io/proxy-body-size: "0"
kubernetes.io/ingress.class: f5
virtual-server.f5.com/ip: 172.27.15.242
virtual-server.f5.com/partition: kubernetes
virtual-server.f5.com/balance: round-robin
virtual-server.f5.com/http-port: "80"
virtual-server.f5.com/https-port: "443"
virtual-server.f5.com/health: |
[
{
"path": "core.skt.ai/",
"send": "HTTP GET /",
"interval": 5,
"timeout": 10
}, {
"path": "core.skt.ai/v2/",
"send": "HTTP GET /v2/",
"interval": 5,
"timeout": 10
}
]
clusterIP:
# The name of ClusterIP service
name: harbor
ports:
# The service port Harbor listens on when serving with HTTP
httpPort: 80
# The service port Harbor listens on when serving with HTTPS
httpsPort: 443
# The service port Notary listens on. Only needed when notary.enabled
# is set to true
notaryPort: 4443
nodePort:
# The name of NodePort service
name: harbor
ports:
http:
# The service port Harbor listens on when serving with HTTP
port: 80
# The node port Harbor listens on when serving with HTTP
nodePort: 30002
https:
# The service port Harbor listens on when serving with HTTPS
port: 443
# The node port Harbor listens on when serving with HTTPS
nodePort: 30003
# Only needed when notary.enabled is set to true
notary:
# The service port Notary listens on
port: 4443
# The node port Notary listens on
nodePort: 30004
loadBalancer:
# The name of LoadBalancer service
name: harbor
# Set the IP if the LoadBalancer supports assigning IP
IP: ""
ports:
# The service port Harbor listens on when serving with HTTP
httpPort: 80
# The service port Harbor listens on when serving with HTTPS
httpsPort: 443
# The service port Notary listens on. Only needed when notary.enabled
# is set to true
notaryPort: 4443
annotations: {}
sourceRanges: []

The external URL for Harbor core service. It is used to

1) populate the docker/helm commands showed on portal

2) populate the token service URL returned to docker/notary client

Format: protocol://domain[:port]. Usually:

1) if "expose.type" is "ingress", the "domain" should be

the value of "expose.ingress.hosts.core"

2) if "expose.type" is "clusterIP", the "domain" should be

the value of "expose.clusterIP.name"

3) if "expose.type" is "nodePort", the "domain" should be

the IP address of k8s node

If Harbor is deployed behind the proxy, set it as the URL of proxy

externalURL: https://core.skt.ai

The persistence is enabled by default and a default StorageClass

is needed in the k8s cluster to provision volumes dynamicly.

Specify another StorageClass in the "storageClass" or set "existingClaim"

if you have already existing persistent volumes to use

For storing images and charts, you can also use "azure", "gcs", "s3",

"swift" or "oss". Set it in the "imageChartStorage" section

persistence:
enabled: true

Setting it to "keep" to avoid removing PVCs during a helm delete

operation. Leaving it empty will delete PVCs after the chart deleted

resourcePolicy: ""
persistentVolumeClaim:
registry:
# Use the existing PVC which must be created manually before bound,
# and specify the "subPath" if the PVC is shared with other components
existingClaim: ""
# Specify the "storageClass" used to provision the volume. Or the default
# StorageClass will be used(the default).
# Set it to "-" to disable dynamic provisioning
storageClass: "nfs-client"
subPath: ""
accessMode: ReadWriteOnce
size: 5Gi
chartmuseum:
existingClaim: ""
storageClass: "nfs-client"
subPath: ""
accessMode: ReadWriteOnce
size: 5Gi
jobservice:
existingClaim: ""
storageClass: "nfs-client"
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
# If external database is used, the following settings for database will
# be ignored
database:
existingClaim: ""
storageClass: "nfs-client"
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
# If external Redis is used, the following settings for Redis will
# be ignored
redis:
existingClaim: ""
storageClass: "nfs-client"
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi

Define which storage backend is used for registry and chartmuseum to store

images and charts. Refer to

https://github.com/docker/distribution/blob/master/docs/configuration.md#storage

for the detail.

imageChartStorage:
# Specify whether to disable redirect for images and chart storage, for
# backends which not supported it (such as using minio for s3 storage type), please disable
# it. To disable redirects, simply set disableredirect to true instead.
# Refer to
# https://github.com/docker/distribution/blob/master/docs/configuration.md#redirect
# for the detail.
disableredirect: false
# Specify the "caBundleSecretName" if the storage service uses a self-signed certificate.
# The secret must contain keys named "ca.crt" which will be injected into the trust store
# of registry's and chartmuseum's containers.
# caBundleSecretName:

# Specify the type of storage: "filesystem", "azure", "gcs", "s3", "swift",
# "oss" and fill the information needed in the corresponding section. The type
# must be "filesystem" if you want to use persistent volumes for registry
# and chartmuseum
type: filesystem
filesystem:
  rootdirectory: /storage
  #maxthreads: 100
azure:
  accountname: accountname
  accountkey: base64encodedaccountkey
  container: containername
  #realm: core.windows.net
gcs:
  bucket: bucketname
  # The base64 encoded json file which contains the key
  encodedkey: base64-encoded-json-key-file
  #rootdirectory: /gcs/object/name/prefix
  #chunksize: "5242880"
s3:
  region: us-west-1
  bucket: bucketname
  #accesskey: awsaccesskey
  #secretkey: awssecretkey
  #regionendpoint: http://myobjects.local
  #encrypt: false
  #keyid: mykeyid
  #secure: true
  #v4auth: true
  #chunksize: "5242880"
  #rootdirectory: /s3/object/name/prefix
  #storageclass: STANDARD
swift:
  authurl: https://storage.myprovider.com/v3/auth
  username: username
  password: password
  container: containername
  #region: fr
  #tenant: tenantname
  #tenantid: tenantid
  #domain: domainname
  #domainid: domainid
  #trustid: trustid
  #insecureskipverify: false
  #chunksize: 5M
  #prefix:
  #secretkey: secretkey
  #accesskey: accesskey
  #authversion: 3
  #endpointtype: public
  #tempurlcontainerkey: false
  #tempurlmethods:
oss:
  accesskeyid: accesskeyid
  accesskeysecret: accesskeysecret
  region: regionname
  bucket: bucketname
  #endpoint: endpoint
  #internal: false
  #encrypt: false
  #secure: true
  #chunksize: 10M
  #rootdirectory: rootdirectory

imagePullPolicy: IfNotPresent

Use this set to assign a list of default pullSecrets

imagePullSecrets:

- name: docker-registry-secret

- name: internal-registry-secret

The update strategy for deployments with persistent volumes(jobservice, registry

and chartmuseum): "RollingUpdate" or "Recreate"

Set it as "Recreate" when "RWM" for volumes isn't supported

updateStrategy:
type: RollingUpdate

debug, info, warning, error or fatal

logLevel: info

The initial password of Harbor admin. Change it from portal after launching Harbor

harborAdminPassword: "Harbor12345"

The secret key used for encryption. Must be a string of 16 chars.

secretKey: "not-a-secure-key"

The proxy settings for updating clair vulnerabilities from the Internet and replicating

artifacts from/to the registries that cannot be reached directly

proxy:
httpProxy:
httpsProxy:
noProxy: 127.0.0.1,localhost,.local,.internal
components:
- core
- jobservice
- clair

UAA Authentication Options

If you're using UAA for authentication behind a self-signed

certificate you will need to provide the CA Cert.

Set uaaSecretName below to provide a pre-created secret that

contains a base64 encoded CA Certificate named ca.crt.

uaaSecretName:

If expose the service via "ingress", the Nginx will not be used

nginx:
image:
repository: goharbor/nginx-photon
tag: dev
replicas: 1

resources:

requests:

memory: 256Mi

cpu: 100m

nodeSelector: {}
tolerations: []
affinity: {}

Additional deployment annotations

podAnnotations: {}

portal:
image:
repository: goharbor/harbor-portal
tag: dev
replicas: 1

resources:

requests:

memory: 256Mi

cpu: 100m

nodeSelector: {}
tolerations: []
affinity: {}

Additional deployment annotations

podAnnotations: {}

core:
image:
repository: goharbor/harbor-core
tag: dev
replicas: 1

Liveness probe values

livenessProbe:
initialDelaySeconds: 300

resources:

requests:

memory: 256Mi

cpu: 100m

nodeSelector: {}
tolerations: []
affinity: {}

Additional deployment annotations

podAnnotations: {}

Secret is used when core server communicates with other components.

If a secret key is not specified, Helm will generate one.

Must be a string of 16 chars.

secret: ""

Fill the name of a kubernetes secret if you want to use your own

TLS certificate and private key for token encryption/decryption.

The secret must contain keys named:

"tls.crt" - the certificate

"tls.key" - the private key

The default key pair will be used if it isn't set

secretName: ""

The XSRF key. Will be generated automatically if it isn't specified

xsrfKey: ""

jobservice:
image:
repository: goharbor/harbor-jobservice
tag: dev
replicas: 1
maxJobWorkers: 10

The logger for jobs: "file", "database" or "stdout"

jobLogger: file

resources:

requests:

memory: 256Mi

cpu: 100m

nodeSelector: {}
tolerations: []
affinity: {}

Additional deployment annotations

podAnnotations: {}

Secret is used when job service communicates with other components.

If a secret key is not specified, Helm will generate one.

Must be a string of 16 chars.

secret: ""

registry:
registry:
image:
repository: goharbor/registry-photon
tag: dev

# resources:
#  requests:
#    memory: 256Mi
#    cpu: 100m

controller:
image:
repository: goharbor/harbor-registryctl
tag: dev

# resources:
#  requests:
#    memory: 256Mi
#    cpu: 100m

replicas: 1
nodeSelector: {}
tolerations: []
affinity: {}

Additional deployment annotations

podAnnotations: {}

Secret is used to secure the upload state from client

and registry storage backend.

See: https://github.com/docker/distribution/blob/master/docs/configuration.md#http

If a secret key is not specified, Helm will generate one.

Must be a string of 16 chars.

secret: ""

If true, the registry returns relative URLs in Location headers. The client is responsible for resolving the correct URL.

relativeurls: false
middleware:
enabled: false
type: cloudFront
cloudFront:
baseurl: example.cloudfront.net
keypairid: KEYPAIRID
duration: 3000s
ipfilteredby: none
# The secret key that should be present is CLOUDFRONT_KEY_DATA, which should be the encoded private key
# that allows access to CloudFront
privateKeySecret: "my-secret"

chartmuseum:
enabled: true

Harbor defaults ChartMuseum to returning relative urls, if you want using absolute url you should enable it by change the following value to 'true'

absoluteUrl: false
image:
repository: goharbor/chartmuseum-photon
tag: dev
replicas: 1

resources:

requests:

memory: 256Mi

cpu: 100m

nodeSelector: {}
tolerations: []
affinity: {}

Additional deployment annotations

podAnnotations: {}

clair:
enabled: true
clair:
image:
repository: goharbor/clair-photon
tag: dev
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
adapter:
image:
repository: goharbor/clair-adapter-photon
tag: dev
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
replicas: 1

The interval of clair updaters, the unit is hour, set to 0 to

disable the updaters

updatersInterval: 12
nodeSelector: {}
tolerations: []
affinity: {}

Additional deployment annotations

podAnnotations: {}

notary:
enabled: true
server:
image:
repository: goharbor/notary-server-photon
tag: dev
replicas: 1
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
signer:
image:
repository: goharbor/notary-signer-photon
tag: dev
replicas: 1
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
nodeSelector: {}
tolerations: []
affinity: {}

Additional deployment annotations

podAnnotations: {}

Fill the name of a kubernetes secret if you want to use your own

TLS certificate authority, certificate and private key for notary

communications.

The secret must contain keys named ca.crt, tls.crt and tls.key that

contain the CA, certificate and private key.

They will be generated if not set.

secretName: ""

database:

if external database is used, set "type" to "external"

and fill the connection informations in "external" section

type: internal
internal:
image:
repository: goharbor/harbor-db
tag: dev
# the image used by the init container
initContainerImage:
repository: busybox
tag: latest
# The initial superuser password for internal database
password: "changeit"
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
nodeSelector: {}
tolerations: []
affinity: {}
external:
host: "192.168.0.1"
port: "5432"
username: "user"
password: "password"
coreDatabase: "registry"
clairDatabase: "clair"
notaryServerDatabase: "notary_server"
notarySignerDatabase: "notary_signer"
# "disable" - No SSL
# "require" - Always SSL (skip verification)
# "verify-ca" - Always SSL (verify that the certificate presented by the
# server was signed by a trusted CA)
# "verify-full" - Always SSL (verify that the certification presented by the
# server was signed by a trusted CA and the server host name matches the one
# in the certificate)
sslmode: "disable"

The maximum number of connections in the idle connection pool.

If it <=0, no idle connections are retained.

maxIdleConns: 50

The maximum number of open connections to the database.

If it <= 0, then there is no limit on the number of open connections.

Note: the default number of connections is 100 for postgre.

maxOpenConns: 100

Additional deployment annotations

podAnnotations: {}

redis:

if external Redis is used, set "type" to "external"

and fill the connection informations in "external" section

type: internal
internal:
image:
repository: goharbor/redis-photon
tag: dev
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
nodeSelector: {}
tolerations: []
affinity: {}
external:
host: "192.168.0.2"
port: "6379"
# The "coreDatabaseIndex" must be "0" as the library Harbor
# used doesn't support configuring it
coreDatabaseIndex: "0"
jobserviceDatabaseIndex: "1"
registryDatabaseIndex: "2"
chartmuseumDatabaseIndex: "3"
clairAdapterIndex: "4"
password: ""

Additional deployment annotations

podAnnotations: {}

[yohan-couppey]

I ran into the same issue with a nginx ingress controller yesterday. I was working with "dev" tag for all the images on my values.yaml. I moved to a stable release, basically git checkout v1.2.4, changed the values.yaml to fit my needs and it works well now.

[teramucho]

I ran into the same issue with a nginx ingress controller yesterday. I was working with "dev" tag for all the images on my values.yaml. I moved to a stable release, basically git checkout v1.2.4, changed the values.yaml to fit my needs and it works well now.

after you edited values "tag: dev" to "tag: stable", i did same work on my values.yaml, but it was not work...
like this
$ kubectl get pod -n harbor-stable -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
harbor-stable-harbor-chartmuseum-6945c7b5fb-m2dnb 0/1 ImagePullBackOff 0 6m52s 10.233.84.178 ddp-deveco-gpu02
harbor-stable-harbor-clair-85675877c7-cttkp 0/2 ImagePullBackOff 0 6m52s 10.233.84.176 ddp-deveco-gpu02
harbor-stable-harbor-core-695bcc4bc-kxcvw 0/1 ImagePullBackOff 0 6m52s 10.233.65.199 ddp-deveco-master02
harbor-stable-harbor-database-0 0/1 Init:ErrImagePull 0 6m52s 10.233.104.236 ddp-deveco-master03
harbor-stable-harbor-jobservice-78cd5cf4df-jc44x 0/1 ImagePullBackOff 0 6m52s 10.233.65.202 ddp-deveco-master02
harbor-stable-harbor-notary-server-5997b55894-8hwfv 0/1 ImagePullBackOff 0 6m52s 10.233.104.235 ddp-deveco-master03
harbor-stable-harbor-notary-signer-7c9f557c9f-vfpwt 0/1 ImagePullBackOff 0 6m52s 10.233.92.32 ddp-deveco-gpu01
harbor-stable-harbor-portal-7bdbb6b6d5-ks6mf 0/1 ImagePullBackOff 0 6m52s 10.233.84.177 ddp-deveco-gpu02
harbor-stable-harbor-redis-0 0/1 ImagePullBackOff 0 6m52s 10.233.65.200 ddp-deveco-master02
harbor-stable-harbor-registry-565b98d9dc-2hw86 0/2 ImagePullBackOff 0 6m52s 10.233.92.34 ddp-deveco-gpu01

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.