[Harbor] e2e test is failing for harbor

[aman556]

Bug Report

While running e2e test for harbor by following the steps mention in description of PR 2278 it is failing with the below logs.

STEP: validating harbor components are healthy
Failure [1068.336 seconds]
[BeforeSuite] BeforeSuite 
/Users/amansha/go/src/github.com/vmware-tanzu/community-edition/addons/packages/harbor/2.2.3/test/e2e/harbor_suite_test.go:83

  Timed out after 647.877s.
  harbor is unhealthy
  Expected success, but got an error:
      <*errors.errorString | 0xc000097f60>: {
          s: "Assertion in callback at /Users/amansha/go/src/github.com/vmware-tanzu/community-edition/addons/packages/harbor/2.2.3/test/e2e/harbor_suite_test.go:324 failed:\nUnexpected error:\n    <*url.Error | 0xc000304300>: {\n        Op: \"Get\",\n        URL: \"https://harbor.172.21.0.2.nip.io/api/v2.0/health\",\n        Err: <*net.OpError | 0xc0000b05a0>{\n            Op: \"dial\",\n            Net: \"tcp\",\n            Source: nil,\n            Addr: <*net.TCPAddr | 0xc00019e480>{IP: [172, 21, 0, 2], Port: 443, Zone: \"\"},\n            Err: <*os.SyscallError | 0xc0000d8200>{\n                Syscall: \"connect\",\n                Err: <syscall.Errno>0x3c,\n            },\n        },\n    }\n    Get \"https://harbor.172.21.0.2.nip.io/api/v2.0/health\": dial tcp 172.21.0.2:443: connect: operation timed out\noccurred",
      }
      Assertion in callback at /Users/amansha/go/src/github.com/vmware-tanzu/community-edition/addons/packages/harbor/2.2.3/test/e2e/harbor_suite_test.go:324 failed:
      Unexpected error:
          <*url.Error | 0xc000304300>: {
              Op: "Get",
              URL: "https://harbor.172.21.0.2.nip.io/api/v2.0/health",
              Err: <*net.OpError | 0xc0000b05a0>{
                  Op: "dial",
                  Net: "tcp",
                  Source: nil,
                  Addr: <*net.TCPAddr | 0xc00019e480>{IP: [172, 21, 0, 2], Port: 443, Zone: ""},
                  Err: <*os.SyscallError | 0xc0000d8200>{
                      Syscall: "connect",
                      Err: <syscall.Errno>0x3c,
                  },
              },
          }
          Get "https://harbor.172.21.0.2.nip.io/api/v2.0/health": dial tcp 172.21.0.2:443: connect: operation timed out
      occurred

  /Users/amansha/go/src/github.com/vmware-tanzu/community-edition/addons/packages/harbor/2.2.3/test/e2e/harbor_suite_test.go:326
------------------------------
STEP: cleaning up harbor addon package
+ tanzu package installed delete harbor --poll-interval 5s --poll-timeout 10m --namespace default --yes
| Deleting service account 'harbor-default-sa' 
STEP: cleaning up contour addon package
+ tanzu package installed delete contour --poll-interval 5s --poll-timeout 10m --namespace default --yes
| Deleting admin role 'contour-default-cluster-role' 
STEP: cleaning up cert-manager addon package
+ tanzu package installed delete cert-manager --poll-interval 5s --poll-timeout 10m --namespace default --yes
| Deleting service account 'cert-manager-default-sa' 
STEP: validating the harbor package install no longer exists
+ kubectl -n default get packageinstall harbor
Error from server (NotFound): packageinstalls.packaging.carvel.dev "harbor" not found

STEP: cleaning up goharbor namespace
+ kubectl delete ns goharbor
namespace "goharbor" deleted

Expected Behavior

The e2e-test has to be pass without any failure.

Steps to Reproduce the Bug

Follow steps in PR 2278 description to test harbor e2e.

Screenshots or additional information and context

Environment Details

• Build version (tanzu version): v0.10.1
• Deployment (Managed/Standalone cluster): Unmanaged cluster
• Infrastructure Provider (Docker/AWS/Azure/vSphere): Docker
• Operating System (client): Mac

Diagnostics and log bundle

[aman556]

Did we have to use any configuration file for harbor in docker environment ?

[joshrosso]

@vmware-tanzu/pkg-harbor-owners && @heww, looks like this is an issue with the tests, curious if you could take a look.

[heww]

@MinerYang could you have a look at this issue?

[joshrosso]

@karuppiah7890 @aman556 how severe is this? I'm guessing it'll break our e2e validation? Do we need to escalate?

Trying to get a sense for whether this should block our next release, v0.11.

[aman556]

Hello @joshrosso so up to my understanding the issue is at the code level in e2e-test for Harbor we can do the test manually so it is not like a blocker. I will test it manually and update you here.
Thanks!

[joshrosso]

Thanks @aman556! I'll still make sure this gets visibility internally so you're not on the hook perpetually to validate this package.

For now, moving to v0.12.0 since it won't block v0.11.0

[MinerYang]

Hi @aman556 ,

Sorry for the late, and I can't reproduce the same issue that you mentioned in the PR by running cd test/e2e && ginkgo -v -- --packages=harbor --version=2.2.3 --guest-cluster-name="tce". All the test passed successfully.

+ tanzu package install harbor --poll-interval 5s --poll-timeout 10m --namespace default --package-name harbor.community.tanzu.vmware.com --version 2.2.3 --values-file /tmp/harbor-1841825547.yaml
- 'PackageInstall' resource install status: Reconciling
+ kubectl get apps
NAME                 DESCRIPTION           SINCE-DEPLOY   AGE
cert-manager         Reconcile succeeded   32s            3m32s
contour              Reconcile succeeded   14s            3m5s
harbor               Reconcile succeeded   2m13s          2m15s
local-path-storage   Reconcile succeeded   11s            2m28s

+ tanzu package installed list
\ Retrieving installed packages...
  NAME                PACKAGE-NAME                                   PACKAGE-VERSION  STATUS
  cert-manager        cert-manager.community.tanzu.vmware.com        1.6.1            Reconcile succeeded
  contour             contour.community.tanzu.vmware.com             1.19.1           Reconcile succeeded
  harbor              harbor.community.tanzu.vmware.com              2.2.3            Reconcile succeeded
  local-path-storage  local-path-storage.community.tanzu.vmware.com  0.0.20           Reconcile succeeded

STEP: validating harbor package is ready
+ kubectl -n default get packageinstalls harbor -o jsonpath={.status.conditions[?(@.type == 'ReconcileSucceeded')].status}
True
STEP: validating harbor components are healthy
Harbor Addon E2E Test
  push image to the library project
  /home/ubuntu/tce/community-edition/addons/packages/harbor/2.2.3/test/e2e/harbor_test.go:53
time="2022-03-08T01:27:21Z" level=warning msg="reference for unknown type: tce-x-harbor" digest="sha256:835921565879266b4b53f10b5e2cc78fd0194c011b4e86a8ff864e6e704da5ef" mediatype=tce-x-harbor size=14
...

Ran 2 of 2 Specs in 515.033 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Environment Details

• Build version (tanzu version): v0.10.1
• Deployment (Managed/Standalone cluster): Unmanaged cluster
• Infrastructure Provider (Docker/AWS/Azure/vSphere): Docker
• Operating System (client): Ubuntu 20.04.3 LTS

[MinerYang]

Hi @aman556，

Here's some updates for this issue. I have just reproduce the same failure as you mentioned in a Mac Operator System.
The root cause for this is that container IP 172.21.0.2 is not accessible on Docker for Mac., because it is running at a lightweight virtual machine instead of handled directly by the docker, which is different from Linux. You can refer to this issue docker/for-mac#155.
So, for the harbor package e2e test itself, how do we find a Reachable IP for harbor when installing?
First, it will look for an IP address of the Load Balancer when it's public. Otherwise use a container IP by running kubectl get pod -l app=envoy -n projectcontour. Normally, in a standalone docker cluster, harbor would looking for the second one which is not accessible in a Mac system.
It is a problem that might not been solved by us, and we highly recommend running this test on a Linux environment.

ping 172.18.0.2
PING 172.18.0.2 (172.18.0.2): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3

STEP: validating harbor components are healthy
Failure [1331.202 seconds]
[BeforeSuite] BeforeSuite
/Users/yminer/tce_worksapce/community-edition/addons/packages/harbor/2.2.3/test/e2e/harbor_suite_test.go:83

  Timed out after 643.459s.
  harbor is unhealthy
  Expected success, but got an error:
      <*errors.errorString | 0xc000542480>: {
          s: "Assertion in callback at /Users/yminer/tce_worksapce/community-edition/addons/packages/harbor/2.2.3/test/e2e/harbor_suite_test.go:325 failed:\nUnexpected error:\n    <*url.Error | 0xc0001108a0>: {\n        Op: \"Get\",\n        URL: \"https://harbor.172.18.0.2.nip.io/api/v2.0/health\",\n        Err: <*net.OpError | 0xc0003c4000>{\n            Op: \"dial\",\n            Net: \"tcp\",\n            Source: nil,\n            Addr: <*net.TCPAddr | 0xc000310330>{IP: [172, 18, 0, 2], Port: 443, Zone: \"\"},\n            Err: <*os.SyscallError | 0xc00008e000>{\n                Syscall: \"connect\",\n                Err: <syscall.Errno>0x3c,\n            },\n        },\n    }\n    Get \"https://harbor.172.18.0.2.nip.io/api/v2.0/health\": dial tcp 172.18.0.2:443: connect: operation timed out\noccurred",
      }
      Assertion in callback at /Users/yminer/tce_worksapce/community-edition/addons/packages/harbor/2.2.3/test/e2e/harbor_suite_test.go:325 failed:
      Unexpected error:
          <*url.Error | 0xc0001108a0>: {
              Op: "Get",
              URL: "https://harbor.172.18.0.2.nip.io/api/v2.0/health",
              Err: <*net.OpError | 0xc0003c4000>{
                  Op: "dial",
                  Net: "tcp",
                  Source: nil,
                  Addr: <*net.TCPAddr | 0xc000310330>{IP: [172, 18, 0, 2], Port: 443, Zone: ""},
                  Err: <*os.SyscallError | 0xc00008e000>{
                      Syscall: "connect",
                      Err: <syscall.Errno>0x3c,
                  },
              },
          }
          Get "https://harbor.172.18.0.2.nip.io/api/v2.0/health": dial tcp 172.18.0.2:443: connect: operation timed out
      occurred

Hope this can help you well and have a good day!

Best,
Miner

[aman556]

Hey @MinerYang thanks for following up and for the details. from now-onwards we will test on linux.
Thanks