Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deployment replicas syncer testing #5136

Open
RainbowMango opened this issue Jul 4, 2024 · 6 comments
Open

deployment replicas syncer testing #5136

RainbowMango opened this issue Jul 4, 2024 · 6 comments
Assignees
@chaosi-zju
Labels
kind/flake Categorizes issue or PR as related to a flaky test.

Comments

@RainbowMango
Copy link
Member

Which jobs are flaking:

Which test(s) are flaking:

https://github.com/karmada-io/karmada/actions/runs/9788476477/job/27026725265

Reason for failure:

Anything else we need to know:

Timeline >>
  STEP: Creating PropagationPolicy(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:42:38.3
  STEP: Creating Deployment(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:42:38.31
  STEP: Creating HPA(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:42:38.315
  STEP: step1: propagate 4 replicas to two clusters @ 07/04/24 04:42:38.323
  STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:38.323
  STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:43.332
  STEP: step2: hpa scale each member cluster replicas from 2 to 3 @ 07/04/24 04:42:43.349
  STEP: Updating HPA(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:42:43.349
  STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:43.369
  STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:58.399
  STEP: step3: manually add deployment template replicas from 6 to 10 @ 07/04/24 04:42:58.422
  STEP: Updating Deployment(karmadatest-m5dpb/deploy-g69xd)'s replicas to 10 @ 07/04/24 04:42:58.422
  STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:58.447
  STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:58.461
  [FAILED] in [It] - /home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 @ 07/04/24 04:43:58.472
  STEP: Removing PropagationPolicy(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:43:58.473
  STEP: Removing Deployment(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:43:58.479
  STEP: Removing HPA(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:43:58.483
  STEP: Check if deployment(karmadatest-m5dpb/deploy-g69xd) disappears on member clusters @ 07/04/24 04:43:58.489
  << Timeline

  [FAILED] Timed out after 60.000s.
  Expected
      <bool>: false
  to equal
      <bool>: true
  In [It] at: /home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 @ 07/04/24 04:43:58.472

  Full Stack Trace
    github.com/karmada-io/karmada/test/e2e.assertDeploymentTemplateReplicas({0xc00051eca8, 0x11}, {0xc0001f5744, 0xc}, 0x6)
    	/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 +0x257
    github.com/karmada-io/karmada/test/e2e.init.func13.4.2.3()
    	/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:142 +0x1de
    github.com/karmada-io/karmada/test/e2e.init.func13.4.2()
    	/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:139 +0x2fa
@RainbowMango RainbowMango added the kind/flake Categorizes issue or PR as related to a flaky test. label Jul 4, 2024
@chaosi-zju
Copy link
Member

/assign

@chaosi-zju
Copy link
Member

how many times does this error occur?

@chaosi-zju
Copy link
Member

the log of controller-manager is:

controller-manager.log 
$ cat controller-manager.log | grep deployment_replicas_syncer_controller.go | grep -A 200 '2024-07-04T04:42:38.348681435Z stderr F I0704 04:42:38.348611'
2024-07-04T04:42:38.348681435Z stderr F I0704 04:42:38.348611       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:38.349314846Z stderr F I0704 04:42:38.349251       1 deployment_replicas_syncer_controller.go:163] wait until scheduler observed generation (0) equal to generation in binding (1)
2024-07-04T04:42:38.633299953Z stderr F I0704 04:42:38.633098       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:38.633305023Z stderr F I0704 04:42:38.633147       1 deployment_replicas_syncer_controller.go:190] wait until aggregated status replicas of cluster member1 collected
2024-07-04T04:42:38.75664433Z stderr F I0704 04:42:38.756584       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:38.756797706Z stderr F I0704 04:42:38.756727       1 deployment_replicas_syncer_controller.go:129] replicas in spec field (4) already equal to in status field (4)
2024-07-04T04:42:39.350750574Z stderr F I0704 04:42:39.350615       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:39.350765843Z stderr F I0704 04:42:39.350711       1 deployment_replicas_syncer_controller.go:129] replicas in spec field (4) already equal to in status field (4)
2024-07-04T04:42:41.788647528Z stderr F I0704 04:42:41.788411       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:41.788685509Z stderr F I0704 04:42:41.788652       1 deployment_replicas_syncer_controller.go:163] wait until scheduler observed generation (0) equal to generation in binding (1)
2024-07-04T04:42:42.032515467Z stderr F I0704 04:42:42.032456       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:42.034628338Z stderr F I0704 04:42:42.034574       1 deployment_replicas_syncer_controller.go:190] wait until aggregated status replicas of cluster member2 collected
2024-07-04T04:42:42.058527425Z stderr F I0704 04:42:42.058452       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:42.058737587Z stderr F I0704 04:42:42.058661       1 deployment_replicas_syncer_controller.go:129] replicas in spec field (2) already equal to in status field (2)
2024-07-04T04:42:42.793182239Z stderr F I0704 04:42:42.792535       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:42.79323648Z stderr F I0704 04:42:42.792589       1 deployment_replicas_syncer_controller.go:129] replicas in spec field (2) already equal to in status field (2)
2024-07-04T04:42:46.790275125Z stderr F I0704 04:42:46.789925       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:46.790279754Z stderr F I0704 04:42:46.790064       1 deployment_replicas_syncer_controller.go:156] wait until replicas of binding (2) equal to replicas of deployment (1)
2024-07-04T04:42:46.941194503Z stderr F I0704 04:42:46.941089       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:46.941763033Z stderr F I0704 04:42:46.941702       1 deployment_replicas_syncer_controller.go:129] replicas in spec field (1) already equal to in status field (1)
2024-07-04T04:42:47.790522419Z stderr F I0704 04:42:47.790373       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:47.790545312Z stderr F I0704 04:42:47.790446       1 deployment_replicas_syncer_controller.go:129] replicas in spec field (1) already equal to in status field (1)
2024-07-04T04:42:54.537885193Z stderr F I0704 04:42:54.536588       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:54.601312782Z stderr F I0704 04:42:54.601238       1 deployment_replicas_syncer_controller.go:146] successfully update deployment (karmadatest-m5dpb/deploy-g69xd) replicas from 4 to 5
2024-07-04T04:42:54.601813556Z stderr F I0704 04:42:54.601572       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:54.60183161Z stderr F I0704 04:42:54.601651       1 deployment_replicas_syncer_controller.go:129] replicas in spec field (5) already equal to in status field (5)
2024-07-04T04:42:54.631012895Z stderr F I0704 04:42:54.630939       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:54.631175348Z stderr F I0704 04:42:54.631121       1 deployment_replicas_syncer_controller.go:156] wait until replicas of binding (4) equal to replicas of deployment (5)
2024-07-04T04:42:55.633735224Z stderr F I0704 04:42:55.633616       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:55.658987001Z stderr F I0704 04:42:55.658890       1 deployment_replicas_syncer_controller.go:146] successfully update deployment (karmadatest-m5dpb/deploy-g69xd) replicas from 5 to 6
2024-07-04T04:42:55.65915775Z stderr F I0704 04:42:55.659098       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:55.659294865Z stderr F I0704 04:42:55.659247       1 deployment_replicas_syncer_controller.go:129] replicas in spec field (6) already equal to in status field (6)
2024-07-04T04:42:58.44880422Z stderr F I0704 04:42:58.448717       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:58.45040973Z stderr F I0704 04:42:58.450326       1 deployment_replicas_syncer_controller.go:156] wait until replicas of binding (6) equal to replicas of deployment (10)
2024-07-04T04:42:59.483885582Z stderr F I0704 04:42:59.475252       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:59.483954461Z stderr F I0704 04:42:59.483873       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:42:59.483962024Z stderr F I0704 04:42:59.483914       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:59.48402365Z stderr F I0704 04:42:59.483939       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:00.511409144Z stderr F I0704 04:43:00.511009       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:00.511540069Z stderr F I0704 04:43:00.511491       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:01.536768598Z stderr F I0704 04:43:01.534993       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:01.536789396Z stderr F I0704 04:43:01.535103       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:02.579765844Z stderr F I0704 04:43:02.579690       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:02.600448542Z stderr F I0704 04:43:02.600279       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:03.643271921Z stderr F I0704 04:43:03.602373       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:03.643306346Z stderr F I0704 04:43:03.602467       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:04.615004062Z stderr F I0704 04:43:04.609571       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:04.615016696Z stderr F I0704 04:43:04.609625       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:05.610707101Z stderr F I0704 04:43:05.610588       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:05.610844317Z stderr F I0704 04:43:05.610786       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:06.612604682Z stderr F I0704 04:43:06.612479       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:06.612708105Z stderr F I0704 04:43:06.612591       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:07.617791715Z stderr F I0704 04:43:07.617742       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:07.618290706Z stderr F I0704 04:43:07.618243       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:08.995610749Z stderr F I0704 04:43:08.623748       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:08.995760749Z stderr F I0704 04:43:08.992365       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:09.993049138Z stderr F I0704 04:43:09.992990       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:09.993207694Z stderr F I0704 04:43:09.993163       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:10.994547653Z stderr F I0704 04:43:10.994465       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:10.994724032Z stderr F I0704 04:43:10.994683       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:11.994980148Z stderr F I0704 04:43:11.994878       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:11.995400993Z stderr F I0704 04:43:11.995050       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:13.001582738Z stderr F I0704 04:43:13.001435       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:13.009395901Z stderr F I0704 04:43:13.009305       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:14.009906666Z stderr F I0704 04:43:14.009715       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:14.032956097Z stderr F I0704 04:43:14.009832       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:15.010596957Z stderr F I0704 04:43:15.010545       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:15.012511178Z stderr F I0704 04:43:15.010688       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:16.013161271Z stderr F I0704 04:43:16.013056       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:16.013324646Z stderr F I0704 04:43:16.013279       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:17.013842394Z stderr F I0704 04:43:17.013733       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:17.014036596Z stderr F I0704 04:43:17.013971       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:18.206775381Z stderr F I0704 04:43:18.199304       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:18.20678021Z stderr F I0704 04:43:18.199394       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:19.225669668Z stderr F I0704 04:43:19.225582       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:19.230145606Z stderr F I0704 04:43:19.228308       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:20.230956422Z stderr F I0704 04:43:20.230853       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:20.231144663Z stderr F I0704 04:43:20.231090       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:21.232135366Z stderr F I0704 04:43:21.232021       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:21.232332634Z stderr F I0704 04:43:21.232274       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:22.233620913Z stderr F I0704 04:43:22.233483       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:22.233655998Z stderr F I0704 04:43:22.233584       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:23.234693049Z stderr F I0704 04:43:23.234511       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:23.235077005Z stderr F I0704 04:43:23.235007       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:24.238761179Z stderr F I0704 04:43:24.238678       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:24.238910217Z stderr F I0704 04:43:24.238831       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:25.239605969Z stderr F I0704 04:43:25.239499       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:25.23981552Z stderr F I0704 04:43:25.239759       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:26.240888078Z stderr F I0704 04:43:26.240759       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:26.241134488Z stderr F I0704 04:43:26.241048       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:27.24218126Z stderr F I0704 04:43:27.242049       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:27.242206257Z stderr F I0704 04:43:27.242146       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:28.253078689Z stderr F I0704 04:43:28.245814       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:28.253088727Z stderr F I0704 04:43:28.251382       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:29.256729894Z stderr F I0704 04:43:29.256600       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:29.256766452Z stderr F I0704 04:43:29.256703       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:30.283301444Z stderr F I0704 04:43:30.283119       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:30.283443859Z stderr F I0704 04:43:30.283382       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:31.283734007Z stderr F I0704 04:43:31.283598       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:31.284009511Z stderr F I0704 04:43:31.283944       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:32.284739839Z stderr F I0704 04:43:32.284626       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:32.284822934Z stderr F I0704 04:43:32.284769       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:33.28553309Z stderr F I0704 04:43:33.285426       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:33.285755475Z stderr F I0704 04:43:33.285703       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:34.286656118Z stderr F I0704 04:43:34.286518       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:34.286816767Z stderr F I0704 04:43:34.286764       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:35.294992759Z stderr F I0704 04:43:35.294006       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:35.295366987Z stderr F I0704 04:43:35.294119       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:36.29508643Z stderr F I0704 04:43:36.294961       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:36.29512414Z stderr F I0704 04:43:36.295045       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:37.295342322Z stderr F I0704 04:43:37.295189       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:37.295365095Z stderr F I0704 04:43:37.295264       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:38.295741609Z stderr F I0704 04:43:38.295643       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:38.295814394Z stderr F I0704 04:43:38.295753       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:39.296305295Z stderr F I0704 04:43:39.296180       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:39.296399281Z stderr F I0704 04:43:39.296276       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:40.29744568Z stderr F I0704 04:43:40.297331       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:40.297479062Z stderr F I0704 04:43:40.297401       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:41.298582571Z stderr F I0704 04:43:41.298433       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:41.299078476Z stderr F I0704 04:43:41.298987       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:42.299877223Z stderr F I0704 04:43:42.299756       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:42.29991297Z stderr F I0704 04:43:42.299844       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:43.300724312Z stderr F I0704 04:43:43.300619       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:43.300750721Z stderr F I0704 04:43:43.300688       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:44.301558372Z stderr F I0704 04:43:44.301432       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:44.3015843Z stderr F I0704 04:43:44.301503       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:45.30220186Z stderr F I0704 04:43:45.302082       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:45.302316327Z stderr F I0704 04:43:45.302211       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:46.303055614Z stderr F I0704 04:43:46.302959       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:46.303111756Z stderr F I0704 04:43:46.303032       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:47.303599296Z stderr F I0704 04:43:47.303435       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:47.303653994Z stderr F I0704 04:43:47.303544       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:48.304638729Z stderr F I0704 04:43:48.304493       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:48.304683642Z stderr F I0704 04:43:48.304616       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:49.309119683Z stderr F I0704 04:43:49.304715       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:49.309310148Z stderr F I0704 04:43:49.309258       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:50.310571176Z stderr F I0704 04:43:50.310451       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:50.310641647Z stderr F I0704 04:43:50.310574       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:51.311769002Z stderr F I0704 04:43:51.311664       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:51.311797284Z stderr F I0704 04:43:51.311739       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:52.312894451Z stderr F I0704 04:43:52.312782       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:52.312931872Z stderr F I0704 04:43:52.312877       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:53.313885509Z stderr F I0704 04:43:53.313756       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:53.313998073Z stderr F I0704 04:43:53.313907       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:54.324535097Z stderr F I0704 04:43:54.324396       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:54.32454234Z stderr F I0704 04:43:54.324495       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:55.32563716Z stderr F I0704 04:43:55.325532       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:55.325670692Z stderr F I0704 04:43:55.325606       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:56.326297593Z stderr F I0704 04:43:56.326190       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:56.326351323Z stderr F I0704 04:43:56.326284       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:57.327330613Z stderr F I0704 04:43:57.327239       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:57.327368715Z stderr F I0704 04:43:57.327317       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:58.328475923Z stderr F I0704 04:43:58.328375       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:58.328504757Z stderr F I0704 04:43:58.328457       1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:59.328768022Z stderr F I0704 04:43:59.328640       1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:59.328786577Z stderr F I0704 04:43:59.328685       1 deployment_replicas_syncer_controller.go:107] no need to update deployment replicas for binding not found

the key error log is deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected, the corresponding code is:

karmada/pkg/controllers/deploymentreplicassyncer/deployment_replicas_syncer_controller.go

Lines 175 to 181 in c4a81bf

// make sure the status.replicas in binding has been collected from work to binding.
bindingStatusSumReplicas := int32(0)
for _, status := range binding.Status.AggregatedStatus {
if status.Status == nil {
klog.V(4).Infof("wait until aggregated status of cluster %s collected", status.ClusterName)
return false
}

So, when we manually modify deployment template's spec.replicas from 6 to 10, the controller is expected to update it back to 6 on condition of member clusters' status collected, however, it's strange that binding.Status.AggregatedStatus["member1"].Status is nil, which caused wait loop keep executing.

@RainbowMango
Copy link
Member Author

how many times does this error occur?

Just found it once on the master branch.

@chaosi-zju
Copy link
Member

So, when we manually modify deployment template's spec.replicas from 6 to 10, the controller is expected to update it back to 6 on condition of member clusters' status collected, however, it's strange that binding.Status.AggregatedStatus["member1"].Status is nil, which caused wait loop keep executing.

Considering the low probability of occurrence and there are no major errors in logic, shall we do thing to handle it? @RainbowMango

@RainbowMango
Copy link
Member Author

Found another failure at PR 5146 with the similar output:
https://github.com/karmada-io/karmada/actions/runs/9917886710/job/27401983740?pr=5146

[FAILED] in [It] - /home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 @ 07/13/24 06:38:03.994
  STEP: Removing PropagationPolicy(karmadatest-8jmgv/deploy-6pbdk) @ 07/13/24 06:38:03.995
  STEP: Removing Deployment(karmadatest-8jmgv/deploy-6pbdk) @ 07/13/24 06:38:03.999
  STEP: Removing HPA(karmadatest-8jmgv/deploy-6pbdk) @ 07/13/24 06:38:04.002
  STEP: Check if deployment(karmadatest-8jmgv/deploy-6pbdk) disappears on member clusters @ 07/13/24 06:38:04.009
  << Timeline

  [FAILED] Timed out after 60.001s.
  Expected
      <bool>: false
  to equal
      <bool>: true
  In [It] at: /home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 @ 07/13/24 06:38:03.994

  Full Stack Trace
    github.com/karmada-io/karmada/test/e2e.assertDeploymentTemplateReplicas({0xc000b22798, 0x11}, {0xc000899224, 0xc}, 0x6)
    	/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 +0x257
    github.com/karmada-io/karmada/test/e2e.init.func13.4.2.3()
    	/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:142 +0x1de
    github.com/karmada-io/karmada/test/e2e.init.func13.4.2()
    	/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:139 +0x2fa

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chaosi-zju chaosi-zju

Labels
kind/flake Categorizes issue or PR as related to a flaky test.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@RainbowMango @chaosi-zju