Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to push artifacts to the project #19902

Closed
microyahoo opened this issue Jan 24, 2024 · 26 comments
Closed

Failed to push artifacts to the project #19902

microyahoo opened this issue Jan 24, 2024 · 26 comments
Assignees
@wy65701436
Labels
area/performance more-info-needed The issue author need to provide more details and context to the issue Stale

Comments

@microyahoo
Copy link
Contributor

microyahoo commented Jan 24, 2024 •
edited
Loading

When push artifacts to the project we met a lot of errors, no response, 500 Internal Server Error, 502 Bad Gateway, http: proxy error: context canceled or other errors. The registry could not find any useful infos, seem like it failed to update quota usage. How can I figure out what's wrong with harbor? Thanks a lot.

harbor version: harbor-core:v2.10.0

ERRO[0445] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0542, error: failed commit on ref "config--sha256:33a99864ad06dde5266521987f02bdf4a296af6ead7df4d3cdc3f422f0e8": unexpected status: 500 Internal Server Error  source=console                  
ERRO[0446] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0544, error: failed commit on ref "unknown-sha256:90c7451250a331ea12e0e3724f980646eb385d630334d8e9547838066dda5474": unexpected status: 500 Internal Server Error  source=console                 
ERRO[0459] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0551, error: failed commit on ref "unknown-sha256:864afecfab23de142c1056b0fe96f923afe8416eebb033ed02a23349a34b9264": unexpected status: 500 Internal Server Error  source=console                 
ERRO[0485] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0566, error: failed commit on ref "unknown-sha256:b5ac28930b97687ea019c0e60b5d618f54ac707e3218d9d3400c9669f1b3371b": unexpected status: 500 Internal Server Error  source=console                 
ERRO[0487] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0556, error: failed commit on ref "config-sha256:a9f81ec686250237c70af838d41db0b4671b8c0adc42f58d36a8453bb0055166": unexpected status: 500 Internal Server Error  source=console                  
ERRO[0487] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0557, error: failed commit on ref "config-sha256:6d4e1ce65eeacd8e5eee1cb7433ce52d026a6027faf1a1d48b3e637f2ba17e63": unexpected status: 500 Internal Server Error  source=console                  
ERRO[0489] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0550, error: failed commit on ref "unknown-sha256:b74eafac9eb23196e290c4a18878746562478fae426161ef63bb70683304f470": unexpected status: 500 Internal Server Error  source=console                 
ERRO[0504] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0562, error: failed commit on ref "unknown-sha256:a19be41fe0fcca49f09d3cd983292127b336108262e31df8ecdb1a7dda40004a": unexpected status: 500 Internal Server Error  source=console                 
ERRO[0544] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0605, error: failed commit on ref "unknown-sha256:dba337de1501f715a50e470973bd459542bc60c46bd8e4e44b4db8b99d468f70": unexpected status: 500 Internal Server Error  source=console                 
ERRO[0587] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0615, error: failed commit on ref "config-sha256:435c75ba3fd76cf6a732d95c394d12c27a8283fd71940aab8add65af2ef31930": unexpected status: 500 Internal Server Error  source=console                  
ERRO[0587] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0592, error: failed commit on ref "unknown-sha256:d202ee1f8add7720d8091e703e374bac189ebf8a3b3f3a19106869277b3ae9ef": unexpected status: 500 Internal Server Error  source=console                 
ERRO[0587] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0581, error: failed commit on ref "config-sha256:8f318000015cbd907be21db84036ac84e895fb513697f403a0e9e3d58743beee": unexpected status: 500 Internal Server Error  source=console                  
ERRO[0635] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0949, error: failed commit on ref "manifest-sha256:2ade5784ce8b40860a69c6b4c922322a0c789bb44e47756491e7e08fc4f4859a": no response  source=console                                                 
ERRO[0635] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0789, error: failed commit on ref "manifest-sha256:fe2328b64ead028480953e44444dcfd21f21e3fcaf6567f1896b1990c1222306": no response  source=console
ERRO[0635] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0739, error: failed commit on ref "config-sha256:2cb5c81e8d94a426e22ff66a6d7e1648541984fe0f1609106d81fbef5e571b7b": no response  source=console
ERRO[0635] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0937, error: failed commit on ref "manifest-sha256:03f18fb651039485742523299b5411b05374f13630d673094451de00b552ee9a": no response  source=console
ERRO[0635] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0991, error: failed commit on ref "config-sha256:4fab1702e004d795a0975784a5922ef9092a0b345246290f4a2d5710afe43954": no response  source=console
ERRO[0635] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0896, error: failed commit on ref "config-sha256:6998186021c5aed46a6342d5bc68727c2ea31e7a21a4ed8f6a3a44efb524a57b": no response  source=console
ERRO[0635] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0992, error: failed commit on ref "manifest-sha256:227fd68a1cefe12e2d32b014bf6b49fb2b59ba81b3bd60e198c20dbb39c1ce39": no response  source=console                                                 
ERRO[0635] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0928, error: failed commit on ref "manifest-sha256:99ba5bc6424be48afaa25ddb83afa2f431974d686056041eebf8e17be9c39bb2": no response  source=console                                                 
ERRO[0635] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0960, error: failed commit on ref "manifest-sha256:65f621b3572ea69622d77ad5505e9a219ccc3ac47e4e54f15ad62770a8b16ffe": no response  source=console                                                 
ERRO[0635] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0938, error: failed commit on ref "config-sha256:090cdf1d3c04b438133536b0ce3960573069031b537a84ed3eae9bc7690f7cf3": no response  source=console                                                   
ERRO[0635] failed to push reg.srv.deeproute.cn/project-031/repository-1706081006002:tag-0830, error: failed commit on ref "manifest-sha256:6726d5f7629dbef3f25cef948733b3d3cd07f3c2294543ac2cb8415be3ee31b9": no response  source=console                                                 
                                                                                                                                             
                                                                                                                                             
     █ setup                                                                                                                                 
                                                                                                                                             
     █ teardown                                                                                                                              

     data_received............: 0 B    0 B/s
     data_sent................: 0 B    0 B/s
     iteration_duration.......: avg=4m7s     min=175.42ms med=4m39s    max=6m58s    p(90)=5m33s    p(95)=5m39s   
     ✓ { group:::setup }......: avg=4.57s    min=4.57s    med=4.57s    max=4.57s    p(90)=4.57s    p(95)=4.57s   
     ✓ { group:::teardown }...: avg=175.42ms min=175.42ms med=175.42ms max=175.42ms p(90)=175.42ms p(95)=175.42ms
     ✓ { scenario:default }...: avg=4m7s     min=6.65s    med=4m40s    max=6m58s    p(90)=5m33s    p(95)=5m39s   
     iterations...............: 878    1.383173/s
     success..................: 61.27% ✓ 538      ✗ 340  
     vus......................: 123    min=0      max=500
     vus_max..................: 500    min=500    max=500


running (10m34.8s), 000/500 VUs, 878 complete and 122 interrupted iterations
default ✗ [================================>-----] 500 VUs  10m30.0s/10m0s  0878/1000 shared iters
ERRO[0636] GoError: failed to delete artifact project-031/repository-1706081006002:tag-0000, error: [DELETE /projects/{project_name}/reposito
ries/{repository_name}/artifacts/{reference}][404] deleteArtifactNotFound  &{Errors:[0xc0087e6000]}
        at reflect.methodValueCall (native)
        at teardown (file:///root/perf/scripts/test/push-artifacts-to-same-project.js:67:30(36))  hint="script exception"
Error: running "k6-harbor run ./scripts/test/push-artifacts-to-same-project.js --no-usage-report --vus 500 --iterations 1000" failed with exi
t code 107
Run       [======================================] setup()
default   [--------------------------------------]
ERRO[0147] GoError: failed to push reg.srv.deeproute.cn/project-060/repository-1706015003594:tag-1706015003594, error: failed commit on ref "manifest-sha256:9b15eb416511109d4d79ae529633de73b7264c0c6f6361f517cba690cc66d058": unexpected status: 502 Bad Gateway
        at reflect.methodValueCall (native)
        at setup (file:///root/go/src/github.com/goharbor/perf/scripts/test/pull-artifacts-from-different-projects.js:38:20(51))  hint="script exception"
Error: running "k6-harbor run scripts/test/pull-artifacts-from-different-projects.js --no-usage-report" failed with exit code 107
exit status 107
~/go/src/github.com/goharbor/perf (main ✗) go run mage.go run push-artifacts-to-different-projects

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: ./scripts/test/push-artifacts-to-different-projects.js
     output: -

  scenarios: (100.00%) 1 scenario, 500 max VUs, 24h0m30s max duration (incl. graceful stop):
           * default: 1000 iterations shared among 500 VUs (maxDuration: 24h0m0s, gracefulStop: 30s)

ERRO[0026] failed to push reg.srv.deeproute.cn/project-100/repository-1706023860548:tag-0504, error: failed commit on ref "config-sha256:64e128
9fef1160c02bd921bf3e885aca0d73812cb7156c0a34e0727178066fa5": unexpected status: 500 Internal Server Error  source=console
ERRO[0027] failed to push reg.srv.deeproute.cn/project-018/repository-1706023860548:tag-0507, error: failed commit on ref "unknown-sha256:4d8ca
5fcc0b85c46f2b8d7d735685a234ea63badc29f568248576e89439ea432": unexpected status: 500 Internal Server Error  source=console
ERRO[0028] failed to push reg.srv.deeproute.cn/project-030/repository-1706023860549:tag-0515, error: failed commit on ref "config-sha256:fce9a6
8a4ce2d7556be09ec968970ab10a8c8db41929675470a9f809c587bac6": unexpected status: 500 Internal Server Error  source=console

deploy:

~ kubectl --kubeconfig ~/.kube/dr-config get pods -n harbor-server -o wide                                 
NAME                                 READY   STATUS    RESTARTS   AGE   IP               NODE                                    NOMINATED NODE   READINESS GATES
harbor-core-5f6547b76c-2cmf8         1/1     Running   0          40m   10.233.116.188   node02.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-core-5f6547b76c-8ldtx         1/1     Running   0          40m   10.233.94.179    node04.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-core-5f6547b76c-g687l         1/1     Running   0          40m   10.233.107.184   node06.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-exporter-8584c59495-dvgbv     1/1     Running   0          20h   10.233.102.226   node03.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-jobservice-5f878d4bf8-c698f   1/1     Running   0          20h   10.233.94.249    node04.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-jobservice-5f878d4bf8-cdq5w   1/1     Running   0          20h   10.233.102.45    node03.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-portal-758585db78-trhm5       1/1     Running   0          20h   10.233.125.239   node05.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-registry-6b77c9d778-855cg     2/2     Running   0          40m   10.233.106.76    node01.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-registry-6b77c9d778-zbhpb     2/2     Running   0          40m   10.233.112.22    node07.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-registry-6b77c9d778-zh47w     2/2     Running   0          40m   10.233.120.57    node08.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-trivy-0                       1/1     Running   0          20h   10.233.102.87    node03.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>

log:
harbor.tar.gz
@microyahoo microyahoo changed the title Failed to push artifacts to the same project Failed to push artifacts to the project Jan 24, 2024
@chlins
Copy link
Member

chlins commented Jan 25, 2024

Please attach the harbor database logs as well.

@microyahoo
Copy link
Contributor Author

microyahoo commented Jan 25, 2024 •
edited
Loading

We use the Aliyun pg database, which can not be downloaded. Viewing the log is anti-human.

2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 400
-- | --
2024年1月25日14:10:39 | LOG: process 3993 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.259 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3904 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.192 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3542 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.649 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3989 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.112 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3991 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.017 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3992 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.950 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3879 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.868 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3983 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.790 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3746 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.715 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3980 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.045 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3985 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.606 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3984 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.542 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3986 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.467 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.

2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
-- | --
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3944 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.400 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3687 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.817 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3943 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.296 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3829 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.364 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3383 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.146 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3928 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.186 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3886 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1002.299 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3028 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.561 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3835 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.713 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3413 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.412 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3673 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1001.430 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3913 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.267 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 4010 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.275 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3998 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.187 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3997 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.172 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3918 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.121 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3909 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.701 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3906 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.109 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3907 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.465 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3162 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.106 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3559 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.092 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3841 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.086 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3978 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.214 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3982 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.107 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3977 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.205 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3324 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.071 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3782 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.091 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 4102 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.069 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3106 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.038 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3982 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1508.515 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3907 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1432.161 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3324 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1379.138 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3978 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1327.658 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3977 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1272.214 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3909 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1222.151 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5

2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
-- | --
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3877 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1028.971 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3728 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1028.456 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3843 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1027.949 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3863 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1027.438 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3722 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1026.937 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3654 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1026.453 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3842 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1025.959 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3735 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1025.499 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3933 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1024.995 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3803 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1024.544 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3571 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1024.053 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3778 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1023.613 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3157 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1023.161 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3437 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1022.729 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3387 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1022.293 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3988 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1021.872 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3435 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1021.459 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3229 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1021.054 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3336 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1020.644 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3637 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1020.227 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3798 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1019.838 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3636 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1019.461 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3613 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1019.088 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3609 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1018.739 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3058 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1018.374 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3979 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1019.701 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3271 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1017.977 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3704 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1017.619 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3103 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1017.271 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3464 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1016.940 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3503 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1016.601 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 4102 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1018.196 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3119 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1016.172 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.

@microyahoo
Copy link
Contributor Author

The failure rate is high when push artifacts to same project due to db lock

🍺 /home/DEEPROUTE/liangzheng/go/src/github.com/goharbor/perf ☞ git:(main) ✗ go run mage.go run push-artifacts-to-same-project      

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: ./scripts/test/push-artifacts-to-same-project.js
     output: -

  scenarios: (100.00%) 1 scenario, 500 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 1000 iterations shared among 500 VUs (maxDuration: 10m0s, gracefulStop: 30s)

ERRO[0417] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0511, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0424] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0355, error: failed commit on ref "manifest-sha256:17263e1b56449c2da9b46d05a0d347a2cd5cc2d4c113146c3556c072fea53b70": unexpected status: 500 Internal Server Error  source=console
ERRO[0424] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0518, error: failed commit on ref "config-sha256:ad36a672e2e7eb51fec8e3b02489569ca75f73953d349617ba6a4a930f00665c": unexpected status: 500 Internal Server Error  source=console
ERRO[0424] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0336, error: failed commit on ref "manifest-sha256:8d5ff1ef3e7f6f95ad6245a698b608a62a28125e386278f7c152bf66c82a16e4": unexpected status: 500 Internal Server Error  source=console
ERRO[0424] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0514, error: failed commit on ref "config-sha256:88d554378c8356d4a84865c20c7f2bb464f087abbcb1ee902c885925b4ffaa3a": unexpected status: 500 Internal Server Error  source=console
ERRO[0424] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0113, error: failed commit on ref "manifest-sha256:99af304c45bff60789594d2bdf1cd51702304d65380179dd6ad1eee386433495": unexpected status: 500 Internal Server Error  source=console
ERRO[0427] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0295, error: failed commit on ref "manifest-sha256:ffb8572bb947b57ccd22bd768a14c6ace293a9628501d67869937e36f90f4859": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0345, error: failed commit on ref "manifest-sha256:3465648934477b6e21d69e221feb6b7cd2812cc1c38d1c9bb6ea518b636ce735": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0082, error: failed commit on ref "manifest-sha256:2406c9a257ca8c0a258edd4501ebe42c6b8bbb3cd20de5ba309ef74df471f990": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0133, error: failed commit on ref "manifest-sha256:7f5db2e3fcb83b256609ae6e8aa7853279ff493ad93644dc021d8b07447ebcc8": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0148, error: failed commit on ref "manifest-sha256:dcd5c9f53ca1d3d6cf6393dbe0eb1249afaeddf5ba994e8eeffec47bb0668f7f": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0305, error: failed commit on ref "manifest-sha256:0a402ed28939acc0adca283a8a1cab1871b85899257ac2c064be9adc1d39716b": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0056, error: failed commit on ref "manifest-sha256:6e17ff3eb86d67d101c3fc6b71d6c472d5b026a59d18d2a8678d505790aa3ee5": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0237, error: failed commit on ref "manifest-sha256:7b494be7bd831226100b1f2d0d771c2cc77034ed16f5f24c6348c6b79e114651": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0149, error: failed commit on ref "manifest-sha256:f60af2805bb29b3f566479c2b8f0cf87cd719b3e78d9c0b98609fa28dbf86ae0": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0327, error: failed commit on ref "manifest-sha256:86dab818e990813711640ab74a0251c5f9a180950ae32256c28334fb5398d1f1": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0073, error: failed commit on ref "manifest-sha256:28d63a012c46712a73052bb00c893c3b01f38ad6ca4befff4e6c84c9da580888": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0176, error: failed commit on ref "manifest-sha256:fe1cc89255037d70a765819ce8848984afa00e46c440f5e7bfc362cd349bcb07": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0157, error: failed commit on ref "manifest-sha256:a02d47b7ed6cb628939a6c1c04a9916b8b4f5ac4c2382dc7c70db8fa929751ec": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0131, error: failed commit on ref "manifest-sha256:fd284c2565c288aed95b987de078d58115decfc140efb96d042df94f5697a513": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0296, error: failed commit on ref "manifest-sha256:9216ffb0362bd193b8b9ed916703781b082208969a91f554879b4a0821cc9c78": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0101, error: failed commit on ref "manifest-sha256:678ae08528cbc780333dd1acf499190e8ce4af3d971a2623133546e3349284d2": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0524, error: failed commit on ref "config-sha256:b8d24567e7997cb96761a951a006f4ef90da767c09dfd5642a7f5cff76b2a0ff": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0416, error: failed commit on ref "manifest-sha256:3735ae81d5f8d761b6c52d5a4d5559de3eb9d99f59b5040ba5c8fd957ca63746": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0180, error: failed commit on ref "manifest-sha256:1989b7538c7225adb1ae1e2994e9cd10cd53eb425f005dba3669dc65f0e28e0e": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0323, error: failed commit on ref "manifest-sha256:87d248ce8da84a2b4333ddd27dddef08fe7e4eb0fe57dc0d06623a90fbf720ab": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0513, error: failed commit on ref "config-sha256:f0a3d60b2db4c24226c50e80c2cd7efcfed85915a8cbb1f63a38038e27b56fbd": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0308, error: failed commit on ref "manifest-sha256:9da1db507739d0dd64fb121989a5bc1f29c636456eae25a5f39452f326d1196e": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0437, error: failed commit on ref "manifest-sha256:631e8203a8fc809c9894da1c2c933571679bf3882ec84cd2375b982508bf2e61": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0076, error: failed commit on ref "manifest-sha256:c1249c83ee47aaa5967daa79d3a3377cfd3699b51040dc01b2feff57ec4383bb": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0349, error: failed commit on ref "manifest-sha256:03ad104a3f8327c88555c60cf68d49a25b1ac10b9e0810186d8791ad2e1d7335": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0529, error: failed commit on ref "config-sha256:a78feb8f4f0c09a6982ed950d00d5269622a92182c9eae29cb092c022c81823b": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0543, error: failed commit on ref "config-sha256:7910a5581e482972b51cacdade4f1b3e1d428b110e69d1fa397a156b2f651917": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0187, error: failed commit on ref "manifest-sha256:18c17731ec5ce07560193a8ce787d3db018fc40aebd1519951387c3a27af596a": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0531, error: failed commit on ref "config-sha256:9bf9233f8a116e7af6d8b8adfba5ede7196625ca0501511eef106d6095024799": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0094, error: failed commit on ref "manifest-sha256:eff5657b392b0e8b3766c0df0e3f9465fac4da262002a979da95e4a993b4a3ae": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0487, error: failed commit on ref "manifest-sha256:267d5a9a002d0c31c5a892e8b58263c5aff359d50b48f94c2f6e68433b2ed91e": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0516, error: failed commit on ref "config-sha256:23eff14183848416f89001bc5cea0ae4bf2f93ec6f707f667a238751f0d223cf": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0055, error: failed commit on ref "manifest-sha256:09f281e15f37af6d7cbf9509eb71b5858d7a64d8488825d5e712e9ccd4cc2af3": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0412, error: failed commit on ref "manifest-sha256:d195036926225418dc07b1e669a17960fdca63d567ec6d403870f55d6ac99d28": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0304, error: failed commit on ref "manifest-sha256:d8445a6875f7a672123d6650b23ef53a8fa8b3eb656da9ed8ea405e60dbcad4a": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0217, error: failed commit on ref "manifest-sha256:fd5b75907618517c1aa908957d116bd62357563f8a893179c35c6ca1c11358f9": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0538, error: failed commit on ref "config-sha256:b80816852552346c218e9f103ba72b7a90ccc588b40d2fea79e046bf61d6aff3": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0464, error: failed commit on ref "manifest-sha256:1a13ff5419f9cf7adf56709493d62b7af18a09cb943f8584bf8df79ef17827bf": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0166, error: failed commit on ref "manifest-sha256:e5c976345d4518013d768d3512db45e4f8cd1be1e765677c519bd38b18536655": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0002, error: failed commit on ref "manifest-sha256:de6e4cb3feef46c16196588cc3667f6349c7a77ee8b1295d24a95e9c9bb8f9db": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0151, error: failed commit on ref "manifest-sha256:ae7c0b0a36f7025e54eb167defa4b9087bd38f7e2cad67db72bff2f9c807b2ec": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0303, error: failed commit on ref "manifest-sha256:c366ab6bab90d113c3bbf0f9c3e2a7e2347b77bfd3323d5c2cf36d8b83d7e3b5": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0322, error: failed commit on ref "manifest-sha256:f7f5bcb91f7276ad64c1e7c0435fb0fdd2f9b191dbfccc95798c8e6dc3d97f5f": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0449, error: failed commit on ref "manifest-sha256:b904d880740afef32a5c05c366c23346e8262626b14abc2270ecaa43299a8d12": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0306, error: failed commit on ref "manifest-sha256:8b5b0e3a2c85b3956f9aebe2fa30a9f9262bd48ff3625b8ffa1f29077be14183": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0520, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0539, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0052, error: failed commit on ref "manifest-sha256:78195992b34d6d4021f8491e8974790b247fc0d5d179f48dbb38b04e4b8545fd": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0326, error: failed commit on ref "manifest-sha256:f9d3926603d72ad85bc364cebcbf340099c40ee8215795ad7080c5fbbe9f243a": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0525, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0481, error: failed commit on ref "manifest-sha256:42f3853aed18ce4c6abbdeb8132229e4ac57c9dc67288b9da652bf6796fb4e3f": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0519, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0522, error: failed commit on ref "config-sha256:b8bb9cd006b016fb523370326fc73ff1987f47f1c0c264be2e4c728562aa51d8": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0179, error: failed commit on ref "manifest-sha256:c695aa6d5a5009215a71079a81b901f01d18c68bba048eae895a7431f5d1adda": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0255, error: failed commit on ref "manifest-sha256:f3671dea8b10f109543dcb6d16c5ccdd554277a9a4f4451901605939b8f12008": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0497, error: failed commit on ref "manifest-sha256:24500a461a3781f0bf36f5b9f41b68a96ccadfd0f3510a23427ae0ae5a83c019": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0542, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0475, error: failed commit on ref "manifest-sha256:ccf886a5691b8282c1e6cba76f4cc56b35c7686ffe1e800844ee5d790e83e836": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0120, error: failed commit on ref "manifest-sha256:839132229aa55a77c1a7c8aab8214187d3dc432cea6c2ec5ef9ebf61e662b9f9": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0234, error: failed commit on ref "manifest-sha256:eabcc03f4ec0643899a37a6f4184d5e03f7095dffb93d8af877857faf19769f3": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0049, error: failed commit on ref "manifest-sha256:a1c355f945eddbe7159cb6b3efd11f704b0d0c3d6ce8dcd0a087efdcc3000e9a": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0299, error: failed commit on ref "manifest-sha256:873c9d1759f0d80ef3cbff6a0548b39fe21a93aeb15627ce549a6018461b3fd2": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0265, error: failed commit on ref "manifest-sha256:cbc387aba1bc93903be3d2797779c0205a2ec80146174fe202920d65a9e572d7": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0271, error: failed commit on ref "manifest-sha256:01198418e69aaab9e75a8fcc39fd4344f17279049a43e9883276b70d730fb72c": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0310, error: failed commit on ref "manifest-sha256:36f0c02f9d87f81c3cb7be80113541b7fb3ae4527936527b3b4eebb12b9f8021": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0454, error: failed commit on ref "manifest-sha256:bc5f7fc520aef0b03e2571e6c0b7b05afd6c176cc0f9f658dfe3f36bbf799da6": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0407, error: failed commit on ref "manifest-sha256:6a8ffd15bbff2180e6ed0b304cf58b77c299c99d85b7988661afc428411f3181": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0429, error: failed commit on ref "manifest-sha256:a94f01125f3a3b32dd2a7e00456af48227c749a8ca0b264da72bdb781fb5fba1": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0535, error: failed commit on ref "config-sha256:cade28afe2a5f2aaf67ff996e18290c8b2a1258a2854936a88120ccda9367e46": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0314, error: failed commit on ref "manifest-sha256:76b56af91a674692d586892a12aa408a64a73711c2e02be30f284e92a55cb76c": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0425, error: failed commit on ref "manifest-sha256:42cd412fdb1223ea83d06e27e1e54f58aa54714786e96221ec1a7666016fd904": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0391, error: failed commit on ref "manifest-sha256:a312a2252d146595d6fc71e19938d58599e6feecae79ede005b5456b584d5730": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0331, error: failed commit on ref "manifest-sha256:1680a34498bfafea9ecb499b473d6adc3aabd39efb57b467fb366acf22c89593": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0256, error: failed commit on ref "manifest-sha256:6027deb5915fae7981d9dad12e24c047f36d693cd8c29029421bd551adb5eb83": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0122, error: failed commit on ref "manifest-sha256:fce3c3d70c0ac101a02166b565cc8d867db32ebf17ffb97bc1ce77670c8896fb": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0160, error: failed commit on ref "manifest-sha256:58ffb22aade12e6509dea19ccdc9a3f14f42c71c67256ca9fa8f7b8caa760541": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0278, error: failed commit on ref "manifest-sha256:0a8661bb08501a8d07e8a6878cf3b9015c9fd86e101be1eb412948aa6c10fa92": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0540, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0335, error: failed commit on ref "manifest-sha256:b90164de30796fd57715862df6c398ed235fcedaeebd4c464f7c0b6a09578a34": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0275, error: failed commit on ref "manifest-sha256:503636bf93f1ef977296dbe30e1647dd22ab819be54e99de64ada91661b4ee67": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0328, error: failed commit on ref "manifest-sha256:e4f05c8f2d5d62d9a87d01ae000eb17cc70317f34c2084b58dbaf6db38eff3f0": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0526, error: failed commit on ref "config-sha256:06f3f353d3472054261c53d9a4d79042976f0cc4dc73bb71936f7709f02bb817": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0245, error: failed commit on ref "manifest-sha256:82bd67a7f9582514d7c815bfb5613db7032b561d94b557a47883492e57fabb49": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0249, error: failed commit on ref "manifest-sha256:b9af0b0dcfb58a626f58c80b76da2e10b5a3b71710709a99e9ac54d2760f047f": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0291, error: failed commit on ref "manifest-sha256:38a77a83c78b1552ebe4350b55be41c7f5ae3d531d984c3a96e8c42efbd165eb": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0512, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0506, error: failed commit on ref "manifest-sha256:75d7f3f1d27c6cbe165f8cf900faaa43ac1ba39823e06fa17ca927aa525b95b4": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0515, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0072, error: failed commit on ref "manifest-sha256:7db9376226f3c11d78a2d03f79564d7d9d486f8fbf181f1cb881cab798c919c2": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0205, error: failed commit on ref "manifest-sha256:cabbe1b9ac8ffd6999debb567966bdb7f0e7d7e11c836cbb27b76dd25a2a25ec": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0510, error: failed commit on ref "config-sha256:ca86bbaf1a0321cf39607cce6621e0c026622322e7600a68d6a630b92ed5ab9e": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0532, error: failed commit on ref "config-sha256:62b80052c371e9d92f0839aa49883877a000e8a5d7a1b7ec9ae2a1256e8f37ad": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0380, error: failed commit on ref "manifest-sha256:13786d65d87bbd45e5da5974c0949d5217a9416e18edb0655f8306f0eb68d561": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0479, error: failed commit on ref "manifest-sha256:a6b54a0f683d597337f2f13d9f1a09ed4a44d177a562178bee82b8e96e5cd2e6": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0517, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0408, error: failed commit on ref "manifest-sha256:a72d00fb5d185c01e14f770c296a70f5b313d787289e9b38bfe72c68d5c50360": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0435, error: failed commit on ref "manifest-sha256:1365cfc8f8f17ef6fbf0f725306f51ec02a21926e1a963928d793384a3c66556": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0541, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0530, error: failed commit on ref "config-sha256:11f2695280b5a0e9d48399b8f58af75440bd2ee6a051537eaf9b936bcf7718fa": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0528, error: failed commit on ref "config-sha256:2a3f90e935a390968082bfebe911327fd5d14872121aeb1450a24a52ee05f85f": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0534, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0041, error: failed commit on ref "manifest-sha256:4b726cc20af511d2e44181d24ff0216b5b2501dee2ad00b94282e3613861e4ea": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0459, error: failed commit on ref "manifest-sha256:f729c185311bfab22c6ee90627806a7274a33cb78a5db1c7ab2b99c8707ff8b0": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0373, error: failed commit on ref "manifest-sha256:5efc615ec0affd00c375c5eefc50577decc62ee0f70b51e023269d4b1a7a87b2": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0228, error: failed commit on ref "manifest-sha256:9ac800c13563885e545b3b7b17723def1d029ff0d6437e2231ca1edab3b20124": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0023, error: failed commit on ref "manifest-sha256:4462aff82f9e7447eeb055f62cc29ce287644a77001d7b20096e14b2a97bb09f": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0130, error: failed commit on ref "manifest-sha256:48cb2e1e38693e7e01b3ce5d46be2200feb8d80ef5fc636298754183487f81cf": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0182, error: failed commit on ref "manifest-sha256:9b2ef6b4abec6ab67703b031f8b0e434699fe5e39c52019bd43e7f8d77f23060": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0527, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0276, error: failed commit on ref "manifest-sha256:f1444ef7ac5a43b127c1e002a4cae0d5163acca4869fafbe7d4e4785a2997676": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0401, error: failed commit on ref "manifest-sha256:1243d172be35055f4b162761c7cf05fa5841b771301d7558f133e2bb37d4364b": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0458, error: failed commit on ref "manifest-sha256:9e1bb0dc3f5bf58a0ebd80785ecfc41c40d30abfedf97b3d97359e92b5772771": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0365, error: failed commit on ref "manifest-sha256:3f30af72412e21763e8a610d99417e33e79334fb3ccde23ddbb3d72dc1d06011": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0125, error: failed commit on ref "manifest-sha256:10428d60d448e5094bf152795811a9a9d8b2a8cfbbd47be61feb2d765983f906": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0033, error: failed commit on ref "manifest-sha256:d4605cf3ddeddeb2aa15c32cc7407ef9c71a197d65604462e661748749083589": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0274, error: failed commit on ref "manifest-sha256:5d4b0982628d52fdf89f21eecabae321a34a7aa49f72975ddf5dc17f405fb7e4": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0096, error: failed commit on ref "manifest-sha256:7578ca7756b55654ae87bd285f0246684a2ef974c9d2d8ef3457c02545539c41": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0090, error: failed commit on ref "manifest-sha256:346f2b2a22a1d50d51fcc0fac2c80334f1177e5f01f01e17524eac14f5a0a642": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0302, error: failed commit on ref "manifest-sha256:14a801e1fa90b5c43b97057a604c97a1327020a2912af31f8bfcf170fb700ca7": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0171, error: failed commit on ref "manifest-sha256:8a6eb59cc87279e05fffee4cd87cd4a03961f9509b149b23eb17758f6a0b33f1": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0162, error: failed commit on ref "manifest-sha256:a3bde802d99d84a12bbc2eeba34a3a4cab23581d54cca66741bd25331cd10596": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0357, error: failed commit on ref "manifest-sha256:bc55a87bfedd7e0d16af10a2051e61ee2f0e4c033bcbe5535968f0cc9779fc1a": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0164, error: failed commit on ref "manifest-sha256:19edaefe5f01538e9522f6b68b3b54ddc990b6012e9c1041c41e22ba223e16ef": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0226, error: failed commit on ref "manifest-sha256:7ded8785c8956bcf6ca47306de0b6488b8df9ce679db6bb3da8a9bb17144c822": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0286, error: failed commit on ref "manifest-sha256:1b972266d14e4e5cc27cb7847ff356e86783757d7edb9e6fbd619cbf6b1f4d5e": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0246, error: failed commit on ref "manifest-sha256:c77a7b81d5bcb63d445bd7b3e8ef10d9d5ca26a73d2ac6bd47ea89ebc181ea58": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0312, error: failed commit on ref "manifest-sha256:f56ef3b70ce3b47bb657efaadf28464a9128ba0311bea14c850c15c86c0e369e": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0478, error: failed commit on ref "manifest-sha256:9c68630a1ab35d77453514cd6dee0efd6b3144efb847e1a453df72486667a841": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0088, error: failed commit on ref "manifest-sha256:5104ce5e3e058976ed18b05cc568fe38587acef648f65edcd9eba7fb74863c5f": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0521, error: failed commit on ref "config-sha256:01390157cef7afcfc05a78c3d97a08c1b0a804f149e49e51e5e6117f2a4db59b": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0290, error: failed commit on ref "manifest-sha256:215ed7a2a9368a5f1337de7a4e4c1a319251d76f4d0eaf4242c5df3fc72f6fec": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0272, error: failed commit on ref "manifest-sha256:f17c700bed48460c7b2a7e9b3f6e402261cf79d8a4f4dee17c581806dedb0f93": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0285, error: failed commit on ref "manifest-sha256:34ec8fd54aecdd994f8a5ce27372a789c13e1b242cf62797b6535896e3957774": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0165, error: failed commit on ref "manifest-sha256:62045737ce51926ebbf59f5fd157204b0f10ce1859cee335d9d9f657ccb37e22": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0198, error: failed commit on ref "manifest-sha256:9b774980bb26d8180b38d165d84bba78c4c1662613addcd4fe3ac3d93f671ea1": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0545, error: failed commit on ref "config-sha256:a43636293d914cd9fc581cbed39892db264d3d544befb06cf791f987abd61792": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0394, error: failed commit on ref "manifest-sha256:b602d44bfa1620208414f54ac21ad73945b3345122bc8e21a2406a2c4beaabfc": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0537, error: failed commit on ref "config-sha256:4ce29dd59b6e3786225b0272eaa40c5fb4d9fefc30d5d28808503f922737e53c": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0075, error: failed commit on ref "manifest-sha256:1ca997ff0b05566e4304cdaddc1086096544543b6b2ab7ba725ef1c34263ba91": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0093, error: failed commit on ref "manifest-sha256:93e52eb6ac248f03898e5b326c6249ed8e8850c10280d7a4caa2389ff84acc97": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0472, error: failed commit on ref "manifest-sha256:01dc3b40cdf6ef33320979cb77adfcc1592c5ee4dc9ecea77ebb401d9da6410c": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0186, error: failed commit on ref "manifest-sha256:b2b5c2522be33167b3ae4912d458378237e90f49debca8b8569574585ead3cbe": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0190, error: failed commit on ref "manifest-sha256:f10b0d42d4a1a112cf6806c06b034036e098c00e6dae5fc620c63843496099fe": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0546, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0231, error: failed commit on ref "manifest-sha256:5ece6ea8611f3d88bd76de671d5c73cffbfdcc59f7fdca7209af0904dc092dcc": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0132, error: failed commit on ref "manifest-sha256:df1e172c1444482cc1d0c03ace4f7cb4fa45152bc0f4080d709ca38d8f2a658e": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0447, error: failed commit on ref "manifest-sha256:c95fd7cc8e382c5e7775595e5a18c7a74ced2ce5f0c8a93b323037c57013971d": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0112, error: failed commit on ref "manifest-sha256:06b7f0a7174277c738f573b0f3c357e5a26b0bb7977043446cc5884d16aeb917": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0230, error: failed commit on ref "manifest-sha256:34d1edc3db4fe1aaba5b64e37f9cc38394b7ca153af96a0dd12c6b6a47843d5d": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0270, error: failed commit on ref "manifest-sha256:9b28367e7047ab231b693aa1afc1bfa25e40bd4952a99b2c799bdaa670ff2d1d": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0224, error: failed commit on ref "manifest-sha256:f2ffc822fb4c5fa512ed7ece770ea798f4377b37e9aef90fdf0d776681698362": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0207, error: failed commit on ref "manifest-sha256:f904af368c37a9aa96cdbcefe7896a5ed3b0e99fb84ce4b111d94d5367d58a8a": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0397, error: failed commit on ref "manifest-sha256:7a8eb42c3e0340cc0ded0a6dd580517607d66eaaa30edfb6dd79b038341091a3": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0533, error: failed commit on ref "config-sha256:c8c5cfce47254161078fa683caf49356490d4c75da35fe03fe52382181d609fb": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0334, error: failed commit on ref "manifest-sha256:2c591fbfcb990a540c907ac5f98754cabcb4261045de35694a4ee526326bf741": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0434, error: failed commit on ref "manifest-sha256:d94c83023b5d2beff49319aa7de2e526cd8fb1c223b9bed4d74be2adb0b65c54": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0450, error: failed commit on ref "manifest-sha256:c3b25a7263d27e5bdf6c59bcc547fcb2f81ced620e23cd30e53f91f6d7824184": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0220, error: failed commit on ref "manifest-sha256:207ada37501e2905344a1c68079d2e939be59cfe338ac3836ea3dd4385b3d6ce": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0124, error: failed commit on ref "manifest-sha256:e3daab9ffb20669e502588d5c84d9bde406d0fb6fbde5d4f9d6a614bc50a1e8e": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0037, error: failed commit on ref "manifest-sha256:adcb6d9b3a356393e8532397f6e5901a3966573b51a586c837ce096f9eda4f5a": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0211, error: failed commit on ref "manifest-sha256:65f339fb257b4266e12772d90b6cc2101f0ef76eb9c2bc72708b38fe3e9cd79b": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0463, error: failed commit on ref "manifest-sha256:fab23292a7dc82aa6dc0b41f5b259b377b859fe4bf8bae3f68ba7a5b29be2ba1": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0006, error: failed commit on ref "manifest-sha256:d4382da4591838715d4b8c2dc71c50e67fbd78afd24f116be395138b404e3e82": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0008, error: failed commit on ref "manifest-sha256:efc7be4c11b98f37052c33a57b9d05ffeb04858af3361377de45e133de0f1e4f": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0411, error: failed commit on ref "manifest-sha256:89ddc75cfc9cc6e0c11d3296781b4ea4557eee94ca329ff212992653725367be": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0243, error: failed commit on ref "manifest-sha256:a101aeb11de4916be2c99295310798ccb5ebb8900aa59021a071a4b47a849fdd": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0500, error: failed commit on ref "manifest-sha256:c85b3c4317837e6b76cae720ae240edc327110d5f45c3f22f40e365d502ca88a": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0150, error: failed commit on ref "manifest-sha256:cae612d67fe702539fe8a89cbd456f3bbf238d19b9871cfce9e70a892c753339": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0293, error: failed commit on ref "manifest-sha256:1c8a72bf2485ef8702c8d7ed27c41502ddd6fb4ee7820c6f14d6faf869997b2f": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0204, error: failed commit on ref "manifest-sha256:b7dca79fbc95a852df48c5d357bb9aba0436ccabdc0f071cda6e9e2925537130": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0185, error: failed commit on ref "manifest-sha256:88f8fe5dd01a941d6a8cbc04f369f27eaccc79a7c453232c1378b5bb0c1a5235": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0523, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0007, error: failed commit on ref "manifest-sha256:b93e0a3cc204631ed07eed20bee6a3976e10c8aeae637184fb7ca2ec9e043d55": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0369, error: failed commit on ref "manifest-sha256:eb77e90ed9b8fab498025677bec65872b24075f0b236bf8532f15dcaf06cdb83": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0128, error: failed commit on ref "manifest-sha256:22b833dec1971a495e78ef06c6bbe9c616981ee58517ebcc6234a3ce362535e6": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0352, error: failed commit on ref "manifest-sha256:7b5e6c0e2e77934caa1b1b12226ed6f134f4eabd5bfabe0d4be1ae0c2d42ed4e": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0390, error: failed commit on ref "manifest-sha256:7c50c5c00f55181f374f7837906a9ded6c8876224df92577cb272d875924d152": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0152, error: failed commit on ref "manifest-sha256:4d6a9437ce58978c82f88f8d391734dd466c805369c2df2c00ba6edd61813f6e": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0277, error: failed commit on ref "manifest-sha256:63465c55576e8bf750a38bc6f3646233c65523522c82e6ac22dd369ff8d8ab8c": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0126, error: failed commit on ref "manifest-sha256:8273906aea34817fdfb611edb057ca2befef760a517ac402095d10532990f7a8": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0547, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0462, error: failed commit on ref "manifest-sha256:55e02acb2386b5b693982954f93670dc5a5d644abbb7ed934343c5b13086433a": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0421, error: failed commit on ref "manifest-sha256:3b1c3df1f7d8089b38204c8d96245a1222c9d31fea4be4dbd1f14e4f9dd292cc": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0086, error: failed commit on ref "manifest-sha256:91593695f7370af9d1a9862ab1875245af13e607c942d719e6306ac8735d11df": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0199, error: failed commit on ref "manifest-sha256:5ff2ff065d0a62c2f746792a0a1f6f8b0498f10bfcce23e92f69788c632b5a0b": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0413, error: failed commit on ref "manifest-sha256:d22d3c1dd7df962f92e3649361495cc765f289518bc75abe4cfa08d73ba484c8": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0544, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0117, error: failed commit on ref "manifest-sha256:c290ff020ef3e528425f055e13d73638301391436edbc36b3b9b159cc463fdef": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0047, error: failed commit on ref "manifest-sha256:5594d4d4eda4def52a63e6687444ffcaba15b060f3c4940900b158b7a7b3bb6f": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0483, error: failed commit on ref "manifest-sha256:5e90873e625feca940381d338a03260915cd837a874fbcf0f296ea23d608de4b": unexpected status: 500 Internal Server Error  source=console
ERRO[0450] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0085, error: failed commit on ref "manifest-sha256:6150348c5bc181fec5bb20a0b7eb1470c44f901446750c4a4377be4ac094d8e5": unexpected status: 500 Internal Server Error  source=console
ERRO[0450] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0536, error: failed commit on ref "config-sha256:c52862699bb2ffe24edb070873e6b3a5ab37484cd0973378e5fcb0766e78305c": unexpected status: 500 Internal Server Error  source=console
ERRO[0451] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0473, error: failed commit on ref "manifest-sha256:86c1ed5a892a7144dd31568f5fd3fc1e7817ac07da442fc097e827b528b00551": unexpected status: 500 Internal Server Error  source=console
ERRO[0451] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0548, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0451] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0430, error: failed commit on ref "manifest-sha256:033ee48b9125e2e7080a3d596e41a58673375f06ec13c6c217af7be31de0575b": unexpected status: 500 Internal Server Error  source=console
ERRO[0451] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0502, error: failed commit on ref "manifest-sha256:d995da7ae1c0eef4db6fd7b2eddb3274095867981ae5d07b86eaa2b28e9a6f85": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0480, error: failed commit on ref "manifest-sha256:d1692f8bcdbc2f1fe8b4f843857aa70ed1773e646e84916d2a1b95e4050916db": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0346, error: failed commit on ref "manifest-sha256:02f247335d02ae0caece1278d2c7ac6f13a4a91cedfc93de333889f67b4c008b": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0550, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0038, error: failed commit on ref "manifest-sha256:e3409a0659b81ad759929672659d6b0bbe624455d6185e29cbd3144305c9a007": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0549, error: failed commit on ref "config-sha256:b2a46b56e3601034fa0e6c7f545e595b53b9176519485689a5942ddf52fcc601": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0206, error: failed commit on ref "manifest-sha256:19dfbd073aa9177151a23758a8775e801965d421c3d25b4c42c02286752dff06": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0551, error: failed commit on ref "config-sha256:5d8f3041403f4bf037afcd83d5762501532707ee0087dc875902e3c0cf2b715c": unexpected status: 500 Internal Server Error  source=console
ERRO[0454] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0552, error: failed commit on ref "config-sha256:40aa6ec3788bb0d59310e6115f1a91f1308938142becf169ee0d808b9a7b5072": unexpected status: 500 Internal Server Error  source=console
ERRO[0454] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0553, error: failed commit on ref "config-sha256:3dec38f9570c170834953eecaa76a1a9555260d110389d36cc4390b1485e0c5e": unexpected status: 500 Internal Server Error  source=console
ERRO[0566] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0579, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0569] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0615, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0571] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0591, error: failed commit on ref "config-sha256:f48ded328cc5169b4e5ca322aa64d09219c99a6720acaa278886783ce381f82d": unexpected status: 500 Internal Server Error  source=console
ERRO[0572] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0640, error: failed commit on ref "config-sha256:971de18fc22c521d0c824107f88654ea27abb7adc85b0adf7af95ee4445571cd": unexpected status: 500 Internal Server Error  source=console
ERRO[0573] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0583, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0573] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0664, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0585] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0679, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0585] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0643, error: failed commit on ref "config-sha256:e20356f33e555734b826dc11f8d64818582f17e1a407e36d6771f3c8b7de2ebe": unexpected status: 500 Internal Server Error  source=console
ERRO[0594] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0577, error: failed commit on ref "config-sha256:3d936251e9f69a0374a77773aae9516d67ccf07f0a698ef7a8969c690173eaf6": unexpected status: 500 Internal Server Error  source=console
ERRO[0621] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0675, error: failed commit on ref "config-sha256:0dab025f4d80c0ce808a4f425911db3449c5e9c273045e7a4934becc4f101ce5": unexpected status: 500 Internal Server Error  source=console
ERRO[0636] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0715, error: failed commit on ref "config-sha256:ff695f8cccf063f9ee7100795b7be641a1c682b83686d9a22570d786eee5e79b": unexpected status: 500 Internal Server Error  source=console
ERRO[0639] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0722, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0639] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0730, error: failed commit on ref "config-sha256:75ca02f0a8faaa941af9e65d7e91c9af659af2663345fffd6e76104e17c7d7d8": unexpected status: 500 Internal Server Error  source=console
ERRO[0640] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0704, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0891, error: failed commit on ref "manifest-sha256:31f198fd4c94ae50fa347f34d4fa3b7356d082e54f1c3989c18e61815241e3d1": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0934, error: failed commit on ref "manifest-sha256:d1da42342613a6d21b5524a6489d790e27d2aae1f974ae7351797709da895581": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0607, error: failed commit on ref "manifest-sha256:c1bc71e321568af0110af547c4ec6f338ee7e43dc3ee15e39496c121887021f9": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0811, error: failed commit on ref "manifest-sha256:d8afd6d05f0fefc5cc96a08d5b5f498cb6de9bfc61430c40ad886fffa14bccea": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0916, error: failed commit on ref "config-sha256:375955231417102363aa98e4bb85c8ff405077ac452e4a9de0b4083a6163def6": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0869, error: failed commit on ref "manifest-sha256:f4c84900880d107b2279b3536fc026384ba8d63f507f1b99bc7c3acbee2929fc": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0898, error: failed commit on ref "manifest-sha256:e88b4bbea7e7f59035dada5421701e97a7e15fa8e32c04cacc299631501b9ec6": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0978, error: failed commit on ref "manifest-sha256:5e5a3f96d33720727eed5e4d6b126d0e9ff241eb5831a80bc9f659588489f494": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0667, error: failed commit on ref "manifest-sha256:aa2310b9823cce901288550e7d9a7622ff38a53c1da049ef551a23a1f39fb060": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0879, error: failed commit on ref "manifest-sha256:84dec2b04ad823c81b9ae026da763fa1adfc8661a38aa1778bf81893d6c1164f": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0966, error: failed commit on ref "manifest-sha256:09f5efe408d11d60a6c16ab51586d8b81547ff6cbb3bd90c6dc7eec75c9db573": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0587, error: failed commit on ref "manifest-sha256:3c77b2befa6d0ee4bcc620322be04a45ee5e84d73fb4e65e3d0fb4fc05db2b01": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0570, error: failed commit on ref "manifest-sha256:8efd949c18e8bdd17bdbcb316eb2decbae398afc874e17f9efe1ef7c69ec4d62": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0682, error: failed commit on ref "manifest-sha256:3619cffa7bf082abed50580bfa716015f4fd915967535199e385a22a0ae96207": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0967, error: failed commit on ref "manifest-sha256:838a31bca59041022c9d41ce473c6ed777910d9f0321a93b9f3e45eaabd9ff6f": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0866, error: failed commit on ref "manifest-sha256:90c9463699c7862a66787f23e2b146ccf998479445e50a0854b00e172b2f461f": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0972, error: failed commit on ref "manifest-sha256:36124e47a39e9e7664583733a33cb96384eb6821c7402ead8c056377a97247ba": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0782, error: failed commit on ref "manifest-sha256:3c521b1338ea6430bcc57319216d1ac8830ce768a9a0f5fe66b9ebc97822c649": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0805, error: failed commit on ref "manifest-sha256:9e3558a70e5f24e85d8976822502d8a2bc2c1bcd6bda34791b1f7b6f9906d4fa": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0783, error: failed commit on ref "manifest-sha256:e76b6d8588ef16ae4673d19dd63aacdd1faeda1e3580bae8530eb504dda47732": no response  source=console


     █ setup

     █ teardown

     data_received............: 0 B    0 B/s
     data_sent................: 0 B    0 B/s
     iteration_duration.......: avg=4m9s  min=7.31s  med=4m22s max=8m56s p(90)=6m8s  p(95)=6m28s
     ✓ { group:::setup }......: avg=1m18s min=1m18s  med=1m18s max=1m18s p(90)=1m18s p(95)=1m18s
     ✓ { group:::teardown }...: avg=7.31s min=7.31s  med=7.31s max=7.31s p(90)=7.31s p(95)=7.31s
     ✓ { scenario:default }...: avg=4m9s  min=22.59s med=4m22s max=8m56s p(90)=6m8s  p(95)=6m28s
     iterations...............: 860    1.201053/s
     success..................: 73.83% ✓ 635      ✗ 225  
     vus......................: 0      min=0      max=500
     vus_max..................: 500    min=500    max=500


running (11m56.0s), 000/500 VUs, 860 complete and 140 interrupted iterations
default ✗ [===============================>------] 500 VUs  10m30.0s/10m0s  0860/1000 shared iters
ERRO[0717] GoError: failed to delete artifact project-020/repository-1706166862428:tag-0002, error: [DELETE /projects/{project_name}/repositories/{repository_name}/artifacts/{reference}][404] deleteArtifactNotFound  &{Errors:[0xc007aa6ec0]}
	at reflect.methodValueCall (native)
	at teardown (file:///home/DEEPROUTE/liangzheng/go/src/github.com/goharbor/perf/scripts/test/push-artifacts-to-same-project.js:67:30(36))  hint="script exception"
Error: running "k6-harbor run ./scripts/test/push-artifacts-to-same-project.js --no-usage-report --vus 500 --iterations 1000" failed with exit code 107
exit status 107
➜ /home/DEEPROUTE/liangzheng/go/src/github.com/goharbor/perf ☞ git:(main) ✗ cat outputs/push-artifacts-to-same-projects.summary.json| jq     
{
  "timestamp": 1706167578474,
  "description": "push-artifacts-to-same-projects",
  "avg": "249.88s",
  "min": "22.59s",
  "med": "262.71s",
  "max": "536.57s",
  "p90": "368.85s",
  "p95": "388.22s",
  "successRate": "73.83%",
  "iterationsRate": "1.20/s"
}

@chlins
Copy link
Member

chlins commented Jan 26, 2024

The original quota update relies on the optimistic lock based on Postgres, which may affect performance under high concurrency. However, you might want to try the quota update by redis introduced since version v2.9. It uses redis to implement the optimistic lock to reduce the pressure on Postgres, refer to https://github.com/goharbor/perf/wiki/Quota-Update see more details.

@microyahoo
Copy link
Contributor Author

microyahoo commented Jan 27, 2024 •
edited
Loading

Thanks for your nice reminder, I will test it and give you feedback. Is harbor 2.10.0 report under QUOTA_UPDATE_PROVIDER="redis"? @chlins

@wy65701436 wy65701436 self-assigned this Jan 29, 2024
@wy65701436 wy65701436 added area/performance more-info-needed The issue author need to provide more details and context to the issue labels Jan 29, 2024
@microyahoo
Copy link
Contributor Author

microyahoo commented Jan 29, 2024 •
edited
Loading

hi @chlins, the success rate increased significantly to 100% after setting QUOTA_UPDATE_PROVIDER to redis. But push artifacts to different projects still has some errors. And there is no error info in redis log. It looks like the client actively terminated the context.

[root@bd-ssd07-node01 perf]# cat outputs/push-artifacts-to-different-projects.summary.json | jq
{
  "timestamp": 1706520431766,
  "description": "push-artifacts-to-different-projects",
  "avg": "22.25s",
  "min": "2.24s",
  "med": "20.38s",
  "max": "44.37s",
  "p90": "34.7s",
  "p95": "36.52s",
  "successRate": "95.8%",
  "iterationsRate": "4.82/s"
}
[root@bd-ssd07-node01 perf]# date; time go run mage.go run push-artifacts-to-different-projects; date                             [109/11816]
Mon Jan 29 17:23:42 CST 2024                                                                                                                 
                                                                                                                                             
          /\      |‾‾| /‾‾/   /‾‾/                                                                                                           
     /\  /  \     |  |/  /   /  /                                                                                                            
    /  \/    \    |     (   /   ‾‾\                                                                                                          
   /          \   |  |\  \ |  (‾)  |                                                                                                         
  / __________ \  |__| \__\ \_____/ .io                                                                                                      
                                                                                                                                             
  execution: local                                                                                                                           
     script: ./scripts/test/push-artifacts-to-different-projects.js                                                                          
     output: -                                                                                                                               
                                                                                                                                             
  scenarios: (100.00%) 1 scenario, 500 max VUs, 10m30s max duration (incl. graceful stop):                                                   
           * default: 1000 iterations shared among 500 VUs (maxDuration: 10m0s, gracefulStop: 30s)                                           

ERRO[0063] failed to push reg.srv.deeproute.cn/project-089/repository-1706520246968:tag-0508, error: failed commit on ref "config-sha256:e410ae7e8f6d2f2eb84b5de65873ed708d10f257b2f6189c0028e52991447456": unexpected status: 500 Internal Server Error  source=console
ERRO[0064] failed to push reg.srv.deeproute.cn/project-009/repository-1706520247715:tag-0523, error: failed to copy: unexpected response: 500
 Internal Server Error  source=console                                 
ERRO[0065] failed to push reg.srv.deeproute.cn/project-090/repository-1706520247472:tag-0518, error: failed commit on ref "config-sha256:a0896a329efde033ebe1583144326b65eb5d8efbbcbc86d8b074af0e6bbf94f5": unexpected status: 500 Internal Server Error  source=console
ERRO[0066] failed to push reg.srv.deeproute.cn/project-041/repository-1706520247767:tag-0524, error: failed commit on ref "config-sha256:5a311a8a5641b4c439dc767b8600e6c4123a6cef2a39b086a84432af2e553a59": unexpected status: 500 Internal Server Error  source=console
ERRO[0066] failed to push reg.srv.deeproute.cn/project-015/repository-1706520248470:tag-0538, error: failed commit on ref "config-sha256:f4bfba168847f2d2c4adbc595693556d5bca829ad7ea9bb658b0bdc0804e116a": unexpected status: 500 Internal Server Error  source=console
ERRO[0067] failed to push reg.srv.deeproute.cn/project-059/repository-1706520248651:tag-0542, error: failed commit on ref "config-sha256:9e72c1ad14f2772e723fc7f2505e5ebb73ee3365e0a4a0757857b36853021eac": unexpected status: 500 Internal Server Error  source=console
ERRO[0067] failed to push reg.srv.deeproute.cn/project-068/repository-1706520248059:tag-0530, error: failed commit on ref "config-sha256:581157990da18658c71ff6b653be1eba059dd8a262aef2a55b3adca509ae0fcf": unexpected status: 500 Internal Server Error  source=console
ERRO[0068] failed to push reg.srv.deeproute.cn/project-057/repository-1706520248516:tag-0539, error: failed commit on ref "config-sha256:349fb819b03580f298d7b1f678c51acc81728e3905f4aea5e747524057526b03": unexpected status: 500 Internal Server Error  source=console
ERRO[0070] failed to push reg.srv.deeproute.cn/project-061/repository-1706520250099:tag-0572, error: failed commit on ref "config-sha256:28a42f4ffa5ce488e6ba19cf50b102c37d066f153551117401ecbf14e3ae916a": unexpected status: 500 Internal Server Error  source=console
ERRO[0072] failed to push reg.srv.deeproute.cn/project-049/repository-1706520251723:tag-0608, error: failed commit on ref "config-sha256:540a23173af2f607450cd15f2a05231e65823edd45f48f1ac23c353062e71d3a": unexpected status: 500 Internal Server Error  source=console
...

harbor core log snippet

22695 2024-01-29T07:25:11Z [DEBUG] [/pkg/notifier/notifier.go:220]: Handle notification with Handler 'Replication' on topic 'PUSH_ARTIFACT': ID-6868, Repository-project-012/repository-1706513014512 Tags-[tag-0129] Digest-sha256:08555fb6aad81d47f4b19765f84bc0caf3b404eda1aa7cb80104b1eac919cde7 Operator-admin OccurAt-2024-01-29 07:25:11
22696 2024-01-29T07:25:11Z [DEBUG] [/pkg/allowlist/manager.go:75]: No CVE allowlist found for project 0, returning empty list.
22697 2024-01-29T07:25:11Z [DEBUG] [/pkg/cached/project_metadata/redis/manager.go:85]: get project 80 metadata from cache error: key not found, will query from database.
22698 2024-01-29T07:25:11Z [DEBUG] [/controller/p2p/preheat/enforcer.go:301]: No preheat policy matched for the artifact project-012/repository-1706513014512@sha256:08555fb6aad81d47f4b19765f84bc0caf3b404eda1aa7cb80104b1eac919cde7
22699 2024-01-29T07:25:11Z [DEBUG] [/pkg/notifier/notifier.go:220]: Handle notification with Handler 'P2PPreheat' on topic 'PUSH_ARTIFACT': ID-6868, Repository-project-012/repository-1706513014512 Tags-[tag-0129] Digest-sha256:08555fb6aad81d47f4b19765f84bc0caf3b404eda1aa7cb80104b1eac919cde7 Operator-admin OccurAt-2024-01-29 07:25:11
22700 2024/01/29 07:25:11 http: proxy error: context canceled                                             
22701 2024-01-29T07:25:11Z [WARNING] [/controller/quota/controller.go:402][requestID="4d3a7c1c-1fe9-43f0-88f8-c46668f506ac"]: rollback resources {"storage":10485760} for project 120 failed, error: context canceled
22702 2024/01/29 07:25:12 http: proxy error: context canceled                                             
22703 2024-01-29T07:25:12Z [WARNING] [/controller/quota/controller.go:402][requestID="f20e7050-77b5-4149-91b6-22b8b0941737"]: rollback resources {"storage":10485760} for project 56 failed, error: context canceled
22704 2024/01/29 07:25:12 http: proxy error: context canceled                                             
22705 2024-01-29T07:25:12Z [WARNING] [/controller/quota/controller.go:402][requestID="4f1d9a88-91d9-45e7-b87d-102b63b9be29"]: rollback resources {"storage":10485760} for project 75 failed, error: context canceled
22706 2024/01/29 07:25:12 http: proxy error: context canceled                                             
22707 2024-01-29T07:25:12Z [WARNING] [/controller/quota/controller.go:402][requestID="bc4d4329-d253-4b06-9cbc-96f3401156ed"]: rollback resources {"storage":10485760} for project 94 failed, error: context canceled
22708 2024/01/29 07:25:12 http: proxy error: context canceled                                             
22709 2024-01-29T07:25:12Z [WARNING] [/controller/quota/controller.go:402][requestID="5bf7f46c-deb5-4161-bb20-6003035a8800"]: rollback resources {"storage":10485760} for project 116 failed, error: context canceled 
22710 2024-01-29T07:25:12Z [DEBUG] [/pkg/cached/project/redis/manager.go:127]: get project project-029 from cache error: key not found:context canceled, will query from database.
22711 2024-01-29T07:25:12Z [DEBUG] [/pkg/cached/project_metadata/redis/manager.go:85]: get project 89 metadata from cache error: key not found:context canceled, will query from database.
22712 2024-01-29T07:25:12Z [DEBUG] [/server/middleware/quota/quota.go:138][action="request" middleware="quota" requestID="65b0eb58-5ec0-4f30-87cb-6fb04522254d" url="/v2/project-029/repository-1706513029439/manifests/tag-0389"]: not warning resources found
22713 2024-01-29T07:25:12Z [DEBUG] [/pkg/notifier/event/event.go:112]: event PUSH_ARTIFACT published

registry log

146100 10.233.106.119 - - [29/Jan/2024:07:25:01 +0000] "PUT /v2/project-059/repository-1706513021492/manifests/tag-0247 HTTP/1.1" 201 0 "" "containerd/1.3.2+unknown"
146101 2024/01/29 07:25:12 http: superfluous response.WriteHeader call from github.com/docker/distribution/vendor/github.com/gorilla/handlers.(*responseLogger).WriteHeader (handlers.go:125)                                                                                      
146102 time="2024-01-29T07:25:12.325658472Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code=unknown err.detail="client disconnected" err.message="unknown error" go.version=go1.21.4 http.request.host=reg.srv.deeproute.cn http.request.id=d4a47f4b-d1fd-42ba-aefe-8d1c1c1a67d7 http.request.method=PUT http.request.remoteaddr=10.10.12.31 http.request.uri="/v2/project-005/repository-1706513057097/blobs/uploads/ad0c34a4-681b-4060-8812-773bace159bd?_state=THmaShtyHrYfjMIYyyz5g0coS7giPrqnzBLayP2Dz817Ik5hbWUiOiJwcm9qZWN0LTAwNS9yZXBvc2l0b3J5LTE3MDY1MTMwNTcwOTciLCJVVUlEIjoiYWQwYzM0YTQtNjgxYi00MDYwLTg4MTItNzczYmFjZTE1OWJkIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDI0LTAxLTI5VDA3OjI1OjA4LjgzOTAyMzIwM1oifQ%3D%3D&digest=sha256%3Ad1721ef657242932b5439db9ecd6e6208e53e5ba0257715511f7f92856b1dab5" http.request.useragent="containerd/1.3.2+unknown" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.308967875s http.response.status=500 http.response.written=89 vars.name="project-005/repository-1706513057097" vars.uuid=ad0c34a4-681b-4060-8812-773bace159bd 
146103 10.233.94.55 - - [29/Jan/2024:07:25:09 +0000] "PUT /v2/project-005/repository-1706513057097/blobs/uploads/ad0c34a4-681b-4060-8812-773bace159bd?_state=THmaShtyHrYfjMIYyyz5g0coS7giPrqnzBLayP2Dz817Ik5hbWUiOiJwcm9qZWN0LTAwNS9yZXBvc2l0b3J5LTE3MDY1MTMwNTcwOTciLCJVVUlEIjoiYWQwYzM0YTQtNjgxYi00MDYwLTg4MTItNzczYmFjZTE1OWJkIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDI0LTAxLTI5VDA3OjI1OjA4LjgzOTAyMzIwM1oifQ%3D%3D&dig       est=sha256%3Ad1721ef657242932b5439db9ecd6e6208e53e5ba0257715511f7f92856b1dab5 HTTP/1.1" 500 89 "" "containerd/1.3.2+unknown"
146104 time="2024-01-29T07:25:12.3261603Z" level=error msg="client disconnected during blob PUT" auth.user.name="harbor_registry_user" contentLength=10485760 copied=4724492 error="unexpected EOF" go.version=go1.21.4 http.request.host=reg.srv.deeproute.cn http.request.id=52525813-40a7-4d2a-ad1a-a7851db29cb0 http.request.method=PUT http.request.remoteaddr=10.10.12.31 http.request.uri="/v2/project-049/repository-1706513057321/blobs/uploads/3bfe34ec-e17e-4800-add7-3e6caa0679be?_state=JSA7glgfoiHWwTD6KaAQAumOpr6Ow3f0n6RHusuC4r57Ik5hbWUiOiJwcm       9qZWN0LTA0OS9yZXBvc2l0b3J5LTE3MDY1MTMwNTczMjEiLCJVVUlEIjoiM2JmZTM0ZWMtZTE3ZS00ODAwLWFkZDctM2U2Y2FhMDY3OWJlIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDI0LTAxLTI5VDA3OjI1OjA4Ljg2NjMyNDI1NloifQ%3D%3D&digest=sha256%3Afeba9bf8492c45d739983df53f3a475562909172e6af2c6d9125e23357ae869d" http.request.useragent="containerd/1.3.2+unknown" vars.name="project-049/repository-1706513057321" vars.uuid=3bfe34ec-e17e-4800-add7-3e6caa0679be

@microyahoo microyahoo mentioned this issue Jan 29, 2024
Closed
@microyahoo
Copy link
Contributor Author

hi @chlins, @wy65701436, happy Chinese new year! Is there any progress on this issue?

@microyahoo
Copy link
Contributor Author

The issue still persists

~/go/src/github.com/goharbor/perf (main ✗) go run mage.go run push-artifacts-to-different-projects                              
                                                                                                                                             
          /\      |‾‾| /‾‾/   /‾‾/               
     /\  /  \     |  |/  /   /  /                        
    /  \/    \    |     (   /   ‾‾\                      
   /          \   |  |\  \ |  (‾)  |                     
  / __________ \  |__| \__\ \_____/ .io

  execution: local                                                                                                                           
     script: ./scripts/test/push-artifacts-to-different-projects.js                                                                          
     output: -                                                                                                                                                                                                                                                                            
  scenarios: (100.00%) 1 scenario, 500 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 1000 iterations shared among 500 VUs (maxDuration: 10m0s, gracefulStop: 30s)                                                                              
ERRO[0156] failed to push reg.srv.deeproute.cn/project-072/repository-1708504916925:tag-0546, error: failed to copy: unexpected response: 500 Internal Server Error  source=console                                 
ERRO[0159] failed to push reg.srv.deeproute.cn/project-100/repository-1708504920884:tag-0582, error: failed to copy: unexpected response: 500
 Internal Server Error  source=console                                                                       
ERRO[0159] failed to push reg.srv.deeproute.cn/project-046/repository-1708504920140:tag-0575, error: failed to copy: unexpected response: 502
 Bad Gateway  source=console                                           
ERRO[0160] failed to push reg.srv.deeproute.cn/project-024/repository-1708504925567:tag-0619, error: failed commit on ref "config-sha256:b1d8
fc6db087b8ac5c064153a31a8fe6be6c105beb91359d7193e0a156afe8f7": unexpected status: 500 Internal Server Error  source=console
ERRO[0164] failed to push reg.srv.deeproute.cn/project-003/repository-1708504930727:tag-0656, error: failed to copy: unexpected response: 500
 Internal Server Error  source=console

@microyahoo
Copy link
Contributor Author

microyahoo commented Mar 17, 2024 •
edited
Loading

I met a lot of redis: transaction failed and rollback resources {"storage":47} for project 133 failed, error: context canceled error, even through I set QUOTA_UPDATE_PROVIDER="redis". This can lead to a drastic drop in success rates, how can this be mitigated? Please help to look at again, thanks. @wy65701436 @chlins. Below is logs.

harbor.log.tar.gz

@chlins
Copy link
Member

chlins commented Mar 18, 2024

@microyahoo Hi, could you please provide the hardware configuration such as CPU/memory/disk info of the harbor deployment environment? so that can help us to reproduce the issue.

@microyahoo
Copy link
Contributor Author

microyahoo commented Mar 18, 2024 •
edited
Loading

@microyahoo Hi, could you please provide the hardware configuration such as CPU/memory/disk info of the harbor deployment environment? so that can help us to reproduce the issue.

Thanks for your quick response. @chlins

node exporter:
node

redis: Aliyun cluodbox redis, below is metric when running performance test.
redis
redis info:

🍺 /home/DEEPROUTE/liangzheng ☞ redis-cli -h <redis host> -a <password> info
# Server
redis_version:5.0.0
redis_git_sha1:3d1b465b
redis_git_dirty:1
redis_build_id:1ae5f854cb4a62e4
redis_mode:standalone
os:Linux  
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:0.0.0
process_id:19
run_id:5e782aa338a437171e51f34e2d84649fd371deb7
tcp_port:6379
uptime_in_seconds:4844273
uptime_in_days:56
hz:10
configured_hz:10
lru_clock:16236228
executable:
support_ptod:1
config_file:

# Clients
connected_clients:30
client_recent_max_input_buffer:8
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
pubsub_clients:0
clients_in_timeout_table:0

# Memory
used_memory:120468480
used_memory_human:114.89M
used_memory_rss:154132480
used_memory_rss_human:146.99M
used_memory_peak:802205936
used_memory_peak_human:765.04M
used_memory_peak_perc:15.02%
used_memory_overhead:97004728
used_memory_startup:58737976
used_memory_dataset:23463752
used_memory_dataset_perc:38.01%
allocator_allocated:121233776
allocator_active:143593472
allocator_resident:164933632
used_memory_lua:77824
used_memory_lua_human:76.00K
used_memory_scripts:7152
used_memory_scripts_human:6.98K
number_of_cached_scripts:6
maxmemory:2147483648
maxmemory_human:2.00G
maxmemory_policy:volatile-lru
allocator_frag_ratio:1.18
allocator_frag_bytes:22359696
allocator_rss_ratio:1.15
allocator_rss_bytes:21340160
rss_overhead_ratio:0.93
rss_overhead_bytes:-10801152
mem_fragmentation_ratio:1.28
mem_fragmentation_bytes:33850712
mem_not_counted_for_evict:3424
mem_replication_backlog:33554432
mem_clients_slaves:20512
mem_clients_normal:642144
mem_aof_buffer:3584
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfreed_objects:0
lazyfree_pending_objects:0
mem_overhead_hashtable_main:3687888
mem_overhead_hashtable_expires:351040
oom_err_count:0

# Stats
total_connections_received:2481509
total_commands_processed:1408303735
instantaneous_ops_per_sec:327
instantaneous_write_ops_per_sec:1
instantaneous_read_ops_per_sec:289
instantaneous_other_ops_per_sec:37
instantaneous_sync_write_ops_per_sec:0
total_net_input_bytes:208859095323
total_net_output_bytes:123982011288
total_net_repl_input_bytes:0
total_net_repl_output_bytes:12050215712
instantaneous_input_kbps:50.50
instantaneous_output_kbps:13.67
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.24
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:0
expired_keys:11846344
expired_stale_perc:0.04
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:695189
evicted_keys:0
keyspace_hits:620400703
keyspace_misses:545676998
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:3991
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:190
instantaneous_error_replies_ops_per_sec:0
total_reads_processed:438853812
total_writes_processed:408688526
io_threaded_reads_processed:0
io_threaded_writes_processed:0
slot_psync_ok:0
slot_psync_err:0
total_commands_received:403594937
commands_received_per_sec:42
evicted_keys_per_sec:0
hits_per_sec:142
misses_per_sec:146
hit_rate_percentage:49.31
cmd_slowlog_count:5
traffic_control_input:0
traffic_control_input_status:0
traffic_control_output:0
traffic_control_output_status:0
stat_avg_rt:18
stat_max_rt:1250

# Replication
role:master

# CPU
used_cpu_sys:5918.144167
used_cpu_user:32298.789347
used_cpu_sys_children:8.529888
used_cpu_user_children:33.265656

# Errorstats
errorstat_ERR:count=20
errorstat_NOAUTH:count=1
errorstat_WRONGPASS:count=168
errorstat_WRONGTYPE:count=1

# Cluster
cluster_enabled:0
databases:256
nodecount:1

# Keyspace
db0:keys=3602,expires=3602,avg_ttl=28209546
db1:keys=100,expires=58,avg_ttl=56702724
db2:keys=56696,expires=0,avg_ttl=0
db5:keys=5,expires=0,avg_ttl=0

# mio
main_io:conn=18,ncommands=204923374,total_input_bytes=191627162785,total_output_bytes=103737733019,ops=42,input_kbps=50.50,output_kbps=13.67
follower_io_0:conn=3,ncommands=47556705,total_input_bytes=4087644351,total_output_bytes=4880824679,ops=0,input_kbps=0.00,output_kbps=0.00
follower_io_1:conn=4,ncommands=50548295,total_input_bytes=4443605741,total_output_bytes=5135831050,ops=0,input_kbps=0.00,output_kbps=0.00
follower_io_2:conn=2,ncommands=48177131,total_input_bytes=4195807063,total_output_bytes=4857888184,ops=0,input_kbps=0.00,output_kbps=0.00
follower_io_3:conn=6,ncommands=52389435,total_input_bytes=4504875383,total_output_bytes=5369734088,ops=0,input_kbps=0.00,output_kbps=0.00

# Capability
control_subsystem_enabled:1
latency_metric_enabled:1
mio_enabled:4
bigkey_enabled:1
hotkey_enabled:1
tde_enabled:0
ptod_enabled:1
slotmig_enabled:1
seamlessha_enabled:0
cluster_dht_tls_enabled:1

harbor core:
core

harbor registry:
harbor registry

~/go/src/deeproute/harbor-cli (main ✗) kubectl --kubeconfig ~/.kube/dr-config get nodes -o wide
NAME                                    STATUS   ROLES    AGE      VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
master1                                 Ready    master   3y253d   v1.17.6   10.10.12.11   <none>        Ubuntu 18.04.4 LTS   4.15.0-197-generic   docker://18.9.7
master2                                 Ready    master   3y253d   v1.17.6   10.10.12.12   <none>        Ubuntu 18.04.4 LTS   4.15.0-142-generic   docker://18.9.7
master3                                 Ready    master   3y253d   v1.17.6   10.10.12.13   <none>        Ubuntu 18.04.4 LTS   4.15.0-191-generic   docker://18.9.7
node01.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y191d   v1.17.6   10.10.12.31   <none>        Ubuntu 18.04.5 LTS   4.15.0-156-generic   docker://18.9.7
node02.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y214d   v1.17.6   10.10.12.32   <none>        Ubuntu 18.04.5 LTS   4.15.0-153-generic   docker://18.9.7
node03.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y214d   v1.17.6   10.10.12.33   <none>        Ubuntu 18.04.5 LTS   4.15.0-197-generic   docker://18.9.7
node04.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y212d   v1.17.6   10.10.12.34   <none>        Ubuntu 18.04.5 LTS   4.15.0-153-generic   docker://18.9.7
node05.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y207d   v1.17.6   10.10.12.35   <none>        Ubuntu 18.04.5 LTS   4.15.0-153-generic   docker://18.9.7
node06.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y205d   v1.17.6   10.10.12.36   <none>        Ubuntu 18.04.5 LTS   4.15.0-197-generic   docker://18.9.7
node07.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y200d   v1.17.6   10.10.12.37   <none>        Ubuntu 18.04.5 LTS   4.15.0-197-generic   docker://18.9.7
node08.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y200d   v1.17.6   10.10.12.38   <none>        Ubuntu 18.04.5 LTS   4.15.0-197-generic   docker://18.9.7

~/go/src/deeproute/harbor-cli (main ✗) kubectl --kubeconfig ~/.kube/dr-config -n harbor-server get pods -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP               NODE                                    NOMINATED NODE   READINESS GATES
harbor-core-66588fc7bd-8b9tn         1/1     Running   0          2d15h   10.233.106.243   node01.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-core-66588fc7bd-hs6qq         1/1     Running   0          2d15h   10.233.125.7     node05.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-core-66588fc7bd-mnqsb         1/1     Running   0          2d15h   10.233.112.80    node07.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-exporter-9d46fd77-r9z2p       1/1     Running   0          5d19h   10.233.102.32    node03.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-jobservice-64644b6477-lnvpf   1/1     Running   0          2d16h   10.233.120.146   node08.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-jobservice-64644b6477-n5mql   1/1     Running   0          2d16h   10.233.102.188   node03.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-jobservice-64644b6477-vrl2d   1/1     Running   0          2d16h   10.233.94.112    node04.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-portal-6ffd5f95f5-s6tpj       1/1     Running   0          2d16h   10.233.102.26    node03.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-registry-6df6b8c894-hs8vx     2/2     Running   0          2d16h   10.233.107.205   node06.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-registry-6df6b8c894-md8fd     2/2     Running   0          2d16h   10.233.120.49    node08.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-registry-6df6b8c894-vhs62     2/2     Running   0          2d16h   10.233.102.137   node03.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-trivy-0                       1/1     Running   0          5d19h   10.233.125.146   node05.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>

registry.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    dep.configmap.hash/app-conf: 5098cc7a4b4970489b11ff4a4d176adaa5c2538c66e330d71c241ae199b20368
    dep.configmap.hash/jobservice-conf: a789a925b9310cb71746003deae0b4bf662f283c42853053696f2a94fa6905a6
    dep.configmap.hash/portal-conf: 4978cdfbc877e6b6015a90557f4603cd506bfe64d16986b79904ecd97a3c4297
    dep.configmap.hash/registry-conf: 7ab3b982a3c9728bff492128abd438d977e1fb0c0935c755b5d1744152d9adc9
    dep.configmap.hash/registry-ctl: 585c3753f4f7d155458b73e9501c0fd6d3002636aa3470ba079ef18ef7ecfcab
    dep.secret.hash/ceph-secret: 44a441dad46bec23b59e3ee2a4c46bcf0188d77207ee7e081c744535e725dc9d
    dep.secret.hash/core-env: 60b03d02d86d64d6de48e1341dee37862b2b9ce5042d89c54d0f80a3c014c8dc
    dep.secret.hash/exporter-env: f218b81357a05155188b3e80578e79159a5c06f62b89e65b99a5788249955a02
    dep.secret.hash/harbor-env: 7effc6edfb35fa76eb6274db2f0c25c3bd663765d61ad28111402e42ba0f76b4
    dep.secret.hash/jobservice-env: 65545ed53a39b359edcb4f5da591bbbbf6cd8b5e33522b50f57ed1fdf465b443
    dep.secret.hash/private-key: 97736853235d1705c806e9b852bb19a20278b66f6cb6e484229a6beebf2dbd2a
    dep.secret.hash/registry-env: 688328dafac719b43d1e58fcb9117639e3f9c88cb2e6e6b6da1e0ab8789b24b5
    dep.secret.hash/registry-htpwd: 3084e97b8b36708eade0a4868b6fad186b99b2bccd235c59ef4467c65fa69647
    dep.secret.hash/secret-key: d67cffe0dcf29c9a0e8a28e167a787925e223dea0499230d802b63c30e521750
    dep.secret.hash/trivy-env: 151c72f6b00798cb8c2af3d153bd14ca54d839e39d065c7bcacc783bd97d0268
  creationTimestamp: "2024-03-15T09:41:57Z"
  generateName: harbor-registry-6df6b8c894-
  labels:
    app: harbor-registry
    pod-template-hash: 6df6b8c894
    project: harbor
  name: harbor-registry-6df6b8c894-hs8vx
  namespace: harbor-server
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: harbor-registry-6df6b8c894
    uid: 8d5cbaf4-0614-451a-baea-eed958b38648
  resourceVersion: "1109916475"
  selfLink: /api/v1/namespaces/harbor-server/pods/harbor-registry-6df6b8c894-hs8vx
  uid: d191640b-fe2e-4f97-abc1-99dc9f40b115
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - harbor-registry
          topologyKey: kubernetes.io/hostname
        weight: 100
  containers:
  - args:
    - serve
    - /etc/registry/config.yml
    env:
    - name: DR_META_K8S_CLUSTER_ENV
      value: production
    - name: DR_META_K8S_CLUSTER_NAME
      value: prod-k8s-cluster
    - name: STAKATER_REGISTRY_CONF_CONFIGMAP
      value: 72130abe884a7db5d543dfaeec65aac74b83bf4a
    envFrom:
    - secretRef:
        name: harbor-env
    - secretRef:
        name: registry-env
    image: reg.deeproute.ai/deeproute-public/registry-photon:v2.10.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 2
      httpGet:
        path: /
        port: registrya
        scheme: HTTP
      initialDelaySeconds: 300
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: registry
    ports:
    - containerPort: 5000
      name: registrya
      protocol: TCP
    - containerPort: 5001
      name: registryb
      protocol: TCP
    - containerPort: 8080
      name: controller
      protocol: TCP
    - containerPort: 8001
      name: metrics
      protocol: TCP
    readinessProbe:
      failureThreshold: 2
      httpGet:
        path: /
        port: registrya
        scheme: HTTP
      initialDelaySeconds: 1
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "16"
        memory: 64Gi
      requests:
        cpu: "16"
        memory: 64Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/registry//config.yml
      name: registry-conf
      subPath: config.yml
    - mountPath: /etc/registry//passwd
      name: registry-htpwd
      subPath: passwd
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-6nfll
      readOnly: true
  - env:
    - name: DR_META_K8S_CLUSTER_ENV
      value: production
    - name: DR_META_K8S_CLUSTER_NAME
      value: prod-k8s-cluster
    envFrom:
    - secretRef:
        name: harbor-env
    - secretRef:
        name: registry-env
    image: reg.deeproute.ai/deeproute-public/harbor-registryctl:v2.10.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 2
      httpGet:
        path: /api/health
        port: registryctl
        scheme: HTTP
      initialDelaySeconds: 300
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: registryctl
    ports:
    - containerPort: 8080
      name: registryctl
      protocol: TCP
    readinessProbe:
      failureThreshold: 2
      httpGet:
        path: /api/health
        port: registryctl
        scheme: HTTP
      initialDelaySeconds: 1
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "2"
        memory: 4Gi
      requests:
        cpu: "2"
        memory: 4Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/registry//config.yml
      name: registry-conf
      subPath: config.yml
    - mountPath: /etc/registryctl//config.yml
      name: registry-ctl
      subPath: config.yml
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-6nfll
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: image-pull-secret
  nodeName: node06.prod-k8s.bm.pd.sz.deeproute.ai
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - configMap:
      defaultMode: 420
      name: registry-conf
    name: registry-conf
  - configMap:
      defaultMode: 420
      name: registry-ctl
    name: registry-ctl
  - name: registry-htpwd
    secret:
      defaultMode: 420
      secretName: registry-htpwd
  - name: default-token-6nfll
    secret:
      defaultMode: 420
      secretName: default-token-6nfll

core.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    dep.configmap.hash/app-conf: 5098cc7a4b4970489b11ff4a4d176adaa5c2538c66e330d71c241ae199b20368
    dep.configmap.hash/jobservice-conf: a789a925b9310cb71746003deae0b4bf662f283c42853053696f2a94fa6905a6
    dep.configmap.hash/portal-conf: 4978cdfbc877e6b6015a90557f4603cd506bfe64d16986b79904ecd97a3c4297
    dep.configmap.hash/registry-conf: 7ab3b982a3c9728bff492128abd438d977e1fb0c0935c755b5d1744152d9adc9
    dep.configmap.hash/registry-ctl: 585c3753f4f7d155458b73e9501c0fd6d3002636aa3470ba079ef18ef7ecfcab
    dep.secret.hash/ceph-secret: 44a441dad46bec23b59e3ee2a4c46bcf0188d77207ee7e081c744535e725dc9d
    dep.secret.hash/core-env: 60b03d02d86d64d6de48e1341dee37862b2b9ce5042d89c54d0f80a3c014c8dc
    dep.secret.hash/exporter-env: f218b81357a05155188b3e80578e79159a5c06f62b89e65b99a5788249955a02
    dep.secret.hash/harbor-env: 7effc6edfb35fa76eb6274db2f0c25c3bd663765d61ad28111402e42ba0f76b4
    dep.secret.hash/jobservice-env: 65545ed53a39b359edcb4f5da591bbbbf6cd8b5e33522b50f57ed1fdf465b443
    dep.secret.hash/private-key: 97736853235d1705c806e9b852bb19a20278b66f6cb6e484229a6beebf2dbd2a
    dep.secret.hash/registry-env: 688328dafac719b43d1e58fcb9117639e3f9c88cb2e6e6b6da1e0ab8789b24b5
    dep.secret.hash/registry-htpwd: 3084e97b8b36708eade0a4868b6fad186b99b2bccd235c59ef4467c65fa69647
    dep.secret.hash/secret-key: d67cffe0dcf29c9a0e8a28e167a787925e223dea0499230d802b63c30e521750
    dep.secret.hash/trivy-env: 151c72f6b00798cb8c2af3d153bd14ca54d839e39d065c7bcacc783bd97d0268
  creationTimestamp: "2024-03-15T10:40:15Z"
  generateName: harbor-core-66588fc7bd-
  labels:
    app: harbor-core
    pod-template-hash: 66588fc7bd
    project: harbor
  name: harbor-core-66588fc7bd-8b9tn
  namespace: harbor-server
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: harbor-core-66588fc7bd
    uid: e5c76cb3-230b-4b45-a53d-0e5f96012d33
  resourceVersion: "1109966187"
  selfLink: /api/v1/namespaces/harbor-server/pods/harbor-core-66588fc7bd-8b9tn
  uid: 242d70b6-fa9e-4271-9c6a-cd8f724d570f
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - harbor-core
          topologyKey: kubernetes.io/hostname
        weight: 100
  containers:
  - env:
    - name: DR_META_K8S_CLUSTER_ENV
      value: production
    - name: DR_META_K8S_CLUSTER_NAME
      value: prod-k8s-cluster
    - name: METRIC_SUBSYSTEM
      value: core
    - name: PORT
      value: "8080"
    - name: STAKATER_CORE_ENV_SECRET
      value: 59c3b877edcfe3a23bfe5112f6940687f072418c
    envFrom:
    - secretRef:
        name: harbor-env
    - secretRef:
        name: core-env
    image: reg.deeproute.ai/deeproute-public/harbor-core:v2.10.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 2
      httpGet:
        path: /api/v2.0/ping
        port: core
        scheme: HTTP
      initialDelaySeconds: 15
      periodSeconds: 120
      successThreshold: 1
      timeoutSeconds: 1
    name: core
    ports:
    - containerPort: 8080
      name: core
      protocol: TCP
    - containerPort: 8001
      name: metrics
      protocol: TCP
    readinessProbe:
      failureThreshold: 2
      httpGet:
        path: /api/v2.0/ping
        port: core
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 60
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "8"
        memory: 16Gi
      requests:
        cpu: "8"
        memory: 16Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/core//app.conf
      name: app-conf
      subPath: app.conf
    - mountPath: /etc/core//private_key.pem
      name: private-key
      subPath: private_key.pem
    - mountPath: /etc/core/token
      name: psc
    - mountPath: /etc/core//key
      name: secret-key
      subPath: key
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-6nfll
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: image-pull-secret
  nodeName: node01.prod-k8s.bm.pd.sz.deeproute.ai
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 10000
    runAsUser: 10000
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 120
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: psc
  - configMap:
      defaultMode: 420
      name: app-conf
    name: app-conf
  - name: private-key
    secret:
      defaultMode: 420
      secretName: private-key
  - name: secret-key
    secret:
      defaultMode: 420
      secretName: secret-key
  - name: default-token-6nfll
    secret:
      defaultMode: 420
      secretName: default-token-6nfll

jobservice.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    dep.configmap.hash/app-conf: 5098cc7a4b4970489b11ff4a4d176adaa5c2538c66e330d71c241ae199b20368
    dep.configmap.hash/jobservice-conf: a789a925b9310cb71746003deae0b4bf662f283c42853053696f2a94fa6905a6
    dep.configmap.hash/portal-conf: 4978cdfbc877e6b6015a90557f4603cd506bfe64d16986b79904ecd97a3c4297
    dep.configmap.hash/registry-conf: 7ab3b982a3c9728bff492128abd438d977e1fb0c0935c755b5d1744152d9adc9
    dep.configmap.hash/registry-ctl: 585c3753f4f7d155458b73e9501c0fd6d3002636aa3470ba079ef18ef7ecfcab
    dep.secret.hash/ceph-secret: 44a441dad46bec23b59e3ee2a4c46bcf0188d77207ee7e081c744535e725dc9d
    dep.secret.hash/core-env: 60b03d02d86d64d6de48e1341dee37862b2b9ce5042d89c54d0f80a3c014c8dc
    dep.secret.hash/exporter-env: f218b81357a05155188b3e80578e79159a5c06f62b89e65b99a5788249955a02
    dep.secret.hash/harbor-env: 7effc6edfb35fa76eb6274db2f0c25c3bd663765d61ad28111402e42ba0f76b4
    dep.secret.hash/jobservice-env: 65545ed53a39b359edcb4f5da591bbbbf6cd8b5e33522b50f57ed1fdf465b443
    dep.secret.hash/private-key: 97736853235d1705c806e9b852bb19a20278b66f6cb6e484229a6beebf2dbd2a
    dep.secret.hash/registry-env: 688328dafac719b43d1e58fcb9117639e3f9c88cb2e6e6b6da1e0ab8789b24b5
    dep.secret.hash/registry-htpwd: 3084e97b8b36708eade0a4868b6fad186b99b2bccd235c59ef4467c65fa69647
    dep.secret.hash/secret-key: d67cffe0dcf29c9a0e8a28e167a787925e223dea0499230d802b63c30e521750
    dep.secret.hash/trivy-env: 151c72f6b00798cb8c2af3d153bd14ca54d839e39d065c7bcacc783bd97d0268
  creationTimestamp: "2024-03-15T09:41:57Z"
  generateName: harbor-jobservice-64644b6477-
  labels:
    app: harbor-jobservice
    pod-template-hash: 64644b6477
    project: harbor
  name: harbor-jobservice-64644b6477-lnvpf
  namespace: harbor-server
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: harbor-jobservice-64644b6477
    uid: 08ca140d-00cb-4a71-bb07-e21c22b3e9b6
  resourceVersion: "1109916751"
  selfLink: /api/v1/namespaces/harbor-server/pods/harbor-jobservice-64644b6477-lnvpf
  uid: 14962de8-4c58-497e-a0fe-a25a29e70fc2
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - harbor-jobservice
          topologyKey: kubernetes.io/hostname
        weight: 100
  containers:
  - env:
    - name: DR_META_K8S_CLUSTER_ENV
      value: production
    - name: DR_META_K8S_CLUSTER_NAME
      value: prod-k8s-cluster
    - name: METRIC_SUBSYSTEM
      value: jobservice
    - name: STAKATER_JOBSERVICE_ENV_SECRET
      value: 94af87eb4dc056ddd0adfb26fbb4f9d3dbcbfb8a
    - name: STAKATER_JOBSERVICE_CONF_CONFIGMAP
      value: 725d96e8e74f6dc70f7edb0b544b1a525d6dbeb3
    envFrom:
    - secretRef:
        name: harbor-env
    - secretRef:
        name: jobservice-env
    image: reg.deeproute.ai/deeproute-public/harbor-jobservice:v2.10.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /api/v1/stats
        port: jobservice
        scheme: HTTP
      initialDelaySeconds: 300
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: jobservice
    ports:
    - containerPort: 8080
      name: jobservice
      protocol: TCP
    - containerPort: 8001
      name: metrics
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /api/v1/stats
        port: jobservice
        scheme: HTTP
      initialDelaySeconds: 20
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "2"
        memory: 4Gi
      requests:
        cpu: "2"
        memory: 4Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/log/jobs
      name: job-logs
    - mountPath: /etc/jobservice//config.yml
      name: jobservice-conf
      subPath: config.yml
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-6nfll
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: image-pull-secret
  initContainers:
  - command:
    - sh
    - -c
    - for i in $(seq 1 60); do nc -z -w3 harbor-core:80  && exit 0 || sleep 5; done;
      exit 1
    env:
    - name: DR_META_K8S_CLUSTER_ENV
      value: production
    - name: DR_META_K8S_CLUSTER_NAME
      value: prod-k8s-cluster
    image: reg.deeproute.ai/deeproute-public/busybox:latest
    imagePullPolicy: IfNotPresent
    name: wait-for-core
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 200m
        memory: 256Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-6nfll
      readOnly: true
  nodeName: node08.prod-k8s.bm.pd.sz.deeproute.ai
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 10000
    runAsUser: 10000
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 120
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - cephfs:
      monitors:
      - <mon-ip>:6789
      path: /volumes/<cephfs-path>
      secretRef:
        name: ceph-secret
      user: csi-cephfs-node
    name: job-logs
  - configMap:
      defaultMode: 420
      name: jobservice-conf
    name: jobservice-conf
  - name: default-token-6nfll
    secret:
      defaultMode: 420
      secretName: default-token-6nfll

@microyahoo
Copy link
Contributor Author

microyahoo commented Mar 18, 2024 •
edited
Loading

Hi @chlins, I filtered the two request ids in the core log and found that it only took less than 1min to go from attach request ID to rollback. However, when review code of updating the quota usage, there will be a retry defaultRetryTimeout of 5 minutes. I am not sure whether this has an impact. Please help to confirm and give some suggestions, thank you.

~/go/src/deeproute/harbor-cli (main ✗) ag d9e69564-3e7b-4f2f-801d-066ac2e35c96 core.*
core.3
36770:2024-03-15T10:43:21Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id d9e69564-3e7b-4f2f-801d-066ac2e35c96 to the logger for the request PUT /v2/project-015/repository-1710499314344/blobs/uploads/48fc0d3c-dde7-4a6c-bf44-7f94aa48356c
36912:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
36952:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
36991:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37033:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37076:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37113:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37151:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37186:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37220:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37254:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37291:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37326:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37359:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37393:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37424:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37456:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37488:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37518:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37545:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37570:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37597:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37628:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37653:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37679:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37706:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37731:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37757:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37782:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37805:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37833:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37861:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37887:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37910:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37934:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37958:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37982:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38005:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38027:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38048:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38068:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38088:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38107:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38126:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38144:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38161:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38178:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38193:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38207:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38222:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38236:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38252:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38265:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38282:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38295:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38308:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38320:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38336:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38351:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38364:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38376:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38388:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38398:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38411:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38426:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38438:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38451:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38462:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38475:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38488:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38498:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38508:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38517:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38531:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38539:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38547:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38553:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38561:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38567:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38572:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38577:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38580:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38583:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38586:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38588:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38590:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38595:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38602:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
39203:2024-03-15T10:43:45Z [WARNING] [/controller/quota/controller.go:402][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: rollback resources {"storage":47} for project 133 failed, error: context canceled
~/go/src/deeproute/harbor-cli (main ✗) ag 96b18f4d-bcf4-426f-b7a2-383e24e0f65b core.*
core.3
37842:2024-03-15T10:43:21Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id 96b18f4d-bcf4-426f-b7a2-383e24e0f65b to the logger for the request PUT /v2/project-015/repository-1710499314344/blobs/uploads/a7f354c6-b70c-4bae-b588-f908a753ff85
37932:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
37957:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
37981:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38004:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38026:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38047:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38066:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38084:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38103:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38121:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38139:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38156:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38171:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38185:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38200:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38215:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38230:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38246:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38261:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38275:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38290:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38305:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38319:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38331:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38347:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38362:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38373:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38382:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
39516:2024-03-15T10:44:09Z [WARNING] [/controller/quota/controller.go:402][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: rollback resources {"storage":10485760} for project 133 failed, error: context canceled

@microyahoo
Copy link
Contributor Author

I try to set quota_per_project_enable to false to deactivating update project quota usage, but there is still no improvement in the success rate.

insert into properties (k, v) values('quota_per_project_enable', false);

@chlins
Copy link
Member

chlins commented Mar 18, 2024 •
edited
Loading

There should be a debug log message if you disabled the quota_per_project_enable if you set the log level to debug, could you confirm it?

harbor/src/server/middleware/quota/quota.go

Line 88 in 69fc957

logger.Debugf("quota is deactivated for %s %s, so direct to next handler", reference, referenceID)

BTW, is the error message same with previous after you set the quota_per_project_enable to false?

@microyahoo
Copy link
Contributor Author

microyahoo commented Mar 18, 2024 •
edited
Loading

There should be a debug log message if you disabled the quota_per_project_enable if you set the log level to debug, could you confirm it?

harbor/src/server/middleware/quota/quota.go

Line 88 in 69fc957

logger.Debugf("quota is deactivated for %s %s, so direct to next handler", reference, referenceID)

BTW, is the error message same with previous after you set the quota_per_project_enable to false?

yes, I already toggle on the core debug log, and the message for quota is deactivated for project xxx, so direct to next handler appears in large numbers in the core log. The previous error message redis: transaction failed and rollback resources {"storage":47} for project 133 failed, error: context canceled have all gone. Therefore, it can be proven that the error has nothing to do with redis. There are many 500 errors in the registry, such as Path not found, client disconnected, etc.

In addition, I found a clue, taking upload id 056f13cc-f188-4959-aa83-5c6fcadef6c7 as an example, while the registry uses s3 as the backend storage.
2024-03-18T06:50:37.686 s3 received the POST request for multipart upload and returned the s3 upload id, but did not receive the data to upload.

[root@cicd-node01 ~]# for i in `seq 141 146`; do echo 10.3.9.$i; ssh 10.3.9.$i 'ag 056f13cc-f188-4959-aa83-5c6fcadef6c7 /var/lib/rook/rook-ceph/log/ceph-client.rgw.os.*';  done
10.3.9.141
2024-03-18T06:50:36.411+0000 7fb7e8be0700  1 beast: 0x7fb750d88620: 10.10.12.33 - harbor [18/Mar/2024:06:50:36.081 +0000] "PUT /harbor/docker/registry/v2/repositories/project-062/repository-1710744554486/_uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7/startedat HTTP/1.1" 200 20 - "aws-sdk-go/1.15.11 (go1.21.4; linux; amd64)" - latency=0.330001682s
2024-03-18T06:50:37.686+0000 7fb75cac8700  1 beast: 0x7fb750d88620: 10.10.12.33 - harbor [18/Mar/2024:06:50:36.458 +0000] "POST /harbor/docker/registry/v2/repositories/project-062/repository-1710744554486/_uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7/data?uploads= HTTP/1.1" 200 356 - "aws-sdk-go/1.15.11 (go1.21.4; linux; amd64)" - latency=1.228006363s
2024-03-18T06:50:37.692+0000 7fb802c14700  1 beast: 0x7fb74c06f620: 10.10.12.33 - harbor [18/Mar/2024:06:50:37.688 +0000] "PUT /harbor/docker/registry/v2/repositories/project-062/repository-1710744554486/_uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7/hashstates/sha256/0 HTTP/1.1" 200 108 - "aws-sdk-go/1.15.11 (go1.21.4; linux; amd64)" - latency=0.004000021s
10.3.9.142
10.3.9.143
10.3.9.144
10.3.9.145
2024-03-18T06:50:37.898+0000 7f3f06920700  1 beast: 0x7f3e51f26620: 10.10.12.36 - harbor [18/Mar/2024:06:50:37.896 +0000] "GET /harbor/docker/registry/v2/repositories/project-062/repository-1710744554486/_uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7/startedat HTTP/1.1" 206 20 - "aws-sdk-go/1.15.11 (go1.21.4; linux; amd64)" bytes=0- latency=0.002000014s
2024-03-18T06:50:48.032+0000 7f3eab069700  1 beast: 0x7f3e506f6620: 10.10.12.36 - harbor [18/Mar/2024:06:50:37.940 +0000] "GET /harbor?prefix=docker%2Fregistry%2Fv2%2Frepositories%2Fproject-062%2Frepository-1710744554486%2F_uploads%2F056f13cc-f188-4959-aa83-5c6fcadef6c7%2Fdata&uploads= HTTP/1.1" 200 375 - "aws-sdk-go/1.15.11 (go1.21.4; linux; amd64)" - latency=10.092069626s
10.3.9.146

2024-03-18T06:50:48.061206954Z registry calls s3.stat to query 056f13cc-f188-4959-aa83-5c6fcadef6c7/data and finds that it does not exist and returns 500.

time="2024-03-18T06:50:48.061206954Z" level=error msg="error resolving upload: s3aws: Path not found: /docker/registry/v2/repositories/project-062/repository-1710744554486/_uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7/data" auth.user.name="harbor_registry_user" go.version=go1.21.4 http.request.host=reg.srv.deeproute.cn http.request.id=6dcc0b71-ea73-490e-9002-438a39b67061 http.request.method=PUT http.request.remoteaddr=10.10.12.31 http.request.uri="/v2/project-062/repository-1710744554486/blobs/uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7?_state=e4olaHxH5oH62wPrPgfZxg32hYqUOQCRz-ZhRFI7FQt7Ik5hbWUiOiJwcm9qZWN0LTA2Mi9yZXBvc2l0b3J5LTE3MTA3NDQ1NTQ0ODYiLCJVVUlEIjoiMDU2ZjEzY2MtZjE4OC00OTU5LWFhODMtNWM2ZmNhZGVmNmM3IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDI0LTAzLTE4VDA2OjUwOjM2LjA4ODg4MjYzOVoifQ%3D%3D&digest=sha256%3Abc927d17069db75a4708a98a8e3ca8fb1a29683bff3ecbd6efae0cde2ba0f18d" http.request.useragent="containerd/1.3.2+unknown" vars.name="project-062/repository-1710744554486" vars.uuid=056f13cc-f188-4959-aa83-5c6fcadef6c7

harbor.log.tar.gz

@microyahoo
Copy link
Contributor Author

hi @chlins, I notice another clue, the perf blob file is not existed when I comment out perf teardown to keep blob digest file.

screenshot-20240318-193008
screenshot-20240318-193312

@microyahoo
Copy link
Contributor Author

Hi @chlins, I hacked harbor-core code and add detailed log for some middlewares, take PUT v2_manifest for example. Because I noticed that putting manifest sometimes takes a long time.

For the registry, the time spent putting manifest is negligible.
registry-tag-088 png

For harbor-core, The total time spent on put manifest is 1m, of which putManifest takes 17s and PutManifestMiddleware after response takes 29s.
core-tag-088

@microyahoo
Copy link
Contributor Author

microyahoo commented Mar 20, 2024 •
edited
Loading

I roughly counted one of the put manifest requests. The time overhead in each stage is as follows:
WX20240320-235223@2x

metric                                                  50.833120332s   2024-03-20T11:29:42
    repoproxy.DisableBlobAndManifestUploadMiddleware    1.647785287s    2024-03-20T11:29:03
    immutable.Middleware                                2.444636649s    2024-03-20T11:29:05
    quota.PutManifestMiddleware                         36.183096215s   2024-03-20T11:29:41
        blob.PutManifestMiddleware before request       208.525058ms    2024-03-20T11:29:09
        putManifest                                     6.580389936s    2024-03-20T11:29:16
        blob.PutManifestMiddleware after response       24.024166654s   2024-03-20T11:29:40
        subject.Middleware                              1.382551771s    2024-03-20T11:29:41
        cosign.SignatureMiddleware                      12.553us        2024-03-20T11:29:41

In the above, the entire request takes about 50 seconds, of which the actual registry processing time is less than 1 second.
20240320-235127

@chlins
Copy link
Member

chlins commented Mar 21, 2024 •
edited
Loading

Could you try to deploy the original registry https://github.com/distribution/distribution on your environment and run the performance push case to it? In theory, if the quota is turned off, then the harbor core and the registry only use golang's reverse proxy to forward traffic, and there should not be such a big gap.

@microyahoo
Copy link
Contributor Author

Could you try to deploy the original registry https://github.com/distribution/distribution on your environment and run the performance push case to it? In theory, if the quota is turned off, then the harbor core and the registry only use golang's reverse proxy to forward traffic, and there should not be such a big gap.

hi @chlins, yes, the analysis result of the above comment is to use the original registry. I added some logs on it and recompiled and deployed it. However, the results when running with harbor perf are similar. The performance is improved, but the improvement is not significant.

@microyahoo
Copy link
Contributor Author

microyahoo commented Mar 21, 2024 •
edited
Loading

I analyzed one of the put v2_blob_upload requests, and the specific time cost is as follows:

metric                                                  1m3.340869412s  2024-03-20T11:29:55
    quota.PutManifestMiddleware                         55.734914587s   2024-03-20T11:29:55
        blob.PutManifestMiddleware before request       682.602959ms    2024-03-20T11:29:03
        proxy                                           44.444999102s   2024-03-20T11:29:48
            s3                                          0.066000909s    2024-03-20T11:29:48
        blob.PutManifestMiddleware after response       5.063451415s    2024-03-20T11:29:55

Not sure where the extra >10s of overhead is? The above put manifest request is also similar too.

screenshot-20240321-152339
20240321-152238

@microyahoo
Copy link
Contributor Author

Hi @chlins, is it possible that the perf client is processing requests too slowly? For example, it takes 44 seconds to push 10MB of data, but the actual io.copy to s3 storage takes less than one second.

@microyahoo
Copy link
Contributor Author

microyahoo commented Mar 24, 2024 •
edited
Loading

The following is my test result after turning off redis quota.

push-artifacts-to-different-projects

VUS BLOB SIZE(kbyte) Avg Duration Min Duration Max Duration Med Duration P90 P95 SUCCESS RATE ITERATIONS RATE
50 4 4.38s 2.18s 4.79s 4.41s 4.71s 4.74s 100% 3.40/s
50 512 4.69s 4.15s 4.93s 4.71s 4.87s 4.9s 100% 3.22/s
50 1024 4.48s 3.3s 4.78s 4.49s 4.7s 4.73s 100% 3.26/s
50 5120 4.54s 3.84s 4.82s 4.53s 4.79s 4.79s 100% 2.68/s
50 10240 4.4s 3.04s 4.94s 4.38s 4.88s 4.92s 100% 2.35/s
50 20480 4.57s 2.46s 8.41s 4.29s 6.12s 6.8s 100% 1.76/s
50 40960 6.36s 2.95s 12.72s 5.41s 9.84s 11.52s 100% 1.16/s
50 102400 11.94s 5.58s 20.16s 12.83s 16.13s 16.87s 100% 0.57/s
100 4 7.15s 2.96s 8.29s 7.22s 7.87s 8.03s 100% 3.42/s
100 512 7.8s 6.03s 8.58s 7.75s 8.42s 8.47s 100% 3.33/s
100 1024 7.87s 6s 8.7s 7.7s 8.57s 8.61s 100% 3.23/s
100 5120 7.8s 4.9s 8.88s 7.73s 8.41s 8.62s 100% 2.66/s
100 10240 8s 6.44s 8.9s 7.91s 8.82s 8.84s 100% 2.27/s
100 20480 8.16s 5.46s 10.09s 8.24s 8.79s 8.94s 100% 1.75/s
100 40960 9.01s 3.57s 23.53s 8.36s 14.46s 16.77s 100% 1.21/s
100 102400 24.31s 8.45s 1m13.99s 18.2s 47.56s 59.53s 100% 0.53/s
200 4 15.45s 5.02s 18.25s 15.2s 17.13s 17.55s 100% 3.51/s
200 512 15.21s 11.42s 18.84s 15.12s 16.48s 16.91s 100% 3.37/s
200 1024 14.77s 8.94s 20.98s 14.66s 16.78s 17.56s 100% 3.47/s
200 5120 15.54s 11.99s 17.38s 15.46s 16.57s 16.68s 100% 2.72/s
200 10240 14.32s 7.93s 23.44s 14.04s 18.45s 19.5s 100% 2.29/s
200 20480 15.53s 7.24s 20.13s 15.52s 18.23s 18.75s 100% 1.63/s
200 40960 18.97s 4.07s 52.09s 15.76s 34.82s 41.62s 100% 1.14/s
200 102400 1m14.43s 22.22s 2m16.78s 1m12.56s 1m51.58s 1m59.24s 100% 0.53/s
300 4 20.44s 1.69s 30.99s 20.38s 26.34s 27.73s 99.33% 3.61/s
300 512 20.42s 7.59s 28.34s 18.78s 25.56s 26.16s 100% 3.49/s
300 1024 20.87s 1.76s 30.79s 20.89s 26.06s 27.25s 99.33% 3.43/s
300 5120 19.86s 1.87s 31.07s 20.02s 25.58s 26.34s 99.5% 2.83/s
300 10240 19.99s 2.11s 33.73s 19.37s 26.44s 28.14s 98.16% 2.45/s
300 20480 20.19s 3.59s 33.34s 20.07s 27.44s 28.92s 99.66% 1.84/s
300 40960 39.37s 10.46s 1m30.68s 38.18s 1m0.03s 1m8.77s 100% 1.19/s
300 102400 1m50.89s 17.99s 3m48.34s 1m51.11s 2m32.67s 2m52.09s 100% 0.56/s
400 4 29.85s 10.33s 41.13s 29.81s 32.08s 32.62s 100% 3.49/s
400 512 27.99s 1.12s 51.09s 27.81s 33.93s 36.2s 96.5% 5.03/s
400 1024 26.86s 1.71s 46.31s 27.06s 31.89s 32.64s 97% 4.84/s
400 5120 28.53s 1.39s 46.63s 28.75s 34.32s 36.72s 97.5% 3.87/s
400 10240 28.85s 1.21s 46.99s 28.54s 35.66s 37.6s 99.5% 3.06/s
400 20480 29.09s 1.8s 1m12.35s 28.19s 42.65s 53.6s 99.62% 2.08/s
400 40960 57.76s 7.17s 2m5.21s 55.57s 1m30.39s 1m43.55s 100% 1.15/s
400 102400 2m21.81s 12.95s 5m7.45s 2m19.01s 3m39.94s 4m3.83s 100% 0.56/s
500 4 32.18s 1.5s 54.21s 31.98s 40.73s 42.8s 97.5% 13.18/s
500 512 27.69s 3.31s 57.55s 26.03s 42.59s 46.15s 96.4% 13.91/s
500 1024 31.22s 1.72s 55.97s 31s 40.27s 43.12s 93.9% 11.70/s
500 5120 32.51s 2.27s 56.4s 32.35s 40.43s 42.34s 97.8% 6.70/s
500 10240 31.79s 1.6s 56.97s 31.12s 42.7s 46.45s 97.9% 3.17/s
500 20480 32.38s 2.94s 1m1.66s 31.76s 44.25s 45.88s 99.2% 1.94/s
500 40960 1m13s 9.46s 2m30.95s 1m10.53s 1m52.51s 2m0.67s 100% 1.14/s
500 102400 3m1.27s 13.7s 6m50.15s 2m57.85s 4m14.3s 4m40.32s 100% 0.55/s
1000 4 47.14s 307.48ms 1m42.59s 45.03s 1m18.18s 1m25.66s 85.25% 17.41/s
1000 512 52.53s 515.47ms 1m55.77s 45.88s 1m24.78s 1m32.46s 90.5% 14.31/s
1000 1024 51.41s 312.84ms 1m48.92s 45.97s 1m25.46s 1m31.33s 85.95% 13.52/s
1000 5120 49.83s 328.2ms 1m44.26s 48.69s 1m16.78s 1m23.47s 88.2% 7.82/s
1000 10240 55.11s 589.08ms 1m52.43s 52.13s 1m23.79s 1m30.69s 91.9% 5.11/s
1000 20480 1m8.97s 5.74s 2m35.85s 1m4.5s 1m47.31s 1m59.7s 98.1% 2.27/s
1000 40960 2m26.36s 17.45s 5m38.16s 2m19.13s 4m11.1s 4m41.72s 99.55% 1.34/s
1000 102400 6m28.78s 53.38s 10m25.54s 6m29.55s 9m18.23s 9m51.78s 100% 0.31/s

push-artifacts-to-same-projects

VUS BLOB SIZE(kbyte) Avg Duration Min Duration Max Duration Med Duration P90 P95 SUCCESS RATE ITERATIONS RATE
50 4 4.74s 2.11s 5.21s 4.58s 5.17s 5.18s 100% 3.11/s
50 512 4.98s 3.54s 6.12s 5.18s 5.82s 5.94s 100% 2.90/s
50 1024 4.86s 2.4s 5.47s 5.1s 5.31s 5.42s 100% 2.88/s
50 5120 6.51s 3.48s 8.83s 6.16s 8.75s 8.77s 100% 2.27/s
50 10240 4.93s 2.76s 7.08s 5.32s 5.43s 5.64s 100% 2.17/s
50 20480 5.83s 2.18s 14.41s 4.77s 9.65s 11.29s 100% 1.61/s
50 40960 6.44s 2.55s 17.01s 5.55s 11s 12.82s 100% 1.11/s
50 102400 20.32s 3.85s 1m7.35s 11.06s 55.02s 57.86s 100% 0.45/s
100 4 7.99s 4.34s 9.28s 7.9s 8.61s 8.89s 100% 3.24/s
100 512 8.36s 2.87s 9.8s 8.37s 9.15s 9.31s 100% 3.25/s
100 1024 8.08s 3.86s 8.86s 8.11s 8.45s 8.56s 100% 3.04/s
100 5120 7.95s 3.51s 13.21s 7.96s 9.05s 9.55s 100% 2.75/s
100 10240 9.51s 2.66s 22.08s 8.72s 14.58s 16.27s 100% 2.20/s
100 20480 11.09s 2.91s 29.01s 9.23s 18.81s 20.51s 100% 1.60/s
100 40960 19.68s 3s 1m4.45s 13.36s 43.17s 49.85s 100% 0.94/s
100 102400 1m11.02s 4.23s 3m22.94s 35.41s 3m13.42s 3m22.8s 76.5% 0.45/s
200 4 14.96s 4.85s 26.02s 14.28s 22.12s 22.65s 100% 3.24/s
200 512 15.67s 5.43s 26.09s 15.74s 18.68s 20.89s 100% 3.12/s
200 1024 14.35s 6.17s 24.49s 14.21s 18.36s 19.33s 100% 3.18/s
200 5120 19.32s 1.88s 41.62s 18.74s 27.62s 34.76s 100% 2.21/s
200 10240 18.73s 2.56s 56.4s 16.88s 33.76s 41.05s 100% 1.90/s
200 20480 48.54s 1.94s 2m35.81s 37.72s 1m44.99s 2m5.05s 100% 1.15/s
200 40960 32.28s 3.24s 2m3.22s 25.4s 1m1.33s 1m21.78s 100% 0.95/s
200 102400 1m27.57s 15.01s 4m51.97s 1m11.8s 2m37.25s 3m18.62s 100% 0.46/s
300 4 19.45s 4.84s 31.51s 18.74s 26.25s 27.31s 100% 3.10/s
300 512 18.98s 2.84s 32.79s 18.31s 25.12s 26.45s 99.66% 3.15/s
300 1024 20.04s 4.64s 33.62s 18.78s 27.68s 28.74s 99.83% 3.25/s
300 5120 19.55s 4.68s 40.68s 18.36s 28.5s 33.17s 99.83% 2.43/s
300 10240 22.79s 3.36s 1m8.75s 20.29s 37.27s 48.88s 100% 1.90/s
300 20480 40.12s 2.34s 2m37.96s 26.65s 1m31.58s 1m48.56s 100% 1.29/s
300 40960 53.26s 7.83s 3m11.42s 44.97s 1m47.03s 2m5.72s 100% 0.95/s
300 102400 2m50.62s 8.59s 9m8.57s 1m49.71s 7m3.64s 7m44.12s 89.33% 0.47/s
400 4 25.8s 2.13s 40.64s 26.32s 31.28s 31.7s 97.62% 4.43/s
400 512 28.52s 1.09s 41.72s 28.21s 32.54s 33.16s 98.87% 11.54/s
400 1024 29.68s 1.8s 40s 29.72s 33.23s 35.72s 99.87% 4.15/s
400 5120 26.99s 1.78s 43.4s 26.17s 33.56s 34.64s 98.25% 3.67/s
400 10240 30.23s 9.67s 1m4.26s 29.8s 40.18s 44.97s 100% 2.15/s
400 20480 41.29s 2.96s 2m0.12s 35.19s 1m15.6s 1m24.8s 100% 1.53/s
400 40960 1m30.17s 2.6s 5m11.48s 1m15.42s 2m41.71s 3m22.19s 100% 0.88/s
400 102400 2m24.7s 2.89s 5m26.1s 1m57.51s 3m57.75s 3m58.01s 50.12% 0.60/s
500 4 31.58s 1.77s 1m0.5s 31.42s 39.44s 42.54s 97% 13.10/s
500 512 26.41s 1.9s 52.11s 26.45s 36.58s 38s 96.8% 9.11/s
500 1024 30.29s 555.07ms 57.54s 29.12s 43.78s 46.7s 96.2% 11.58/s
500 5120 32.11s 1.73s 57.83s 32.01s 39.29s 41.8s 97.6% 6.67/s
500 10240 31.46s 2.24s 56.39s 31.48s 39.87s 42.47s 98.4% 3.03/s
500 20480 33.39s 7.21s 1m15.49s 32.11s 50.87s 56.74s 100% 1.84/s
500 40960 1m11.91s 8.71s 2m29.33s 1m9.86s 1m48.74s 1m58.35s 100% 1.13/s
500 102400 3m4.63s 13.89s 6m49.78s 2m57.55s 4m52.42s 5m37.59s 100% 0.55/s
1000 4 51.03s 318.37ms 1m48.99s 46.51s 1m26.75s 1m34.58s 88.75% 15.81/s
1000 512 46.45s 296.48ms 1m43.93s 43.64s 1m13.06s 1m20.76s 87.65% 15.42/s
1000 1024 41.23s 314.81ms 1m30.02s 41.33s 1m2.35s 1m8.32s 87% 14.28/s
1000 5120 46.42s 331.87ms 1m40.58s 45.98s 1m10.04s 1m19.85s 86.6% 7.71/s
1000 10240 45.85s 553.95ms 1m38.19s 44.8s 1m11.57s 1m18.68s 87.15% 4.99/s
1000 20480 1m14.08s 5.68s 2m51.33s 1m10.98s 1m47.42s 2m4.8s 92.65% 2.77/s
1000 40960 2m32.44s 7.56s 5m43.41s 2m23.58s 4m21.09s 4m53.73s 96.7% 1.41/s
1000 102400 6m23.78s 42.64s 10m26.23s 6m34.98s 9m25.75s 9m46.52s 100% 0.29/s

pull-artifacts-from-different-projects

VUS BLOB SIZE(kbyte) Avg Duration Min Duration Max Duration Med Duration P90 P95 SUCCESS RATE ITERATIONS RATE
50 4 905.89ms 549.64ms 1.65s 890.71ms 1.09s 1.32s 100% 0.45/s
50 512 919.14ms 561.73ms 1.5s 933.08ms 1.2s 1.4s 100% 0.51/s
50 1024 929.66ms 476.97ms 2.22s 949.02ms 1.27s 1.29s 100% 0.50/s
50 5120 1.81s 890.85ms 3.95s 1.73s 2.69s 2.8s 100% 0.47/s
50 10240 3.62s 670.83ms 9.7s 2.83s 7.17s 7.68s 100% 0.42/s
50 20480 5.37s 1.46s 16.54s 3.54s 12.04s 14.21s 100% 0.36/s
50 40960 9.59s 2.46s 29.41s 6.39s 19.88s 23.78s 100% 0.27/s
50 102400 22.71s 4.97s 1m12.59s 15.28s 58.42s 1m5.77s 100% 0.13/s
100 4 1.7s 599.71ms 3.17s 1.66s 2.54s 2.73s 100% 0.90/s
100 512 1.23s 607.68ms 2.37s 1.17s 1.7s 1.95s 100% 0.84/s
100 1024 3.51s 719.66ms 7.51s 2.35s 6s 6.24s 100% 0.79/s
100 5120 2.85s 787.43ms 7.42s 2.57s 4.81s 5.41s 100% 0.88/s
100 10240 4.23s 1.58s 9.47s 4.05s 5.79s 6.11s 100% 0.79/s
100 20480 7.83s 3.54s 18.26s 7.13s 11.89s 14.05s 100% 0.77/s
100 40960 18.9s 2.75s 57.54s 13.46s 39.95s 48.57s 100% 0.51/s
100 102400 41.33s 9.61s 2m30.11s 26.82s 1m34.66s 1m42.67s 100% 0.27/s
200 4 1.85s 803.68ms 4.46s 1.7s 2.9s 3.3s 100% 1.93/s
200 512 2.72s 896.46ms 7.02s 2.51s 4.3s 5.1s 100% 1.92/s
200 1024 2.87s 1.09s 7.12s 2.66s 4.25s 4.85s 100% 1.87/s
200 5120 6.24s 1.21s 16.72s 5.86s 9.74s 11.12s 100% 1.73/s
200 10240 7.55s 2.26s 18.95s 7.07s 11.2s 14.04s 100% 1.58/s
200 20480 14.53s 1.67s 38.99s 12.42s 24.7s 29.88s 100% 1.39/s
200 40960 27.77s 6.83s 1m11.48s 24.57s 47.95s 55.61s 100% 1.04/s
200 102400 1m12.39s 11.96s 3m25.04s 1m1.22s 2m18.94s 2m33.71s 100% 0.60/s
300 4 3.17s 981.03ms 7.2s 2.88s 5.19s 5.88s 100% 2.78/s
300 512 3.11s 857.19ms 8.77s 2.73s 5.22s 6.17s 100% 2.81/s
300 1024 3.46s 980.37ms 8.61s 3.13s 5.84s 6.88s 100% 2.80/s
300 5120 8.04s 1.45s 21.28s 7.25s 13.03s 16.1s 100% 2.47/s
300 10240 11.28s 2.63s 30.32s 9.72s 19.73s 23.72s 100% 2.10/s
300 20480 21.01s 4.73s 54.69s 18s 38.26s 43.72s 100% 1.93/s
300 40960 39.06s 5.24s 1m41.75s 33.05s 1m12.87s 1m25.23s 100% 1.42/s
300 102400 1m37.05s 22.82s 4m14.15s 1m24.01s 3m5.44s 3m33.67s 100% 0.83/s
400 4 3.55s 1s 9.44s 3.11s 6.02s 7.21s 100% 3.79/s
400 512 3.6s 998.07ms 9.53s 3.18s 6.01s 7.15s 100% 3.79/s
400 1024 6.91s 1.32s 16.66s 6.34s 12.08s 13.8s 100% 3.61/s
400 5120 10.64s 1.72s 26.9s 9.76s 18.06s 21.07s 100% 3.31/s
400 10240 15.41s 2.74s 38.7s 13.5s 27.17s 31.59s 100% 2.98/s
400 20480 27.88s 6.11s 1m10.95s 24.97s 51.12s 58.39s 100% 2.44/s
400 40960 52.82s 7.18s 2m17.31s 44.93s 1m41.49s 1m57s 100% 1.74/s
400 102400 2m8.79s 1.6s 5m37.22s 1m48.55s 4m16.98s 4m45.3s 97.12% 0.98/s
500 4 4.45s 896.87ms 11.56s 3.91s 8.03s 9.18s 100% 4.76/s
500 512 5.9s 1.36s 15.37s 5.23s 10.69s 12.41s 100% 4.42/s
500 1024 8.73s 1.86s 21.89s 7.61s 14.86s 18.06s 100% 4.21/s
500 5120 31.63s 655.32ms 1m40.06s 25.77s 1m5.96s 1m13.83s 100% 2.85/s
500 10240 48.04s 754.34ms 2m30.51s 41.05s 1m34.79s 1m47.66s 100% 2.30/s
500 20480 47.48s 34.06ms 2m27.31s 47.34s 1m23.25s 1m49.9s 58.8% 2.22/s
500 40960 1m1.87s 30.19ms 2m59.56s 53.84s 2m9.57s 2m28.35s 78.1% 1.80/s
500 102400 2m1.92s 76.68ms 6m12.34s 1m38.31s 4m30.7s 5m0.92s 74.7% 0.89/s
1000 4 9.63s 1.1s 25.27s 9.16s 16.3s 19.57s 100% 7.70/s
1000 512 10.28s 1.03s 25.95s 9.07s 19.16s 22.25s 100% 7.67/s
1000 1024 18.95s 1.46s 44.72s 18.44s 30.87s 37.54s 100% 7.07/s
1000 5120 34.57s 1.03s 1m43.5s 30.11s 1m5.84s 1m16.84s 100% 5.57/s
1000 10240 58.56s 1.02s 2m28.98s 50.64s 1m57.02s 2m7.29s 96.85% 4.62/s
1000 20480 1m8.26s 2.73s 3m15.7s 58.81s 2m18.68s 2m36.42s 100% 3.98/s
1000 40960 3m44.71s 161.81ms 9m50.72s 3m21.81s 7m46.59s 8m30.53s 84.45% 2.02/s
1000 102400 3m32.82s 2.97s 10m25.6s 2m48.56s 7m33.87s 8m59.61s 94.87% 0.72/s

pull-artifacts-from-same-project

VUS BLOB SIZE(kbyte) Avg Duration Min Duration Max Duration Med Duration P90 P95 SUCCESS RATE ITERATIONS RATE
50 4 9.74s 553.59ms 22.78s 8.47s 20.16s 21.19s 100% 3.76/s
50 512 10.16s 662.95ms 24.88s 8.6s 20.91s 22.57s 100% 3.59/s
50 1024 10.98s 650.07ms 28.41s 8.93s 23.24s 25.52s 100% 3.11/s
50 5120 14.05s 618.32ms 37.42s 11.88s 27.33s 33.64s 100% 2.38/s
50 10240 18.2s 1.04s 47.6s 17.04s 35.35s 39.85s 100% 1.94/s
50 20480 23.67s 1.06s 1m4.14s 20.26s 47.78s 53.92s 100% 1.46/s
50 40960 46.57s 1.57s 2m3.46s 37.95s 1m36.6s 1m51.36s 100% 0.76/s
50 102400 1m42.44s 1.75s 4m39.63s 1m23.01s 3m49.76s 4m9.6s 100% 0.35/s
100 4 18.88s 666.62ms 45.47s 17.05s 35.84s 39.94s 100% 4.02/s
100 512 20.32s 1.3s 50.45s 17.36s 40.33s 44.3s 100% 3.59/s
100 1024 21.88s 824.46ms 53.05s 19.9s 43.02s 47.25s 100% 3.25/s
100 5120 28.8s 892.43ms 1m15.26s 26.17s 1m1.22s 1m7s 100% 2.45/s
100 10240 36.26s 679.41ms 1m33.77s 34.42s 1m13.09s 1m19.51s 100% 2.00/s
100 20480 1m0.14s 1.14s 2m43.9s 52.82s 2m8.27s 2m20.67s 100% 1.16/s
100 40960 1m36.22s 1.38s 4m30.53s 1m24.45s 3m13.94s 3m56.24s 100% 0.72/s
100 102400 1m52.76s 1.88s 4m54.07s 1m42.2s 3m48.03s 4m9.64s 100% 0.63/s
200 4 38.06s 809.6ms 1m29.51s 34.55s 1m14.06s 1m22.7s 100% 4.10/s
200 512 38.64s 849.02ms 1m31.87s 34.45s 1m15.82s 1m23.67s 100% 4.01/s
200 1024 41.66s 1.11s 1m44.51s 38.66s 1m20.33s 1m27.81s 100% 3.49/s
200 5120 59.33s 1.55s 2m28.27s 54.89s 1m55.64s 2m10.38s 100% 2.50/s
200 10240 1m4.03s 920.08ms 2m46.36s 58.22s 2m10.48s 2m28.14s 100% 2.28/s
200 20480 2m20.2s 1.61s 6m4.15s 2m6.47s 4m50.35s 5m17.46s 100% 1.04/s
200 40960 3m47.48s 1.48s 10m4.33s 3m14.51s 8m2.14s 8m45.93s 100% 0.64/s
200 102400 3m21.47s 2.35s 10m27.52s 2m41.11s 7m20.53s 8m14.15s 100% 0.33/s
300 4 55.24s 947.87ms 2m10.02s 49.55s 1m46.52s 2m0.3s 100% 4.13/s
300 512 1m0.95s 939.97ms 2m29.64s 55.73s 1m59.85s 2m12.37s 100% 3.76/s
300 1024 1m10.47s 991.62ms 3m3.87s 1m2.4s 2m21.42s 2m40.36s 100% 3.10/s
300 5120 1m55.33s 935.28ms 5m6s 1m43.41s 3m55.27s 4m26.14s 100% 1.91/s
300 10240 2m46.12s 884.21ms 7m27.34s 2m17.75s 5m49.54s 6m21.01s 100% 1.31/s
300 20480 2m50.13s 842.53ms 6m58.22s 2m28.28s 5m47.92s 6m24.18s 100% 1.39/s
300 40960 3m31.36s 1.22s 10m23.55s 2m45.32s 7m59.27s 9m18.85s 100% 0.85/s
300 102400 3m51.39s 2.06s 10m22.59s 3m22.69s 8m2.04s 9m8.46s 100% 0.68/s
400 4 1m13.45s 998.68ms 2m52.36s 1m6.08s 2m21.43s 2m35.33s 100% 4.31/s
400 512 1m20.6s 649.55ms 3m22.2s 1m12.89s 2m36.38s 2m53.99s 100% 3.76/s
400 1024 1m27.84s 1.6s 3m48.39s 1m21.08s 2m46.21s 3m7.93s 100% 3.34/s
400 5120 2m9.35s 591.15ms 5m54.09s 1m51.64s 4m18.01s 5m2.73s 100% 2.17/s
400 10240 2m37.68s 672.52ms 7m10.24s 2m10.1s 5m20.13s 6m3.23s 100% 1.80/s
400 20480 3m47.37s 1.56s 10m11.56s 3m14.86s 8m14.37s 8m57.83s 100% 0.90/s
400 40960 4m7.84s 1.3s 10m29.66s 3m31.46s 8m27.02s 9m15.26s 100% 0.55/s
400 102400 4m0.38s 2.45s 10m25s 3m33.75s 8m30.22s 9m22.19s 100% 0.35/s
500 4 1m34.57s 1.86s 3m43.29s 1m23s 3m2.69s 3m23.26s 100% 4.23/s
500 512 1m41.8s 789.48ms 4m24.4s 1m32.92s 3m15.59s 3m39.61s 100% 3.62/s
500 1024 1m58.43s 849.62ms 5m15.91s 1m46.03s 3m44.21s 4m28.05s 100% 3.04/s
500 5120 3m31.35s 1.75s 9m28.47s 2m58.93s 7m11.37s 8m13.06s 100% 1.70/s
500 10240 3m40.59s 707.84ms 10m27.8s 3m1.85s 7m36.95s 9m12.85s 100% 1.23/s
500 20480 3m54.34s 853.71ms 10m25.1s 3m22.99s 8m18.77s 9m9.16s 100% 0.86/s
500 40960 3m42.55s 1.48s 10m29.03s 3m2.46s 8m11.57s 9m29.09s 100% 0.70/s
500 102400 4m45.36s 4.15s 10m29.76s 4m31.71s 9m0.88s 9m42.84s 100% 0.33/s
1000 4 3m6.07s 1.89s 7m19.05s 2m48.86s 5m59.39s 6m34.52s 100% 4.33/s
1000 512 3m45.75s 1.04s 9m40.91s 3m29.11s 7m3.64s 8m4.6s 100% 3.31/s
1000 1024 3m57.91s 1.21s 10m18.6s 3m36.21s 7m38.78s 8m58.79s 100% 3.09/s
1000 5120 3m55.67s 550.33ms 10m28.94s 3m30.54s 7m51.64s 9m7.79s 100% 2.13/s
1000 10240 3m58.34s 625.94ms 10m27.1s 3m35.09s 8m5.12s 9m16.33s 100% 1.47/s
1000 20480 4m8.95s 1.26s 10m29.59s 3m44.75s 8m26.72s 9m29.06s 100% 1.00/s
1000 40960 4m36.18s 2.01s 10m28.81s 4m12.9s 9m7.83s 9m50.91s 100% 0.68/s
1000 102400 5m25.2s 4.75s 10m22.27s 5m19.89s 9m39.35s 10m0.14s 100% 0.22/s

@github-actions GitHub Actions
Copy link

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

@github-actions github-actions bot added the Stale label May 24, 2024
@github-actions GitHub Actions
Copy link

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jun 23, 2024
@ianseyer
Copy link

ianseyer commented Nov 7, 2024

It may be worth noting that the current nginx config does not specify a timeout, and therefore defaults to 60s. Specifying this to something like 5m made a lot of the "invisible" 502 bad gateway and context canceled errors go away

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wy65701436 wy65701436

Labels
area/performance more-info-needed The issue author need to provide more details and context to the issue Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wy65701436 @ianseyer @microyahoo @chlins