custom gitlab still no syncing #165

bash-stack · 2020-05-19T19:30:44Z

#158 followup ...

the first entry is https://username:token , worked ,
custom gitlab entries get stuck ( and yes bin/console messenger:consume async is running )

Logs:

{"time":"2020-05-19T19:23:34.667Z","severity":"INFO","duration":234.94,"db":21.03,"view":213.91,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"simple","value":"true"},{"key":"order_by","value":"last_activity_at"},{"key":"owned","value":"true"}],"host":"somegitlab.tld","remote_ip":"0.0.0.0, 0.0.0.0","ua":"php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)","route":"/api/:version/projects","user_id":23,"username":"someuser","queue_duration":21.02,"gitaly_calls":1,"gitaly_duration":4.54,"correlation_id":"U7pPs8VDRH5"}
{"time":"2020-05-19T19:23:34.853Z","severity":"INFO","duration":149.55,"db":12.67,"view":136.88000000000002,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"membership","value":"false"},{"key":"order_by","value":"last_activity_at"},{"key":"owned","value":"true"},{"key":"page","value":"2"},{"key":"per_page","value":"20"},{"key":"simple","value":"true"},{"key":"sort","value":"desc"},{"key":"starred","value":"false"},{"key":"statistics","value":"false"},{"key":"with_custom_attributes","value":"false"},{"key":"with_issues_enabled","value":"false"},{"key":"with_merge_requests_enabled","value":"false"}],"host":"somegitlab.tld","remote_ip":"0.0.0.0, 0.0.0.0","ua":"php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)","route":"/api/:version/projects","user_id":23,"username":"someuser","queue_duration":23.96,"gitaly_calls":2,"gitaly_duration":12.32,"correlation_id":"ppcecS9JOB3"}
Started GET "/api/v4/projects?simple=true&order_by=last_activity_at&owned=true" for 0.0.0.0 at 2020-05-19 21:23:34 +0200
Started GET "/api/v4/projects?membership=false&order_by=last_activity_at&owned=true&page=2&per_page=20&simple=true&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false" for 0.0.0.0 at 2020-05-19 21:23:34 +0200
Started GET "/api/v4/projects?simple=true&order_by=last_activity_at&membership=true" for 0.0.0.0 at 2020-05-19 21:23:34 +0200
0.0.0.0 - - [19/May/2020:21:23:34 +0200] "GET /api/v4/projects?simple=true&order_by=last_activity_at&owned=true HTTP/2.0" 200 2347 "" "php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)"
0.0.0.0 - - [19/May/2020:21:23:34 +0200] "GET /api/v4/projects?membership=false&order_by=last_activity_at&owned=true&page=2&per_page=20&simple=true&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false HTTP/2.0" 200 1786 "" "php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)"
{"time":"2020-05-19T19:23:35.043Z","severity":"INFO","duration":167.44,"db":12.04,"view":155.4,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"simple","value":"true"},{"key":"order_by","value":"last_activity_at"},{"key":"membership","value":"true"}],"host":"somegitlab.tld","remote_ip":"0.0.0.0, 0.0.0.0","ua":"php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)","route":"/api/:version/projects","user_id":23,"username":"someuser","queue_duration":13.93,"gitaly_calls":1,"gitaly_duration":4.41,"correlation_id":"d4nM4bsioW2"}
{"time":"2020-05-19T19:23:35.476Z","severity":"INFO","duration":398.42,"db":10.77,"view":387.65000000000003,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"membership","value":"true"},{"key":"order_by","value":"last_activity_at"},{"key":"owned","value":"false"},{"key":"page","value":"2"},{"key":"per_page","value":"20"},{"key":"simple","value":"true"},{"key":"sort","value":"desc"},{"key":"starred","value":"false"},{"key":"statistics","value":"false"},{"key":"with_custom_attributes","value":"false"},{"key":"with_issues_enabled","value":"false"},{"key":"with_merge_requests_enabled","value":"false"}],"host":"somegitlab.tld","remote_ip":"0.0.0.0, 0.0.0.0","ua":"php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)","route":"/api/:version/projects","user_id":23,"username":"someuser","queue_duration":26.82,"gitaly_calls":2,"gitaly_duration":10.08,"correlation_id":"Bm2ICu7Fht7"}
Started GET "/api/v4/projects?membership=true&order_by=last_activity_at&owned=false&page=2&per_page=20&simple=true&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false" for 0.0.0.0 at 2020-05-19 21:23:35 +0200
0.0.0.0 - - [19/May/2020:21:23:35 +0200] "GET /api/v4/projects?simple=true&order_by=last_activity_at&membership=true HTTP/2.0" 200 2630 "" "php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)"
0.0.0.0 - - [19/May/2020:21:23:35 +0200] "GET /api/v4/projects?membership=true&order_by=last_activity_at&owned=false&page=2&per_page=20&simple=true&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false HTTP/2.0" 200 2304 "" "php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)"
{"time":"2020-05-19T19:23:36.869Z","severity":"INFO","duration":16.15,"db":3.11,"view":13.04,"status":200,"method":"GET","path":"/api/v4/projects/162/hooks","params":[],"host":"somegitlab.tld","remote_ip":"0.0.0.0, 0.0.0.0","ua":"php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)","route":"/api/:version/projects/:id/hooks","user_id":23,"username":"someuser","queue_duration":12.03,"correlation_id":"juHkkUtLjS8"}
{"time":"2020-05-19T19:23:36.936Z","severity":"INFO","duration":32.77,"db":5.28,"view":27.490000000000002,"status":201,"method":"POST","path":"/api/v4/projects/162/hooks","params":[{"key":"push_events","value":"1"},{"key":"tag_push_events","value":"1"},{"key":"url","value":"https://repman.some.where.else/hook/82d60d69-edb7-434b-8f35-d60fb5326e85"}],"host":"somegitlab.tld","remote_ip":"0.0.0.0, 0.0.0.0","ua":"php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)","route":"/api/:version/projects/:id/hooks","user_id":23,"username":"someuser","queue_duration":13.79,"correlation_id":"g7KNez7wdk3"}
Started GET "/api/v4/projects/162/hooks" for 0.0.0.0 at 2020-05-19 21:23:36 +0200
Started POST "/api/v4/projects/162/hooks" for 0.0.0.0 at 2020-05-19 21:23:36 +0200
0.0.0.0 - - [19/May/2020:21:23:36 +0200] "GET /api/v4/projects/162/hooks HTTP/2.0" 200 2 "" "php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)"
0.0.0.0 - - [19/May/2020:21:23:36 +0200] "POST /api/v4/projects/162/hooks HTTP/2.0" 201 492 "" "php-gitlab-api (http://github.com/m4tthumphrey/php-gitlab-api)"

The text was updated successfully, but these errors were encountered:

akondas · 2020-05-20T05:40:15Z

Next things you can check:

Failed message queue: bin/console messenger:failed:show
Message consumer logs var/log/consumer.log
Supervisor logs /var/log/supervisor/message-consumer_00-stderr ...

bash-stack · 2020-05-20T05:42:15Z

Next things you can check:

hit:

Failed message queue: bin/console messenger:failed:show

52 Buddy\Repman\Message\Organization\SynchronizePackage 2020-05-19 20:46:11 Too few arguments to function Buddy\Repman\Service\PackageSynchronizer\ComposerPackageSynchronizer::__construct(), 4 passed in /var/www/repman/var/cache/local/ContainerX8StnRW/Buddy_Repman_KernelLocalContainer.php on line 1690 and exactly 5 expected
41 Buddy\Repman\Message\Organization\SynchronizePackage 2020-05-19 20:41:25 Too few arguments to function Buddy\Repman\Service\PackageSynchronizer\ComposerPackageSynchronizer::__construct(), 4 passed in /var/www/repman/var/cache/local/ContainerX8StnRW/Buddy_Repman_KernelLocalContainer.php on line 1690 and exactly 5 expected
64 Buddy\Repman\Message\Organization\SynchronizePackage 2020-05-19 21:25:43 Too few arguments to function Buddy\Repman\Service\PackageSynchronizer\ComposerPackageSynchronizer::__construct(), 4 passed in /var/www/repman/var/cache/local/ContainerX8StnRW/Buddy_Repman_KernelLocalContainer.php on line 1690 and exactly 5 expected
69 Buddy\Repman\Message\Organization\SynchronizePackage 2020-05-19 21:38:02 Too few arguments to function Buddy\Repman\Service\PackageSynchronizer\ComposerPackageSynchronizer::__construct(), 4 passed in /var/www/repman/var/cache/local/ContainerX8StnRW/Buddy_Repman_KernelLocalContainer.php on line 1690 and exactly 5 expected
75 Buddy\Repman\Message\Organization\SynchronizePackage 2020-05-19 21:48:15 Too few arguments to function Buddy\Repman\Service\PackageSynchronizer\ComposerPackageSynchronizer::__construct(), 4 passed in /var/www/repman/var/cache/local/ContainerX8StnRW/Buddy_Repman_KernelLocalContainer.php on line 1690 and exactly 5 expected
46 Buddy\Repman\Message\Organization\SynchronizePackage 2020-05-19 20:41:41 Too few arguments to function Buddy\Repman\Service\PackageSynchronizer\ComposerPackageSynchronizer::__construct(), 4 passed in /var/www/repman/var/cache/local/ContainerX8StnRW/Buddy_Repman_KernelLocalContainer.php on line 1690 and exactly 5 expected
58 Buddy\Repman\Message\Organization\SynchronizePackage 2020-05-19 20:49:54 Too few arguments to function Buddy\Repman\Service\PackageSynchronizer\ComposerPackageSynchronizer::__construct(), 4 passed in /var/www/repman/var/cache/local/ContainerX8StnRW/Buddy_Repman_KernelLocalContainer.php on line 1690 and exactly 5 expected
81 Buddy\Repman\Message\Organization\SynchronizePackage 2020-05-19 21:52:20 Too few arguments to function Buddy\Repman\Service\PackageSynchronizer\ComposerPackageSynchronizer::__construct(), 4 passed in /var/www/repman/var/cache/local/ContainerX8StnRW/Buddy_Repman_KernelLocalContainer.php on line 1690 and exactly 5 expected

akondas · 2020-05-20T05:57:11Z

Looks like you need to clear cache (bin/console cache:clear). You probably did the code update without clearing your cache afterwards.

Next you can run: bin/console messenger:failed:retry to retry failed messages.

Tip: if you do manual update please check ansible/deploy.yml for list of steps you need to do:

composer install
clear cache
run migrations
restart consumers

(it is for now, but it can always change)

bash-stack · 2020-05-20T18:46:56Z

now it fails with

Error: No driver found to handle VCS repository https://somegitlab.tld:8888/mygroup/myproject

akondas · 2020-05-21T05:14:17Z

What is the type of package? Because authorization for this type of files is set only for packages added by the oauth integration. Otherwise, you need to either add the repman's ssh key to your gitlab instance and use the git repository link, or use the token link as you wrote earlier.

bash-stack · 2020-05-22T14:39:36Z

oauth ( clicking through → organization → add package → gitlab → "repository/path" )

nickygerritsen · 2020-05-27T13:08:34Z

So this might be related, but when I use a custom GitLab host and add a repository, I get that the bin/console messenger:consume async command asks for a username/password for my custom gitlab URL (and indeed the UI shows the above message).

Screenshot of the messenger:consume with -vv:

My guess: the git clone doesn't use the correct authentication?

nickygerritsen · 2020-05-27T14:08:59Z

Did some debugging and it turns out Composer uses the VcsRepository::getDriver and loops over drivers. It then comes to the GitDriver and asks for a username/password.

First of all I think the last two arguments of https://github.com/repman-io/repman/blob/master/src/Service/PackageSynchronizer/ComposerPackageSynchronizer.php#L121 need to be swapped? Although Composer itself in their Gitlab driver also uses this order 🤔 .

But even with that it doesn't work. Composer uses the GitDriver to fetch the data. I tried updating ComposerPackageSynchronizer::createConfig to also add gitlab-domains and gitlab-token to force it to use the GitLab API to fetch data but couldn't get that to work either, since it seems to not be able to authenticate then either.

karniv00l · 2020-05-27T14:16:28Z

Hi @nickygerritsen,

is this on your local machine? Would you try reproducing that using docker?

nickygerritsen · 2020-05-27T14:20:01Z

It is on my machine with a clone from master without Docker.
Can I easily run master in Docker? Since the releases don't have the latest code yet.

karniv00l · 2020-05-27T14:28:07Z

Sure, just create docker-compose.override.yml file containing:

version: '3.4'

services:
  app:
    image: repman:beta
    build: .

  consumer:
    image: repman:beta
    build: .

  cron:
    image: repman:beta
    build: .

And run docker-compose up --build

although you could start with removing your GitLab token (Profile -> Unlink GitLab) and trying again?

nickygerritsen · 2020-05-27T14:37:25Z

Just tried Docker, that gives the same issue indeed.

…an-io#165.

nickygerritsen · 2020-05-27T18:39:04Z

See the PR I just created, that seems to fix it for me at least.

…an-io#165.

akondas · 2020-05-27T20:27:10Z

@bash-stack please try to update (#179) and check if this fix helps

akondas · 2020-06-26T21:16:53Z

@bash-stack do you think we can close this issue now?

bash-stack mentioned this issue May 19, 2020

private repositories from custom gitlab pulled without auth ( fails with 401) #158

Closed

akondas added the question Further information is requested label May 20, 2020

nickygerritsen added a commit to Lets-Talk-NL/repman that referenced this issue May 27, 2020

Write custom Gitlab URL to gitlab-domains composer option. Fixes repm…

980527e

…an-io#165.

nickygerritsen added a commit to Lets-Talk-NL/repman that referenced this issue May 27, 2020

Write custom Gitlab URL to gitlab-domains composer option. Fixes repm…

3b40a3c

…an-io#165.

akondas closed this as completed in 22dbc71 May 27, 2020

akondas reopened this May 27, 2020

akondas closed this as completed Sep 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

custom gitlab still no syncing #165

custom gitlab still no syncing #165

bash-stack commented May 19, 2020 •

edited

Loading

akondas commented May 20, 2020

bash-stack commented May 20, 2020 •

edited

Loading

akondas commented May 20, 2020

bash-stack commented May 20, 2020

akondas commented May 21, 2020

bash-stack commented May 22, 2020

nickygerritsen commented May 27, 2020

nickygerritsen commented May 27, 2020 •

edited

Loading

karniv00l commented May 27, 2020

nickygerritsen commented May 27, 2020

karniv00l commented May 27, 2020

nickygerritsen commented May 27, 2020

nickygerritsen commented May 27, 2020

akondas commented May 27, 2020

akondas commented Jun 26, 2020

custom gitlab still no syncing #165

custom gitlab still no syncing #165

Comments

bash-stack commented May 19, 2020 • edited Loading

akondas commented May 20, 2020

bash-stack commented May 20, 2020 • edited Loading

akondas commented May 20, 2020

bash-stack commented May 20, 2020

akondas commented May 21, 2020

bash-stack commented May 22, 2020

nickygerritsen commented May 27, 2020

nickygerritsen commented May 27, 2020 • edited Loading

karniv00l commented May 27, 2020

nickygerritsen commented May 27, 2020

karniv00l commented May 27, 2020

nickygerritsen commented May 27, 2020

nickygerritsen commented May 27, 2020

akondas commented May 27, 2020

akondas commented Jun 26, 2020

bash-stack commented May 19, 2020 •

edited

Loading

bash-stack commented May 20, 2020 •

edited

Loading

nickygerritsen commented May 27, 2020 •

edited

Loading