http error code 400 when bitbucket-server running behind awselb

[mwiede]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

bitbucket-server 8.9.14

Please tell us more about your question or problem

We have changed our configuration from RENOVATE_PASSWORD to RENOVATE_TOKEN and with persistRepoData: true, with any next run, we retrieve http 400 errors from bitbucket server.

I activated the simple-git debug to see what is happening and I could see, that http.extraheader=Authorization: Bearer xxx was set. Nothing wrong so far. I also set GIT_CURL_VERBOSE=true and now could see the issue: The Authoriation header is sent twice

12:57:24.040455 http.c:802              => Send header: Host: bitbucket-host
12:57:24.040457 http.c:802
     => Send header: User-Agent: git/2.45.2
12:57:24.040458 http.c:802              => Send header: Accept: */*
12:57:24.040459 http.c:802
   => Send header: Accept-Encoding: deflate, gzip, br
12:57:24.040460 http.c:802              => Send header: Authorization: Bearer <redacted>
12:57:24.040461 http.c:802              => Send header: Authorization: Bearer <redacted>
12:57:24.040462 http.c:802              => Send header: Accept-Language: C, *;q=0.9
12:57:24.040463 http.c:802              => Send header: Pragma: no-cache
12:57:24.040463 http.c:802              => Send header: Git-Protocol: version=2
12:57:24.040464 http.c:802              => Send header:

But bitbucket was not returning the 400 response, it was the awsalb, which does not accept that. I asked AWS support, and they confirmed:

Multiple Authorization headers are not compliant with the HTTP Semantics RFC: https://www.rfc-editor.org/rfc/rfc9110.html#fields.order.

The issue seems to be related to #22894 and git-lfs/git-lfs#4031 because in this setup we have two locations where the header is set:

1. Once the repo is checked out with http.extraheader, the repo folder contains a .git/config containing the auth header
2. When renovate runs again, it again overwrites http.extraheader and therefore, git just sends both headers, the one from local repo config and the one from command.

So basically, one can reproduce this issue with or without renovate:

first run, repo does not exist on disk: ✅

2024-07-15T06:24:03.135Z simple-git [GitExecutor] [SPAWN] git [ '-c', 'core.quotePath=false', 'clone', '--filter=blob:none', '-c', 'http.extraheader=Authorization: Bearer xxxx', 'https://bitbucket-host/scm/~mwiede/renovate-debug.git', '.' ]

second run, repo is already on disk, renovate fails while doing git ls-remote: ❌

2024-07-15T06:25:49.529Z simple-git [GitExecutor] [SPAWN] git [ '-c', 'core.quotePath=false', '-c', 'http.extraheader=Authorization: Bearer xxxx', 'ls-remote', '--heads', 'https://bitbucket-host/scm/~mwiede/renovate-debug.git' ]

I think the issue can be solved by setting the http.extraheader before clone command and not after, so that it does't get persisted in the local git repo config file. I suggest to divide clone options and auth options in the related code.

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks

[ESloman]

I'm experiencing the same issue with a very similar setup. Bitbucket DC hosted in AWS and with renovate config persistRepoData: true. First run executes absolutely fine - subsequent runs always fail with 400 errors. Good to have an explanation as to why!