Sync - fix 404 errors

[ewels]

See #709 (comment)

Some pipelines get a 404 error when we try to make the automatic PR:

ERROR     Sync failed for nf-core/deepvariant:                       sync.py:385
GitHub API returned code 404:
{
    "message": "Not Found",
    "documentation_url": "https://developer.github.com/v3/pulls/#create-a-pull-request"
}

I don't know why this is happening, but we can try looping a few times with a sleep if we get specifically this error. Hopefully they will work on another go...?

[ewels]

Should compare these 404 error pipelines with the v1.9 release sync to see if there is any overlap

[ewels]

Issue for the v1.9 release sync is here: #548

Four out of the five repos that gave 404 errors this time also did for v1.9:

• deepvariant - 1.9 and 1.10
• hlatyping - just 1.9
• mag - 1.9 and 1.10
• mhcquant - 1.9 and 1.10
• nascent - 1.9 and 1.10

So this is not random as I had initially thought. Need to look into these pipelines to see what makes them different from the rest.

[ewels]

Ok, if you try to make this PR manually for deepvariant you see that there is no shared history between the branches, so that might explain that one.

The others look fine though, so that doesn't explain everything.

[ewels]

Trying to list PRs between these two branches on these repos also gives a 404, eg: https://api.github.com/repos/nf-core/mhcquan/pulls?head=nf-core:TEMPLATE&base=dev

[ewels]

Wait, what?? The URL is missing a t - mhcquan instead of mhcquant

The logs show that the code has the correct repo name, as it's pushing the template branch properly:

$  nf-core --log-file sync_log.txt sync nf-core/mhcquant --from-branch dev --pull-request --username nf-core-bot --repository nf-core/mhcquant

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 1.11.dev0



INFO      Pipeline directory: /Users/philewels/GitHub/nf-core/TEST_SYNC/nf-core/mhcquant                                                                                                                 sync.py:79
INFO      Using branch `dev` to fetch workflow variables                                                                                                                                                 sync.py:81
INFO      Will attempt to automatically create a pull request                                                                                                                                            sync.py:83
INFO      Original pipeline repository branch is 'master'                                                                                                                                               sync.py:124
INFO      Checking out workflow branch 'dev'                                                                                                                                                            sync.py:139
INFO      Deleting all files in TEMPLATE branch                                                                                                                                                         sync.py:200
INFO      Making a new template pipeline using pipeline variables                                                                                                                                       sync.py:218
INFO      Committed changes to TEMPLATE branch                                                                                                                                                          sync.py:245
INFO      Pushing TEMPLATE branch to remote: 'mhcquant'                                                                                                                                                 sync.py:255
INFO      Submitting a pull request via the GitHub API                                                                                                                                                  sync.py:285
WARNING   Could not list open PRs ('404')                                                                                                                                                               sync.py:363
          https://api.github.com/repos/nf-core/mhcquan/pulls?head=nf-core:TEMPLATE&base=dev
          {
              "message": "Not Found",
              "documentation_url": "https://developer.github.com/v3/pulls/#list-pull-requests"
          }
INFO      Checking out original branch: 'master'                                                                                                                                                        sync.py:406
ERROR     GitHub API returned code 404:                                                                                                                                                             __main__.py:577
          {
              "message": "Not Found",
              "documentation_url": "https://developer.github.com/v3/pulls/#create-a-pull-request"
          }

Code that builds this URL:
https://github.com/ewels/nf-core-tools/blob/866a591fbf50f71166439874c8f014a81dcd0f97/nf_core/sync.py#L312-L315

# Look for existing pull-requests
list_prs_url = "https://api.github.com/repos/{}/{}/pulls?head=nf-core:TEMPLATE&base={}".format(
    self.gh_username, self.gh_repo, self.from_branch
)

This should be impossible. What the hell is going on??

[ewels]

Ok wait, I think I might have found it:

Buried in the verbose log:

Found username and repo from remote: nf-core, mhcquan - git@github.com:nf-core/mhcquant.git

The code overwrites the git repo name from the checked out repo (bad) here - need to skip this if it's suppled (or just delete this chunk of code and make it required).

There must be something about the regex github\.com[:\/]([^\/]+)/([^\/]+)$ and this particular remote (git@github.com:nf-core/mhcquant.git) - but I can't see what.

[ewels]

Ok, I don't know what was wrong with the regex but it doesn't matter - I have stripped that from the code as we didn't need it anyway and now everything seems to work 🎉

[ewels]

Fixed in #716