Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix parent pipeline number env on restarts #3683

Merged
merged 4 commits into from
May 6, 2024

Conversation

qwerty287
Copy link
Contributor

Closes #3680

@qwerty287 qwerty287 added the bug Something isn't working label May 6, 2024
@qwerty287 qwerty287 added this to the 2.5.0 milestone May 6, 2024
@qwerty287 qwerty287 requested a review from a team May 6, 2024 06:06
@qwerty287
Copy link
Contributor Author

What can we do about this failure? https://ci.woodpecker-ci.org/repos/3780/pipeline/15704/27#L183

@xoxys
Copy link
Member

xoxys commented May 6, 2024

Not that simple. I had this issue quite often with Drone and Woodpecker while using the autoscaler. Summary from my observation:

  • Some US cloud platforms do heavy IP based geo/- spam blocking
  • With an autoscaler setup, you get a random IP from the Hetzner pool
  • If a former owner of the IP has used it for bad habit, they get tainted and blocked by some providers

Solutions:

  • Quick and dirty: Change the used Hetzer DC in the autoscaler deployment. The IP ranges seem to be different for the DCs, so temporarily switching to e.g. NUE, FSN or HEL worked for me in most cases.
  • Use IPv6 only. It is quite difficult to perform IP-based filtering with IPv6. However, every component in the Woodpecker stack must support this. With Drone this was not possible, not sure about Woodpecker.
  • Use an HTTP proxy server with a static IP and set HTTP_PROXY/HTTPS_PROXY env on the affected steps. However, most HTTP_PROXY servers cannot be used with authentication. Binding to a public IP allows anyone to use your proxy.
  • Create a wiregurad network and connect all autoscaler hosts to this network. Use a persistent host with a static IP as a wiregurad gateway and route all Internet traffic via this gateway.

@anbraten
Copy link
Member

anbraten commented May 6, 2024

Would caching using sth like the s3-cache help with this at least for the majority of executions? Otherwise as this problem seems to just occur rarely we are probably best to just throw away the agent and try again for now.

@xoxys
Copy link
Member

xoxys commented May 6, 2024

Would caching using sth like the s3-cache help with this at least for the majority of executions

In many cases, no. It is in the nature of a CI system to contain dynamic parts and, at least in my experience, it is quite difficult to make pipelines work completely offline.

Otherwise as this problem seems to just occur rarely we are probably best to just throw away the agent and try again for now.

Keep in mind that the same IP address will be assigned to your machine again if it has not already been assigned to someone else. Just re-creating the agent might not work, that's why I had to switch the DC to ensure to get a new IP from another pool.

@qwerty287
Copy link
Contributor Author

@anbraten Maybe you can just force-merge this for now?

@anbraten anbraten merged commit 8b785be into woodpecker-ci:main May 6, 2024
4 of 7 checks passed
@anbraten anbraten changed the title Fix parent pipeline env Fix parent pipeline number env on restarts May 6, 2024
@qwerty287 qwerty287 deleted the fix-parent branch May 6, 2024 09:48
@woodpecker-bot woodpecker-bot mentioned this pull request May 6, 2024
1 task
6543 pushed a commit to 6543-forks/woodpecker that referenced this pull request Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The value of CI_PIPELINE_PARENT is the ID of the parent pipeline
3 participants