proxy.golang.org: reports of failing requests in the southeastern US area

[heschi]

Edit: We believe the issue is resolved as of 8AM US Eastern time.

proxy.golang.org's monitoring, confirmed by widespread user reports, shows that Google is having an infrastructure failure that causes requests to proxy.golang.org to fail. Unfortunately there's nothing the Go team can directly do to fix this problem, but we are monitoring it and escalating internally.

[heschi]

As a workaround, users can set GOPROXY=direct to bypass the mirror. However, that won't help for modules that happen to be hosted on Google's infrastructure, e.g. golang.org/x modules.

[Raffo]

@heschi thank you for opening this issue and for reaching out internally to get it fixed. I have one question: would it be possible to have a status page up for proxy.golang.org for the community to check in case an outage is suspected? I spent quite some time figuring out what was wrong with some of our builds and a status page would have been handy.

[heschi]

@Raffo That's fair. We should definitely consider it. I believe this is the first really major outage we've had in the ~4 years since the proxy launched, so we don't have a lot of practice managing them. Thanks for bearing with us.

[or-shachar]

Is there a a status page for the Google infrastructure service that is malfunctioning ?

[OrBin]

Thanks for the report, just want to note that it's happening not only in southeastern US, but also in Europe and the Middle East (Israel).

[xymon2]

Asia(Korea) has same problem too.

[heschi]

@or-shachar not directly. If it affected a major Google product you might see the status page for those products updated, but it appears that that's not the case. proxy.golang.org's use case is different from most Google services and it's possible that a common outage is affecting us worse than others.

@OrBin @xymon2 That's quite surprising based on my current understanding of the issue. Thank you.

[heschi]

All: please do not post +1s for the eastern US. I appreciate new information, but we are well aware of the issues in the eastern US :)

[heschi]

The proxy.golang.org front page should start showing a message pointing to this issue soon. Unfortunately it's cached for 30 minutes, so it'll take a while to be visible everywhere, but it is better than nothing. Thanks again for bearing with us.

[cbowlby-bt]

I am seeing responses back from the proxy as of 2 minutes ago, most of my actions are working once again (for US and Canada)

[heschi]

I'm told a mitigation is in process and we may see an improvement in the next few minutes.

[heschi]

Our monitoring shows recovery as of a couple of minutes ago. DNS propagation delay may vary somewhat, but generally the issue appears to be resolved. Continuing to monitor.

[RayMeibergen]

@heschi do you have health dashboard page where we can check?

[skport]

I am using proxy.golang.org from CircleCI and the error was resolved just now.

[heschi]

@RayMeibergen if you're asking for a user-facing status page, we don't have one other than the note I added to proxy.golang.org; this issue is the authoritative communication channel. If you're asking if you can see our monitoring dashboards, I'm afraid they're private. Apologies.

[heschi]

Our monitoring has been clean for 20 minutes and I'm comfortable calling this issue resolved. If you're still seeing errors of the form Get "https://proxy.golang.org/...": EOF, please report them here. Thanks.

proxy.golang.org was essentially an innocent victim of the underlying outage, so a post mortem from our perspective will probably not be very exciting. But if there's interest we can consider writing something up.

Again, thanks to all for bearing with us.