I can't build main branch because of ssl issue with the restore #52201
Labels
area-Infrastructure
discussion
tracking-external-issue
The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly
Milestone
Comments
|
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
dotnet-issue-labeler
bot
added
the
untriaged
|
Tagging subscribers to this area: @dotnet/runtime-infrastructure Issue DetailsDescriptionI can't build main branch from this repo, as the restore stage fails with various SSL errors: I the error messages is not always the sames, so it does look like a network issue, but could conceivably be something I'm doing wrong too. Configuration
Windows 10 -10.0.19042 Build 19042
x64 Regression?This is the first time I've attempted to build on this laptop< Other informationI noticed it looks similar to this issue: #52053 , but I don't think it's the same issue as the download of the initial sdk seems to work okay.
|
|
Wasn't that linux only @hoyosjs ? |
|
#51640 was with package signing, this appears to be happen earlier than that, when Nuget is connecting to AzDo feed.
Download of the initial SDK doesn't happen through latest .NET runtime, it uses PowerShell which will use .NETFramework in OS. It could be the same issue. cc @dotnet/ncl |
|
Indeed. My bad, I somehow read the message and missed the path completely. This seems like a rude closed connection. The only thing that I could find around this is https://developercommunity.visualstudio.com/t/deployment-failing-in-release-pipeline-1/932190 and microsoft/azure-pipelines-tasks#12444 (comment). Not sure if it applies as I don't see a TLS version warning on yours, but this is easy to check. |
|
I checked that I'm using a recent version of powershell, it seems it meets the minimum specified: I also entered the following values into my registry: This didn't seem to make any difference. If I try to manually download the url causing an issue, it seems to work fine: |
|
I decided to check with fiddle which URLs were failing, but this seemed to cause the error message to change, to variations of the following: I noticed that while some failures were for .json index files, others where for actual nuget packages, for example: I installed this packages locally via: Once the following packages were installed locally, the build seems to get a little further: However the build still fails with this error: Examining the http requests with fiddler, I see the download of this package fails repeatedly: Since I have no way to install |
|
How many SDKs do you currently have installed? In case you have old 2.x or 3.x ones installed, you could try deleting those. Also we are updating the SDK to Preview 3 later today, that might also help you to get further. |
|
I have quite a few old sdks installed: But I'd be reluctant to remove them, since I need to be able to test with a wide range of frameworks. I don't really see how it would help, since fiddle seems to show it's very recent version of nuget that are making the failing requests. I just tried building the v5.0.5 tag and that fails with similar issues, which makes me think the problem related to nuget servers supporting this build, although if that's the case, I don't really understand why other people aren't complaining about it. |
|
I recommended to delete these old SDKs as IIRC @tmds it a similar issues and deleting old SDKs helped him. Maybe you could give it a try and afterwards reinstall the old ones? |
|
I removed all sdks apart from When I try to build tag v5.0.5, I get this error: If I try and build the latest main I get: |
This indicates that the nuget sdk resolver silently failed when restoring the SDK package. cc @nkolev92
Hmm that's unfortunate. As a last resort, could you please try to build again from a clean repo (git clean -xdf) with the following change:
Thanks a lot |
|
I tried to clean build (git clean -xdf) tag v5.0.5 with the modifications suggested, but still get similar error messages: |
|
Thanks for trying the suggested workarounds out. In this case I don't know how to help further. Let me ping the @dotnet/nuget-team who are hopefully able to help. |
|
So it turns out the issue is my wifi router or ISP. After a brief automated reboot of my wifi router made me think, what if the problem is the network, but my network infrastructure, so I switched by laptop to use my phones 4G connection and the restore phase worked just fine (if a little slower than I might have hoped). The actual build phase is running now, so all good. I'd be very interested into digging into why the ssl connections to |
|
@johnterickson for the AzDO packaging related question. Maybe you can help @robertpi diagnose this further? |
|
From NuGet's perspective, we use Given the error message "An existing connection was forcibly closed by the remote host", it sounds to me like the operating system was seeing a TCP packet to close the TCP connection, while the SSL stream was expecting it to remain open. WireShark might help you see a TCP FIN packet, if that's really the case. Whether that originated from your router or your ISP or a pkgs.dev.azure.com CDN endpoint, I'm not sure if it's possible to determine. This is very much outside my expertise. All I can say as a NuGet expert is that the .NET HttpClient didn't give us a successful connection to do transfers over. |
|
I know it may be hard to know where this is coming from but the Aside from firewalls and real network failures, it may be possible that one provider uses IPv6 and the other does not. Also with global load ballancing using different ISP may connect you to different region. Wireshark packet capture would show either one. |
|
What doesn't make sense to me is that it works with PowerShell but not with NuGet. We're not doing any User-Agent trickery or anything like that 🤷♂️ |
ViktorHofer
added
discussion
tracking-external-issue
|
@johnterickson I later found out that |
|
Looks like there is not much we can do here ... this indeed looks like external problem, beyond the local machine (given it fails as well with curl, PS, etc.). Should we close it or is there anything left here to troubleshoot or help with? |
|
My suspicion is this is global load balancing sending me to a dodgy set of nuget servers, which is one of the things @wfurt mentioned, in that case there would be some infrastructure issues to correct. But it could be lots of other things too, so feel free to close if there's nothing that can be done on this front. |
|
Thanks @robertpi. Closing. |
ghost
locked as resolved and limited conversation to collaborators
|
@karelz I have started to get the same problem on all my local physical devices (macOS, Linux, Windows). And after cleaning the nuget cache on one of the devices, I cannot build anything from the runtime repo anymore. This is what happens: Using browser / wget on most of these URLs works fine, so I assume it is some issue in the HttpClient w.r.t. my internet provider infra or something like that. Close before the issue started to happen, I got my internet modem replaced. |
|
Can you check what site it resolves to @janvorli ? This seems like some CDN. Also HttpClient would prefer IPv6 if available. In this case the name does not seems to resolve to IPv6 (at least for me) so it should not problem but it something to remember. |
|
The same as yours: |
|
I have also tried to create a simple test app like this and compile it with the latest .NET 6. All of the URLs above return HTTP error 404, which is kind of strange by itself, but none complains about no route to host. And the errors above are 100% repeatable. using System;
namespace nuget
{
class Program
{
static void Main(string[] args)
{
var task = new System.Net.Http.HttpClient().GetAsync(args[0]);
task.Wait();
task.Result.EnsureSuccessStatusCode();
}
}
}So I wonder if it is possible that the nuget version used by the build uses some old .NET runtime that had some issue that is resolved in the latest. |
|
Btw, opening these URLs return 404 even in Chrome on Windows, the contents of the response is similar for all of them, one example being: |
|
Btw, in the Wireshark log, I can see quite a number of "TCP Retransmission", "TCP Spurious Retransmission" and "TCP Dup ACK 508680". I wonder if it could be related in some way and our HttpClient hickups on these. I can also see many logs as the following ones that Wireshark marks with red color, so I assume there is something wrong. Maybe the Win/Len being zero? But my knowledge of the TCP protocol has faded over the years, so I am not sure. |
|
And also this one: |
|
The 13.107.43.20 is the server ... or networking gear in the middle. All the TCP is handle by the Linux kernel. What matters is stream of data we read from the socket. |
|
BTW there also seems to be some general outage https://github.com/dotnet/core-eng/issues/13691 |
|
It seems it is even more involved. Rerunning the build of clr+libs multiple times ended up passing, so it seems that the problem is really intermittent. |
|
It is possible that your ISP has some transparent caches and once they cannot fetch the file they give you different error. With CDN in place networking is way more complicated. In your case this looks like some packet loss at least. |
|
I've just learned that the 404 error codes are benign. Nuget tries to fetch packages from all the configured sources and the 404 just means that the intermittent issue happened when trying to fetch it from other sources than the one it was in. |
|
In the intermittent cases, there are no HTTP response codes, as the communication gets broken in various ways. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
I can't build main branch from this repo, as the restore stage fails with various SSL errors:
I the error messages is not always the sames, so it does look like a network issue, but could conceivably be something I'm doing wrong too.
Configuration
Windows 10 -10.0.19042 Build 19042
x64
Regression?
This is the first time I've attempted to build on this laptop<
Other information
I noticed it looks similar to this issue: #52053 , but I don't think it's the same issue as the download of the initial sdk seems to work okay.
The text was updated successfully, but these errors were encountered: