Problem to download large files (hash256 and close connection remote host)

[josediazramon]

Hello everyone,

In recent days, I have been trying to solve a problem, but I don't have any idea about what is causing it. I have an application to download all files from a SharePoint library, and for large files, I use the following code (https://pnp.github.io/pnpcore/using-the-sdk/files-large.html):

// Start the download
Stream downloadedContentStream = await fileToDownload.GetContentAsync(true);

// Obtain the hash before downloading the file
StringBuilder hashFileBefore = ObtainHash256Stream(downloadedContentStream);

// Download the file bytes in 2MB chunks and immediately write them to a file on disk
// This approach avoids loading the entire file into process memory
var bufferSize = 2 * 1024 * 1024; // 2 MB buffer
using (var content = System.IO.File.Create(path))
{
var buffer = new byte[bufferSize];
int read;
while ((read = await downloadedContentStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
content.Write(buffer, 0, read);
}
}

static StringBuilder ObtainHash256Stream(Stream downloadedContentStream)
{
SHA256 sha256 = SHA256.Create();
byte[] hash = sha256.ComputeHash(downloadedContentStream);

StringBuilder sBuilder = new StringBuilder();
for (int i = 0; i < hash.Length; i++)
{
    sBuilder.Append(hash[i].ToString("x2"));
}

return sBuilder;
}

I have two problems. First, if I obtain the hash and then try to download the file, the Stream doesn't have the internal property "_connection," and there is an error because I can't download the file to the specific path. Is there a way to download the file's content, obtain the hash, and save it to C:\ without downloading the same file twice?

The second problem I have, is that sometimes when I try to download large files from a SharePoint Online site, I get this error: "Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host."

In my program.cs, I have a timeout set to 6,000 seconds:

.ConfigureServices((context, services) =>
{
    // Add the PnP Core SDK library
    services.AddPnPCore(options =>
    {
        options.PnPContext.GraphFirst = true;
        options.HttpRequests.UserAgent = "GetDocumentosSitio/1.0";
        options.HttpRequests.Timeout = 6000;

        options.Sites.Add("SiteURL", new PnPCoreSiteOptions
        {
            SiteUrl = importacion.siteUrl
        });
    });
});

In my class, I have the following:

HttpClient client = new HttpClient(new ThrottlingHandler(useRateLimiter)
{
    InnerHandler = new HttpClientHandler()
});

client.Timeout = TimeSpan.FromMinutes(100);

Can anyone help me? Thank you very much.

[jansenbe]

@josediazramon : did you consider requesting the hash from SharePoint (https://learn.microsoft.com/en-us/graph/api/resources/driveitem?view=graph-rest-1.0) using the file facet which contains a collection of hashes (https://learn.microsoft.com/en-us/graph/api/resources/hashes?view=graph-rest-1.0).

Regarding the second ask: you're not using the custom created HttpClient to download the file, that's done using the standard approach in the SDK? Given this is an intermittent problem I'm suspecting this to be service issue, does the error happen around the set timeouts?