Remove enormous array allocations in tests causing instability in CI #1367
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I have opened a ticket for lack of ideas https://help.appveyor.com/discussions/problems/35588-net-tests-silently-exiting-on-ubuntu2204 |
string token = ""; // https://ci.appveyor.com/api-keys
var client = new HttpClient();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
JsonDocument x = JsonDocument.Parse(await client.GetStringAsync("https://ci.appveyor.com/api/projects/drieseng/ssh-net/history?recordsNumber=200"));
int buildCount = 0;
int badCount = 0;
foreach (JsonElement build in x.RootElement.GetProperty("builds").EnumerateArray())
{
buildCount++;
long buildId = build.GetProperty("buildId").GetInt64();
string status = build.GetProperty("status").GetString();
if (status != "success")
{
continue;
}
foreach (var job in JsonDocument.Parse(
await client.GetStringAsync($"https://ci.appveyor.com/api/projects/drieseng/ssh-net/builds/{buildId}"))
.RootElement.GetProperty("build").GetProperty("jobs").EnumerateArray())
{
string jobId = job.GetProperty("jobId").GetString();
int testsCount = job.GetProperty("testsCount").GetInt32();
if (testsCount < 1000)
{
badCount++;
Console.WriteLine($"Bad: {testsCount} tests in https://ci.appveyor.com/project/drieseng/ssh-net/builds/{buildId}");
Console.WriteLine(string.Join(Environment.NewLine,
ReadLines(await client.GetStreamAsync($"https://ci.appveyor.com/api/buildjobs/{jobId}/log"))
.TakeLast(5)));
Console.WriteLine();
Console.WriteLine();
}
}
}
Console.WriteLine($"{badCount} bad of {buildCount}");
static IEnumerable<string> ReadLines(Stream s)
{
using StreamReader sr = new(s);
string line;
while ((line = sr.ReadLine()) != null)
{
yield return line;
}
} |
|
Memory explodes: |
Some of which are causing giant array allocations unnecessarily. Should stabilise CI.
Since these changes, I have run the CI 13 times successfully without any silent crashes at the ~840 test mark (the latest build failed one of the usual integration tests afterwards). At a 21 / 100 prior occurrence rate, the chance of 13 passes would be (1 - 0.21) ^ 13 ~= 4.66%. So I think we can say that this is the fix and that the process was crashing due to out-of-memory from these randomly-sized array allocations. Still a bit of a mystery why the dotnet process was showing exit code 0 though |
Rob-Hague
changed the title
scott-xu
approved these changes
Apr 6, 2024
WojciechNagorski
approved these changes
Apr 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.