Allow runner to check service connection in background. #3542
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TingluoHuang
commented
Nov 7, 2024
| namespace GitHub.DistributedTask.WebApi | ||
| { | ||
| [DataContract] | ||
| public class ServiceConnectivityCheckInput |
There was a problem hiding this comment.
i will copy the contract to the service side.
TingluoHuang
commented
Nov 7, 2024
src/Runner.Worker/JobExtension.cs
Outdated
Comment on lines
833
to
838
| var foo = new ServiceConnectivityCheckInput() { IntervalInSecond = 30, RequestTimeoutInSecond = 30 }; | ||
| foo.Endpoints.Add("pipelines", "https://pipelinesghubeus20.actions.githubusercontent.com/_apis/connectiondata"); | ||
| foo.Endpoints.Add("broker", "https://broker.actions.githubusercontent.com/health"); | ||
| foo.Endpoints.Add("results", "https://results-receiver.actions.githubusercontent.com/health"); | ||
| var fooJson = StringUtil.ConvertToJson(foo); | ||
| context.Global.Variables.Set(WellKnownDistributedTaskVariables.RunnerServiceConnectivityTest, fooJson); |
There was a problem hiding this comment.
this is an example input, i will remove this before merge the PR.
TingluoHuang
commented
Nov 7, 2024
| @@ -717,33 +739,58 @@ public async Task FinalizeJob(IExecutionContext jobContext, Pipelines.AgentJobRe | |||
| } | |||
| } | |||
|
|
|||
| private async Task<string> CheckConnectivity(string endpointUrl) | |||
| private async Task<CheckResult> CheckConnectivity(string endpointUrl, string accessToken, int timeoutInSeconds, CancellationToken token) | |||
There was a problem hiding this comment.
I make this more generic so i can reuse between the current 1 time connection check and the backend long-running connection check.
TingluoHuang
commented
Nov 7, 2024
| using (var timeoutTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(5))) | ||
| CheckResult result = new CheckResult() { EndpointUrl = endpointUrl }; | ||
| var stopwatch = Stopwatch.StartNew(); | ||
| using (var timeoutTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(timeoutInSeconds))) |
There was a problem hiding this comment.
the existing use case will pass 5 as method input.
TingluoHuang
commented
Nov 7, 2024
| CheckResult result = new CheckResult() { EndpointUrl = endpointUrl }; | ||
| var stopwatch = Stopwatch.StartNew(); | ||
| using (var timeoutTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(timeoutInSeconds))) | ||
| using (var linkedTokenSource = CancellationTokenSource.CreateLinkedTokenSource(token, timeoutTokenSource.Token)) |
There was a problem hiding this comment.
the existing use code will pass CancellationToken.None
TingluoHuang
commented
Nov 7, 2024
| { | ||
| try | ||
| { | ||
| using (var httpClientHandler = HostContext.CreateHttpClientHandler()) | ||
| using (var httpClient = new HttpClient(httpClientHandler)) | ||
| { | ||
| httpClient.DefaultRequestHeaders.UserAgent.AddRange(HostContext.UserAgents); | ||
| var response = await httpClient.GetAsync(endpointUrl, timeoutTokenSource.Token); | ||
| result = $"{endpointUrl}: {response.StatusCode}"; | ||
| if (!string.IsNullOrEmpty(accessToken)) |
There was a problem hiding this comment.
existing code will pass empty string.
TingluoHuang
commented
Nov 7, 2024
| catch (Exception ex) when (ex is OperationCanceledException && timeoutTokenSource.IsCancellationRequested) | ||
| { | ||
| Trace.Error($"Request timeout during connectivity check: {ex}"); | ||
| result = $"{endpointUrl}: timeout"; |
There was a problem hiding this comment.
we will reconstruct the same result from the caller.
TingluoHuang
commented
Nov 7, 2024
| context.Global.Variables.Set(WellKnownDistributedTaskVariables.RunnerServiceConnectivityTest, fooJson); | ||
|
|
||
| var connectionTest = context.Global.Variables.Get(WellKnownDistributedTaskVariables.RunnerServiceConnectivityTest); | ||
| if (string.IsNullOrEmpty(connectionTest)) |
There was a problem hiding this comment.
this is the featureflag to control the test.
TingluoHuang
commented
Nov 7, 2024
| } | ||
| } | ||
|
|
||
| var telemetryData = StringUtil.ConvertToJson(testResult, Formatting.None); |
There was a problem hiding this comment.
don't add indent to save space.
TingluoHuang
force-pushed
the
users/tihuang/connectioncheck
branch
from
53a41fb
to
896f513
Compare
rentziass
previously approved these changes
Nov 7, 2024
TingluoHuang
force-pushed
the
users/tihuang/connectioncheck
branch
from
f4385f5
to
f931e1a
Compare
TingluoHuang
force-pushed
the
users/tihuang/connectioncheck
branch
from
0a55fc0
to
56372d2
Compare
rentziass
approved these changes
Nov 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The server can request runner to do connectivity check as need to debug network issue between the runner and the service.