The JSON value could not be converted to k8s.Models.V1ServiceStatus

[arkadiya-msft]

Describe the bug
When trying to start a debugging session in VS Code, I see following error in B2K VS Code output:

Failed to establish a connection. Error: Connect operation failed.
An unexpected error occurred: 'The JSON value could not be converted to k8s.Models.V1ServiceStatus. Path: $.status | LineNumber: 0 | BytePositionInLine: 67.'

If I try again, it works as expected.

Mention the platform you are using
VS Code v1.72.2
Extension v1.0.120221031
MacOS 13.0

To Reproduce
Steps to reproduce the behavior:

1. Run and Debug
2. Click on Start Debugging for a predefined launch configuration for B2K
3. Put in your credentials when prompted
4. See error

Expected behavior
Debugging works the first time, without need to try again after first attempt fails.

Additional context

Stacktrace from MindaroCli logs:

ServiceConnectCommand.ExecuteInnerAsync caught exception System.Text.Json.JsonException: The JSON value could not be converted to k8s.Models.V1ServiceStatus. Path: $.status | LineNumber: 0 | BytePositionInLine: 67.\n at System.Text.Json.ThrowHelper.ThrowJsonException_DeserializeUnableToConvertValue(Type propertyType)\n at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)\n at System.Text.Json.Serialization.JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)\n at System.Text.Json.Serialization.Metadata.JsonPropertyInfo1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)\n at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)\n at System.Text.Json.Serialization.JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)\n at System.Text.Json.Serialization.JsonConverter1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)\n at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)\n at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)\n at System.Text.Json.JsonSerializer.ContinueDeserialize[TValue](ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, JsonConverter converter, JsonSerializerOptions options)\n at System.Text.Json.JsonSerializer.ReadAll[TValue](Stream utf8Json, JsonTypeInfo jsonTypeInfo)\n at System.Text.Json.JsonSerializer.ReadAllUsingOptions[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options)\n at System.Text.Json.JsonSerializer.Deserialize[TValue](Stream utf8Json, JsonSerializerOptions options)\n at k8s.KubernetesJson.Deserialize[TValue](Stream json)\n at k8s.Kubernetes.CreateResultAsync[T](HttpRequestMessage httpRequest, HttpResponseMessage httpResponse, Nullable1 watch, CancellationToken cancellationToken)\n at k8s.AbstractKubernetes.DeleteNamespacedServiceWithHttpMessagesAsync(String name, String namespaceParameter, V1DeleteOptions body, String dryRun, Nullable1 gracePeriodSeconds, Nullable1 orphanDependents, String propagationPolicy, Nullable1 pretty, IReadOnlyDictionary2 customHeaders, CancellationToken cancellationToken)\n at k8s.KubernetesExtensions.DeleteNamespacedServiceAsync(IBasicKubernetes operations, String name, String namespaceParameter, V1DeleteOptions body, String dryRun, Nullable1 gracePeriodSeconds, Nullable1 orphanDependents, String propagationPolicy, Nullable1 pretty, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.<>c__DisplayClass39_0.<<CreateOrReplaceV1ServiceAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.<>c__DisplayClass75_01.<b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Common.Utilities.WebUtilities.RetryUntilTimeAsync(Func2 action, TimeSpan maxWaitTime, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.ClientInvokeWrapperAsync[T](Func1 handler, String operation, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.CreateOrReplaceV1ServiceAsync(String namespaceName, V1Service service, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.RoutingManagementClient.<>c__DisplayClass11_0.<b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.RoutingManagementClient.DeployRoutingManagerAsync(CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteInnerAsync(IConnectManagementClient connectManagementClient, Action workloadStartedHandler, CancellationToken cancellationToken, IRoutingManagementClient routingManagementClient)

and

Logging handled exception: System.Text.Json.JsonException: {"ClassName":"System.Text.Json.JsonException","Message":"The JSON value could not be converted to k8s.Models.V1ServiceStatus.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":" at System.Text.Json.ThrowHelper.ThrowJsonException_DeserializeUnableToConvertValue(Type propertyType)\n at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)\n at System.Text.Json.Serialization.JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)\n at System.Text.Json.Serialization.Metadata.JsonPropertyInfo1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)\n at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)\n at System.Text.Json.Serialization.JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)\n at System.Text.Json.Serialization.JsonConverter1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)\n at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)\n at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)\n at System.Text.Json.JsonSerializer.ContinueDeserialize[TValue](ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, JsonConverter converter, JsonSerializerOptions options)\n at System.Text.Json.JsonSerializer.ReadAll[TValue](Stream utf8Json, JsonTypeInfo jsonTypeInfo)\n at System.Text.Json.JsonSerializer.ReadAllUsingOptions[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options)\n at System.Text.Json.JsonSerializer.Deserialize[TValue](Stream utf8Json, JsonSerializerOptions options)\n at k8s.KubernetesJson.Deserialize[TValue](Stream json)\n at k8s.Kubernetes.CreateResultAsync[T](HttpRequestMessage httpRequest, HttpResponseMessage httpResponse, Nullable1 watch, CancellationToken cancellationToken)\n at k8s.AbstractKubernetes.DeleteNamespacedServiceWithHttpMessagesAsync(String name, String namespaceParameter, V1DeleteOptions body, String dryRun, Nullable1 gracePeriodSeconds, Nullable1 orphanDependents, String propagationPolicy, Nullable1 pretty, IReadOnlyDictionary2 customHeaders, CancellationToken cancellationToken)\n at k8s.KubernetesExtensions.DeleteNamespacedServiceAsync(IBasicKubernetes operations, String name, String namespaceParameter, V1DeleteOptions body, String dryRun, Nullable1 gracePeriodSeconds, Nullable1 orphanDependents, String propagationPolicy, Nullable1 pretty, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.<>c__DisplayClass39_0.<<CreateOrReplaceV1ServiceAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.<>c__DisplayClass75_01.<b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Common.Utilities.WebUtilities.RetryUntilTimeAsync(Func2 action, TimeSpan maxWaitTime, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.ClientInvokeWrapperAsync[T](Func1 handler, String operation, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.CreateOrReplaceV1ServiceAsync(String namespaceName, V1Service service, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.RoutingManagementClient.<>c__DisplayClass11_0.<b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.RoutingManagementClient.DeployRoutingManagerAsync(CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteInnerAsync(IConnectManagementClient connectManagementClient, Action workloadStartedHandler, CancellationToken cancellationToken, IRoutingManagementClient routingManagementClient)\n at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteInnerAsync(IConnectManagementClient connectManagementClient, Action workloadStartedHandler, CancellationToken cancellationToken, IRoutingManagementClient routingManagementClient)\n at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteAsync()\n at Microsoft.BridgeToKubernetes.Exe.CliApp.RunCommandAsync(String[] args, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Exe.CliApp.ExecuteAsync(String[] args, CancellationToken cancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233088,"Source":"System.Text.Json","WatsonBuckets":null,"LineNumber":0,"BytePositionInLine":67,"Path":"$.status","ActualMessage":"The JSON value could not be converted to k8s.Models.V1ServiceStatus. Path: $.status | LineNumber: 0 | BytePositionInLine: 67."}
2022-11-06T08:33:35.4283620Z | MindaroCli | TRACE | Event: Command.End {"properties":{"arguments":"connect --service controlplane-external --env /var/folders/19/mk67cjtd32gbvlqdv0k_t_v40000gn/t/tmp-63295avqm1lro1o3p.env --script /var/folders/19/mk67cjtd32gbvlqdv0k_t_v40000gn/t/tmp-63295avqm1lro1o3p.env.cmd --control-port 51747 --ppid 9376 --namespace dev --elevation-requests [{"requesttype":"edithostsfile"}] --routing arkadiya --local-port 5100","result":"Failed","failureReason":"The JSON value could not be converted to k8s.Models.V1ServiceStatus. Path: $.status | LineNumber: 0 | BytePositionInLine: 67."},"metrics":{"duration":58411.0}}

[elenavillamil]

Hi, yes this issue is tracked by #55. We believe this happens when application starts before routing manager is started on the cluster (race condition). Retry helps because routing manager is only deployed once, so next attempt routing manager is already there.

[hsubramanianaks]

@elenavillamil FYI - seems like KubernetesClient has an issue open where SDK 7.0 it is v1.service instead of v1.Status, Production version of Bridge uses version 7.12 and I guess that's why this is causing some issues.

Issue - kubernetes-client/csharp#824
another recent issue - kubernetes-client/csharp#1154

Upgrading to latest K8Client for csharp would resolve the issue I guess, already we have updated them to 8+. with dotnet 6 upgrade PR.

[elenavillamil]

(Edit) I was going to close this, but from metrics I still see this error happening even after the dotnet6 upgrade. We will need to verify version 8+ of kubernetes csharp api is used.

[elenavillamil]

Hi everyone, fix for this got deployed today. Please feel free to re-open if you still the issue after upgrading to latest version. I will also keep an eye at our internal metrics.