Strange null reference exception in Ether code

[knocte]

CI of today started failing with:

https://github.com/nblockchain/geewallet/actions/runs/9815030138

Quoting error:

Unhandled exception. System.AggregateException: One or more errors occurred. (Some problem when connecting to 'besu-at.etc-network.info')
 ---> System.Exception: Some problem when connecting to 'besu-at.etc-network.info'
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at GWallet.Backend.ServerManager.web3Func@131-54.Invoke(HexBigInteger _arg2) in /Users/runner/work/geewallet/geewallet/src/GWallet.Backend/ServerManager.fs:line 131
   at Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvokeNoHijackCheck[a,b](AsyncActivation`1 ctxt, b result1, FSharpFunc`2 userCode) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 528
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112
--- End of stack trace from previous location ---
   at Fsdk.FSharpUtil.ReRaise(Exception ex)
   at GWallet.Backend.Ether.Server.HandlePossibleEtherFailures@418-6.Invoke(Exception _arg2) in /Users/runner/work/geewallet/geewallet/src/GWallet.Backend/Ether/EtherServer.fs:line 425
   at GWallet.Backend.Ether.Server.HandlePossibleEtherFailures@418-8.Invoke(Exception exn)
   at GWallet.Backend.Ether.Server.HandlePossibleEtherFailures@418-10.Invoke(Exception edi)
   at Microsoft.FSharp.Control.AsyncPrimitives.CallFilterThenInvoke[T](AsyncActivation`1 ctxt, FSharpFunc`2 filterFunction, ExceptionDispatchInfo edi) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 547
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112
   --- End of inner exception stack trace ---
   at GWallet.Backend.Ether.Server.Web3ServerToRetrievalFunc@437-2.Invoke(Exception _arg3) in /Users/runner/work/geewallet/geewallet/src/GWallet.Backend/Ether/EtherServer.fs:line 446
   at GWallet.Backend.Ether.Server.Web3ServerToRetrievalFunc@437-5.Invoke(Exception exn)
   at GWallet.Backend.Ether.Server.Web3ServerToRetrievalFunc@437-7.Invoke(Exception edi)
   at Microsoft.FSharp.Control.AsyncPrimitives.CallFilterThenInvoke[T](AsyncActivation`1 ctxt, FSharpFunc`2 filterFunction, ExceptionDispatchInfo edi) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 547
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112
--- End of stack trace from previous location ---
   at Fsdk.FSharpUtil.ReRaise(Exception ex)
   at <StartupCode$GWallet-Backend>.$FaultTolerantParallelClient.Run@158-8.Invoke(Exception _arg2) in /Users/runner/work/geewallet/geewallet/src/GWallet.Backend/FaultTolerantParallelClient.fs:line 187
   at <StartupCode$GWallet-Backend>.$FaultTolerantParallelClient.Run@158-15.Invoke(Exception exn)
   at <StartupCode$GWallet-Backend>.$FaultTolerantParallelClient.Run@158-17.Invoke(Exception edi)
   at Microsoft.FSharp.Control.AsyncPrimitives.CallFilterThenInvoke[T](AsyncActivation`1 ctxt, FSharpFunc`2 filterFunction, ExceptionDispatchInfo edi) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 547
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at <StartupCode$GWallet-Backend>.$FaultTolerantParallelClient.clo@334-26.Invoke(ServerTask`2 _arg1) in /Users/runner/work/geewallet/geewallet/src/GWallet.Backend/FaultTolerantParallelClient.fs:line 338
   at Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvokeNoHijackCheck[a,b](AsyncActivation`1 ctxt, b result1, FSharpFunc`2 userCode) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 528
   at <StartupCode$GWallet-Backend>.$FaultTolerantParallelClient.WhenAny@140-2.Invoke(AsyncActivation`1 ctxt)
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112
--- End of stack trace from previous location ---
   at Microsoft.FSharp.Control.AsyncResult`1.Commit() in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 454
   at Microsoft.FSharp.Control.AsyncPrimitives.QueueAsyncAndWaitForResultSynchronously[a](CancellationToken token, FSharpAsync`1 computation, FSharpOption`1 timeout) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 1139
   at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronously[T](CancellationToken cancellationToken, FSharpAsync`1 computation, FSharpOption`1 timeout) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 1166
   at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync`1 computation, FSharpOption`1 timeout, FSharpOption`1 cancellationToken) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 1515
   at GWallet.Backend.ServerManager.UpdateServersStats() in /Users/runner/work/geewallet/geewallet/src/GWallet.Backend/ServerManager.fs:line 172
   at GWallet.Frontend.Console.Program.UpdateServersStats() in /Users/runner/work/geewallet/geewallet/src/GWallet.Frontend.Console/Program.fs:line 522
   at GWallet.Frontend.Console.Program.Main(String[] argv) in /Users/runner/work/geewallet/geewallet/src/GWallet.Frontend.Console/Program.fs:line 537
   at GWallet.Frontend.ConsoleApp.Program.main(String[] argv) in /Users/runner/work/geewallet/geewallet/src/GWallet.Frontend.ConsoleApp/Program.fs:line 13

When running geewallet locally, I got the same problem but with slightly different stacktrace:

---> System.Exception: Weird null response from balance job
   at GWallet.Backend.Ether.Server.web3Func@563-9.Invoke(HexBigInteger _arg2) in /Users/knocte/Documents/Code/geewalletMASTERclean/src/GWallet.Backend/Ether/EtherServer.fs:line 564
   at Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvokeNoHijackCheck[a,b](AsyncActivation`1 ctxt, b result1, FSharpFunc`2 userCode) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 528
   at GWallet.Backend.Ether.Server.GetConfirmedEtherBalanceInternal@533-4.Invoke(AsyncActivation`1 ctxt)
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112
--- End of stack trace from previous location ---
   at Fsdk.FSharpUtil.ReRaise(Exception ex)
   at GWallet.Backend.Ether.Server.HandlePossibleEtherFailures@418-6.Invoke(Exception _arg2) in /Users/knocte/Documents/Code/geewalletMASTERclean/src/GWallet.Backend/Ether/EtherServer.fs:line 425
   at GWallet.Backend.Ether.Server.HandlePossibleEtherFailures@418-8.Invoke(Exception exn)
   at GWallet.Backend.Ether.Server.HandlePossibleEtherFailures@418-10.Invoke(Exception edi)
   at Microsoft.FSharp.Control.AsyncPrimitives.CallFilterThenInvoke[T](AsyncActivation`1 ctxt, FSharpFunc`2 filterFunction, ExceptionDispatchInfo edi) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 547
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112
   --- End of inner exception stack trace ---
   at GWallet.Backend.Ether.Server.Web3ServerToRetrievalFunc@437-2.Invoke(Exception _arg3) in /Users/knocte/Documents/Code/geewalletMASTERclean/src/GWallet.Backend/Ether/EtherServer.fs:line 446
   at GWallet.Backend.Ether.Server.Web3ServerToRetrievalFunc@437-5.Invoke(Exception exn)
   at GWallet.Backend.Ether.Server.Web3ServerToRetrievalFunc@437-7.Invoke(Exception edi)
   at Microsoft.FSharp.Control.AsyncPrimitives.CallFilterThenInvoke[T](AsyncActivation`1 ctxt, FSharpFunc`2 filterFunction, ExceptionDispatchInfo edi) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 547
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112
System.Exception: Some problem when connecting to 'besu-at.etc-network.info'
 ---> System.Exception: Weird null response from balance job
   at GWallet.Backend.Ether.Server.web3Func@563-9.Invoke(HexBigInteger _arg2) in /Users/knocte/Documents/Code/geewalletMASTERclean/src/GWallet.Backend/Ether/EtherServer.fs:line 564
   at Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvokeNoHijackCheck[a,b](AsyncActivation`1 ctxt, b result1, FSharpFunc`2 userCode) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 528
   at GWallet.Backend.Ether.Server.GetConfirmedEtherBalanceInternal@533-4.Invoke(AsyncActivation`1 ctxt)
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112
--- End of stack trace from previous location ---
   at Fsdk.FSharpUtil.ReRaise(Exception ex)
   at GWallet.Backend.Ether.Server.HandlePossibleEtherFailures@418-6.Invoke(Exception _arg2) in /Users/knocte/Documents/Code/geewalletMASTERclean/src/GWallet.Backend/Ether/EtherServer.fs:line 425
   at GWallet.Backend.Ether.Server.HandlePossibleEtherFailures@418-8.Invoke(Exception exn)
   at GWallet.Backend.Ether.Server.HandlePossibleEtherFailures@418-10.Invoke(Exception edi)
   at Microsoft.FSharp.Control.AsyncPrimitives.CallFilterThenInvoke[T](AsyncActivation`1 ctxt, FSharpFunc`2 filterFunction, ExceptionDispatchInfo edi) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 547
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112
   --- End of inner exception stack trace ---
   at GWallet.Backend.Ether.Server.Web3ServerToRetrievalFunc@437-2.Invoke(Exception _arg3) in /Users/knocte/Documents/Code/geewalletMASTERclean/src/GWallet.Backend/Ether/EtherServer.fs:line 446
   at GWallet.Backend.Ether.Server.Web3ServerToRetrievalFunc@437-5.Invoke(Exception exn)
   at GWallet.Backend.Ether.Server.Web3ServerToRetrievalFunc@437-7.Invoke(Exception edi)
   at Microsoft.FSharp.Control.AsyncPrimitives.CallFilterThenInvoke[T](AsyncActivation`1 ctxt, FSharpFunc`2 filterFunction, ExceptionDispatchInfo edi) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 547
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112

Both problems seem to be caused by a null value being returned from web3.Eth.GetBalance.SendRequestAsync() method.

[knocte]

I've digged a little bit in our Nethereum fork (https://github.com/nblockchain/Nethereum) and so far I think I found that the culprit is actually in JsonRpcSharp, in particular here:

https://github.com/nblockchain/JsonRpcSharp/blob/d6738480b27802efffeddcaa44b4e6a1e289124a/src/JsonRpcSharp.Client/RpcMessages/RpcMessages.cs#L53

Seems that in order to understand the problem better, we would need to first change returnDefaultIfNull param to default to false, and test again to see if we get the exception "Unable to convert the result (null) to type ...". If that is the case, then it means that somehow response.Result property was null, and so the next step is to figure out why.

[webwarrior-ws]

Indeed response.Result is null in our case, although returnDefaultIfNull does nothing in this case since default of Nethereum.Hex.HexTypes.HexBigInteger is null also.

Enabling debug output showed that response json was:

{"jsonrpc":"2.0","id":1,"result":null}

In other responses, result field contains string with number in hex notation.

[webwarrior-ws]

So I think this should be handled on geewallet side. But I don't know which exception type to use here to signal invalid response from serever.