Retry GetExecBatchResults on ApiExceptions caused by GSB Errors

[azawlocki]

Fixes #587

Implements the following behaviour for handling ApiExceptions
raised by get_exec_batch_results():

call get_exec_batch_results()
if the call raises ApiException:
    if the activity is terminated:
        raise BatchError("Activity terminated by provider")
    if the exception is due to GSB endpoint not found:
        wait for 3 seconds
        repeat
    else:
        re-raise the exception

When repeating, the maximum number of attempts is 3.

[filipgolem]

It looks like the message was not finished.

[azawlocki]

Indeed! Thanks, fixed in c6356d9.

[filipgolem]

This conflicts with changes in #548 , which should be merged first.

[azawlocki]

#548 was a PR to master, and the current PR is to b0.6. When we will merge changes in b0.6 to master we'll have to merge those two sets of changes, but no need to worry about this now. I don't think we plan to backport #548 to b0.6, do we?

[filipgolem]

A similar PR will be a part of the next yajsapi release. IMO it would be great to backport #548.

[mfranciszkiewicz]

In place of 500s I get

[2021-08-10T17:53:49.010+0200 DEBUG yapapi.rest] Cannot query activity state
Traceback (most recent call last):
  File "/home/mf/Projects/yapapi/venv/lib/python3.9/site-packages/yapapi/rest/activity.py", line 210, in _get_results
    results = await self._activity._api.get_exec_batch_results(
  File "/home/mf/Projects/yapapi/venv/lib/python3.9/site-packages/ya_activity/api_client.py", line 205, in __call_api
    raise e
  File "/home/mf/Projects/yapapi/venv/lib/python3.9/site-packages/ya_activity/api_client.py", line 193, in __call_api
    response_data = await self.request(
  File "/home/mf/Projects/yapapi/venv/lib/python3.9/site-packages/ya_activity/rest.py", line 192, in GET
    return await self.request(
  File "/home/mf/Projects/yapapi/venv/lib/python3.9/site-packages/ya_activity/rest.py", line 180, in request
    raise ApiException(http_resp=r)
ya_activity.exceptions.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: <CIMultiDictProxy('Content-Length': '213', 'Content-Type': 'application/json', 'Date': 'Tue, 10 Aug 2021 15:53:48 GMT')>
HTTP response body: {"message":"GSB error: Remote service at `/net/0xb921cb630d07152c73640a175555ed970f5413c4/exeunit/7fcffb7659624af8be0be43eb1c567b5/GetExecBatchResults` error: GSB failure: Bad request: endpoint address not found"}


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mf/Projects/yapapi/venv/lib/python3.9/site-packages/yapapi/rest/activity.py", line 187, in _activity_terminated
    state_list = await self._activity.state().state  # type: ignore
AttributeError: 'coroutine' object has no attribute 'state'
[2021-08-10T17:53:49.011+0200 DEBUG yapapi.rest] Destroying activity 7fcffb7659624af8be0be43eb1c567b5 on error:
Traceback (most recent call last):
  File "/home/mf/Projects/yapapi/venv/lib/python3.9/site-packages/yapapi/executor/__init__.py", line 357, in start_worker
    await self._engine.process_batches(agreement.id, act, batch_generator)
  File "/home/mf/Projects/yapapi/venv/lib/python3.9/site-packages/yapapi/engine.py", line 562, in process_batches
    item = await command_generator.athrow(*sys.exc_info())
  File "/home/mf/Projects/yapapi/tests/drone/drone.py", line 48, in worker
    yield ctx.commit(timeout=timedelta(minutes=10))
  File "/home/mf/Projects/yapapi/venv/lib/python3.9/site-packages/yapapi/engine.py", line 556, in process_batches
    results = await get_batch_results()
  File "/home/mf/Projects/yapapi/venv/lib/python3.9/site-packages/yapapi/engine.py", line 537, in get_batch_results
    async for evt_ctx in remote:
  File "/home/mf/Projects/yapapi/venv/lib/python3.9/site-packages/yapapi/rest/activity.py", line 240, in __aiter__
    results = results[last_idx:]
TypeError: 'NoneType' object is not subscriptable