Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Retries with exp backoff and messages to Datadog #16

Open
george-zubrienko opened this issue Oct 18, 2022 · 1 comment
Open
Assignees
Labels
code/new-feature New feature or request

Comments

@george-zubrienko
Copy link
Contributor

Is your feature request related to a problem? Please describe.
We observe some random failures on reading Beast responses from time to time:

[2022-10-18, 14:27:53 UTC] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 171, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 189, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/esd_services_api_client/beast/_connector.py", line 124, in run_job
    request_lifecycle = self.http.get(f"{self.base_url}/job/requests/{request_id}").json()['lifeCycleStage']
  File "/home/airflow/.local/lib/python3.9/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

https://airflow.production.sneaksanddata.com/log?dag_id=eccoCdpSilverRealtime&task_id=inventory_position_d365_europe&execution_date=2022-10-18T14%3A00%3A00%2B00%3A00

Describe the solution you'd like
Errors such as these should be:

  • Retried with backoff until certain limit
  • Logged to Datadog as warnings, so we can investigate and resolve the root cause

fyi @s-vitaliy @jeppe742

@george-zubrienko
Copy link
Contributor Author

I'll replace all prints with Datadog logging in a separate PR fyi @jeppe742

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code/new-feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant