Log batch job failures to Slack

[iamleeg]

• script doesn't depend on reaching an event from lambda
• copes with being rate limited by Slack
• Documentation
• Dockerfile
• Deploy to ECR
• Define job in Batch
• Schedule from EventBridge

[abhidg]

I'm not familiar with boto3 logs module, but will this only return the latest logs? We wouldn't want earlier logs coming in. Also would be good to hook it up to EventBridge event on Batch job failure -- may need some changes to handle the input event which is usually passed as a JSON payload.

Can add the batch job as a target for https://console.aws.amazon.com/events/home?region=us-east-1#/eventbus/default/rules/batch-job-failure once it can accept the event payload.

[iamleeg]

I'm not familiar with boto3 logs module, but will this only return the latest logs? We wouldn't want earlier logs coming in.

Do you mean to say that there isn't a separate log stream for each run of a parser? This makes it more difficult, as I need to know what the start and end dates are of the parser run I care about to filter the log stream. Do you know how I can get those?

Also would be good to hook it up to EventBridge event on Batch job failure -- may need some changes to handle the input event which is usually passed as a JSON payload.

Ah OK, I was hoping I'd be able to define the environment variables in the event but I'll look into doing that.

[abhidg]

Do you mean to say that there isn't a separate log stream for each run of a parser? This makes it more difficult, as I need to know what the start and end dates are of the parser run I care about to filter the log stream. Do you know how I can get those?

Re-reading that bit, it is fine - as there is a separate log stream with the same prefix for each parser (x-x-ingestor-prod/*), which can be obtained from the event.

[abhidg]

3.10?

[iamleeg]

I had problems installing poetry in the image when I based it on python:3.10-slim so I punted it. I will check again, it may have been a transient issue.

[iamleeg]

Poetry does not yet support Python 3.10

[iamleeg]

oh. maybe it does

[abhidg]

This appears to be the new way of installing poetry https://github.com/python-poetry/install.python-poetry.org -- I'm wary of the direct call to install.python-poetry.org, but should be possible to pin to a git tag

[iamleeg]

Good catch. We were calling get-poetry from master too so we probably need to change this throughout the repo.