Added localstack batch and locale2e

[jim-sheldon]

• Create S3 buckets
• Create Batch Job Queue
• Add email address to SES
• Curator AWS clients use localstack when using locale2e docker-compose stack
• Curator async download uses localstack when using locale2e docker-compose stack

[codecov-commenter]

Codecov Report

Merging #2001 (aa35ca8) into main (999e0a1) will decrease coverage by 19.12%.
The diff coverage is 71.42%.

@@             Coverage Diff             @@
##             main    #2001       +/-   ##
===========================================
- Coverage   81.40%   62.28%   -19.13%     
===========================================
  Files          56       96       +40     
  Lines        1845     3516     +1671     
  Branches      337      987      +650     
===========================================
+ Hits         1502     2190      +688     
- Misses        343     1326      +983     

Impacted Files	Coverage Δ
verification/curator-service/api/src/index.ts	88.80% <ø> (ø)
...ation/curator-service/api/src/util/validate-env.ts	100.00% <ø> (ø)
...tion/curator-service/ui/src/components/App/App.tsx	79.60% <ø> (ø)
...rator-service/api/src/clients/aws-lambda-client.ts	62.50% <60.00%> (-1.14%)	⬇️
...urator-service/api/src/clients/aws-batch-client.ts	95.65% <75.00%> (-2.03%)	⬇️
...rator-service/api/src/clients/aws-events-client.ts	96.96% <80.00%> (-3.04%)	⬇️
...g/data-service/src/model/preexisting-conditions.ts
data-serving/data-service/src/controllers/home.ts
data-serving/data-service/src/util/case.ts
data-serving/data-service/src/model/date-range.ts
... and 101 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4221aaf...aa35ca8. Read the comment docs.

[iamleeg]

When I try running dev/run_full_stack.sh locally I hit this exception, is there any setup I need to do? I've set LOCALSTACK_API_KEY in dev/.env.

[jim-sheldon]

I suspect a timing issue. I increased the number of attempts to 20.

[abhidg]

Can't reproduce this, but I got this error message:

Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /tmp/localstack

Creating /tmp/localstack fixed this

[iamleeg]

Creating /tmp/localstack fixed this

Should that ideally go in the host's /tmp or the docker image's /tmp? Either way, it needs to be scripted.

[jim-sheldon]

Strange, I didn't run into that. FWIW /tmp exists in the docker image, but I'm not using it for anything--the volume declared in the compose file was a default--so I removed it.

[iamleeg]

I still see that the timeout exception is raised before the localstack has launched:

setup-localstack_1  | Waiting for localstack
setup-localstack_1  | Traceback (most recent call last):
setup-localstack_1  |   File "./setup_localstack.py", line 177, in <module>
setup-localstack_1  |     wait_for_localstack()
setup-localstack_1  |   File "./setup_localstack.py", line 172, in wait_for_localstack
setup-localstack_1  |     raise Exception("Localstack not available")
setup-localstack_1  | Exception: Localstack not available
setup-localstack_1 exited with code 1
localstack_1        | Waiting for all LocalStack services to be ready
localstack_1        | 2021-07-16T08:25:19:INFO:botocore.credentials: Found credentials in environment variables.
localstack_1        | Waiting for all LocalStack services to be ready
localstack_1        | Ready.
localstack_1        | 2021-07-16 08:25:24,808:API: 127.0.0.1 - - [16/Jul/2021 08:25:24] "GET / HTTP/1.1" 200 -
localstack_1        | 2021-07-16T08:25:24:INFO:localstack.utils.analytics.profiler: Execution of "start_api_services" took 12558.80ms

Could the setup image use wait-for-it.sh and run when the localstack service is finally accepting connections?

[abhidg]

Couple of small changes, looks good otherwise!

[iamleeg]

I see the same test failure in FF on a different PR. I'll try to fix and get a commit for you to cherry-pick.

[iamleeg]

I'm not proud but df56a02 works.