Spring Batch on ECS vs AWS Batch

[VARSHAJOSHY]

Hi, I’m working on designing a use case where I need to run a batch of 10 jobs in parallel every day, processing 100 million records. I’m trying to decide between using Spring Batch on ECS or AWS Batch. Which option would be more suitable for this scenario?

[fmbenhassine]

It really depends on the business context. If you are already running your workloads in (and tied to) an AWS infrastructure, then going with AWS Batch could be a good consistent option.

However, if you would like to be platform agnostic , then it would be better to containerize your spring batch jobs and run them in ECS (and then EKS if more scale is needed).

If it were up to me, I would go with option 2, regardless, because this is the way I believe modern batch infrastructures should be designed and operated. I wrote about this here: https://spring.io/blog/2021/01/27/spring-batch-on-kubernetes-efficient-batch-processing-at-scale

I hope this helps, but let me know if you need more details that might help you make a decision.