A large number of Spring Batch Jobs would lead to the exhaustion of database connections?!

[wondywang]

Discussed in #4745

^{Originally posted by wondywang January 16, 2025}
Recently, I've been considering developing a batch processing system using Spring Batch. However, while looking into the Spring Batch architecture, I found a potential issue.

Each batch job depends on database records for task information. Through use cases, it was verified that one Job establishes 11 persistent connections to the database. So, if I'm designing a Spring Batch system where all Jobs connect to the same database, does this limit the number of Jobs that can run in parallel?

For example, if 1000 Jobs are running at the same time, would these 1000 Jobs establish 11,000 persistent connections to the database? This could quickly exhaust the database connection pool, leading to systemic failures.

I hope my understanding is incorrect.

[wondywang]

Maybe we could add a ProxySQL in front of MySQL to solve this problem?

[wondywang]

Alternatively, we could refactor the JobRepository to direct it to report task status data to a centralized service.

This approach involves customizing the JobRepository so that instead of storing job execution and task status information rely on a database, it forwards this information to an external, centralized system for aggregation and monitoring.

Is this approach feasible?