feat: job system

[PeterBaker0]

feat: job system

Description

Job system MVP including the following new components

• additional DB models representing jobs, job assignments, job results and job types
• jobs are defined as an input payload - results are through an assignment and can have files/result payload
• a storage system which uses s3 - each job gets an assigned storage location (though technically right now it could write to anything in the bucket!) and is allowed to write to it - then the API endpoint can presign a map of relative file paths -> presigned URLs which the frontend can use to download code
• CRUD type endpoints for interacting with job system including poll, list, get, assign, submit result etc
• a prototype 'manager' service which runs as a 24/7 service polling the queue - it's job is to spin up nodes to consume the jobs from the queue
• an example 'worker' service which is a proof of concept - later we can easily swap out this built CDK asset with a real image which consumes jobs - it provides a model for doing so
• CDK code to deploy a storage bucket, ECS services, tasks etc
• added a sequence diagram in README describing the interactions

TODOs later

• validate long term behaviour by running a service which creates jobs every now and again overnight and make sure service is stable
• better auto scaling management e.g. inspecting current number of containers rather than just using a cooldown period

[github-actions]

❌ Code formatting check failed. Please run npm run format:write locally and commit the changes.

[github-actions]

❌ Code formatting check failed. Please run npm run format:write locally and commit the changes.

[github-actions]

❌ Code formatting check failed. Please run npm run format:write locally and commit the changes.

[github-actions]

❌ Code formatting check failed. Please run npm run format:write locally and commit the changes.

[github-actions]

❌ Code formatting check failed. Please run npm run format:write locally and commit the changes.

[github-actions]

❌ Code formatting check failed. Please run npm run format:write locally and commit the changes.

[arlowhite]

looks good!

[github-actions]

❌ Code formatting check failed. Please run npm run format:write locally and commit the changes.

[github-actions]

❌ Code formatting check failed. Please run npm run format:write locally and commit the changes.

[github-actions]

❌ Code formatting check failed. Please run npm run format:write locally and commit the changes.

[github-actions]

❌ Code formatting check failed. Please run npm run format:write locally and commit the changes.