feat: EMR Serverless Stage

[malachi-constant]

Closes #384

Todo

• Implementation
• Testing
• Docs
• Unit tests

Details

• Adds SFN stage to orchestrate trigger and polling of a job running against a serverless EMR cluster

Testing

Tested with infrastructure created in #385
and the following code

const app = new cdk.App();
const stack = new cdk.Stack(app, "EmrServerlessTest");
const emrCluster = new ddk.EMRServerlessCluster(stack, "MyEmrApp", {
  releaseLabel: "emr-6.11.0",
  type: "SPARK",
  vpcCidr: "10.60.0.0/16"
})
new ddk.EMRServerlessJobStage(
  stack,
  "EMR Serverless Job",
  {
    applicationId: emrCluster.emrServerlessApplication.ref,
    executionRoleArn: emrCluster.role.roleArn,
    jobDriver: { 
      SparkSubmit: {
        EntryPoint: `s3://${emrCluster.s3Bucket.bucketName}/scripts/wordcount.py`,
        EntryPointArguments: [`s3://${emrCluster.s3Bucket.bucketName}/emr-serverless-spark/output`],
        SparkSubmitParameters: "--conf spark.executor.cores=1 --conf spark.executor.memory=4g --conf spark.driver.cores=1 --conf spark.driver.memory=4g --conf spark.executor.instances=1"
      }
    }
  }
)

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: DDKIntegrationTestsD8472609-5SCHWjldmWcf
• Commit ID: 89d80f1
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: DDKIntegrationTestsD8472609-5SCHWjldmWcf
• Commit ID: 9af8149
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: DDKIntegrationTestsD8472609-5SCHWjldmWcf
• Commit ID: 00b9d06
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository