Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for glue_version to resource aws_glue_job #10237

Merged
merged 2 commits into from
Oct 30, 2019
Merged

Add support for glue_version to resource aws_glue_job #10237

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cb8453f
Add support for glue version on aws_glue_job
NSjogren Sep 25, 2019
f875f4c
Update resource aws_glue_job documentation
NSjogren Sep 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions aws/resource_aws_glue_job.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ func resourceAwsGlueJob() *schema.Resource {
Type: schema.TypeString,
Optional: true,
},
"glue_version": {
Type: schema.TypeString,
Optional: true,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new argument is currently causing a difference for existing configurations since its set to perform drift detection and Glue assigns a default value, e.g.

--- PASS: TestAccAWSGlueJob_GlueVersion (31.34s)
--- FAIL: TestAccAWSGlueJob_Command (31.54s)
--- FAIL: TestAccAWSGlueJob_DefaultArguments (45.49s)
--- FAIL: TestAccAWSGlueJob_MaxRetries (47.57s)
--- FAIL: TestAccAWSGlueJob_Description (54.14s)
--- FAIL: TestAccAWSGlueJob_SecurityConfiguration (56.34s)
--- FAIL: TestAccAWSGlueJob_ExecutionProperty (59.86s)
--- PASS: TestAccAWSGlueJob_Basic (65.38s)
--- FAIL: TestAccAWSGlueJob_PythonShell (67.25s)
--- FAIL: TestAccAWSGlueJob_MaxCapacity (74.10s)
--- FAIL: TestAccAWSGlueJob_Timeout (74.40s)
--- FAIL: TestAccAWSGlueJob_AllocatedCapacity (81.45s)

These failures look like:

--- FAIL: TestAccAWSGlueJob_Description (54.14s)
    testing.go:569: Step 1 error: After applying this step, the plan was not empty:
        
        DIFF:
        
        UPDATE: aws_glue_job.test
... omitted for clarity ...
          glue_version:                             "0.9" => ""
... omitted for clarity ...

While we could set Default: "0.9" on this attribute, I'm actually going to suggest Computed: true since I would imagine that over time, Glue will update the default version number assigned to Jobs that omit the parameter.

Suggested change
Optional: true,
Optional: true,
Computed: true,

This should get the existing testing passing.

},
"execution_property": {
Type: schema.TypeList,
Optional: true,
Expand Down Expand Up @@ -155,6 +159,10 @@ func resourceAwsGlueJobCreate(d *schema.ResourceData, meta interface{}) error {
input.Description = aws.String(v.(string))
}

if v, ok := d.GetOk("glue_version"); ok {
input.GlueVersion = aws.String(v.(string))
}

if v, ok := d.GetOk("execution_property"); ok {
input.ExecutionProperty = expandGlueExecutionProperty(v.([]interface{}))
}
Expand Down Expand Up @@ -213,6 +221,7 @@ func resourceAwsGlueJobRead(d *schema.ResourceData, meta interface{}) error {
return fmt.Errorf("error setting default_arguments: %s", err)
}
d.Set("description", job.Description)
d.Set("glue_version", job.GlueVersion)
if err := d.Set("execution_property", flattenGlueExecutionProperty(job.ExecutionProperty)); err != nil {
return fmt.Errorf("error setting execution_property: %s", err)
}
Expand Down Expand Up @@ -267,6 +276,10 @@ func resourceAwsGlueJobUpdate(d *schema.ResourceData, meta interface{}) error {
jobUpdate.Description = aws.String(v.(string))
}

if v, ok := d.GetOk("glue_version"); ok {
jobUpdate.GlueVersion = aws.String(v.(string))
}

if v, ok := d.GetOk("execution_property"); ok {
jobUpdate.ExecutionProperty = expandGlueExecutionProperty(v.([]interface{}))
}
Expand Down
53 changes: 53 additions & 0 deletions aws/resource_aws_glue_job_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,40 @@ func TestAccAWSGlueJob_Description(t *testing.T) {
})
}

func TestAccAWSGlueJob_GlueVersion(t *testing.T) {
var job glue.Job

rName := fmt.Sprintf("tf-acc-test-%s", acctest.RandString(5))
resourceName := "aws_glue_job.test"

resource.ParallelTest(t, resource.TestCase{
PreCheck: func() { testAccPreCheck(t) },
Providers: testAccProviders,
CheckDestroy: testAccCheckAWSGlueJobDestroy,
Steps: []resource.TestStep{
{
Config: testAccAWSGlueJobConfig_GlueVersion(rName, "0.9"),
Check: resource.ComposeTestCheckFunc(
testAccCheckAWSGlueJobExists(resourceName, &job),
resource.TestCheckResourceAttr(resourceName, "glue_version", "0.9"),
),
},
{
Config: testAccAWSGlueJobConfig_GlueVersion(rName, "1.0"),
Check: resource.ComposeTestCheckFunc(
testAccCheckAWSGlueJobExists(resourceName, &job),
resource.TestCheckResourceAttr(resourceName, "glue_version", "1.0"),
),
},
{
ResourceName: resourceName,
ImportState: true,
ImportStateVerify: true,
},
},
})
}

func TestAccAWSGlueJob_ExecutionProperty(t *testing.T) {
var job glue.Job

Expand Down Expand Up @@ -645,6 +679,25 @@ resource "aws_glue_job" "test" {
`, testAccAWSGlueJobConfig_Base(rName), description, rName)
}

func testAccAWSGlueJobConfig_GlueVersion(rName, glueVersion string) string {
return fmt.Sprintf(`
%s

resource "aws_glue_job" "test" {
glue_version = "%s"
name = "%s"
role_arn = "${aws_iam_role.test.arn}"
allocated_capacity = 10

command {
script_location = "testscriptlocation"
}

depends_on = ["aws_iam_role_policy_attachment.test"]
}
`, testAccAWSGlueJobConfig_Base(rName), glueVersion, rName)
}

func testAccAWSGlueJobConfig_ExecutionProperty(rName string, maxConcurrentRuns int) string {
return fmt.Sprintf(`
%s
Expand Down
3 changes: 2 additions & 1 deletion website/docs/r/glue_job.html.markdown
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,8 @@ be removed in future releases, please use `max_capacity` instead.
* `role_arn` – (Required) The ARN of the IAM role associated with this job.
* `timeout` – (Optional) The job timeout in minutes. The default is 2880 minutes (48 hours).
* `security_configuration` - (Optional) The name of the Security Configuration to be associated with the job.

* `glue_version` - (Optional) The version of glue to use, for example "1.0". For information about available versionse see [AWS Glue Release Notes](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Can you please alphabetize this new argument with the others? Thanks.


### command Argument Reference

* `name` - (Optional) The name of the job command. Defaults to `glueetl`
Expand Down