HCL to JSON api endpoint #4138

nickethier · 2018-04-11T20:08:15Z

The parse endpoint accepts a hcl jobspec body within a json object
and returns the parsed json object for the job. This allows users to
register jobs with the nomad json api without specifically needing
a nomad binary to parse their hcl encoded jobspec file.

Fixes #2782

schmichael

Great work! 🎉 🎉 🎉

Go ahead and add a changelog entry like:

IMPROVEMENTS:

...
api: Add /v1/jobs/parse endpoint for rending HCL job files as JSON [GH-2782]

Then run make changelogfmt to replace the issue number with a link.

Changelog entries are sorted core first, then alphabetically by category. Within a category they're roughly sorted shortest-to-longest.

schmichael · 2018-04-16T20:10:27Z

api/jobs.go

+type JobsParseRequest struct {
+	//JobHCL is an hcl jobspec
+	JobHCL string
+	//Canonicalize is a flag as to if the server should return default values


Put a newline between the previous field and the comment on the next one.

schmichael · 2018-04-16T20:12:23Z

api/jobs.go

 // Jobs returns a handle on the jobs endpoints.
 func (c *Client) Jobs() *Jobs {
 	return &Jobs{client: c}
 }

+// Parse is used to convert the HCL repesentation of a Job to JSON server side
+// To parse the HCL client side see package github.com/hashicorp/nomad/jobspec


Put a period between multiple sentences in comments as godoc/godoc.org will run them together into one line.

schmichael · 2018-04-16T20:18:01Z

command/agent/job_endpoint_test.go

+	t.Parallel()
+	httpTest(t, nil, func(s *TestAgent) {
+		buf := encodeReq(api.JobsParseRequest{JobHCL: mock.HCL()})
+		req, err := http.NewRequest("POST", "/v1/jobs/render", buf)


/parse not /render

Although since we're calling the handler directly it doesn't matter.

schmichael · 2018-04-16T20:20:05Z

website/source/api/jobs.html.md

+| Method | Path                      | Produces                   |
+| ------ | ------------------------- | -------------------------- |
+| `POST` | `/v1/jobs/parse`          | `application/json`         |
+


Add the standard blocking/acl section even though neither apply:

| Blocking Queries | ACL Required | | ---------------- | -------------------------- | | `No` | `none` |

schmichael · 2018-04-16T20:21:41Z

api/jobs.go

 // Jobs returns a handle on the jobs endpoints.
 func (c *Client) Jobs() *Jobs {
 	return &Jobs{client: c}
 }

+// Parse is used to convert the HCL repesentation of a Job to JSON server side
+// To parse the HCL client side see package github.com/hashicorp/nomad/jobspec
+func (j *Jobs) Parse(jobHCL string) (*Job, error) {


Add canonicalize as a boolean parameter.

Test it in api/jobs_test.go

The parse endpoint accepts a hcl jobspec body within a json object and returns the parsed json object for the job. This allows users to register jobs with the nomad json api without specifically needing a nomad binary to parse their hcl encoded jobspec file.

dadgar · 2018-04-16T23:34:54Z

api/jobs.go

@@ -36,11 +36,33 @@ type Jobs struct {
 	client *Client
 }

+// JobsParseRequest is used for arguments of the /vi/jobs/parse endpoint
+type JobsParseRequest struct {
+	//JobHCL is an hcl jobspec


Put a space between forward slashes and the first letter

dadgar · 2018-04-16T23:36:33Z

api/jobs.go

 // Jobs returns a handle on the jobs endpoints.
 func (c *Client) Jobs() *Jobs {
 	return &Jobs{client: c}
 }

+// Parse is used to convert the HCL repesentation of a Job to JSON server side.
+// To parse the HCL client side see package github.com/hashicorp/nomad/jobspec
+func (j *Jobs) Parse(jobHCL string, canonicalize bool) (*Job, error) {


Name it ParseHCL so we are futureproofed if we want to add a ParseJSON with canonicalize for people who want to just have defaults set.

dadgar · 2018-04-16T23:57:19Z

api/jobs_test.go

+
+	jobs := c.Jobs()
+
+	checkJob := func(job *Job, expected string) {


expected -> expectedRegion

dadgar · 2018-04-16T23:59:18Z

website/source/api/jobs.html.md

@@ -231,6 +231,83 @@ $ curl \
 }
 ```

+## Parse Job
+
+This endpoint will parse an hcl jobspec and produce the equivalent json encoded


HCL and JSON capitalized

dadgar · 2018-04-17T00:02:12Z

command/agent/job_endpoint.go

+	}
+
+	if args.Canonicalize {
+		jobStruct.Canonicalize()


Why not capture the warnings returned here and return it: https://github.com/hashicorp/nomad/blob/master/nomad/structs/structs.go#L906-L908

I'm not sure I follow. Canonicalize() does not return any errors? https://github.com/hashicorp/nomad/blob/master/api/jobs.go#L585

preetapan · 2018-04-17T14:48:32Z

command/agent/job_endpoint_test.go

+			job.Datacenters[0] != expected.Datacenters[0] {
+			t.Fatalf("job datacenters is '%s', expected '%s'",
+				job.Datacenters[0], expected.Datacenters[0])
+		}


Would suggest checking all fields of the returned job against expected instead of just these three fields.
We have a diff helper in

nomad/nomad/structs/diff.go

Line 59 in 5c8b249

func (j *Job) Diff(other *Job, contextual bool) (*JobDiff, error) {

that we use in nomad plan and other places.

These are api.Job structs. Is there a helper to convert to a structs.Job?

nomad/command/agent/job_endpoint.go

Line 547 in 537b947

func ApiJobToStructJob(job *api.Job) *structs.Job {

cautionary note - not sure if a direct translation like works okay when it comes to unset fields (esp if canonicalize is not called). so it may produce a diff complaining about empty list vs nil. I haven't tried it before, seem worth trying it and using it here.

I spent some time looking at this and there are a few things:

Diff errors if IDs don't match (mock.Job() generates a unique ID)

Lots of unset fields yields a somewhat unuseful diff.

The mock.HCL() method doesn't quite mimic the mock.Job() fully.

At some point it feels like I'm testing more of the jobspec.Parse functionality than the api endpoint. Thoughts?

That's fair. It doesn't seem worth it here to use diff for this test given the above.

Thanks for the pointer though, I learned more about the structs package 😃

schmichael · 2018-04-18T21:52:56Z

api/jobs_test.go

+		region := job.Region
+
+		if region == nil {
+			if expectedRegion != "" {


In the future feel free to use require.NotEmpty to condense these simple test assertions.

testify is kind of our blessed assertion utility library.

No need to change this PR.

github-actions · 2023-03-06T02:17:48Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

nickethier changed the title ~~[WIP] HCL to JSON api endpoint~~ HCL to JSON api endpoint Apr 16, 2018

schmichael requested changes Apr 16, 2018

View reviewed changes

nickethier added 9 commits April 16, 2018 19:21

command/agent: add HCL mock for parse endpoint

03f5c2f

command/agent: add Canonicalize option to parse args

14c44da

website: add api docs for v1/jobs/parse endpoint

a5ffa22

command/agent: fix ptr ref in job endpoint test

c594520

api: add test for canonicalized jobs/parse

4952c52

command/agent: fix url in jobs parse ep test

4303941

website: add standard blocking query/ACL table to jobs/parse ep

2e31bbe

add changelog entry for new jobs/parse endpoint

815c9fc

nickethier force-pushed the i-hcl-json-endpoint branch from 88bb27a to 815c9fc Compare April 16, 2018 23:24

website: fix type in api/jobs/parse

8ef7d20

dadgar requested changes Apr 17, 2018

View reviewed changes

minor code review fixes to api/jobs

059ea94

preetapan reviewed Apr 17, 2018

View reviewed changes

schmichael reviewed Apr 18, 2018

View reviewed changes

schmichael approved these changes Apr 18, 2018

View reviewed changes

nickethier merged commit 95d9eb9 into master Apr 19, 2018

nickethier deleted the i-hcl-json-endpoint branch April 19, 2018 18:18

github-actions bot locked as resolved and limited conversation to collaborators Mar 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HCL to JSON api endpoint #4138

HCL to JSON api endpoint #4138

nickethier commented Apr 11, 2018 •

edited

Loading

schmichael left a comment

schmichael Apr 16, 2018

schmichael Apr 16, 2018

schmichael Apr 16, 2018

schmichael Apr 16, 2018

nickethier Apr 16, 2018

schmichael Apr 16, 2018

dadgar Apr 16, 2018

dadgar Apr 16, 2018

dadgar Apr 16, 2018

dadgar Apr 16, 2018

dadgar Apr 17, 2018

nickethier Apr 17, 2018

preetapan Apr 17, 2018

nickethier Apr 17, 2018

preetapan Apr 17, 2018

preetapan Apr 17, 2018

nickethier Apr 19, 2018

preetapan Apr 19, 2018

nickethier Apr 19, 2018

schmichael Apr 18, 2018

github-actions bot commented Mar 6, 2023


		jobs := c.Jobs()

		checkJob := func(job *Job, expected string) {

HCL to JSON api endpoint #4138

HCL to JSON api endpoint #4138

Conversation

nickethier commented Apr 11, 2018 • edited Loading

schmichael left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 6, 2023

nickethier commented Apr 11, 2018 •

edited

Loading