Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nomad cli job run -detach on promotion-ready, fail exit code if reverted, clarify eval status -t go templates #10810

Open
urusha opened this issue Jun 24, 2021 · 2 comments
Labels
stage/accepted Confirmed, and intend to work on. No timeline committment though. theme/cli theme/deployments type/enhancement

Comments

@urusha
Copy link

urusha commented Jun 24, 2021

Nomad 1.1.2. This is about #6818
We'd like to have an ability to detach nomad job run right after deployment becomes ready for promotion (all allocations are placed and healthy). This is needed for finishing CI deployment job and not locking it for hours/days (if testing before promotion requires long time).
Before -detach option we did it manually by running nomad deployment status and checking its output. But now with -detach option we can only get Evaluation ID but not Deployment id. nomad eval status doesn't print deployment id. It has -t option which is not well documented - we haven't found a way to print deployment id. timeout 2 nomad job run ... without -detach allows to see deployment id, but that's rather a hack.
Another great option would be -fail-on-revert or like so. This is needed to exit with non zero code if the deployment was reverted, currently we always getting 0, even with unsuccessful deployment, so we have to parse nomad job run output to determine this.

@tgross
Copy link
Member

tgross commented Jun 25, 2021

Hi @urusha! Let me walk you thru how the CLI does it's job to give you a little context on my suggested workaround and then we can discuss how we might implement this enhancements.

  • (1) When you submit a job via nomad job run, the CLI calls the Create Job API which immediately returns with the Evaluation ID. If you pass -detach, the CLI exits here.
  • (2) Then the CLI polls the Read Evaluation API until the evaluation is complete and returns a DeploymentID.
  • (3) And now in 1.1.2, the CLI polls the deployment status with all the rich output you're seeing.

So previously you were relying on the CLI to get you the deployment ID because it would stop at step (2), and the -detach flag stops everything at step (1). Effectively what we're looking for here is something like a -wait-for-deployment without the full-on monitoring. I'll mark this issue as accepted and leave it to whomever picks it up to come up with a better name for the flag 😀

Another great option would be -fail-on-revert or like so.

Yes, this is a great idea!


In the meantime, you can workaround your immediate problem by using nomad eval status -json :evalID to get the deployment ID for your nomad deployment status :id command:

$ nomad eval status -json c7b4f20b
{
    "AnnotatePlan": false,
    "BlockedEval": "",
    "ClassEligibility": null,
    "CreateIndex": 10,
    "CreateTime": 1624630187983095996,
    "DeploymentID": "60902747-12fe-3fcb-52c9-79df1b3374a7",
    "EscapedComputedClass": false,
    "FailedTGAllocs": null,
    "ID": "c7b4f20b-69cc-3a57-9cee-ae165d0dccc9",
    "JobID": "example",
    "JobModifyIndex": 10,
    "ModifyIndex": 12,
    "ModifyTime": 1624630187987915757,
    "Namespace": "default",
    "NextEval": "",
    "NodeID": "",
    "NodeModifyIndex": 0,
    "PreviousEval": "",
    "Priority": 50,
    "QueuedAllocations": {
        "cache": 0
    },
    "QuotaLimitReached": "",
    "SnapshotIndex": 10,
    "Status": "complete",
    "StatusDescription": "",
    "TriggeredBy": "job-register",
    "Type": "service",
    "Wait": 0,
    "WaitUntil": null
}

$ nomad eval status -json c7b4f20b | jq -r '.DeploymentID'
60902747-12fe-3fcb-52c9-79df1b3374a7

@tgross tgross added the stage/accepted Confirmed, and intend to work on. No timeline committment though. label Jun 25, 2021
@urusha
Copy link
Author

urusha commented Jun 25, 2021

Thanks for info! -json is exactly what we need, somehow we missed it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stage/accepted Confirmed, and intend to work on. No timeline committment though. theme/cli theme/deployments type/enhancement
Projects
None yet
Development

No branches or pull requests

2 participants