Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

paramaterized jobs will sometimes show as running and sometimes show as dead #3457

Closed
dansteen opened this issue Oct 27, 2017 · 2 comments
Closed

paramaterized jobs will sometimes show as running and sometimes show as dead #3457

dansteen opened this issue Oct 27, 2017 · 2 comments

Comments

@dansteen
Copy link

dansteen commented Oct 27, 2017 •
edited

Nomad version

Nomad v0.6.3

Operating system and Environment details

debian 8

Issue

When creating parameterized jobs, the job itself (not the dispatched jobs) will sometimes show as running and sometimes show as dead. As an example, I have the following jobs:

ID                                     Type                 Priority  Status   Submit Date
profile-service-stag-migrate           batch/parameterized  50        dead     10/16/17 12:21:36 EDT
traitify-api-stag-migrate-with-params  batch/parameterized  50        running  10/25/17 11:01:36 EDT

Notice that the top one shows as "dead" and the bottom one shows as "running"
Here are the detailed inspects of the two jobs:

The dead job:

{
    "Job": {
        "AllAtOnce": false,
        "Constraints": [
            {
                "LTarget": "${meta.role}",
                "Operand": "=",
                "RTarget": "utility"
            },
            {
                "LTarget": "${meta.env}",
                "Operand": "=",
                "RTarget": "stag"
            }
        ],
        "CreateIndex": 1075600,
        "Datacenters": [
            "awse"
        ],
        "ID": "profile-service-stag-migrate",
        "JobModifyIndex": 1144897,
        "Meta": {
            "JOB_ACTION": "describe"
        },
        "ModifyIndex": 1287556,
        "Name": "profile-service-stag-migrate",
        "ParameterizedJob": {
            "MetaOptional": [
                "JOB_ACTION"
            ],
            "MetaRequired": null,
            "Payload": "forbidden"
        },
        "ParentID": "",
        "Payload": null,
        "Periodic": null,
        "Priority": 50,
        "Region": "global",
        "Stable": false,
        "Status": "dead",
        "StatusDescription": "",
        "Stop": false,
        "SubmitTime": 1508170896772455140,
        "TaskGroups": [
            {
                "Constraints": [
                    {
                        "LTarget": "${attr.vault.version}",
                        "Operand": "version",
                        "RTarget": ">= 0.6.1"
                    }
                ],
                "Count": 1,
                "EphemeralDisk": {
                    "Migrate": false,
                    "SizeMB": 300,
                    "Sticky": false
                },
                "Meta": null,
                "Name": "migrate",
                "RestartPolicy": {
                    "Attempts": 2,
                    "Delay": 15000000000,
                    "Interval": 60000000000,
                    "Mode": "fail"
                },
                "Tasks": [
                    {
                        "Artifacts": [
                            {
                                "GetterMode": "any",
                                "GetterOptions": null,
                                "GetterSource": "https://xxxxxxxxx/profile-service/profile-service-0789504767fb3319092d91596b060c9e4c602077.tar.gz",
                                "RelativeDest": "local/profile_service"
                            },
                            {
                                "GetterMode": "any",
                                "GetterOptions": null,
                                "GetterSource": "https://xxxxxxxx/profile-service/profile-service-config-0789504767fb3319092d91596b060c9e4c602077.conf.tmpl",
                                "RelativeDest": "local/"
                            },
                            {
                                "GetterMode": "any",
                                "GetterOptions": null,
                                "GetterSource": "https://xxxxxxx/profile-service/profile-service-vm-0789504767fb3319092d91596b060c9e4c602077.args.tmpl",
                                "RelativeDest": "local/"
                            }
                        ],
                        "Config": {
                            "command": "/local/profile_service/bin/profile_service",
                            "args": [
                                "${NOMAD_META_JOB_ACTION}"
                            ]
                        },
                        "Constraints": null,
                        "DispatchPayload": null,
                        "Driver": "exec",
                        "Env": {
                            "CHEF_ENV": "${meta.env}",
                            "ERL_CRASH_DUMP": "/alloc/logs/erl_crash.dump",
                            "ERL_EPMD_PORT": "${NOMAD_PORT_epmd}",
                            "LOCAL_HOSTNAME": "${node.unique.name}",
                            "PORT": "${NOMAD_PORT_app}",
                            "RELEASE_MUTABLE_DIR": "/local/run_dir",
                            "APP_NAME": "profile-service"
                        },
                        "KillTimeout": 5000000000,
                        "Leader": true,
                        "LogConfig": {
                            "MaxFileSizeMB": 10,
                            "MaxFiles": 10
                        },
                        "Meta": null,
                        "Name": "profile-service-migrate",
                        "Resources": {
                            "CPU": 400,
                            "DiskMB": 0,
                            "IOPS": 0,
                            "MemoryMB": 200,
                            "Networks": [
                                {
                                    "CIDR": "",
                                    "Device": "",
                                    "DynamicPorts": [
                                        {
                                            "Label": "app",
                                            "Value": 0
                                        },
                                        {
                                            "Label": "admin",
                                            "Value": 0
                                        }
                                    ],
                                    "IP": "",
                                    "MBits": 10,
                                    "ReservedPorts": [
                                        {
                                            "Label": "epmd",
                                            "Value": 11002
                                        }
                                    ]
                                }
                            ]
                        },
                        "Services": null,
                        "ShutdownDelay": 0,
                        "Templates": [
                            {
                                "ChangeMode": "noop",
                                "ChangeSignal": "",
                                "DestPath": "local/profile_service/releases/0.1.0/profile_service.conf",
                                "EmbeddedTmpl": "",
                                "Envvars": false,
                                "LeftDelim": "{{",
                                "Perms": "664",
                                "RightDelim": "}}",
                                "SourcePath": "local/profile-service-config-0789504767fb3319092d91596b060c9e4c602077.conf.tmpl",
                                "Splay": 5000000000,
                                "VaultGrace": 300000000000
                            },
                            {
                                "ChangeMode": "noop",
                                "ChangeSignal": "",
                                "DestPath": "local/profile_service/vm.args",
                                "EmbeddedTmpl": "",
                                "Envvars": false,
                                "LeftDelim": "{{",
                                "Perms": "664",
                                "RightDelim": "}}",
                                "SourcePath": "local/profile-service-vm-0789504767fb3319092d91596b060c9e4c602077.args.tmpl",
                                "Splay": 5000000000,
                                "VaultGrace": 300000000000
                            }
                        ],
                        "User": "",
                        "Vault": {
                            "ChangeMode": "noop",
                            "ChangeSignal": "SIGHUP",
                            "Env": true,
                            "Policies": [
                                "stag_sp"
                            ]
                        }
                    },
                    {
                        "Artifacts": [
                            {
                                "GetterMode": "any",
                                "GetterOptions": null,
                                "GetterSource": "https://xxxxxx/profile-service/profile-service-remote-syslog2-0789504767fb3319092d91596b060c9e4c602077.yml.tmpl",
                                "RelativeDest": "local/"
                            }
                        ],
                        "Config": {
                            "args": [
                                "-c",
                                "/local/remote-syslog2.yml",
                                "-D"
                            ],
                            "command": "/usr/local/bin/remote_syslog"
                        },
                        "Constraints": null,
                        "DispatchPayload": null,
                        "Driver": "exec",
                        "Env": {
                            "LOG_TASK_NAME": "profile-service-migrate",
                            "APP_NAME": "profile-service",
                            "CHEF_ENV": "${meta.env}",
                            "LOCAL_HOSTNAME": "${node.unique.name}"
                        },
                        "KillTimeout": 5000000000,
                        "Leader": false,
                        "LogConfig": {
                            "MaxFileSizeMB": 10,
                            "MaxFiles": 10
                        },
                        "Meta": null,
                        "Name": "log-shipper",
                        "Resources": {
                            "CPU": 100,
                            "DiskMB": 0,
                            "IOPS": 0,
                            "MemoryMB": 100,
                            "Networks": null
                        },
                        "Services": null,
                        "ShutdownDelay": 0,
                        "Templates": [
                            {
                                "ChangeMode": "noop",
                                "ChangeSignal": "",
                                "DestPath": "local/remote-syslog2.yml",
                                "EmbeddedTmpl": "",
                                "Envvars": false,
                                "LeftDelim": "{{",
                                "Perms": "664",
                                "RightDelim": "}}",
                                "SourcePath": "local/profile-service-remote-syslog2-0789504767fb3319092d91596b060c9e4c602077.yml.tmpl",
                                "Splay": 5000000000,
                                "VaultGrace": 300000000000
                            }
                        ],
                        "User": "",
                        "Vault": null
                    }
                ],
                "Update": null
            }
        ],
        "Type": "batch",
        "Update": {
            "AutoRevert": false,
            "Canary": 0,
            "HealthCheck": "",
            "HealthyDeadline": 0,
            "MaxParallel": 0,
            "MinHealthyTime": 0,
            "Stagger": 0
        },
        "VaultToken": "",
        "Version": 18
    }
}

The running job:

{
    "Job": {
        "AllAtOnce": false,
        "Constraints": [
            {
                "LTarget": "${meta.role}",
                "Operand": "=",
                "RTarget": "utility"
            },
            {
                "LTarget": "${meta.env}",
                "Operand": "=",
                "RTarget": "stag"
            }
        ],
        "CreateIndex": 1074563,
        "Datacenters": [
            "awse"
        ],
        "ID": "traitify-api-stag-migrate-with-params",
        "JobModifyIndex": 1258273,
        "Meta": null,
        "ModifyIndex": 1258273,
        "Name": "traitify-api-stag-migrate-with-params",
        "ParameterizedJob": {
            "MetaOptional": null,
            "MetaRequired": [
                "MIGRATION_CONTEXT"
            ],
            "Payload": "forbidden"
        },
        "ParentID": "",
        "Payload": null,
        "Periodic": null,
        "Priority": 50,
        "Region": "global",
        "Stable": false,
        "Status": "running",
        "StatusDescription": "",
        "Stop": false,
        "SubmitTime": 1508943696252941620,
        "TaskGroups": [
            {
                "Constraints": [
                    {
                        "LTarget": "${attr.vault.version}",
                        "Operand": "version",
                        "RTarget": ">= 0.6.1"
                    }
                ],
                "Count": 1,
                "EphemeralDisk": {
                    "Migrate": false,
                    "SizeMB": 300,
                    "Sticky": false
                },
                "Meta": null,
                "Name": "migrate",
                "RestartPolicy": {
                    "Attempts": 2,
                    "Delay": 15000000000,
                    "Interval": 60000000000,
                    "Mode": "fail"
                },
                "Tasks": [
                    {
                        "Artifacts": [
                            {
                                "GetterMode": "any",
                                "GetterOptions": null,
                                "GetterSource": "https://xxxxxxx/traitify-api/traitify-api-config-50d140b5701020a2c41a1a57de5405f3f27a108b.yml.tmpl",
                                "RelativeDest": "local/"
                            },
                            {
                                "GetterMode": "any",
                                "GetterOptions": null,
                                "GetterSource": "https://xxxxxxxx/traitify-api/traitify-api-50d140b5701020a2c41a1a57de5405f3f27a108b.jar",
                                "RelativeDest": "local/"
                            }
                        ],
                        "Config": {
                            "jar_path": "local/traitify-api-50d140b5701020a2c41a1a57de5405f3f27a108b.jar",
                            "args": [
                                "db",
                                "migrate",
                                "-i",
                                "${NOMAD_META_MIGRATION_CONTEXT}",
                                "local/traitify-api-config.yml"
                            ]
                        },
                        "Constraints": null,
                        "DispatchPayload": null,
                        "Driver": "java",
                        "Env": {
                            "JAVA_TOOL_OPTIONS": "-Dcom.sun.management.jmxremote \n                             -Dcom.sun.management.jmxremote.port=${NOMAD_PORT_jmx}\n                             -Dcom.sun.management.jmxremote.local.only=true \n                             -Dcom.sun.management.jmxremote.authenticate=false \n                             -Dcom.sun.management.jmxremote.ssl=false \n                             -Djava.rmi.server.hostname=localhost \n                             -Dnetworkaddress.cache.ttl=60 \n                             -Xms1024M\n                             -XX:-UseConcMarkSweepGC \n                             -Xmx1024M",
                            "LOCAL_HOSTNAME": "${node.unique.name}",
                            "APP_NAME": "traitify-api",
                            "CHEF_ENV": "${meta.env}",
                            "IP_ADDRESS": "${attr.unique.platform.aws.public-ipv4}"
                        },
                        "KillTimeout": 5000000000,
                        "Leader": true,
                        "LogConfig": {
                            "MaxFileSizeMB": 10,
                            "MaxFiles": 10
                        },
                        "Meta": null,
                        "Name": "traitify-api-migrate",
                        "Resources": {
                            "CPU": 400,
                            "DiskMB": 0,
                            "IOPS": 0,
                            "MemoryMB": 1024,
                            "Networks": [
                                {
                                    "CIDR": "",
                                    "Device": "",
                                    "DynamicPorts": [
                                        {
                                            "Label": "app",
                                            "Value": 0
                                        },
                                        {
                                            "Label": "admin",
                                            "Value": 0
                                        },
                                        {
                                            "Label": "jmx",
                                            "Value": 0
                                        }
                                    ],
                                    "IP": "",
                                    "MBits": 10,
                                    "ReservedPorts": null
                                }
                            ]
                        },
                        "Services": null,
                        "ShutdownDelay": 0,
                        "Templates": [
                            {
                                "ChangeMode": "noop",
                                "ChangeSignal": "",
                                "DestPath": "local/traitify-api-config.yml",
                                "EmbeddedTmpl": "",
                                "Envvars": false,
                                "LeftDelim": "{{",
                                "Perms": "664",
                                "RightDelim": "}}",
                                "SourcePath": "local/traitify-api-config-50d140b5701020a2c41a1a57de5405f3f27a108b.yml.tmpl",
                                "Splay": 5000000000,
                                "VaultGrace": 300000000000
                            }
                        ],
                        "User": "",
                        "Vault": {
                            "ChangeMode": "noop",
                            "ChangeSignal": "SIGHUP",
                            "Env": true,
                            "Policies": [
                                "stag_sp"
                            ]
                        }
                    },
                    {
                        "Artifacts": [
                            {
                                "GetterMode": "any",
                                "GetterOptions": null,
                                "GetterSource": "https://xxxxxxx/traitify-api/traitify-api-remote-syslog2-50d140b5701020a2c41a1a57de5405f3f27a108b.yml.tmpl",
                                "RelativeDest": "local/"
                            }
                        ],
                        "Config": {
                            "command": "/usr/local/bin/remote_syslog",
                            "args": [
                                "-c",
                                "/local/remote-syslog2.yml",
                                "-D"
                            ]
                        },
                        "Constraints": null,
                        "DispatchPayload": null,
                        "Driver": "exec",
                        "Env": {
                            "LOCAL_HOSTNAME": "${node.unique.name}",
                            "LOG_TASK_NAME": "traitify-api-migrate",
                            "APP_NAME": "traitify-api",
                            "CHEF_ENV": "${meta.env}"
                        },
                        "KillTimeout": 5000000000,
                        "Leader": false,
                        "LogConfig": {
                            "MaxFileSizeMB": 10,
                            "MaxFiles": 10
                        },
                        "Meta": null,
                        "Name": "log-shipper",
                        "Resources": {
                            "CPU": 100,
                            "DiskMB": 0,
                            "IOPS": 0,
                            "MemoryMB": 100,
                            "Networks": null
                        },
                        "Services": null,
                        "ShutdownDelay": 0,
                        "Templates": [
                            {
                                "ChangeMode": "noop",
                                "ChangeSignal": "",
                                "DestPath": "local/remote-syslog2.yml",
                                "EmbeddedTmpl": "",
                                "Envvars": false,
                                "LeftDelim": "{{",
                                "Perms": "664",
                                "RightDelim": "}}",
                                "SourcePath": "local/traitify-api-remote-syslog2-50d140b5701020a2c41a1a57de5405f3f27a108b.yml.tmpl",
                                "Splay": 5000000000,
                                "VaultGrace": 300000000000
                            }
                        ],
                        "User": "",
                        "Vault": null
                    }
                ],
                "Update": null
            }
        ],
        "Type": "batch",
        "Update": {
            "AutoRevert": false,
            "Canary": 0,
            "HealthCheck": "",
            "HealthyDeadline": 0,
            "MaxParallel": 0,
            "MinHealthyTime": 0,
            "Stagger": 0
        },
        "VaultToken": "",
        "Version": 7
    }
}

Is there any reason these would show up differently?

Thanks!
@dadgar dadgar mentioned this issue Oct 27, 2017
Merged
@dadgar
Copy link
Contributor

dadgar commented Oct 27, 2017

@dansteen Thanks for reporting. Got a fix and will be in 0.7

@github-actions
Copy link

github-actions bot commented Dec 6, 2022

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dansteen @dadgar