Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more logging to restore state errors #2763

Merged
merged 2 commits into from
Jul 3, 2017
Merged

Conversation

schmichael
Copy link
Member

Sample output:

2017/07/03 11:52:19.185581 [ERR] client: failed to restore state for alloc 9f3a1012-b39b-9494-4d31-0fb4992492fe: failed to read allocation state: failed to read alloc runner alloc_dir state: failed to decode data into passed object: EOF
2017/07/03 11:52:19.185597 [ERR] client: failed to restore state: 1 error(s) occurred:

* failed to read allocation state: failed to read alloc runner alloc_dir state: failed to decode data into passed object: EOF
2017/07/03 11:52:19.185601 [ERR] client: Nomad is unable to start due to corrupt state. The safest way to proceed is to manually stop running task processes and remove Nomad's state dir ("/tmp/nomad-devagent/client") before restarting. Lost allocations will be rescheduled.
2017/07/03 11:52:19.185603 [ERR] client: Corrupt state is often caused by a bug, please report as much information as possible to https://github.com/hashicorp/nomad/issues

To reproduce this error I started a client+server agent (not -dev as dev agents don't save/restore state), ran example.nomad, stopped nomad, and used a little tool to corrupt the state file:

boltq /tmp/nomad-devagent/client/state.db allocations.9f3a1012-b39b-9494-4d31-0fb4992492fe alloc-dir ""

https://github.com/schmichael/boltq

client/client.go Outdated
logger.Printf("[ERR] client: Nomad is unable to start due to corrupt state. "+
"The safest way to proceed is to manually stop running task processes "+
"and remove Nomad's state dir (%q) before restarting. Lost allocations "+
"will be rescheduled.", c.config.StateDir)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. Pushed.

@schmichael schmichael merged commit 02691c9 into master Jul 3, 2017
@schmichael schmichael deleted the f-bad-state-help branch July 3, 2017 21:45
@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 31, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants