after restart nomad cleint can't start again

[tantra35]

Nomad version

Nomad v0.6.0-dev (52ffc01+CHANGES)

We try to restart nomad agent(client) on one of our servers and after that it can't start with tons of errors:

==> Caught signal: terminated
    Loaded configuration from /etc/nomad/nomad.json
==> Starting Nomad agent...
==> Error starting agent: client setup failed: failed to restore state: 836 error(s) occurred:

* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable

and another type of errors:

    2017/06/16 16:44:53.846436 [WARN] fingerprint.network: Unable to parse Speed in output of '/sbin/ethtool eth0.109'
    2017/06/16 16:44:57.870764 [WARN] driver.raw_exec: raw exec is enabled. Only enable if needed
    2017/06/16 16:44:57.871488 [ERR] client: failed to restore state for alloc 00d68f09-f7f6-a1e7-9e5b-a559975904f5: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.871522 [ERR] client: failed to restore state for alloc 00d9ad1a-2714-912f-76e9-91f6f6fde103: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.871550 [ERR] client: failed to restore state for alloc 014e20d0-ecd7-89cc-abdb-8db9d0a98900: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.871981 [ERR] client: failed to restore state for alloc 0151a8a5-f285-50b8-cc43-65e936967a1b: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.872189 [ERR] client: failed to restore state for alloc 02098646-9174-8e72-1673-dd967da5f632: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.872235 [ERR] client: failed to restore state for alloc 0239ac96-a7a1-18da-3fe3-cdff88458d8a: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.872794 [ERR] client: failed to restore state for alloc 027fe98b-fc17-0ad2-c864-aadacc04f726: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.873101 [ERR] client: failed to restore state for alloc 02dcc355-f907-93ff-b197-b363f2d9f12d: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.873156 [ERR] client: failed to restore state for alloc 035c3af1-2737-0d81-4ca4-186d3756b8fc: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.873691 [ERR] client: failed to restore state for alloc 03ba54bb-acf8-8d8f-5bc7-7da4a2a69b76: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.873723 [ERR] client: failed to restore state for alloc 03d268e8-e345-895b-7dde-eda5f77f851f: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.873761 [ERR] client: failed to restore state for alloc 04017d07-4c9c-5721-bd19-c83220b6f1d0: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874142 [ERR] client: failed to restore state for alloc 0411fe40-5418-b206-52ef-ed4d217d57fd: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874191 [ERR] client: failed to restore state for alloc 04251de1-63c1-6d90-8f09-85196748928d: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874256 [ERR] client: failed to restore state for alloc 049f3c2e-313d-a164-d922-7724c2f1901b: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874288 [ERR] client: failed to restore state for alloc 04a18531-f021-0242-de01-13aa4d3b8b62: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874348 [ERR] client: failed to restore state for alloc 04ebf645-f090-c2d1-49c4-deec71104281: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874399 [ERR] client: failed to restore state for alloc 04f6a563-e68e-38d9-8078-2857c5c54b82: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874436 [ERR] client: failed to restore state for alloc 054ed0fd-c270-4734-664f-078bd024b4bf: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874461 [ERR] client: failed to restore state for alloc 055c3166-7475-da9a-1de6-ddb1da774fd6: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874502 [ERR] client: failed to restore state for alloc 056e4b82-78a8-08af-142c-996c18fc7bae: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874539 [ERR] client: failed to restore state for alloc 05ad35aa-eed7-20e2-77bf-95f8b1cf368c: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874583 [ERR] client: failed to restore state for alloc 05d7cab8-131d-096d-dace-9149b4b7704b: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874637 [ERR] client: failed to restore state for alloc 064ace42-6430-ac94-4170-82548ba37a68: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874662 [ERR] client: failed to restore state for alloc 06a2f777-764b-c137-3ea3-d8368e8663b5: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.874688 [ERR] client: failed to restore state for alloc 06b58602-e50c-2200-0834-0fdeafe2016a: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.876837 [ERR] client: failed to restore state for alloc 06cd8811-7fb9-8432-819e-f735d30d485e: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.876882 [ERR] client: failed to restore state for alloc 07608cd7-8868-919d-5f64-876649fdf92d: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.876927 [ERR] client: failed to restore state for alloc 0766fedf-6753-1461-17d0-a8a50c79ec1a: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
    2017/06/16 16:44:57.876942 [ERR] client: failed to restore state for alloc 07f4e779-2841-9221-5654-d0f0e0779c26: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable

and after that all repears again and agian

It seems that this problem is the same as #2328

[schmichael]

This appears to be related to #2610 and may be a bug in the 0.5.6 -> 0.6 upgrade path.

I assume you were running these allocations with Nomad 0.5.6 before upgrading to master?

Deleting your state directory will allow this node to startup and the tasks to get rescheduled.

[tantra35]

No this happens second time, and at first time we think that is what you describe(0.5.6 -> 0.6 upgrade), but after that time we remove nomad state dir, regenerate it id(), so this "no_host_uuid": true, so this actions allow us to think that we have almost fresh nomad instalation, but this happens again on the same host

[tantra35]

And i think that #2610 isn't reason, because in #2328 we saw the same but at that time nomad doens't have boltdb as state storage