m_etcd can interpret node key expiration as a command

[schmichael]

From an internal metafora app:

2015/01/01 05:32:01.548419 <autogenerated>:50: [WARN] Unexpected error updating node key: 100: Key not found (/workflows/nodes/XYZ) [1682348]
2015/01/01 05:32:01.549067 <autogenerated>:50: [ERROR] Error deleting handled command /workflows/nodes/XYZ: 100: Key not found (/workflows/nodes/XYZ) [1682349]
2015/01/01 05:32:01.549135 <autogenerated>:50: [ERROR] Invalid command /workflows/nodes/XYZ: unexpected end of JSON input

Due to network issues metafora wasn't able to update its node key before the node key's 20s TTL expired. This is expected behavior during network partitions.

However, the coordinator incorrectly tried to interpret the key expiration event as a command (second line) and obviously couldn't (third line).

So luckily this is a noop, but it's pretty scary that the command watcher can even interpret node keys as commands. The command watcher deletes handled commands, so having it touch the node key at all is scary.

This method should be improved to insure we never try to treat node keys as commands:

metafora/m_etcd/coordinator.go

Lines 385 to 402 in 4862f92

	func (ec *EtcdCoordinator) parseCommand(resp *etcd.Response) metafora.Command {
	if strings.HasSuffix(resp.Node.Key, MetadataKey) {
	// Skip metadata marker
	return nil
	}
	const recurse = false
	if _, err := ec.Client.Delete(resp.Node.Key, recurse); err != nil {
	ec.cordCtx.Log(metafora.LogLevelError, "Error deleting handled command %s: %v", resp.Node.Key, err)
	}
	cmd, err := metafora.UnmarshalCommand([]byte(resp.Node.Value))
	if err != nil {
	ec.cordCtx.Log(metafora.LogLevelError, "Invalid command %s: %v", resp.Node.Key, err)
	return nil
	}
	return cmd
	}