vhost_user: Fix & comment "feature cache"

Added a long description of the "feature cache" and how it supports
reconnecting to a running QEMU.

Added a restriction so that the cache is only considered for the first
feature negotiation performed by a vhost-user app instance. This is to
avoid using the cached features when renegotiating with a guest e.g.
when the guest has switched to a new driver. Not yet tested.

src/apps/vhost/vhost_user.lua

@@ -174,13 +174,49 @@ function VhostUser:set_features (msg)
    self.dev:set_features(features)
 end
 
+-- Feature cache: A kludge to be compatible with a "QEMU reconnect" patch.
+--
+-- QEMU upstream (circa 2015) does not support the vhost-user device
+-- (Snabb) reconnecting to QEMU. That is unfortunate because being
+-- able to reconnect after a restart of either the Snabb process or
+-- simply a vhost-user app is very practical.
+--
+-- Reconnect support can however be enabled in QEMU with a small patch
+-- [1]. Caveat: Feature negotiation does not work reliably on the new
+-- connections and may provide an invalid feature list. Workaround:
+-- Cache the most recently negotiated features for each vhost-user
+-- socket and reuse those when available.
+--
+-- This is far from perfect but it is better than nothing.
+-- Reconnecting to QEMU VMs is very practical and enables faster
+-- development, restart of the Snabb process for recovery or upgrade,
+-- and stop/start of vhost-user app instances e.g. due to
+-- configuration changes.
+--
+-- QEMU upstream seem to be determined to solve the reconnect problem
+-- by requiring changes to the guest drivers so that the device could
+-- request a reset. However, this has the undesirable properties that
+-- it will not be transparent to the guest and nor will it work on
+-- existing guest drivers.
+--
+-- And so for now we have this cache for people who want to patch
+-- reconnect support into their QEMU...
+--
+-- 1: QEMU patch:
+--   https://github.com/SnabbCo/qemu/commit/f393aea2301734647fdf470724433f44702e3fb9.patch
+
+-- Consider using virtio-net feature cache to override negotiated features.
 function VhostUser:update_features (features)
    local stat = syscall.stat(self.socket_path)
    local mtime = ("%d.%d"):format(tonumber(stat.st_mtime),
                                   tonumber(stat.st_mtime_nsec))
    local cachepath = "/tmp/vhost_features_"..string.gsub(self.socket_path, "/", "__")
    local f = io.open(cachepath, 'r')
-   if f then
+   -- Use cached features when:
+   --   Negotiating features for the first time for this app instance
+   --   Cache is populated
+   --   QEMU vhost-user socket file has same timestamp as cache
+   if not self.have_negotiated_features and f then
       local file_features, file_mtime = f:read('*a'):match("features:(.*) mtime:(.*)\n")
       f:close()
       if file_mtime == mtime then
@@ -191,6 +227,12 @@ function VhostUser:update_features (features)
          print(("vhost_user: Skipped old feature cache in %s"):format(cachepath))
       end
    end
+   -- Features are now negotiated for this app instance. If they are
+   -- negotiated again it will presumably be due to guest driver
+   -- restart and in that case we should trust the new features rather
+   -- than overriding them with the cache.
+   self.have_negotiated_features = true
+   -- Cache features after they are negotiated
    f = io.open(cachepath, 'w')
    if f then
       print(("vhost_user: Caching features (0x%s) in %s"):format(