Introduce drenv setup command

Minikube supports injecting configuration files into the cluster nodes.
This is very useful for solving 2 issues:

- Regression in minikube 1.33 enabling DNSSEC by default. This breaks
  pulling images from some registries, breaking us badly. But minikube
  1.33 have other improvements we would like to consume (such as faster
  cluster start) so this change work around this regression.

- Random failures when starting kubevirt VM, failing to create inotify
  watch. This is caused by low fs.inotify limits, and can be fixed by
  increasing the limits. We use the default limits from OpenShift worker
  node.

Creating the configuration files is not enough, since minikube inject
the files to the VM too late. We need to reload daemons or run sysctl to
apply the configuration changes when the cluster starts.

This change adds new "setup" command, that should run once when setting
up a development system. This calls minikube.setup_files(), creating the
drenv configuration drop-in files under $MINIKUBE_HOME/.minikube/files/.
New clusters created after that will use the new configuration.

To ensure that the configuration is loaded we call minikube.load_files()
when creating a new minikube cluster. This does the right thing for the
configuration files created during setup_files().

Issues:

- The setup and clear commands require an envfile but do not use it. We
  need to rework command line parsing to fix this.

Example usage:

    $ tree $MINIKUBE_HOME/.minikube/files
    /data/tmp/.minikube/files

    $ drenv setup envs/regional-dr.yaml
    2024-05-06 23:08:10,047 INFO    Setting up minikube for drenv
    2024-05-06 23:08:10,047 INFO    Minikube was setup for drenv

    $ tree $MINIKUBE_HOME/.minikube/files
    /data/tmp/.minikube/files
    └── etc
        ├── sysctl.d
        │   └── 99-drenv.conf
        └── systemd
            └── resolved.conf.d
                └── 99-drenv.conf

    $ drenv start envs/kubevirt.yaml -v
    ...
    2024-05-06 23:12:51,408 DEBUG   [kubevirt] Running ['minikube', 'ssh', '--profile', 'kubevirt', 'sudo sysctl -p /etc/sysctl.d/99-drenv.conf']
    2024-05-06 23:12:51,594 DEBUG   [kubevirt] fs.inotify.max_user_instances = 8192
    2024-05-06 23:12:51,594 DEBUG   [kubevirt] fs.inotify.max_user_watches = 65536
    2024-05-06 23:12:51,600 DEBUG   Applying drenv systemd-resolved configuration
    2024-05-06 23:12:51,600 DEBUG   [kubevirt] Running ['minikube', 'ssh', '--profile', 'kubevirt', 'sudo systemctl restart systemd-resolved.service']
    ...

    $ minikube ssh -p kubevirt 'cat /etc/systemd/resolved.conf.d/99-drenv.conf'
    # Added by drenv setup
    [Resolve]
    DNSSEC=no

    $ minikube ssh -p kubevirt 'cat /etc/sysctl.d/99-drenv.conf'
    # Added by drenv setup
    fs.inotify.max_user_instances = 8192
    fs.inotify.max_user_watches = 65536

    $ minikube ssh -p kubevirt 'sysctl fs.inotify'
    fs.inotify.max_queued_events = 16384
    fs.inotify.max_user_instances = 8192
    fs.inotify.max_user_watches = 65536

Signed-off-by: Nir Soffer <nsoffer@redhat.com>

test/drenv/__main__.py

@@ -124,6 +124,12 @@ def handle_termination_signal(signo, frame):
     sys.exit(1)
 
 
+def cmd_setup(env, args):
+    logging.info("Setting up minikube for drenv")
+    minikube.setup_files()
+    logging.info("Minikube was setup for drenv")
+
+
 def cmd_clear(env, args):
     start = time.monotonic()
     logging.info("[%s] Clearing cache", env["name"])
@@ -280,6 +286,8 @@ def start_cluster(profile, hooks=(), args=None, **options):
             containerd.configure(profile)
         if is_restart:
             restart_failed_deployments(profile)
+        else:
+            minikube.load_files(profile["name"])
 
     if hooks:
         execute(

test/drenv/minikube.py

@@ -104,6 +104,103 @@ def ssh(profile, command):
     _watch("ssh", command, profile=profile)
 
 
+def setup_files():
+    """
+    Set up minikube to work with drenv. Must be called bore starting the first
+    cluster.
+
+    To load the confiuration you must call load_files() after a cluster is
+    created.
+    """
+    _setup_sysctl()
+    _setup_systemd_resolved()
+
+
+def load_files(profile):
+    """
+    Load configuration done in setup_files() before the kinikube cluster was
+    started.
+
+    Must be called after the cluster is started, before running any addon. Not
+    need when starting a stopped cluster.
+    """
+    _load_sysctl(profile)
+    _load_systemd_resolved(profile)
+
+
+def _setup_sysctl():
+    """
+    Increase fs.inotifiy limits to avoid random timeouts when starting kubevirt
+    VM.
+
+    We use the same configuration as OpenShift worker node.
+    See also https://www.suse.com/support/kb/doc/?id=000020048
+    """
+    path = _sysctl_drenv_conf()
+    data = """# Added by drenv setup
+fs.inotify.max_user_instances = 8192
+fs.inotify.max_user_watches = 65536
+"""
+    logging.debug("Writing drenv sysctl configuration %s", path)
+    _write_file(path, data)
+
+
+def _load_sysctl(profile):
+    if not os.path.exists(_sysctl_drenv_conf()):
+        return
+    logging.debug("Applying drenv sysctl configuration")
+    ssh(profile, "sudo sysctl -p /etc/sysctl.d/99-drenv.conf")
+
+
+def _sysctl_drenv_conf():
+    return _minikube_file("etc", "sysctl.d", "99-drenv.conf")
+
+
+def _setup_systemd_resolved():
+    """
+    Disable DNSSEC in systemd-resolved configuration.
+
+    This is workaround for minikube regression in 1.33.0:
+    https://github.com/kubernetes/minikube/issues/18705
+
+    TODO: Remove when issue is fixed in minikube.
+    """
+    path = _systemd_resolved_drenv_conf()
+    data = """# Added by drenv setup
+[Resolve]
+DNSSEC=no
+"""
+    logging.debug("Writing drenv systemd-resolved configuration %s", path)
+    _write_file(path, data)
+
+
+def _load_systemd_resolved(profile):
+    if not os.path.exists(_systemd_resolved_drenv_conf()):
+        return
+    logging.debug("Applying drenv systemd-resolved configuration")
+    ssh(profile, "sudo systemctl restart systemd-resolved.service")
+
+
+def _systemd_resolved_drenv_conf():
+    return _minikube_file("etc", "systemd", "resolved.conf.d", "99-drenv.conf")
+
+
+def _write_file(path, data):
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    with open(path, "w") as f:
+        f.write(data)
+
+
+def _minikube_file(*names):
+    """
+    Create a path into $MINIKUBE_HOME/.minikube/files/...
+
+    The files are injected into the VM when the VM is created.
+    """
+    base = os.environ.get("MINIKUBE_HOME", os.path.expanduser("~"))
+    return os.path.join(base, ".minikube", "files", *names)
+
+
 def _run(command, *args, profile=None, output=None):
     cmd = ["minikube", command]
     if profile: