Merge 3003 changes forward to the master branch (saltstack#59879)

* Merge 3002.6 bugfix changes (saltstack#59822)

* Pass `CI_RUN` as an environment variable to the test run.

This allows us to know if we're running the test suite under a CI
environment or not and adapt/adjust if needed

* Migrate `unit.setup` to PyTest

* Backport ae36b15 just for test_install.py

* Only skip tests on CI runs

* Always store git sha in _version.py during installation

* Fix PEP440 compliance.

The wheel metadata version 1.2 states that the package version MUST be
PEP440 compliant.

This means that instead of `3002.2-511-g033c53eccb`, the salt version
string should look like `3002.2+511.g033c53eccb`, a post release of
`3002.2` ahead by 511 commits with the git sha `033c53eccb`

* Fix and migrate `tests/unit/test_version.py` to PyTest

* Skip test if `easy_install` is not available

* We also need to be PEP440 compliant when there's no git history

* Allow extra_filerefs as sanitized kwargs for SSH client

* Fix regression on cmd.run when passing tuples as cmd

Co-authored-by: Alexander Graul <agraul@suse.com>

* Add unit tests to ensure cmd.run accepts tuples

* Add unit test to check for extra_filerefs on SSH opts

* Add changelog file

* Fix comment for test case

* Fix unit test to avoid failing on Windows

* Skip failing test on windows

* Fix test to work on Windows

* Add all ssh kwargs to sanitize_kwargs method

* Run pre-commit

* Fix pylint

* Fix cmdmod loglevel and module_names tests

* Fix pre-commit

* Skip ssh tests if binary does not exist

* Use setup_loader for cmdmod test

* Prevent argument injection in restartcheck

* Add changelog for restartcheck fix

* docs_3002.6

* Add back tests removed in merge

Co-authored-by: Pedro Algarvio <pedro@algarvio.me>
Co-authored-by: Megan Wilhite <megan.wilhite@gmail.com>
Co-authored-by: Bryce Larson <brycel@vmware.com>
Co-authored-by: Pablo Suárez Hernández <psuarezhernandez@suse.com>
Co-authored-by: Alexander Graul <agraul@suse.com>
Co-authored-by: Frode Gundersen <fgundersen@saltstack.com>

* Remove glance state module in favor of glance_image

* update wording in changelog

* bump deprecation warning to Silicon.

* Updating warnutil version to Phosphorous.

* Update salt/modules/keystone.py

Co-authored-by: Megan Wilhite <megan.wilhite@gmail.com>

* Check $HOMEBREW_PREFIX when linking against libcrypto

When loading `libcrypto`, Salt checks for a Homebrew installation of `openssl`
at Homebrew's default prefix of `/usr/local`. However, on Apple Silicon Macs,
Homebrew's default installation prefix is `/opt/homebrew`. On all platforms,
the prefix is configurable.  If Salt doesn't find one of those `libcrypto`s,
it will fall back on the un-versioned `/usr/lib/libcrypto.dylib`, which will
cause the following crash:

    Application Specific Information:
    /usr/lib/libcrypto.dylib
    abort() called
    Invalid dylib load. Clients should not load the unversioned libcrypto dylib as it does not have a stable ABI.

This commit checks $HOMEBREW_PREFIX instead of hard-coding `/usr/local`.

* Add test case

* Add changelog for 59808

* Add changelog entry

* Make _find_libcrypto fail on Big Sur if it can't find a library

Right now, if `_find_libcrypto` can't find any externally-managed versions of
libcrypto, it will fall back on the pre-Catalina un-versioned system libcrypto.
This does not exist on Big Sur and it would be better to raise an exception
here rather than crashing later when trying to open it.

* Update _find_libcrypto tests

This commit simplifies the unit tests for _find_libcrypto by mocking out the
host's filesystem and testing the common libcrypto installations (brew, ports,
etc.) on Big Sur. It simplifies the tests for falling back on system versions
of libcrypto on previous versions of macOS.

* Fix description of test_find_libcrypto_with_system_before_catalina

* Patch sys.platform for test_rsax931 tests

* modules/match: add missing "minion_id" in Pillar example

The documented Pillar example for `match.filter_by` lacks the `minion_id` parameter. Without it, the assignment won't work as expected.
- fix documentation
- add tests:
  - to prove the misbehavior of the documented example
  - to prove the proper behaviour when supplying `minion_id`
  - to ensure some misbehaviour observed with compound matchers doesn't occur

* Fix for issue saltstack#59773

- When instantiating the loader grab values of grains and pillars if
  they are NamedLoaderContext instances.
- The loader uses a copy of opts.
- Impliment deepcopy on NamedLoaderContext instances.

* Add changelog for saltstack#59773

* _get_initial_pillar function returns pillar

* Fix linter issues

* Clean up test

* Bump deprecation release for neutron

* Uncomment Sulfur release name

* Removing the _ext_nodes deprecation warning and alias.

* Adding changelog.

* Renaming changelog file.

* Update 59804.removed

* Initial pass at fips_mode config option

* Fix pre-commit

* Fix tests and add changelog

* update docs 3003

* update docs 3003 - newline

* Fix warts in changelog

Co-authored-by: Pedro Algarvio <pedro@algarvio.me>
Co-authored-by: Megan Wilhite <megan.wilhite@gmail.com>
Co-authored-by: Bryce Larson <brycel@vmware.com>
Co-authored-by: Pablo Suárez Hernández <psuarezhernandez@suse.com>
Co-authored-by: Alexander Graul <agraul@suse.com>
Co-authored-by: Frode Gundersen <fgundersen@saltstack.com>
Co-authored-by: Gareth J. Greenaway <gareth@saltstack.com>
Co-authored-by: Gareth J. Greenaway <gareth@wiked.org>
Co-authored-by: Hoa-Long Tam <hoalong@apple.com>
Co-authored-by: krionbsd <krion@freebsd.org>
Co-authored-by: Elias Probst <e.probst@ssc-services.de>
Co-authored-by: Frode Gundersen <frogunder@gmail.com>

CHANGELOG.md

@@ -13,6 +13,9 @@ Salt 3003 (2021-03-05)
 Removed
 -------
 
+- Removed the deprecated glance state and execution module in favor of the glance_image
+  state module and the glanceng execution module. (#59079)
+- Removing the _ext_nodes deprecation warning and alias to the master_tops function.  This change will break compatibility with a Salt master running versions 2017.7.8 and older and Salt minions running versions 3003 and newer. (#59804)
 - removed the arg `managed_private_key` from 'salt.states.x509.certificate_managed' (#59247)
 - Drop support for python 3.5 on Windows (#59479)
 
@@ -22,7 +25,6 @@ Deprecated
 
 - Added deprecation warning for grains.get_or_set_hash (#59425)
 
-
 Changed
 -------
 
@@ -35,6 +37,9 @@ Changed
 Fixed
 -----
 
+- When instantiating the loader grab values of grains and pillars if
+  they are NamedLoaderContext instances. (#59773)
+- Fixed installation on Apple Silicon Macs by checking $HOMEBREW_PREFIX for `libcrypto` instead of assuming /usr/local. (#59808)
 - Fix incorrect documentation for pillar_source_merging_strategy (#26396)
 - Don't iterate through cloud map errors (#34033)
 - Supress noisy warnings when very old pyzmq is used. (#50327)
@@ -135,6 +140,7 @@ Fixed
 Added
 -----
 
+- Added "fips_mode" config option to master and minion configs. (#59427)
 - Adding the ability to clear and show the pillar cache enabled when pillar_cache is True. (#37080)
 - SCRAM-SHA-256 support for PostgreSQL passwords.
   Pass encrypted=scram-sha-256 to the postgres_user.present (or postgres_group.present) state. (#51271)
@@ -169,6 +175,24 @@ Added
   binary ELF files in the package. (#59569)
 
 
+Salt 3002.6 (2021-03-10)
+========================
+
+Changed
+-------
+
+- Store git sha in salt/_version.py when installing from a tag so it can be found if needed later. (#59137)
+
+Fixed
+-----
+
+- Fix argument injection bug in restartcheck.restartcheck. This change hardens
+  the fix for CVE-2020-28243. (#200)
+- Allow "extra_filerefs" as sanitized kwargs for SSH client.
+  Fix regression on "cmd.run" when passing tuples as cmd. (#59664)
+- Allow all ssh kwargs as sanitized kwargs for SSH client. (#59748)
+
+
 Salt 3002.5 (2021-02-25)
 ========================
 
@@ -480,6 +504,18 @@ Added
   This flag will be deprecated in the Phosphorus release when this functionality
   becomes the default. (#58652)
 
+Salt 3001.7 (2021-03-10)
+========================
+
+Fixed
+-----
+
+- Fix argument injection bug in restartcheck.restartcheck. This change hardens
+  the fix for CVE-2020-28243. (#200)
+- Allow "extra_filerefs" as sanitized kwargs for SSH client.
+  Fix regression on "cmd.run" when passing tuples as cmd. (#59664)
+- Allow all ssh kwargs as sanitized kwargs for SSH client. (#59748)
+
 Salt 3001.6 (2021-02-09)
 ========================
 
@@ -971,6 +1007,18 @@ Added
 - [#56637](https://github.com/saltstack/salt/pull/56637) - Add ``win_wua.installed`` to the ``win_wua`` execution module
 - Clarify how to get the master fingerprint (#54699)
 
+Salt 3000.9 (2021-03-10)
+========================
+
+Fixed
+-----
+
+- Allow "extra_filerefs" as sanitized kwargs for SSH client.
+  Fix regression on "cmd.run" when passing tuples as cmd. (#59664)
+- Allow all ssh kwargs as sanitized kwargs for SSH client. (#59748)
+- Fix argument injection bug in restartcheck.restartcheck. This change hardens
+  the fix for CVE-2020-28243.
+
 Salt 3000.8 (2021-02-09)
 ========================
 

doc/topics/releases/3000.9.rst

@@ -0,0 +1,17 @@
+.. _release-3000-9:
+
+===========================
+Salt 3000.9 Release Notes
+===========================
+
+Version 3000.9 is a bug fix release for :ref:`3000 <release-3000>`.
+
+
+Fixed
+-----
+
+- Allow "extra_filerefs" as sanitized kwargs for SSH client.
+  Fix regression on "cmd.run" when passing tuples as cmd. (#59664)
+- Allow all ssh kwargs as sanitized kwargs for SSH client. (#59748)
+- Fix argument injection bug in restartcheck.restartcheck. This change hardens
+  the fix for CVE-2020-28243.

doc/topics/releases/3001.7.rst

@@ -0,0 +1,17 @@
+.. _release-3001-7:
+
+=========================
+Salt 3001.7 Release Notes
+=========================
+
+Version 3001.7 is a bug fix release for :ref:`3001 <release-3001>`.
+
+
+Fixed
+-----
+
+- Allow "extra_filerefs" as sanitized kwargs for SSH client.
+  Fix regression on "cmd.run" when passing tuples as cmd. (#59664)
+- Allow all ssh kwargs as sanitized kwargs for SSH client. (#59748)
+- Fix argument injection bug in restartcheck.restartcheck. This change hardens
+  the fix for CVE-2020-28243.

doc/topics/releases/3002.6.rst

@@ -0,0 +1,24 @@
+.. _release-3002-6:
+
+=========================
+Salt 3002.6 Release Notes
+=========================
+
+Version 3002.6 is a bug fix release for :ref:`3002 <release-3002>`.
+
+
+Changed
+-------
+
+- Store git sha in salt/_version.py when installing from a tag so it can be found if needed later. (#59137)
+
+
+Fixed
+-----
+
+- Fix argument injection bug in restartcheck.restartcheck. This change hardens
+  the fix for CVE-2020-28243. (#200)
+- Allow "extra_filerefs" as sanitized kwargs for SSH client.
+  Fix regression on "cmd.run" when passing tuples as cmd. (#59664)
+- Allow all ssh kwargs as sanitized kwargs for SSH client. (#59748)
+

doc/topics/releases/3003.rst

@@ -4,8 +4,6 @@
 Salt 3003 Release Notes - Codename Aluminium
 ============================================
 
-Salt 3003 is an *unreleased* upcoming feature release.
-
 New Features
 ============
 
@@ -22,6 +20,9 @@ previous storage methods.
 Removed
 -------
 
+- Removed the deprecated glance state and execution module in favor of the glance_image
+  state module and the glanceng execution module. (#59079)
+- Removing the _ext_nodes deprecation warning and alias to the master_tops function.  This change will break compatibility with a Salt master running versions 2017.7.8 and older and Salt minions running versions 3003 and newer. (#59804)
 - removed the arg `managed_private_key` from 'salt.states.x509.certificate_managed' (#59247)
 - Drop support for python 3.5 on Windows (#59479)
 
@@ -50,6 +51,9 @@ Changed
 Fixed
 -----
 
+- When instantiating the loader grab values of grains and pillars if
+  they are NamedLoaderContext instances. (#59773)
+- Fixed installation on Apple Silicon Macs by checking $HOMEBREW_PREFIX for `libcrypto` instead of assuming /usr/local. (#59808)
 - The Google Cloud Engine salt-cloud provider now requires `apache-libcloud>=2.5.0`. Service account authentication is broken on older versions.
 - Fix incorrect documentation for pillar_source_merging_strategy (#26396)
 - Don't iterate through cloud map errors (#34033)
@@ -151,6 +155,7 @@ Fixed
 Added
 -----
 
+- Added "fips_mode" config option to master and minion configs. (#59427)
 - Firewall groups support to Vultr Salt Cloud provider
 - Adding the ability to clear and show the pillar cache enabled when pillar_cache is True. (#37080)
 - SCRAM-SHA-256 support for PostgreSQL passwords.

salt/client/ssh/client.py

@@ -52,11 +52,34 @@ def sanitize_kwargs(self, kwargs):
             ("ssh_identities_only", bool),
             ("ssh_remote_port_forwards", str),
             ("ssh_options", list),
+            ("ssh_max_procs", int),
+            ("ssh_askpass", bool),
+            ("ssh_key_deploy", bool),
+            ("ssh_update_roster", bool),
+            ("ssh_scan_ports", str),
+            ("ssh_scan_timeout", int),
+            ("ssh_timeout", int),
+            ("ssh_log_file", str),
+            ("raw_shell", bool),
+            ("refresh_cache", bool),
+            ("roster", str),
             ("roster_file", str),
             ("rosters", list),
             ("ignore_host_keys", bool),
             ("raw_shell", bool),
             ("extra_filerefs", str),
+            ("min_extra_mods", str),
+            ("thin_extra_mods", str),
+            ("verbose", bool),
+            ("static", bool),
+            ("ssh_wipe", bool),
+            ("rand_thin_dir", bool),
+            ("regen_thin", bool),
+            ("python2_bin", str),
+            ("python3_bin", str),
+            ("ssh_run_pre_flight", bool),
+            ("no_host_keys", bool),
+            ("saltfile", str),
         ]
         sane_kwargs = {}
         for name, kind in roster_vals:

salt/config/__init__.py

@@ -952,6 +952,7 @@ def _gather_buffer_space():
         "disabled_requisites": (str, list),
         # Feature flag config
         "features": dict,
+        "fips_mode": bool,
     }
 )
 
@@ -1254,6 +1255,7 @@ def _gather_buffer_space():
         "ssh_merge_pillar": True,
         "disabled_requisites": [],
         "reactor_niceness": None,
+        "fips_mode": False,
     }
 )
 
@@ -1590,6 +1592,7 @@ def _gather_buffer_space():
         "minion_data_cache_events": True,
         "enable_ssh_minions": False,
         "netapi_allow_raw_shell": False,
+        "fips_mode": False,
     }
 )
 

salt/fileclient.py

@@ -1404,15 +1404,7 @@ def master_tops(self):
         """
         Return the metadata derived from the master_tops system
         """
-        log.debug(
-            "The _ext_nodes master function has been renamed to _master_tops. "
-            "To ensure compatibility when using older Salt masters we will "
-            "continue to invoke the function as _ext_nodes until the "
-            "3002 release."
-        )
-        # TODO: Change back to _master_tops
-        # for 3002 release
-        load = {"cmd": "_ext_nodes", "id": self.opts["id"], "opts": self.opts}
+        load = {"cmd": "_master_tops", "id": self.opts["id"], "opts": self.opts}
         if self.auth:
             load["tok"] = self.auth.gen_token(b"salt")
         return self.channel.send(load)

salt/loader.py

@@ -1314,6 +1314,12 @@ def __init__(
                 self.pack[i] = self.pack[i].value()
         if opts is None:
             opts = {}
+        opts = copy.deepcopy(opts)
+        for i in ["pillar", "grains"]:
+            if i in opts and isinstance(
+                opts[i], salt.loader_context.NamedLoaderContext
+            ):
+                opts[i] = opts[i].value()
         threadsafety = not opts.get("multiprocessing")
         self.context_dict = salt.utils.context.ContextDict(threadsafe=threadsafety)
         self.opts = self.__prep_mod_opts(opts)

salt/loader_context.py

@@ -4,6 +4,7 @@
 import collections.abc
 import contextlib
 import contextvars
+import copy
 
 DEFAULT_CTX_VAR = "loader_ctxvar"
 
@@ -112,6 +113,10 @@ def __setstate__(self, state):
     def __getattr__(self, name):
         return getattr(self.value(), name)
 
+    def __deepcopy__(self, memo):
+        default = copy.deepcopy(self.default)
+        return self.__class__(self.name, self.loader_context, default)
+
     def missing_fun_string(self, name):
         return self.loader().missing_fun_string(name)
 

salt/master.py

@@ -1204,7 +1204,6 @@ class AESFuncs(TransportMethods):
     expose_methods = (
         "verify_minion",
         "_master_tops",
-        "_ext_nodes",
         "_master_opts",
         "_mine_get",
         "_mine",
@@ -1429,9 +1428,6 @@ def _master_tops(self, load):
             return {}
         return self.masterapi._master_tops(load, skip_verify=True)
 
-    # Needed so older minions can request master_tops
-    _ext_nodes = _master_tops
-
     def _master_opts(self, load):
         """
         Return the master options to the minion

salt/modules/nacl.py

@@ -1,4 +1,3 @@
-# -*- coding: utf-8 -*-
 """
 This module helps include encrypted passwords in pillars, grains and salt state files.
 
@@ -150,16 +149,15 @@
 
 """
 
-# Import Python libs
-from __future__ import absolute_import, print_function, unicode_literals
 
-# Import Salt libs
 import salt.utils.nacl
 
 __virtualname__ = "nacl"
 
 
 def __virtual__():
+    if __opts__["fips_mode"] is True:
+        return False, "nacl module not available in FIPS mode"
     return salt.utils.nacl.check_requirements()
 
 

salt/modules/restartcheck.py

@@ -11,7 +11,6 @@
 """
 import os
 import re
-import shlex
 import subprocess
 import sys
 import time
@@ -495,17 +494,17 @@ def restartcheck(ignorelist=None, blacklist=None, excludepid=None, **kwargs):
     verbose = kwargs.pop("verbose", True)
     timeout = kwargs.pop("timeout", 5)
     if __grains__.get("os_family") == "Debian":
-        cmd_pkg_query = "dpkg-query --listfiles "
+        cmd_pkg_query = ["dpkg-query", "--listfiles"]
         systemd_folder = "/lib/systemd/system/"
         systemd = "/bin/systemd"
         kernel_versions = _kernel_versions_debian()
     elif __grains__.get("os_family") == "RedHat":
-        cmd_pkg_query = "repoquery -l "
+        cmd_pkg_query = ["repoquery", "-l"]
         systemd_folder = "/usr/lib/systemd/system/"
         systemd = "/usr/bin/systemctl"
         kernel_versions = _kernel_versions_redhat()
     elif __grains__.get("os_family") == NILRT_FAMILY_NAME:
-        cmd_pkg_query = "opkg files "
+        cmd_pkg_query = ["opkg", "files"]
         systemd = ""
         kernel_versions = _kernel_versions_nilrt()
     else:
@@ -613,8 +612,8 @@ def restartcheck(ignorelist=None, blacklist=None, excludepid=None, **kwargs):
 
     for package in packages:
         _check_timeout(start_time, timeout)
-        cmd = cmd_pkg_query + package
-        cmd = shlex.split(cmd)
+        cmd = cmd_pkg_query[:]
+        cmd.append(package)
         paths = subprocess.Popen(cmd, stdout=subprocess.PIPE)
 
         while True:

salt/modules/state.py

@@ -425,7 +425,7 @@ def _check_queue(queue, kwargs):
 
 def _get_initial_pillar(opts):
     return (
-        __pillar__
+        __pillar__.value()
         if __opts__.get("__cli", None) == "salt-call"
         and opts["pillarenv"] == __opts__["pillarenv"]
         else None