Fixed return and logging of password-like props in clear text

Details:

* Increased minimum version of zhmcclient to 1.8.2 to pick up fixes for no
  longer logging password-like properties in clear-text.

* Fixed that all password-like input parameters that were written in clear text
  to the module entry log are now blanked out. This affected the following
  modules: zhmc_ldap_server_definition, zhmc_lpar, zhmc_partition, zhmc_user.

* Fixed that all password-like input parameters that were added to the
  module return value in clear text for 'state' values that created or updated
  the resource are now removed from the return value. This affected the
  following modules: zhmc_ldap_server_definition, zhmc_lpar, zhmc_partition.

* The 'hmc_auth' input parameter is no longer completely removed from the
  module entry log, but instead its sensitive items 'password' and 'session_id'
  are now blanked out.

* In support of the above, added common functions blanked_params(),
  blanked_dict() and removed_dict(). Added unit tests for these new functions.

* Improved the end2end tests for the affected modules to check that the module
  output does not contain the password-like properties.

Signed-off-by: Andreas Maier <maiera@de.ibm.com>

docs/source/modules/zhmc_ldap_server_definition.rst

@@ -228,7 +228,7 @@ ldap_server_definition
     | **type**: str
 
   {property}
-    Additional properties of the LDAP Server Definition, as described in the data model of the 'LDAP Server Definition' object in the :ref:`HMC API <HMC API>` book. The property names have hyphens (-) as described in that book.
+    Additional properties of the LDAP Server Definition, as described in the data model of the 'LDAP Server Definition' object in the :ref:`HMC API <HMC API>` book. Write-only properties in the data model are not included. The property names have hyphens (-) as described in that book.
 
     | **type**: raw
 

docs/source/modules/zhmc_lpar.rst

@@ -560,7 +560,7 @@ lpar
     | **type**: str
 
   {property}
-    Additional properties of the LPAR, as described in the data model of the 'Logical Partition' object in the :ref:`HMC API <HMC API>` book. The property names have hyphens (-) as described in that book.
+    Additional properties of the LPAR, as described in the data model of the 'Logical Partition' object in the :ref:`HMC API <HMC API>` book. Write-only properties in the data model are not included. The property names have hyphens (-) as described in that book.
 
     | **type**: raw
 

docs/source/modules/zhmc_partition.rst

@@ -515,7 +515,7 @@ partition
     | **type**: str
 
   {property}
-    Additional properties of the partition, as described in the data model of the 'Partition' object in the :ref:`HMC API <HMC API>` book. The property names have hyphens (-) as described in that book.
+    Additional properties of the partition, as described in the data model of the 'Partition' object in the :ref:`HMC API <HMC API>` book. Write-only properties in the data model are not included. The property names have hyphens (-) as described in that book.
 
     | **type**: raw
 

docs/source/modules/zhmc_user.rst

@@ -277,7 +277,7 @@ user
     | **type**: str
 
   {property}
-    Additional properties of the user, as described in the data model of the 'User' object in the :ref:`HMC API <HMC API>` book. The property names have hyphens (-) as described in that book.
+    Additional properties of the user, as described in the data model of the 'User' object in the :ref:`HMC API <HMC API>` book. Write-only properties in the data model are not included. The property names have hyphens (-) as described in that book.
 
     | **type**: raw
 
@@ -348,7 +348,7 @@ user
     | **type**: dict
 
     {property}
-      Properties of the LDAP server definition, as described in the data model of the 'LDAP Server Definition' object in the :ref:`HMC API <HMC API>` book. The property names have hyphens (-) as described in that book.
+      Properties of the LDAP server definition, as described in the data model of the 'LDAP Server Definition' object in the :ref:`HMC API <HMC API>` book. Write-only properties in the data model are not included. The property names have hyphens (-) as described in that book.
 
       | **type**: raw
 

docs/source/release_notes.rst

@@ -33,6 +33,17 @@ Availability: `AutomationHub`_, `Galaxy`_, `GitHub`_
 
 **Bug fixes:**
 
+* Increased zhmcclient version to 1.18.2 to pick up fixes. (issue #1074)
+
+* Fixed that all password-like input parameters that were written in clear text
+  to the module entry log are now blanked out. This affected the following
+  modules: zhmc_ldap_server_definition, zhmc_lpar, zhmc_partition, zhmc_user.
+
+* Fixed that all password-like input parameters that were added to the
+  module return value in clear text for 'state' values that created or updated
+  the resource are now removed from the return value. This affected the
+  following modules: zhmc_ldap_server_definition, zhmc_lpar, zhmc_partition.
+
 * Docs: Ignored www.ansible.com in linkcheck, because it occasionally times out.
 
 **Enhancements:**
@@ -70,6 +81,10 @@ Availability: `AutomationHub`_, `Galaxy`_, `GitHub`_
 
 * Support for ansible-core 2.18, by adding an ignore file for the sanity tests.
 
+* The 'hmc_auth' input parameter is no longer completely removed from the
+  module entry log, but instead its sensitive items 'password' and 'session_id'
+  are now blanked out.
+
 
 Version 1.9.1
 -------------

minimum-constraints-develop.txt

@@ -122,7 +122,7 @@ pip-check-reqs==2.5.3; python_version >= '3.12'
 # Indirect dependencies for development that are not in requirements-develop.txt
 
 alabaster==0.7.9
-attrs==19.2.0
+attrs==22.2.0
 bleach==3.3.0
 certifi==2024.07.04
 chardet==3.0.3

minimum-constraints-install.txt

@@ -38,7 +38,7 @@ requests==2.32.2
 
 pytz==2019.1
 
-zhmcclient==1.17.0
+zhmcclient==1.18.2
 
 
 # Indirect dependencies for install that are needed for some reason (must be consistent with requirements.txt)
@@ -54,10 +54,10 @@ cryptography==43.0.1
 importlib-metadata==4.8.3
 
 packaging==22.0
-PyYAML==6.0.1
+PyYAML==6.0.2
 
 python-dateutil==2.8.2
-jsonschema==4.10.0
+jsonschema==4.18.2
 urllib3==1.26.19
 
 

plugins/module_utils/common.py

@@ -1423,7 +1423,7 @@ def params_deepcopy(params):
     an optional '_faked_session' item with a value that cannot be copied.
 
     Parameters:
-      params (dict): Module input parameters.
+      params (dict): Module input parameters. Must not be None.
 
     Returns:
       dict: Deep copy of params, where possible.
@@ -1435,3 +1435,79 @@ def params_deepcopy(params):
         except TypeError:
             copy_params[key] = value
     return copy_params
+
+
+def blanked_params(params, blanked_properties=None):
+    """
+    Return a copy of the module input parameters, with the following items
+    blanked out:
+
+    * params['properties'][...] according to the blanked_properties list
+    * params['hmc_auth']['password']
+    * params['hmc_auth']['session_id']
+
+    Parameters:
+      params (dict): Module input parameters. Must not be None.
+      blanked_properties (Sequence): List of property names that will be
+        blanked out in the 'properties' item of the module input parameters.
+        Property names that are not in the input properties will be ignored.
+
+    Returns:
+      dict: Deep copy of the input parameters, with blanked out values.
+    """
+    # The params['properties'] dict and the params['hmc_auth'] dict in the
+    # return value will be copies of the corresponding input items, and
+    # therefore it is sufficient to make a shallow copy of params.
+    copied_params = dict(params)
+    if 'properties' in copied_params and copied_params['properties'] \
+            and blanked_properties:
+        copied_params['properties'] = \
+            blanked_dict(copied_params['properties'], blanked_properties)
+    if 'hmc_auth' in copied_params:
+        copied_params['hmc_auth'] = \
+            blanked_dict(copied_params['hmc_auth'], ['password', 'session_id'])
+    return copied_params
+
+
+def blanked_dict(properties, blanked_properties):
+    """
+    Return a shallow copy of the input properties, where the values of the
+    specified properties have been blanked out.
+
+    Parameters:
+      properties (Mapping): Input properties. Must not be None.
+      blanked_properties (Sequence): List of property names that will be
+        blanked out. Property names that are not in the input properties
+        will be ignored. Must not be None.
+
+    Returns:
+      dict: Shallow copy of the input properties, with blanked out values.
+    """
+    copied_properties = dict(properties)
+    for pname in blanked_properties:
+        if pname in copied_properties:
+            copied_properties[pname] = BLANKED_OUT
+    return copied_properties
+
+
+def removed_dict(properties, removed_properties):
+    """
+    Return a shallow copy of the input properties, where the specified
+    properties have been removed.
+
+    Parameters:
+      properties (Mapping): Input properties. Must not be None.
+      removed_properties (Sequence): List of property names that will be
+        removed. Property names that are not in the input properties
+        will be ignored. Must not be None.
+
+    Returns:
+      dict: Shallow copy of the input properties, with removed properties.
+    """
+    copied_properties = dict(properties)
+    for pname in removed_properties:
+        try:
+            del copied_properties[pname]
+        except KeyError:
+            pass
+    return copied_properties

plugins/modules/zhmc_adapter.py

@@ -350,7 +350,7 @@
 from ..module_utils.common import log_init, open_session, close_session, \
     hmc_auth_parameter, Error, ParameterError, to_unicode, \
     process_normal_property, eq_hex, missing_required_lib, \
-    common_fail_on_import_errors, parse_hmc_host  # noqa: E402
+    common_fail_on_import_errors, parse_hmc_host, blanked_params  # noqa: E402
 
 try:
     import urllib3
@@ -966,9 +966,9 @@ def main():
 
     module.params['hmc_host'] = parse_hmc_host(module.params['hmc_host'])
 
-    _params = dict(module.params)
-    del _params['hmc_auth']
-    LOGGER.debug("Module entry: params: %r", _params)
+    if LOGGER.isEnabledFor(logging.DEBUG):
+        LOGGER.debug("Module entry: params: %r",
+                     blanked_params(module.params))
 
     try:
 

plugins/modules/zhmc_adapter_list.py

@@ -279,7 +279,7 @@
 
 from ..module_utils.common import log_init, open_session, close_session, \
     hmc_auth_parameter, Error, ParameterError, \
-    missing_required_lib, parse_hmc_host  # noqa: E402
+    missing_required_lib, parse_hmc_host, blanked_params  # noqa: E402
 
 try:
     import urllib3
@@ -474,9 +474,9 @@ def main():
 
     module.params['hmc_host'] = parse_hmc_host(module.params['hmc_host'])
 
-    _params = dict(module.params)
-    del _params['hmc_auth']
-    LOGGER.debug("Module entry: params: %r", _params)
+    if LOGGER.isEnabledFor(logging.DEBUG):
+        LOGGER.debug("Module entry: params: %r",
+                     blanked_params(module.params))
 
     changed = False
     try:

plugins/modules/zhmc_console.py

@@ -237,7 +237,7 @@
 
 from ..module_utils.common import log_init, open_session, close_session, \
     hmc_auth_parameter, Error, ParameterError, missing_required_lib, \
-    common_fail_on_import_errors, parse_hmc_host  # noqa: E402
+    common_fail_on_import_errors, parse_hmc_host, blanked_params  # noqa: E402
 
 try:
     import urllib3
@@ -413,9 +413,9 @@ def main():
 
     module.params['hmc_host'] = parse_hmc_host(module.params['hmc_host'])
 
-    _params = dict(module.params)
-    del _params['hmc_auth']
-    LOGGER.debug("Module entry: params: %r", _params)
+    if LOGGER.isEnabledFor(logging.DEBUG):
+        LOGGER.debug("Module entry: params: %r",
+                     blanked_params(module.params))
 
     try:
 

plugins/modules/zhmc_cpc.py

@@ -416,7 +416,8 @@
 from ..module_utils.common import log_init, open_session, close_session, \
     hmc_auth_parameter, Error, StatusError, ParameterError, to_unicode, \
     process_normal_property, missing_required_lib, \
-    common_fail_on_import_errors, pull_properties, parse_hmc_host  # noqa: E402
+    common_fail_on_import_errors, pull_properties, parse_hmc_host, \
+    blanked_params  # noqa: E402
 
 try:
     import urllib3
@@ -899,9 +900,9 @@ def main():
 
     module.params['hmc_host'] = parse_hmc_host(module.params['hmc_host'])
 
-    _params = dict(module.params)
-    del _params['hmc_auth']
-    LOGGER.debug("Module entry: params: %r", _params)
+    if LOGGER.isEnabledFor(logging.DEBUG):
+        LOGGER.debug("Module entry: params: %r",
+                     blanked_params(module.params))
 
     try:
 

plugins/modules/zhmc_cpc_capacity.py

@@ -451,7 +451,7 @@
 from ..module_utils.common import log_init, open_session, close_session, \
     hmc_auth_parameter, Error, missing_required_lib, \
     common_fail_on_import_errors, parse_hmc_host, \
-    underscore_properties  # noqa: E402
+    underscore_properties, blanked_params  # noqa: E402
 
 try:
     import urllib3
@@ -835,9 +835,9 @@ def main():
 
     module.params['hmc_host'] = parse_hmc_host(module.params['hmc_host'])
 
-    _params = dict(module.params)
-    del _params['hmc_auth']
-    LOGGER.debug("Module entry: params: %r", _params)
+    if LOGGER.isEnabledFor(logging.DEBUG):
+        LOGGER.debug("Module entry: params: %r",
+                     blanked_params(module.params))
 
     try:
 

plugins/modules/zhmc_cpc_list.py

@@ -222,7 +222,7 @@
 
 from ..module_utils.common import log_init, open_session, close_session, \
     hmc_auth_parameter, Error, missing_required_lib, \
-    common_fail_on_import_errors, parse_hmc_host  # noqa: E402
+    common_fail_on_import_errors, parse_hmc_host, blanked_params  # noqa: E402
 
 try:
     import urllib3
@@ -330,9 +330,9 @@ def main():
 
     module.params['hmc_host'] = parse_hmc_host(module.params['hmc_host'])
 
-    _params = dict(module.params)
-    del _params['hmc_auth']
-    LOGGER.debug("Module entry: params: %r", _params)
+    if LOGGER.isEnabledFor(logging.DEBUG):
+        LOGGER.debug("Module entry: params: %r",
+                     blanked_params(module.params))
 
     changed = False
     try:

plugins/modules/zhmc_crypto_attachment.py

@@ -374,7 +374,7 @@
 
 from ..module_utils.common import log_init, open_session, close_session, \
     hmc_auth_parameter, Error, ParameterError, missing_required_lib, \
-    common_fail_on_import_errors, parse_hmc_host  # noqa: E402
+    common_fail_on_import_errors, parse_hmc_host, blanked_params  # noqa: E402
 
 
 try:
@@ -1091,9 +1091,9 @@ def main():
 
     module.params['hmc_host'] = parse_hmc_host(module.params['hmc_host'])
 
-    _params = dict(module.params)
-    del _params['hmc_auth']
-    LOGGER.debug("Module entry: params: %r", _params)
+    if LOGGER.isEnabledFor(logging.DEBUG):
+        LOGGER.debug("Module entry: params: %r",
+                     blanked_params(module.params))
 
     try:
 

plugins/modules/zhmc_hba.py

@@ -246,7 +246,7 @@
 from ..module_utils.common import log_init, open_session, close_session, \
     hmc_auth_parameter, Error, ParameterError, wait_for_transition_completion, \
     eq_hex, to_unicode, process_normal_property, missing_required_lib, \
-    common_fail_on_import_errors, parse_hmc_host  # noqa: E402
+    common_fail_on_import_errors, parse_hmc_host, blanked_params  # noqa: E402
 
 try:
     import urllib3
@@ -627,9 +627,9 @@ def main():
 
     module.params['hmc_host'] = parse_hmc_host(module.params['hmc_host'])
 
-    _params = dict(module.params)
-    del _params['hmc_auth']
-    LOGGER.debug("Module entry: params: %r", _params)
+    if LOGGER.isEnabledFor(logging.DEBUG):
+        LOGGER.debug("Module entry: params: %r",
+                     blanked_params(module.params))
 
     try: