Skip to content

ops leaking secrets if `subprocess.CalledProcessError` happens with a `secret-*` CLI command

Moderate severity GitHub Reviewed Published Jul 22, 2024 in canonical/operator • Updated Aug 1, 2024

Package

pip ops (pip)

Affected versions

>= 2.0.0, < 2.15.0

Patched versions

2.15.0

Description

Summary

The issue here is that we pass the secret content as one of the args via CLI. This issue may affect any of our charms that are using: Juju (>=3.0), Juju secrets and not correctly capturing and processing subprocess.CalledProcessError.

There are two points that may log this command, in different files:

First, if there is an error during a secret handling, there will be a subprocess.CalledProcessError, which will contain the CLI comand + all its args. This is going to be logged in any logging level. This exception, if not caught by the charm, will bubble up to the /var/log/juju/ logs and syslog journal. Now, on Ubuntu 22.04, these logs are protected with:

$ juju ssh -m controller 0 -- ls -la /var/log/juju/
total 224
drwxr-xr-x 2 syslog adm      4096 Jul 14 10:59 .
drwxrwxr-x 9 root   syslog   4096 Jul 14 10:58 ..
-rw-r----- 1 syslog adm     20124 Jul 14 11:10 audit.log
-rw-r----- 1 syslog adm    110432 Jul 14 11:10 logsink.log
-rw-r----- 1 syslog adm     80783 Jul 14 11:06 machine-0.log
-rw-r----- 1 syslog adm       766 Jul 14 11:10 machine-lock.log
-rw-r--r-- 1 root   root        0 Jul 14 10:59 slow-query.log
-rw-r----- 1 syslog adm      3732 Jul 14 11:10 unit-controller-0.log

Second, certain audit setups may log terminal commands, which would result in this command being logged with its secrets. It is unknown if this is done on ubuntu security benchmarks, such as CIS hardening.

Keep in mind these logs may be copied or even backed up. Which exposes it to more services in the user's environment (e.g. CI runs in GH - although these are dummy password generated per test only).

Passing secrets straight via CLI is not advised. Here are some ways out:

  1. Redacting: which commands and which args represent secrets are known, so they can be redacted. It would also mean capturing a subprocess.CalledProcessError, redacting its content and reissuing the same type of exception; this will not cover the case auditd is set to log CLI commands, if that is a risk
  2. Temp files: secret-add, for example, can use a secret file instead, as can be seen here. However, if ops uses a file, ops will need to be sure to correctly remove it later
  3. stdin: not sure it is accepted by secret-* commands, but generally, secrets are not shown on CLI whilst typing them; auditd may not capture that stdin

Severity Rationale

This is a CWE-532. Potentially, these secrets can lead to privilege escalation but Ubuntu default is to have logs only accessible to adm group users.

Marking this issue as "Moderate", as this report is not presenting a clear way on how to get access to the logs themselves: either getting local access to an adm group user (e.g. ubuntu) or recovering logs stored on a 3rd party service.

Details

2024-07-12T14:27:58.0175389Z unit-opensearch-3: 14:27:53 ERROR unit.opensearch/3.juju-log certificates:11: Uncaught exception while in charm code:
2024-07-12T14:27:58.0175524Z Traceback (most recent call last):
2024-07-12T14:27:58.0175957Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 3180, in _run
2024-07-12T14:27:58.0176165Z     result = subprocess.run(args, **kwargs)  # type: ignore
2024-07-12T14:27:58.0176381Z   File "/usr/lib/python3.10/subprocess.py", line 526, in run
2024-07-12T14:27:58.0176561Z     raise CalledProcessError(retcode, process.args,
2024-07-12T14:27:58.0177749Z subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-opensearch-3/secret-add', '--label', 'opensearch:app:app-admin', '--owner', 'application', 'keystore-password-ca=aUE...')' returned non-zero exit status 1.
2024-07-12T14:27:58.0177765Z
2024-07-12T14:27:58.0178023Z The above exception was the direct cause of the following exception:
2024-07-12T14:27:58.0178038Z
2024-07-12T14:27:58.0178168Z Traceback (most recent call last):
2024-07-12T14:27:58.0178588Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/./src/charm.py", line 213, in <module>
2024-07-12T14:27:58.0178724Z     main(OpenSearchOperatorCharm)
2024-07-12T14:27:58.0179129Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 548, in main
2024-07-12T14:27:58.0179237Z     manager.run()
2024-07-12T14:27:58.0179640Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 527, in run
2024-07-12T14:27:58.0179745Z     self._emit()
2024-07-12T14:27:58.0180150Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 516, in _emit
2024-07-12T14:27:58.0180359Z     _emit_charm_event(self.charm, self.dispatcher.event_name)
2024-07-12T14:27:58.0180840Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 147, in _emit_charm_event
2024-07-12T14:27:58.0180974Z     event_to_emit.emit(*args, **kwargs)
2024-07-12T14:27:58.0181537Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 348, in emit
2024-07-12T14:27:58.0181656Z     framework._emit(event)
2024-07-12T14:27:58.0182091Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 860, in _emit
2024-07-12T14:27:58.0182211Z     self._reemit(event_path)
2024-07-12T14:27:58.0182659Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 950, in _reemit
2024-07-12T14:27:58.0182770Z     custom_handler(event)
2024-07-12T14:27:58.0183492Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/tls_certificates_interface/v3/tls_certificates.py", line 1811, in _on_relation_changed
2024-07-12T14:27:58.0183743Z     self.on.certificate_available.emit(
2024-07-12T14:27:58.0184173Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 348, in emit
2024-07-12T14:27:58.0184291Z     framework._emit(event)
2024-07-12T14:27:58.0184717Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 860, in _emit
2024-07-12T14:27:58.0184887Z     self._reemit(event_path)
2024-07-12T14:27:58.0186171Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 950, in _reemit
2024-07-12T14:27:58.0186306Z     custom_handler(event)
2024-07-12T14:27:58.0187043Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 209, in _on_certificate_available
2024-07-12T14:27:58.0187320Z     self.store_new_ca(self.charm.secrets.get_object(scope, cert_type.val))
2024-07-12T14:27:58.0187942Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 444, in store_new_ca
2024-07-12T14:27:58.0188242Z     self._create_keystore_pwd_if_not_exists(Scope.APP, CertType.APP_ADMIN, "ca")
2024-07-12T14:27:58.0188981Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 432, in _create_keystore_pwd_if_not_exists
2024-07-12T14:27:58.0189119Z     self.charm.secrets.put_object(
2024-07-12T14:27:58.0189738Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line 359, in put_object
2024-07-12T14:27:58.0189944Z     self._add_or_update_juju_secret(scope, key, value, merge)
2024-07-12T14:27:58.0190652Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line 272, in _add_or_update_juju_secret
2024-07-12T14:27:58.0190815Z     return self._add_juju_secret(scope, key, value)
2024-07-12T14:27:58.0191462Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line 228, in _add_juju_secret
2024-07-12T14:27:58.0191660Z     secret = scope_obj.add_secret(safe_value, label=label)
2024-07-12T14:27:58.0192110Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 477, in add_secret
2024-07-12T14:27:58.0192241Z     id = self._backend.secret_add(
2024-07-12T14:27:58.0192686Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 3624, in secret_add
2024-07-12T14:27:58.0192948Z     result = self._run('secret-add', *args, return_output=True)
2024-07-12T14:27:58.0193356Z   File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 3182, in _run
2024-07-12T14:27:58.0193489Z     raise ModelError(e.stderr) from e
2024-07-12T14:27:58.0193685Z ops.model.ModelError: ERROR this unit is not the leader
2024-07-12T14:27:58.0193692Z

From CI: https://github.com/canonical/opensearch-operator/actions/runs/9908987369/job/27376377521?pr=364

PoC

  1. Deploy anything with juju
  2. Run a dummy secret-add call that will fail
  3. See the uncaught subprocess error

Impact

Juju secrets are generally composed of private keys, passwords, etc; generally valuable credentials that, if leaked, will likely allow an attacker to get privileged access to its target or other targets in the environment.

References

@benhoyt benhoyt published to canonical/operator Jul 22, 2024
Published by the National Vulnerability Database Jul 22, 2024
Published to the GitHub Advisory Database Jul 22, 2024
Reviewed Jul 22, 2024
Last updated Aug 1, 2024

Severity

Moderate

CVSS overall score

This score calculates overall vulnerability severity from 0 to 10 and is based on the Common Vulnerability Scoring System (CVSS).
/ 10

CVSS v4 base metrics

Exploitability Metrics
Attack Vector Network
Attack Complexity Low
Attack Requirements None
Privileges Required High
User interaction None
Vulnerable System Impact Metrics
Confidentiality High
Integrity None
Availability None
Subsequent System Impact Metrics
Confidentiality None
Integrity None
Availability None

CVSS v4 base metrics

Exploitability Metrics
Attack Vector: This metric reflects the context by which vulnerability exploitation is possible. This metric value (and consequently the resulting severity) will be larger the more remote (logically, and physically) an attacker can be in order to exploit the vulnerable system. The assumption is that the number of potential attackers for a vulnerability that could be exploited from across a network is larger than the number of potential attackers that could exploit a vulnerability requiring physical access to a device, and therefore warrants a greater severity.
Attack Complexity: This metric captures measurable actions that must be taken by the attacker to actively evade or circumvent existing built-in security-enhancing conditions in order to obtain a working exploit. These are conditions whose primary purpose is to increase security and/or increase exploit engineering complexity. A vulnerability exploitable without a target-specific variable has a lower complexity than a vulnerability that would require non-trivial customization. This metric is meant to capture security mechanisms utilized by the vulnerable system.
Attack Requirements: This metric captures the prerequisite deployment and execution conditions or variables of the vulnerable system that enable the attack. These differ from security-enhancing techniques/technologies (ref Attack Complexity) as the primary purpose of these conditions is not to explicitly mitigate attacks, but rather, emerge naturally as a consequence of the deployment and execution of the vulnerable system.
Privileges Required: This metric describes the level of privileges an attacker must possess prior to successfully exploiting the vulnerability. The method by which the attacker obtains privileged credentials prior to the attack (e.g., free trial accounts), is outside the scope of this metric. Generally, self-service provisioned accounts do not constitute a privilege requirement if the attacker can grant themselves privileges as part of the attack.
User interaction: This metric captures the requirement for a human user, other than the attacker, to participate in the successful compromise of the vulnerable system. This metric determines whether the vulnerability can be exploited solely at the will of the attacker, or whether a separate user (or user-initiated process) must participate in some manner.
Vulnerable System Impact Metrics
Confidentiality: This metric measures the impact to the confidentiality of the information managed by the VULNERABLE SYSTEM due to a successfully exploited vulnerability. Confidentiality refers to limiting information access and disclosure to only authorized users, as well as preventing access by, or disclosure to, unauthorized ones.
Integrity: This metric measures the impact to integrity of a successfully exploited vulnerability. Integrity refers to the trustworthiness and veracity of information. Integrity of the VULNERABLE SYSTEM is impacted when an attacker makes unauthorized modification of system data. Integrity is also impacted when a system user can repudiate critical actions taken in the context of the system (e.g. due to insufficient logging).
Availability: This metric measures the impact to the availability of the VULNERABLE SYSTEM resulting from a successfully exploited vulnerability. While the Confidentiality and Integrity impact metrics apply to the loss of confidentiality or integrity of data (e.g., information, files) used by the system, this metric refers to the loss of availability of the impacted system itself, such as a networked service (e.g., web, database, email). Since availability refers to the accessibility of information resources, attacks that consume network bandwidth, processor cycles, or disk space all impact the availability of a system.
Subsequent System Impact Metrics
Confidentiality: This metric measures the impact to the confidentiality of the information managed by the SUBSEQUENT SYSTEM due to a successfully exploited vulnerability. Confidentiality refers to limiting information access and disclosure to only authorized users, as well as preventing access by, or disclosure to, unauthorized ones.
Integrity: This metric measures the impact to integrity of a successfully exploited vulnerability. Integrity refers to the trustworthiness and veracity of information. Integrity of the SUBSEQUENT SYSTEM is impacted when an attacker makes unauthorized modification of system data. Integrity is also impacted when a system user can repudiate critical actions taken in the context of the system (e.g. due to insufficient logging).
Availability: This metric measures the impact to the availability of the SUBSEQUENT SYSTEM resulting from a successfully exploited vulnerability. While the Confidentiality and Integrity impact metrics apply to the loss of confidentiality or integrity of data (e.g., information, files) used by the system, this metric refers to the loss of availability of the impacted system itself, such as a networked service (e.g., web, database, email). Since availability refers to the accessibility of information resources, attacks that consume network bandwidth, processor cycles, or disk space all impact the availability of a system.
CVSS:4.0/AV:N/AC:L/AT:N/PR:H/UI:N/VC:H/VI:N/VA:N/SC:N/SI:N/SA:N

EPSS score

0.043%
(11th percentile)

Weaknesses

CVE ID

CVE-2024-41129

GHSA ID

GHSA-hcmv-jmqh-fjgm

Source code

Credits

Loading Checking history
See something to contribute? Suggest improvements for this vulnerability.