Unhandled exception using requests-2.28.1

[sguarin]

SUMMARY

When running a simple test of lookup plugin using requests==2.27.1 works as expected.
But using last requests: 2.28.1 I got an unhandled exception.

ISSUE TYPE

• Bug Report

COMPONENT NAME

lookup hashi_vault plugin

ANSIBLE VERSION

ansible [core 2.13.1]
  config file = /home/user/src/test-hashi/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/user/RedHat/src/security-guild/vault/vault-sa-rotate-playbooks/.python/lib/python3.10/site-packages/ansible
  ansible collection location = /home/user/src/test-hashi/collections
  executable location = /home/user/src/test-hashi/.python/bin/ansible
  python version = 3.10.7 (main, Sep  7 2022, 00:00:00) [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)]
  jinja version = 3.0.1
  libyaml = True

COLLECTION VERSION

Collection            Version
--------------------- -------
awx.awx               21.4.0 
community.general     4.8.1  
community.hashi_vault 3.3.1  

Tested with community.hashi_vault 3.2.0 and 3.3.1

CONFIGURATION

COLLECTIONS_PATHS(/home/user/src/test-hashi/ansible.cfg) = ['/home/user/src/test-hashi/collections']
DEFAULT_HOST_LIST(/home/user/src/test-hashi/ansible.cfg) = ['/home/user/src/test-hashi/inventory']
DEFAULT_ROLES_PATH(/home/user/src/test-hashi/ansible.cfg) = ['/home/user/src/test-hashi/roles']

OS / ENVIRONMENT

STEPS TO REPRODUCE

Tested with various ansible version and the issue seems to work with:

pip install 'requests==2.27.1'
but not with:
pip install 'requests==2.28.1'

    - name: debug
      ansible.builtin.debug:
        msg: "{{ lookup('community.hashi_vault.hashi_vault',
             'kv2/data/path/gitlab_token:token',
             namespace='ns',
             token='xxx',
             url='https://x.x.x.x:8200'
             ) }}"

EXPECTED RESULTS

Work retrieving the kv content as it works with previous requests library version.

ACTUAL RESULTS

TASK [debug] *******************************************************************************************************************************************************************************************************************************
task path: /home/user/src/test-hashi/test-hashi.yml:21
Loading collection community.hashi_vault from /home/user/src/test-hashi/collections/ansible_collections/community/hashi_vault
Loading collection community.hashi_vault from /home/user/src/test-hashi/collections/ansible_collections/community/hashi_vault
[DEPRECATION WARNING]: The default value for 'token_validate' will change from True to False. This feature will be removed from community.hashi_vault in version 4.0.0. Deprecation warnings can be disabled by setting 
deprecation_warnings=False in ansible.cfg.
exception during Jinja2 execution: Traceback (most recent call last):
  File "/home/user/src/test-hashi/.python/lib/python3.10/site-packages/ansible/template/__init__.py", line 962, in _lookup
    ran = instance.run(loop_terms, variables=self._available_variables, **kwargs)
  File "/home/user/src/test-hashi/collections/ansible_collections/community/hashi_vault/plugins/lookup/hashi_vault.py", line 273, in run
    self.authenticator.authenticate(self.client)
  File "/home/user/src/test-hashi/collections/ansible_collections/community/hashi_vault/plugins/module_utils/_authenticator.py", line 95, in authenticate
    return method.authenticate(*args, **kwargs)
  File "/home/user/src/test-hashi/collections/ansible_collections/community/hashi_vault/plugins/module_utils/_auth_method_token.py", line 104, in authenticate
    response = client.auth.token.lookup_self()
  File "/home/user/src/test-hashi/.python/lib/python3.10/site-packages/hvac/api/auth_methods/token.py", line 284, in lookup_self
    return self._adapter.get(
  File "/home/user/src/test-hashi/.python/lib/python3.10/site-packages/hvac/adapters.py", line 112, in get
    return self.request("get", url, **kwargs)
  File "/home/user/src/test-hashi/.python/lib/python3.10/site-packages/hvac/adapters.py", line 356, in request
    response = super().request(*args, **kwargs)
  File "/home/user/src/test-hashi/.python/lib/python3.10/site-packages/hvac/adapters.py", line 305, in request
    response = self.session.request(
  File "/home/user/src/test-hashi/.python/lib/python3.10/site-packages/requests/sessions.py", line 573, in request
    prep = self.prepare_request(req)
  File "/home/user/src/test-hashi/.python/lib/python3.10/site-packages/requests/sessions.py", line 484, in prepare_request
    p.prepare(
  File "/home/user/src/test-hashi/.python/lib/python3.10/site-packages/requests/models.py", line 369, in prepare
    self.prepare_headers(headers)
  File "/home/user/src/test-hashi/.python/lib/python3.10/site-packages/requests/models.py", line 491, in prepare_headers
    check_header_validity(header)
  File "/home/user/src/test-hashi/.python/lib/python3.10/site-packages/requests/utils.py", line 1037, in check_header_validity
    raise InvalidHeader(
requests.exceptions.InvalidHeader: Header part ('xxx') from {'X-Vault-Token': 'xxx'} must be of type str or bytes, not <class 'ansible.utils.unsafe_proxy.AnsibleUnsafeText'>
fatal: [host_prod]: FAILED! => {
    "msg": "An unhandled exception occurred while running the lookup plugin 'community.hashi_vault.hashi_vault'. Error was a <class 'requests.exceptions.InvalidHeader'>, original message: Header part ('xxx') from {'X-Vault-Token': 'xxx'} must be of type str or bytes, not <class 'ansible.utils.unsafe_proxy.AnsibleUnsafeText'>. Header part ('xxx') from {'X-Vault-Token': 'xxx'} must be of type str or bytes, not <class 'ansible.utils.unsafe_proxy.AnsibleUnsafeText'>"
}

[briantist]

Thank you for submitting this!

This is most likely the same issue as seen in:

• Unable to use vault_kv2_get lookup with python requests version > 2.27.1 #294
• token must be of type str or bytes, not <class 'ansible.utils.unsafe_proxy.AnsibleUnsafeText'> #289

And fixed in:

• Fix "Unsafe" token values that originate from lookups #291

This should be a matter of similarly forcing the type for this parameter to str/bytes.

---

I will be busy with AnsibleFest for the next week+ so it may take some time for me to get to this, but it should be a straightforward fix.

[briantist]

@sguarin @tobiicerb I believe I have a fix:

• Fix namespace value with newer requests #310

Would you be able to try it out by any chance? Use of namespace is a Vault enterprise feature so even though we have unit tests, I cannot test it with integration.

You can install the collection from the fix branch like this:

ansible-galaxy collection install git+https://github.com/briantist/community.hashi_vault.git,unsafe-namespace

[tobiicerb]

@briantist Hi, I tried it out and it is working perfectly. I installed the collection like you suggested and ran the following playbook successfully:

- hosts: localhost
  vars:
    ansible_hashi_vault_namespace: namespacename
    secrettest: "{{ lookup('community.hashi_vault.vault_kv2_get', 'secretpath', engine_mount_point='kv').secret.secretname }}"
  tasks:
    - name: Debug
      ansible.builtin.debug:
        msg: "{{ secrettest }}"

Thank you so much for this very quick response and fix!

[briantist]

@tobiicerb thanks for trying that out! Could you confirm that the same test playbook you used does indeed fail on the current release?

I ask because as it's written here (I assume it's somewhat redacted), with a literal string namespace name, I would have expected it not to fail, because it should only fail if the namespace value is "unsafe" (usually this is the case when the value was the result of a lookup plugin).

One way to force that without having to use a lookup, is to mark it unsafe in YAML:

- hosts: localhost
  vars:
    ansible_hashi_vault_namespace: !unsafe namespacename
    secrettest: "{{ lookup('community.hashi_vault.vault_kv2_get', 'secretpath', engine_mount_point='kv').secret.secretname }}"
  tasks:
    - name: Debug
      ansible.builtin.debug:
        msg: "{{ secrettest }}"

If you have an example that actually uses a static string and triggers the issues, or your namespace string is not static but comes from something other than a lookup, I would really appreciate knowing about that!

[briantist]

@tobiicerb can you confirm the above?

[tobiicerb]

@briantist I was not certain what you meant with "current" release, so I tried out the version on galaxy first:

Here when it is marked as unsafe the error is Header part ('Namespace') from {'X-Vault_Namespace:' 'Namespace'} must be of type str or bytes, not <class 'ansible.untils.unsafe_proxy.AnsibleUnsafeText'>"

Not marked as unsafe the error is Header part ('Namespace') from {'X-Vault-Namespace': 'Namespace'} must be of type str or bytes, not <class 'ansible.parsing.yaml.objects.AnsibleUnicode'>"

Afterwards I tried your unsafe-namespace branch:

Here when it is marked as unsafe there is no error.

Not marked as unsafe there is also no error.

[briantist]

Thank you @tobiicerb that's very interesting, what is your output from ansible --version?

[tobiicerb]

@briantist ansible [core 2.13.5], python version = 3.8.10

[briantist]

@sguarin @tobiicerb this is now released in version 3.4.0!

[tobiicerb]

@briantist I meant to write this earlier. Awesome work!