python3: broken pillar cache: yaml: TypeError: a bytes-like object is required, not 'str'

[disaster123]

Description of Issue

Since Upgrading from 2017.7 py2 to 2019.2 py3 i can't issue any salt-call - not even test.ping

All result in:

TypeError: a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 431, in salt_call
    client.run()
  File "/usr/lib/python3/dist-packages/salt/cli/call.py", line 47, in run
    caller = salt.cli.caller.Caller.factory(self.config)
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 80, in factory
    return ZeroMQCaller(opts, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 332, in __init__
    super(ZeroMQCaller, self).__init__(opts)
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 106, in __init__
    self.minion = salt.minion.SMinion(opts)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 822, in __init__
    salt.utils.yaml.safe_dump(cache_top, fp_)
  File "/usr/lib/python3/dist-packages/salt/utils/yamldumper.py", line 129, in safe_dump
    return yaml.dump(data, stream, Dumper=SafeOrderedDumper, **kwargs)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 200, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 188, in dump_all
    dumper.represent(data)
  File "/usr/lib/python3/dist-packages/yaml/representer.py", line 27, in represent
    self.serialize(node)
  File "ext/_yaml.pyx", line 1350, in _yaml.CEmitter.serialize (ext/_yaml.c:18773)
  File "ext/_yaml.pyx", line 1525, in _yaml.output_handler (ext/_yaml.c:21549)
TypeError: a bytes-like object is required, not 'str'

Steps to Reproduce Issue

just call salt-call test.ping

Versions Report

salt-call --versions-report

Salt Version:
Salt: 2019.2.0

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.5.3
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.9.4
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.4.3
ZMQ: 4.2.1

System Versions:
dist: debian 9.9
locale: UTF-8
machine: x86_64
release: 4.19.45+26-ph
system: Linux
version: debian 9.9

[disaster123]

I'm able to fix / workaround this by replacing:

821 with salt.utils.files.fopen(ptop, 'wb') as fp_:
with
821 with salt.utils.files.fopen(ptop, 'w') as fp_:

and
825 with salt.utils.files.fopen(cache_sls, 'wb') as fp_:
with
825 with salt.utils.files.fopen(cache_sls, 'w') as fp_:

in minion.py

[disaster123]

this one needs: minion_pillar_cache: True

[dwoz]

@disaster123 Does the minion run fine under python 3 if you remove the cache file?

[disaster123]

No

[dwoz]

@disaster123 Thanks, I've confirmed this is an issue.

>>> a
'abcdé'
>>> with open('test', 'wb') as fp:
...     salt.utils.yaml.safe_dump(a, fp)
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/dwoz/src/salt-forge/neon-tests/salt/salt/utils/yamldumper.py", line 138, in safe_dump
    return yaml.dump(data, stream, Dumper=SafeOrderedDumper, **kwargs)
  File "/home/dwoz/src/salt-forge/neon-tests/venv3/lib/python3.5/site-packages/yaml/__init__.py", line 200, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "/home/dwoz/src/salt-forge/neon-tests/venv3/lib/python3.5/site-packages/yaml/__init__.py", line 188, in dump_all
    dumper.represent(data)
  File "/home/dwoz/src/salt-forge/neon-tests/venv3/lib/python3.5/site-packages/yaml/representer.py", line 27, in represent
    self.serialize(node)
  File "/home/dwoz/src/salt-forge/neon-tests/venv3/lib/python3.5/site-packages/yaml/serializer.py", line 54, in serialize
    self.serialize_node(node, None, None)
  File "/home/dwoz/src/salt-forge/neon-tests/venv3/lib/python3.5/site-packages/yaml/serializer.py", line 90, in serialize_node
    style=node.style))
  File "/home/dwoz/src/salt-forge/neon-tests/venv3/lib/python3.5/site-packages/yaml/emitter.py", line 115, in emit
    self.state()
  File "/home/dwoz/src/salt-forge/neon-tests/venv3/lib/python3.5/site-packages/yaml/emitter.py", line 228, in expect_document_root
    self.expect_node(root=True)
  File "/home/dwoz/src/salt-forge/neon-tests/venv3/lib/python3.5/site-packages/yaml/emitter.py", line 244, in expect_node
    self.expect_scalar()
  File "/home/dwoz/src/salt-forge/neon-tests/venv3/lib/python3.5/site-packages/yaml/emitter.py", line 268, in expect_scalar
    self.process_scalar()
  File "/home/dwoz/src/salt-forge/neon-tests/venv3/lib/python3.5/site-packages/yaml/emitter.py", line 533, in process_scalar
    self.write_plain(self.analysis.scalar, split)
  File "/home/dwoz/src/salt-forge/neon-tests/venv3/lib/python3.5/site-packages/yaml/emitter.py", line 1131, in write_plain
    self.stream.write(data)
TypeError: a bytes-like object is required, not 'str'

[dwoz]

@disaster123 Does the minion run fine without this.

[disaster123]

with my patch it runs fine - yes

[disaster123]

can this please be get fixed for 2019.2.1 ?

[disaster123]

Why is this still unfixed in 2019.2.2?

[disaster123]

@sagetherage salt 2019 is unusable without a fix - why is this a todo in sodium?

[sagetherage]

That is the project of work, and is in "Considering" as I need to have a planning session with the team to understand deeper how, when, what, etc. to get this fixed, not to state that it will be only fixed in Sodium or Master, sorry for the confusion. It is my attempt to move it along.

[disaster123]

ah ok thx

[twangboy]

Looks like this was fixed in #56143. It was part of the 3000.1 release. Could you verify with 3000.2 please?