Unicode paths break file states with masterless minion on Vagrant/Ubuntu 14.04

[twisty7867]

I can't get the yaml_utf8 option working properly at all, specifically with file.directory or file.managed. Unicode characters do work fine for my own custom states, and some other states depending on how they're used. I assembled a simple repro example.

When using unicode in the state ID declaration, like so, vagrant provision fails while calling `salt-call --local 'state.highstate' (and the state is not applied) with the following traceback:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 5: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 225, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 69, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 236, in run
    ret = self.call()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 138, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 515, in highstate
    whitelist=kwargs.get('whitelist')
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2962, in call_highstate
    return self.state.call_high(high)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2031, in call_high
    chunks = self.compile_high_data(high)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1143, in compile_high_data
    chunks = self.order_chunks(chunks)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1082, in order_chunks
    chunks.sort(key=lambda chunk: (chunk['order'], '{0[state]}{0[name]}{0[fun]}'.format(chunk)))
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1082, in <lambda>
    chunks.sort(key=lambda chunk: (chunk['order'], '{0[state]}{0[name]}{0[fun]}'.format(chunk)))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 5: ordinal not in range(128)

Assigning a bogus ID and moving the Unicode text to - name: like so allows the state to be applied, but then, as described in #16139, the salt fails with the following traceback trying to output:

Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 225, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 69, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 246, in run
    self.opts)
  File "/usr/lib/python2.7/dist-packages/salt/output/__init__.py", line 94, in display_output
    display_data = try_printout(data, out, opts)
  File "/usr/lib/python2.7/dist-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout(out, opts)(data).rstrip()
  File "/usr/lib/python2.7/dist-packages/salt/output/highstate.py", line 80, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/dist-packages/salt/output/highstate.py", line 249, in _format_host
    hstrs.extend([sline.format(**svars) for sline in state_lines])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128)

A suggested workaround for this was to use --out pprint - which works for the file.directory state. However, it doesn't work for other states, like file.recurse (example), which fails internally:

[INFO    ] Running state [/opt/répertoire-de-test] at time 17:58:21.882238
[INFO    ] Executing state file.recurse for /opt/répertoire-de-test
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1562, in call
    **cdata['kwargs'])
  File "/usr/lib/python2.7/dist-packages/salt/states/file.py", line 2297, in recurse
    dest = os.path.join(name, relname)
  File "/usr/lib/python2.7/posixpath.py", line 80, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)

These fail in the same way whether calling vagrant provision or sudo salt-call --local 'state.highstate' from within the VM. The same errors occur when using the default Salt supplied by Vagrant (2014.7.5) or when getting v2015.2.0rc2 from git. Based on the notes in #16139, I checked the values of the LANG and LC_ALL environment variables, both of which are set to en_US.UTF-8. Based on the comments in #17227, I tried changing by box to box-cutter/ubuntu1404, but got the same results.

[twisty7867]

It seems like, for the highstate outputter errors, more fields just need sdecode(). I don't feel like I have a good enough understanding of the implications to submit a PR, but I'm running a copy of highstate.py with these changes and it has fixed some of the issues.

It does not, obviously, fix the order_chunks issue or the issues internal to the file state.

[jfindlay]

@twisty7867, I was thinking the same thing. Thanks for the report.

[jfindlay]

@twisty7867, I have fixed these errors in #28803. Are you able to test it? Thanks.

[jfindlay]

I would like to get a testcase for this based on @twisty7867's state:

# -*- coding: utf-8 -*-

lame_hack:
  cmd.run:
    - name: mkdir $DIR_TO_MAKE
    - env:
      - DIR_TO_MAKE: /opt/succès

fails:
  file.directory:
    - name: /opt/échouer

also_fails:
  file.recurse:
    - name: /opt/répertoire-de-test
    - source: salt://files/test-folder

[rallytime]

I'm going to go ahead and close this since we haven't heard back. We will gladly reopen this issue if it pops up again. Thanks!