UnicodeDecodeError in states.file

[bmcorser]

I have a Runner that is calling state.single with file.recurse to copy files from master to minion. I guess the error is occuring on the minion. But I'm a bit unclear as to what could be causing it. I know the files I'm copying with file.recurse have unicode codepoints in them, but would be surprised if this was causing the issue.

minion-x:
    ----------
    file_|-/opt/deploy_|-/opt/deploy_|-recurse:
        ----------
        __run_num__:
            0
        changes:
            ----------
        comment:
            An exception occurred in this state: Traceback (most recent call last):
              File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1560, in call
                **cdata['kwargs'])
              File "/usr/lib/python2.7/dist-packages/salt/states/file.py", line 2317, in recurse
                src = u'salt://{0}'.format(fn_)
            UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 45: ordinal not in range(128)
        duration:
            8868.469
        name:
            /opt/deploy
        result:
            False
        start_time:
            15:56:11.275353

Versions match; master version:

                  Salt: 2015.5.3
                Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
                Jinja2: 2.7.2
              M2Crypto: 0.21.1
        msgpack-python: 0.3.0
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 14.0.1
                  RAET: Not Installed
                   ZMQ: 4.0.4
                  Mako: 0.9.1
               Tornado: Not Installed
 Debian source package: 2015.5.3+ds-1trusty1

Minion version:

salt-minion 2015.5.3 (Lithium)

[bmcorser]

Ok, it appears that unicode in filenames isn't supported somewhere along the way. If I make a small change to the exception that was getting raised:

diff --git a/usr/lib/python2.7/dist-packages/salt/states/file.py b/file.py
index f3bb0db..491a508 100644
--- a/usr/lib/python2.7/dist-packages/salt/states/file.py
+++ b/file.py
@@ -2313,7 +2313,10 @@ def recurse(name,
             # verify the directory perms if they are set
             manage_directory(dirname)
             vdir.add(dirname)
-        src = u'salt://{0}'.format(fn_)
+        try:
+            src = u'salt://{0}'.format(fn_)
+        except Exception as exc:
+            raise Exception(fn_)
         manage_file(dest, src)

     if include_empty:

Then I can see what's causing the barf in my output:

minion-x:
    ----------
    file_|-/opt/deploy_|-/opt/deploy_|-recurse:
        ----------
        __run_num__:
            0
        changes:
            ----------
        comment:
            An exception occurred in this state: Traceback (most recent call last):
              File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1560, in call
                **cdata['kwargs'])
              File "/usr/lib/python2.7/dist-packages/salt/states/file.py", line 2319, in recurse
                raise Exception(fn_)
            Exception: _tmp/5Lmgtm/some/path/components/pACP-ADRß2.fasta
        duration:
            8880.017
        name:
            /opt/deploy
        result:
            False
        start_time:
            16:14:19.999637

There's a ß character in the filename.

[jfindlay]

@bmcorser, thanks for the report. This should be fixed in the pull requests mentioned.

[bmcorser]

👍 Please close when you're happy

[bmcorser]

Hmmm just spotted #23271