Beet crashing when importing unicode paths

[theof]

Problem

Any folder with at least one unicode character could not be imported by beets: see following traceback.

Running this command:

$ mkdir 'ithéof/' && beet -vv import 'ithéof/'

Led to this problem:

user configuration: /home/theophile/.config/beets/config.yaml
data directory: /home/theophile/.config/beets
plugin paths:
Sending event: pluginload
library database: /home/theophile/.config/musiclibrary.db
library directory: /data/music
Sending event: library_opened
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/beets/util/__init__.py", line 362, in bytestring_path
    return path.encode(_fsencoding())
UnicodeEncodeError: 'ascii' codec can't encode characters in position 8-9: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/beet", line 11, in <module>
    load_entry_point('beets==1.4.6', 'console_scripts', 'beet')()
  File "/usr/lib/python3.6/site-packages/beets/ui/__init__.py", line 1256, in main
    _raw_main(args)
  File "/usr/lib/python3.6/site-packages/beets/ui/__init__.py", line 1243, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/lib/python3.6/site-packages/beets/ui/commands.py", line 937, in import_func
    import_files(lib, paths, query)
  File "/usr/lib/python3.6/site-packages/beets/ui/commands.py", line 889, in import_files
    if not os.path.exists(syspath(normpath(path))):
  File "/usr/lib/python3.6/site-packages/beets/util/__init__.py", line 143, in normpath
    return bytestring_path(path)
  File "/usr/lib/python3.6/site-packages/beets/util/__init__.py", line 364, in bytestring_path
    return path.encode('utf-8')
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 8-9: surrogates not allowed

Setup

• OS: Archlinux
• Python version: 3.6.4
• beets version: 1.4.6
• Turning off plugins made problem go away: no

My configuration (output of beet config) is:

directory: /data/music
library: ~/.config/musiclibrary.db

import:
    move: yes

Locale is poorly setup on my system:

> locale
LANG=
LC_CTYPE=
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
> mkdir 'ithéof/' && ls 
'ith'$'\303\251''of'

may be similar to #2585

[sampsyo]

Wow; this is very strange! Especially because I'm unable to reproduce the problem here. Does it seem to work if you temporarily set a UTF-8 locale? (For example, export LC_ALL=en_US.UTF-8 might help you experiment.)

[theof]

To be able to reproduce, maybe you could try with env -i <command>. Anyway when setting LC variables like you said, it is working just fine.

[sampsyo]

Hello! Could you please try out #2803? I can't reproduce this on my dev machine (because macOS forces a UTF-8 locale, I think), but I believe I found the problem.

[theof]

#2803 fixed my issue. Well done on troubleshooting !

[sampsyo]

Great news! Thanks for trying it out. ✨