Duplicate Plugin: beet dup -C "md5sum {file}" doesn't calculate checksum on Win10 cmd (incl ad-hoc fix)

[Madrawn]

I was trying to check for duplicates with md5sum. And ran into several problems.

Problem

Running this command in verbose (-vv) mode:
(copied like its shown in this doc https://beets.readthedocs.io/en/stable/plugins/duplicates.html)

$ beet -vv dup -C 'md5sum {file}'

Led to this problem:

user configuration: F:\Users\yasok\AppData\Roaming\beets\config.yaml
data directory: F:\Users\yasok\AppData\Roaming\beets
plugin paths:
Sending event: pluginload
library database: F:\Users\yasok\AppData\Roaming\beets\library.db
library directory: G:\MusicNoDupes
Sending event: library_opened
Traceback (most recent call last):
  File "f:\users\yasok\anaconda3\envs\tagging\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "f:\users\yasok\anaconda3\envs\tagging\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "F:\Users\yasok\anaconda3\envs\tagging\Scripts\beet.exe\__main__.py", line 7, in <module>
  File "f:\users\yasok\anaconda3\envs\tagging\lib\site-packages\beets\ui\__init__.py", line 1291, in main
    _raw_main(args)
  File "f:\users\yasok\anaconda3\envs\tagging\lib\site-packages\beets\ui\__init__.py", line 1278, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "f:\users\yasok\anaconda3\envs\tagging\lib\site-packages\beetsplug\duplicates.py", line 152, in _dup
    keys = [k]
UnboundLocalError: local variable 'k' referenced before assignment

After debugging I noticed that the command it's trying to run is:

'md5sum

missing the "{file}", so I figured I'll try it with

$ beet -vv dup -C "md5sum {file}"

which didn't crash, but led to:

duplicates: key md5sum on item G:\MusicNoDupes\SWR3.online\Jogis Jungs\00 Jogis Jungs 026 Masern in der Schweiz.mp3 not cached:computing checksum
duplicates: failed to checksum G:\MusicNoDupes\SWR3.online\Jogis Jungs\00 Jogis Jungs 026 Masern in der Schweiz.mp3: Command 'md5sum b'G:\\MusicNoDupes\\SWR3.online\\Jogis Jungs\\00 Jogis Jungs 026 Masern in der Schweiz.mp3'' returned non-zero exit status 1.

I debugged again and realized it tries to run the command as:

md5sum b'G:\\MusicNoDupes\\SWR3.online\\Jogis Jungs\\00 Jogis Jungs 026 Masern in der Schweiz.mp3

The "b' " at the start confuses md5sum and leads to it not finding the file.

ad-hoc fix

I changed the following line:
F:\Users\yasok\anaconda3\envs\tagging\Lib\site-packages\beetsplug\duplicates.py:200
From

        args = [p.format(file=item.path) for p in shlex.split(prog)]

To

        args = [p.format(file=displayable_path(item.path)) for p in shlex.split(prog)]

Now $ beet -vv dup -C "md5sum {file}" works.

Setup

• OS: Windows 10 Pro 20H2
• beets version 1.5.0
• Python version 3.8.8
• plugins: duplicates
• Turning off plugins made problem go away (yes/no): not applicable

My configuration (output of beet config) is:

duplicates:
  tiebreak: 
    items: [bitrate]
directory: G:\MusicNoDupes
import:
    move: yes
plugins: duplicates
terminal_encoding: utf-8

[wisp3rwind]

user configuration: F:\Users\yasok\AppData\Roaming\beets\config.yaml
data directory: F:\Users\yasok\AppData\Roaming\beets
plugin paths:
Sending event: pluginload
library database: F:\Users\yasok\AppData\Roaming\beets\library.db
library directory: G:\MusicNoDupes
Sending event: library_opened
Traceback (most recent call last):
  File "f:\users\yasok\anaconda3\envs\tagging\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "f:\users\yasok\anaconda3\envs\tagging\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "F:\Users\yasok\anaconda3\envs\tagging\Scripts\beet.exe\__main__.py", line 7, in <module>
  File "f:\users\yasok\anaconda3\envs\tagging\lib\site-packages\beets\ui\__init__.py", line 1291, in main
    _raw_main(args)
  File "f:\users\yasok\anaconda3\envs\tagging\lib\site-packages\beets\ui\__init__.py", line 1278, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "f:\users\yasok\anaconda3\envs\tagging\lib\site-packages\beetsplug\duplicates.py", line 152, in _dup
    keys = [k]
UnboundLocalError: local variable 'k' referenced before assignment

Looks like the _dup function needs an early return if the query returns empy, i.e. items is an empty list.

After debugging I noticed that the command it's trying to run is:

'md5sum

missing the "{file}", so I figured I'll try it with

$ beet -vv dup -C "md5sum {file}"

which didn't crash, but led to:

The crash with single quotes seems like a Windows-specific quirk of how the shell interprets the commandline, and not an issue with beets.

duplicates: key md5sum on item G:\MusicNoDupes\SWR3.online\Jogis Jungs\00 Jogis Jungs 026 Masern in der Schweiz.mp3 not cached:computing checksum
duplicates: failed to checksum G:\MusicNoDupes\SWR3.online\Jogis Jungs\00 Jogis Jungs 026 Masern in der Schweiz.mp3: Command 'md5sum b'G:\\MusicNoDupes\\SWR3.online\\Jogis Jungs\\00 Jogis Jungs 026 Masern in der Schweiz.mp3'' returned non-zero exit status 1.

I changed the following line:
F:\Users\yasok\anaconda3\envs\tagging\Lib\site-packages\beetsplug\duplicates.py:200
From

        args = [p.format(file=item.path) for p in shlex.split(prog)]

To

        args = [p.format(file=displayable_path(item.path)) for p in shlex.split(prog)]

That may work in you particular case, but using displayable_path() here is fundamentally wrong, since it is a lossy conversion, which in general won't given the correct file name. In fact, a fix is already in the making, see #3930

[wisp3rwind]

Note that this is not a complete duplicate of #3930 and #2873 due to the crash for the empty items.