Add ability for 'missing' plugin to show missing albums by library artists #2481

qlyoung · 2017-03-18T06:05:17Z

Passing the -a flag shows, for each artist in the library, the albums by that artist which are not in the library.

Also includes a minor enhancement to allow beets.ui.print_ to print to stderr.

Example output (truncated):

(venv) user@host ~/d/p/beets> beet missing -a
Theorem - Fallout
Theorem - Shift
Theorem - Day From Hell
Theorem - Mantra One
Theorem - Embed
Theorem - Cinder
Theorem - [untitled]
Theorem - Nano
Theorem - Recoil
Theorem - Break in at Apartment 205
Theorem - Intervals
Theorem - THX: Experiments in Synchronicity
Tinariwen - Amassakoul
Tinariwen - Aman Iman : L'eau c'est la vie
Tinariwen - Aman Iman: Water Is Life
Tinariwen - The Radio Tisdas Sessions
Tinariwen - Imidiwan: Companions
Tinariwen - Tassili
[!] No musicbrainz ID for artist she, skipping.
Carbon Based Lifeforms - Irdial
Carbon Based Lifeforms - The Path
Carbon Based Lifeforms - ALT:01
Carbon Based Lifeforms - Photosynthesis Remixes
Carbon Based Lifeforms - Twentythree
Carbon Based Lifeforms - Refuge: Original Motion Picture Soundtrack
Carbon Based Lifeforms - MOS 6581 Remixes
Carbon Based Lifeforms - Hydroponic Garden
Carbon Based Lifeforms - VLA
Carbon Based Lifeforms - Interloper
...

mention-bot · 2017-03-18T06:05:19Z

@qlyoung, thanks for your PR! By analyzing the history of the files in this pull request, we identified @sampsyo, @tomjaspers and @geigerzaehler to be potential reviewers.

Kraymer · 2017-03-18T08:17:14Z

beetsplug/missing.py

+def _normalize_caseless(text):
+    """unicode case normalization
+    """
+    return unicodedata.normalize("NFKD", text.casefold())


casefold() requires python 3.3+

fixed, anything else?

sampsyo

This looks awesome! I've actually wanted something like this for a long time. It's like an in-beets version of Muspy. Thanks for getting it started!

I made a few suggestions inline. It also looks like Travis is reporting that some tests as failed. Can you take a look and perhaps update the tests?

sampsyo · 2017-03-18T18:25:59Z

beetsplug/missing.py

+        for artist, albums in albums_by_artist.items():
+            if artist[1] is None or artist[1] == "":
+                print_("[!] No musicbrainz ID for artist {}, skipping."
+                       .format(artist[0]), stream=sys.stderr)


For these prints to stderr, you can just use our logging system instead. Use self._log.info(...).

perfect, will change

sampsyo · 2017-03-18T18:26:22Z

beetsplug/missing.py

@@ -17,6 +18,12 @@
 """
 from __future__ import division, absolute_import, print_function

+import time
+import beets
+import musicbrainzngs as m


As silly as it seems, can you please use the long "musicbrainzngs" name? This helps readability quite a bit.

sampsyo · 2017-03-18T18:28:11Z

beetsplug/missing.py

+            albums_by_artist[artist] = artistalbs
+
+        # build dict mapping artist to list of all albums
+        m.set_useragent('beets', beets.__version__, 'http://beets.io/')


Why is this necessary?

built it into a separate tool, forgot to remove for port into plugin; will remove

sampsyo · 2017-03-18T18:29:08Z

beetsplug/missing.py

+                print_("{}{}".format(prefix_miss, release))
+
+            # respect musicbrainz rate limits
+            time.sleep(1)


The library contains internal rate limiting, so we don't need extra waiting here.

will remove

sampsyo · 2017-03-18T18:30:18Z

beetsplug/missing.py

+            for release in releases:
+                missing.append(release)
+                for alb in albums:
+                    # compare by title, don't care about specific releases


Maybe using the release group ID would be more robust here? For example, that would help you distinguish between Weezer and Weezer.

good idea, was looking for something better than title but not super familiar with mb api -- will implement

sampsyo · 2017-03-18T18:31:16Z

beetsplug/missing.py

+            prefix_miss = "{} - ".format(artist[0])
+
+            for release in missing_titles:
+                print_("{}{}".format(prefix_miss, release))


This (and the previous stanza would be more readable if they just used:

for release in missing_titles: print_("{} - {}".format(artist[0], release))

agreed, had the ability to show both present and missing albums in the same output in a previous version, but this can be done with standard shell utils, so no need for it; will change

qlyoung · 2017-03-18T20:32:39Z

All requested changes made. Regarding tests, apparently they don't like stream.flush(); removing that passes them.

sampsyo

Nice; thanks! And thanks for adding documentation too. I just have a few low-level coding suggestions based on the revisions.

sampsyo · 2017-03-18T21:31:27Z

beets/ui/__init__.py

        else:
            # In our test harnesses (e.g., DummyOut), sys.stdout.buffer
            # does not exist. We instead just record the text string.
-            sys.stdout.write(txt)
+            stream.write(txt)


Great. Now that you're using the logging infrastructure, I believe these changes to print_ are unnecessary. Could you please back them out?

sampsyo · 2017-03-18T21:32:25Z

beetsplug/missing.py

@@ -106,35 +111,95 @@ def __init__(self):
        self._command.parser.add_option(
            u'-t', u'--total', dest='total', action='store_true',
            help=u'count total of missing tracks')
+        self._command.parser.add_option(
+            u'-a', u'--albums', dest='albums', action='store_true',


In other commands, such as beet ls, a similar (but different) flag is always called --album, not --albums. Can we please stick with that name for consistency? (And the same for the config option name, of course.)

sampsyo · 2017-03-18T21:34:52Z

beetsplug/missing.py

-                if count:
-                    if _missing_count(album):
-                        print_(format(album, fmt))
+    def _missing_tracks(self, lib, args):


Instead of taking a parameter called args, which is bytestrings, these two functions should probably take a query called query—the result of decargs(args). This should make the method signatures a little clearer.

sampsyo · 2017-03-18T21:35:23Z

beetsplug/missing.py

-                        print_(format(album, fmt))
+    def _missing_tracks(self, lib, args):
+        """Logic for determining missing tracks from each album in the library
+        (the usual case)


Suggested docstring that uses an imperative voice (the standard in beets):

Print a listing of tracks missing from each album in the library matching query.

sampsyo · 2017-03-18T21:36:05Z

beetsplug/missing.py

        self._command.parser.add_format_option()

    def commands(self):
        def _miss(lib, opts, args):
            self.config.set_args(opts)
-            count = self.config['count'].get()
-            total = self.config['total'].get()


Should we document the fact that count and total are ignored in album mode?

Noted for count, implemented for total

sampsyo · 2017-03-18T21:37:32Z

beetsplug/missing.py

+        """
+        albums = lib.albums(decargs(args))
+        # build dict mapping artist to list of their albums in library
+        albums_by_artist = {}


This might be a good place for a defaultdict(list).

sampsyo · 2017-03-18T21:38:40Z

beetsplug/missing.py

+        for artist, albums in albums_by_artist.items():
+            if artist[1] is None or artist[1] == "":
+                self._log.info(
+                    u"No musicbrainz ID for artist {}, skipping.",


This seems like it will print "no MBID for artist Foo" if "Foo" appears once with an MBID and once without. I suppose that's a reasonable sacrifice, but it seems a little strange.

How about "no MBID for artist Foo in album {album}"? Seems weird too I guess.

Sure; sounds good.

Passing -a to 'beet missing' shows albums missing by all artists in the library. Signed-off-by: Quentin Young <qlyoung@qlyoung.net>

* Use full name for musicbrainzngs import * Use beets internal logging facilities * Match releases by release id * Convert some strings to Unicode * Remove unnecessary MB rate-limiting * Remove unnecessary imports * Follow beets convention for `--album` option * Follow beets convention for imperative docstrings * Simplify method signatures * Use defaultdict(list) where appropriate * Clarify missing MBID log message Signed-off-by: Quentin Young <qlyoung@qlyoung.net>

Signed-off-by: Quentin Young <qlyoung@qlyoung.net>

Produces a count of missing albums without listing them Signed-off-by: Quentin Young <qlyoung@qlyoung.net>

qlyoung · 2017-03-18T23:19:26Z

All suggestions implemented, plus -t for -a.

New message for missing MBID looks like

(venv) user@host ~/d/p/beets> beet missing -a
...
missing: No musicbrainz ID for artist 'Look What I Did' found in album(s) 'Atlas Drugged'; skipping
...

sampsyo · 2017-03-18T23:34:11Z

This looks perfect. Thank you again for implementing this! I'll run this on my own collection immediately. 😃

Add ability for 'missing' plugin to show missing albums by library artists

qlyoung force-pushed the missing-albums branch from f422732 to f907f3d Compare March 18, 2017 06:42

Kraymer reviewed Mar 18, 2017

View reviewed changes

qlyoung force-pushed the missing-albums branch 2 times, most recently from 24e1a38 to 7e76439 Compare March 18, 2017 18:43

sampsyo reviewed Mar 18, 2017

View reviewed changes

qlyoung force-pushed the missing-albums branch from 7e76439 to 502b06b Compare March 18, 2017 19:31

sampsyo reviewed Mar 18, 2017

View reviewed changes

qlyoung added 4 commits March 18, 2017 17:52

beetsplug: add '-a' to show missing albums to 'missing' plugin

339a1ef

Passing -a to 'beet missing' shows albums missing by all artists in the library. Signed-off-by: Quentin Young <qlyoung@qlyoung.net>

docs: add documentation for missing -a

d82ef10

Signed-off-by: Quentin Young <qlyoung@qlyoung.net>

beetsplug: implement missing -a -t

bf87cab

Produces a count of missing albums without listing them Signed-off-by: Quentin Young <qlyoung@qlyoung.net>

qlyoung force-pushed the missing-albums branch from d35442c to bf87cab Compare March 18, 2017 23:09

sampsyo merged commit bf87cab into beetbox:master Mar 18, 2017

sampsyo added a commit that referenced this pull request Mar 18, 2017

Merge pull request #2481 from qlyoung/missing-albums

620710b

Add ability for 'missing' plugin to show missing albums by library artists

sampsyo added a commit that referenced this pull request Mar 18, 2017

Changelog for #2481

bbf54b9

qlyoung deleted the missing-albums branch March 19, 2017 03:14

Add ability for 'missing' plugin to show missing albums by library artists #2481

Add ability for 'missing' plugin to show missing albums by library artists #2481

Conversation

qlyoung commented Mar 18, 2017 • edited Loading

mention-bot commented Mar 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sampsyo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qlyoung Mar 18, 2017 • edited Loading

Choose a reason for hiding this comment

qlyoung commented Mar 18, 2017

sampsyo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qlyoung Mar 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qlyoung commented Mar 18, 2017 • edited Loading

sampsyo commented Mar 18, 2017

qlyoung commented Mar 18, 2017 •

edited

Loading

qlyoung Mar 18, 2017 •

edited

Loading

qlyoung Mar 18, 2017 •

edited

Loading

qlyoung commented Mar 18, 2017 •

edited

Loading