beet stats is slow

[mfiano]

With a collection of about 200k tracks, beet stats takes about 60 seconds to complete. If I comment out this one line to increment the time, it completes in 3-4 seconds. Would it be too much trouble to cache this?

beets/beets/ui/commands.py

Line 1254 in 63f5028

total_time += item.length

[sampsyo]

Wow; that's pretty crazy—maybe we can do some profiling first to figure out why that line is so slow?

[tecosaur]

This may be a silly suggestion, but since the total time only changes when songs are added/removed could we store the total time and modify it when adding/removing tracks?

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Holzhaus]

This is still relevant and should be reopened.

[wisp3rwind]

This is still relevant and should be reopened.

I'm not so sure about this: Speeding this up is certainly desirable, but we don't have a clear path of action (yet). Caching as suggested by @tecosaur is not quite as simple, since this is not a statistics of the full library only, but stat can also show details of a query. I suspect that the issue is that these fields are not stored in the database, but rather obtained by reading the header of each media file. So the fix might be to also store MediaFile based computed fields in the database. But somebody should really investigate whether my suspicion here is correct, so the needinfo tag is still warranted (and consequently this being closed by the bot). I'm also not sure whether storing these fields in the database is really as simple as it might sound: We'd end up with two sources of information: Does one always take precedence? Do we need to invalidate this fields in the database on some occasions?

A PR to address these questions would of course be very welcome ;)

[Holzhaus]

The first step is to acknowledge the problem. This issue has been confirmed by multiple users, so it should be reopened.

For my - admittedly quite large - collection running beet stats takes more than 8 minutes on my armv7l NAS.

I suppose the use case for most people is to flex with the number artists/albums/tracks in their library, so ivd consider the query variant of this command an edge case.
One possibility would be to introduce a separate code path in case no query was specified that just runs SQL SELECT COUNT(*) queries directly without looping over each item.

[tecosaur]

Hmm, I'd think tiered caching (i.e. Merkle tree) could work nicely, no?