Skip to content

Commit

Permalink
perf: improve indexing speed by 250%
Browse files Browse the repository at this point in the history
  • Loading branch information
@MSOB7YY
MSOB7YY committed Oct 14, 2023
1 parent 261e46a commit 5ae0247
Show file tree
Hide file tree
Showing 9 changed files with 547 additions and 262 deletions.
6 changes: 3 additions & 3 deletions lib/class/track.dart
View file
Original file line number Diff line number Diff line change
Expand Up @@ -225,20 +225,20 @@ class TrackExtended {
return TrackExtended(
title: json['title'] ?? '',
originalArtist: json['originalArtist'] ?? '',
artistsList: Indexer.inst.splitArtist(
artistsList: Indexer.splitArtist(
title: json['title'],
originalArtist: json['originalArtist'],
config: artistsSplitConfig,
),
album: json['album'] ?? '',
albumArtist: json['albumArtist'] ?? '',
originalGenre: json['originalGenre'] ?? '',
genresList: Indexer.inst.splitGenre(
genresList: Indexer.splitGenre(
json['originalGenre'],
config: genresSplitConfig,
),
originalMood: json['originalMood'] ?? '',
moodList: Indexer.inst.splitGenre(
moodList: Indexer.splitGenre(
json['originalMood'],
config: genresSplitConfig,
),
Expand Down
4 changes: 2 additions & 2 deletions lib/controller/current_color.dart
View file
Original file line number Diff line number Diff line change
Expand Up @@ -271,15 +271,15 @@ class CurrentColor {

final _colorGenerationTasks = qs.Queue(parallel: 1);

Future<void> reExtractTrackColorPalette({required Track track, required NamidaColor? newNC, required String? imagePath}) async {
Future<void> reExtractTrackColorPalette({required Track track, required NamidaColor? newNC, required String? imagePath, bool useIsolate = true}) async {
assert(newNC != null || imagePath != null, 'a color or imagePath must be provided');

final paletteFile = File("${AppDirs.PALETTES}${track.filename}.palette");
if (newNC != null) {
await paletteFile.writeAsJson(newNC.toJson());
_updateInColorMap(track.filename, newNC);
} else if (imagePath != null) {
final nc = await extractPaletteFromImage(imagePath, forceReExtract: true);
final nc = await extractPaletteFromImage(imagePath, forceReExtract: true, useIsolate: useIsolate);
_updateInColorMap(imagePath.getFilenameWOExt, nc);
}
if (Player.inst.nowPlayingTrack == track) {
Expand Down
23 changes: 12 additions & 11 deletions lib/controller/edit_delete_controller.dart
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,18 +57,19 @@ class EditDeleteController {
final saveDir = await Directory(AppDirs.SAVED_ARTWORKS).create();
final saveDirPath = saveDir.path;
final newPath = "$saveDirPath${Platform.pathSeparator}${track.filenameWOExt}.png";
final imgFile = await Indexer.inst.extractOneArtwork(track.path, albumIdendifier: track.albumIdentifier);
if (imgFile != null) {
try {
// try copying
await imgFile.copy(newPath);
return saveDirPath;
} catch (e) {
printy(e, isError: true);
return null;
}
final imgFiles = await Indexer.inst.extractOneArtwork(
[track.path],
albumIdendifiers: {track.path: track.albumIdentifier},
);
final imgFile = imgFiles.firstOrNull;
try {
// try copying
await imgFile?.copy(newPath);
return saveDirPath;
} catch (e) {
printy(e, isError: true);
return null;
}
return null;
}

Future<void> updateTrackPathInEveryPartOfNamida(Track oldTrack, String newPath) async {
Expand Down
20 changes: 10 additions & 10 deletions lib/controller/ffmpeg_controller.dart
View file
Original file line number Diff line number Diff line change
Expand Up @@ -220,16 +220,16 @@ class NamidaFFMPEG {
if (cachedThumbnail == null) {
currentFailed++;
} else {
final file = await Indexer.inst.extractOneArtwork(
filee.path,
forceReExtract: true,
artworkPath: cachedThumbnail.path,
albumIdendifier: tr.albumIdentifier,
);
if (file != null) {
final didUpdate = await NamidaFFMPEG.inst.editAudioThumbnail(audioPath: filee.path, thumbnailPath: file.path);
if (!didUpdate) currentFailed++;
}
final file = await Indexer.inst
.extractOneArtwork(
[filee.path],
forceReExtract: true,
artworkPaths: {filee.path: cachedThumbnail.path},
albumIdendifiers: {filee.path: tr.albumIdentifier},
)
.then((value) => value.first);
final didUpdate = file == null ? false : await NamidaFFMPEG.inst.editAudioThumbnail(audioPath: filee.path, thumbnailPath: file.path);
if (!didUpdate) currentFailed++;
}

currentOperations[OperationType.ytdlpThumbnailFix]!.value = OperationProgress(
Expand Down
2 changes: 1 addition & 1 deletion lib/controller/generators_controller.dart
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ class NamidaGenerator {
(element) {
final trackFilename = filename;
final fileSystemFilenameCleaned = element.getFilename.cleanUpForComparison;
final l = Indexer.inst.getTitleAndArtistFromFilename(trackFilename);
final l = Indexer.getTitleAndArtistFromFilename(trackFilename);
final trackTitle = l.$1;
final trackArtist = l.$2;
final matching1 = fileSystemFilenameCleaned.contains(trackFilename.cleanUpForComparison);
Expand Down
Loading

3 comments on commit 5ae0247

@fedikhatib
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain how the performance gain works ?

@MSOB7YY
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, it simply takes advantage of concurrent processess and isolates

  • suppose we have 2000 tracks

  1. a chunk size list (which size is calculated depending on free ram) so lets say 64 tracks

    namida/lib/controller/indexer_controller.dart

    Line 850 in dbf8256

    final chunkSize = (freeMemory ~/ 4).clamp(8, 128);

  2. then in the looper, we loop tracks normally without extraction, but adding them to the chunkList, and when the size of the list matches the chunkSize, we sent these tracks (64) to [extractOneTrack (old name)] method, and clear the chunk list

    namida/lib/controller/indexer_controller.dart

    Lines 868 to 881 in dbf8256

    if (chunkExtractList.isNotEmpty && chunkExtractList.length % chunkSize == 0) {
    await extractOneTrack(
    tracksPath: chunkExtractList,
    minDur: minDur,
    minSize: minSize,
    onMinDurTrigger: () => filteredForSizeDurationTracks.value++,
    onMinSizeTrigger: () => filteredForSizeDurationTracks.value++,
    checkForDuplicates: false,
    );
    chunkExtractList.clear();
    }
    chunkExtractList.add(trackPath);
    }

    • [extractOneTrack] is just a gate for [extractOneTrackIsolate] which takes some tracks paths and extract them all concurrently

      namida/lib/controller/indexer_controller.dart

      Lines 468 to 476 in dbf8256

      final results = await extractOneTrackIsolate.thready({
      "token": RootIsolateToken.instance,
      "trackPaths": paths,
      "minDur": minDur,
      "minSize": minSize,
      "tryExtractingFromFilename": tryExtractingFromFilename,
      "faudiomodels": audiomodels.isEmpty ? {} : {for (final m in audiomodels.entries) m.key: m.value?.toMap()},
      "artworks": artworks,
      });
    • the concurrent extraction inside [extractOneTrackIsolate] works like this (depending on completer)

      namida/lib/controller/indexer_controller.dart

      Lines 434 to 446 in dbf8256

      if (trackPaths.isNotEmpty) {
      final all = <Map<String, dynamic>>[];
      final completer = Completer<void>();
      for (final trackPath in trackPaths) {
      extracty(trackPath).then((r) {
      r['path'] = trackPath;
      all.add(r);
      if (all.length == trackPaths.length) completer.completeIfWasnt();
      });
      }
      await completer.future;
      return all;
      }

  4. the result from [extractOneTrackIsolate] is then sent back to [extractOneTrack] to be added to all related lists, while also calling [extractOneArtwork] to save artworks to cache directory (executed in isolate too)

    namida/lib/controller/indexer_controller.dart

    Lines 547 to 558 in dbf8256

    for (final trext in success.values) {
    final tr = trext.toTrack();
    allTracksMappedByPath[tr] = trext;
    _currentFileNamesMap[trext.path.getFilename] = true;
    if (checkForDuplicates) {
    tracksInfoList.addNoDuplicates(tr);
    SearchSortController.inst.trackSearchList.addNoDuplicates(tr);
    } else {
    tracksInfoList.add(tr);
    SearchSortController.inst.trackSearchList.add(tr);
    }
    }

  • this is the basic idea, more code to handle result failures and extract them on ffmpeg on the UI thread (cuz it doesnt support isolates)
  • let me know if didnt get any point
  • other parts of the commit are just adapting some methods to work as static (to be executable in isolates)
  • side note: for some reason, videos info extraction doesnt work as intented and runs on ui thread, which may cause a jank/delay affecting the extraction process, i just discovered and investigating.

@fedikhatib
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your clear explanation.
Happy coding

Please sign in to comment.