Discussion around improving the import of reading lists

[duckpuppy]

I've been looking at DieselTech/CBL-ReadingLists recently, and having some issues importing them has made me wonder about ways to improve the matching of existing books in the library to the ones in the list. I have to spend a while fixing lists where I didn't have a particular set of issues but the import decided that a series with the same name published 10 years later was a close match, and since these are reading order lists - fixing that once I DO obtain the right issues is tedious. I can't just bulk remove/add because each issue needs to go back into the right spot in the list.

I'm going to make a broad assumption that MOST people scrape with ComicVine. The scraper sets two custom values to the CV series and issue. That same data is in the lists at the above repo - each <Book> entry has a <Database> child with attributes containing the same data.

The problem is that there's no "standard" here. Someone else may construct a reading list with different keys. I was wondering if it were worth discussing improvements to sharing reading lists such that matching is more accurate, or maybe having an option for CR to prompt to confirm each "series" at least.

[maforget]

I've checked the code to get an idea on how importing works:

1. Checks if ID is in DB.
2. If a filename exists, tries to find the book base on filename.
3. Find book that match Series & Number.
4. If nothing is still found from above, tries to match series by trying to remove the Volume from the name.
5. If nothing is still found from above, tries to match series by trying to remove the Volume from the name & removing special char & "the" & "and".
6. If there was more than 1 match from the above: Tries to reduce matches with the Year.
7. If there was more than 1 match from the above: Tries to reduce matches with the Volume.
8. If there was more than 1 match from the above & there is a format in the list: Tries to reduce matches with the format.
9. At this point take the first item from what match we got.
10. If there was nothing returned, create a Fileless comic.

Code:

ComicRackCE/ComicRack.Engine/Database/ComicIdListItem.cs

Lines 154 to 233 in 93017d4

	public static ComicIdListItem CreateFromReadingList(ComicBookCollection library, IEnumerable<ComicReadingListItem> readingItems, IList<ComicBook> booksToAdd = null, Func<int, bool> progress = null)
	{
	ComicIdListItem comicIdListItem = new ComicIdListItem();
	int num = readingItems.Count();
	int num2 = 0;
	foreach (ComicReadingListItem crli in readingItems)
	{
	if (progress != null && !progress(num2++ * 100 / num))
	{
	return null;
	}
	ComicBook comicBook = library[crli.Id];
	if (comicBook != null)
	{
	comicIdListItem.BookIds.Add(comicBook.Id);
	continue;
	}
	if (!string.IsNullOrEmpty(crli.FileName))
	{
	comicBook = library.FindItemByFileName(crli.FileName);
	if (comicBook != null)
	{
	comicIdListItem.BookIds.Add(comicBook.Id);
	continue;
	}
	}
	crli.SetFileNameInfo();
	IEnumerable<ComicBook> enumerable = library.Where((ComicBook ci) => ci.ShadowNumber == crli.Number && ComicInfo.SeriesEquals(ci.ShadowSeries, crli.Series, CompareSeriesOptions.None));
	if (enumerable.Count() == 0)
	{
	enumerable = library.Where((ComicBook ci) => ci.ShadowNumber == crli.Number && ComicInfo.SeriesEquals(ci.ShadowSeries, crli.Series, CompareSeriesOptions.IgnoreVolumeInName));
	}
	if (enumerable.Count() == 0)
	{
	enumerable = library.Where((ComicBook ci) => ci.ShadowNumber == crli.Number && ComicInfo.SeriesEquals(ci.ShadowSeries, crli.Series, CompareSeriesOptions.IgnoreVolumeInName | CompareSeriesOptions.StripDown));
	}
	if (enumerable.Count() > 1)
	{
	IEnumerable<ComicBook> enumerable2 = enumerable;
	enumerable = enumerable2.Where((ComicBook ci) => Math.Abs(ci.ShadowYear - crli.Year) <= 1);
	if (enumerable.Count() == 0)
	{
	enumerable = enumerable2;
	}
	}
	if (enumerable.Count() > 1)
	{
	IEnumerable<ComicBook> enumerable3 = enumerable;
	enumerable = enumerable3.Where((ComicBook ci) => ci.ShadowVolume == crli.Volume);
	if (enumerable.Count() == 0)
	{
	enumerable = enumerable3;
	}
	}
	if (enumerable.Count() > 1 && !string.IsNullOrEmpty(crli.Format))
	{
	IEnumerable<ComicBook> enumerable4 = enumerable;
	enumerable = enumerable4.Where((ComicBook ci) => string.Compare(ci.ShadowFormat, crli.Format, StringComparison.OrdinalIgnoreCase) == 0);
	if (enumerable.Count() == 0)
	{
	enumerable = enumerable4;
	}
	}
	comicBook = enumerable.FirstOrDefault();
	if (comicBook == null)
	{
	comicBook = new ComicBook
	{
	AddedTime = DateTime.Now,
	Series = crli.Series,
	Number = crli.Number,
	Volume = crli.Volume,
	Year = crli.Year,
	Format = crli.Format
	};
	booksToAdd?.Add(comicBook);
	}
	comicIdListItem.BookIds.Add(comicBook.Id);
	}
	return comicIdListItem;

So yes based on that, it would match a file based only on Series & Number first. But you cannot rely on ComicVine alone. For example I don't use ComicVine for most of my comics. Half are french using Bédéthèque which wouldn't work if we based it on cv alone. Also ComicVine isn't what it used to be, more and more I need to rely on my Amazon Scrapper because ComicVine just doesn't have them or is updated too slowly. Combine that with the fact that ComicVine is heavily throttling user, I can't even update my mcl file anymore. I don't know how long ComicVine will be used. Might require to use something like leagueofcomicgeeks.

Also the Database field isn't from ComicRack, it was probably added by other tools. And not all their list have them, this one I used for testing doesn't.

I also think that the reason importing was done that way, is because not everyone will use a scrapper. I think it' easy to believe that most use a scrapper and it probably is, but just reading the subreddit is proof enough that some don't even read the manuals. We need to think about those that will put every comics they own in a folder, without any scrapping and except that they can import a list as easily as everyone else. Usually when checking if something is the same series ComicRack will use the Volume also. It's the way it is used with Series Statistics like Series Gaps, etc. But then site like Amazon & Bédéthèque don't have the Volume field.

So I am not certain what is the best way, you have to think about the people that won't use ComicVine or another scrapper. We also need a way that is future proof and not only rely on a dying website. A change could be to base it on the year, along side Series & Number, it would prevent your problem, but create problem when it is not scrapped or a year isn't available. I haven't though about a better way. Maybe one solution is to have a dialog that would let the user decide which book from their library to use. That would probably be annoying when importing though. Maybe an option to only create fileless comics regardless of match?