Steps to modernize the code

[apiweb]

Thank you for providing the code. I was able to successfully compile it on my machine; I appreciate your assistance and work to port the code to .net framework 4.8, it's a huge work

I'm investigating the code to find obsolete references within the project that could potentially be updated or removed. I have compiled a list that could serve as a guide for others interested in upgrading or cleaning the project.

---

• CrashReport
  • Purpose: The code makes references to SoapHttpClientProtocol (deprecated), used for submitting crash reports to a PHP server.
  • Notes: It's likely that the community edition won't send crash reports. Therefore, I suggest considering the removal of this feature.
• Twitter
  • Purpose: The Twitter class is employed to post tweets when adding a review to any series.
  • Notes: Considering that it uses an old API deprecated by X (old Twitter), I recommend removing this functionality.
• BinaryFormatter binaryFormatter
  • Purpose: Used to deserialize binary files, but deprecated by Microsoft due to security flaws.
  • Notes: It is crucial to update the code to address the security concerns associated with the deprecated BinaryFormatter.
• Windows 7 Multitouch References
  • Purpose: Adds support for multitouch on certain elements, such as the Tabbar.
  • Notes: Removing these references is advisable since Windows 7 multitouch is no longer maintained by Microsoft.
• Tao.OpenGl
  • Description: Tao is a considerably old library that has not received updates for the past 14 years. It has been superseded by OpenTK, which includes a compatibility layer with Tao. This layer can be used to port the OpenGL code.
  • Notes: I recommend transitioning from Tao to OpenTK for better compatibility and ongoing support.

We also can convert some project (like the plugin project for example) to a .net standard class library before to migrate to .net core.

---

My focus is on identifying points that can be addressed to make the application easier to update in the future.

If there are additional considerations that need attention, we can use this list as a guide for individuals looking to update the community edition version.

[maforget]

Pretty much every DLL that isn't found on NuGet should be removed, so it included a lot of your proposal.

• CrashReport: Agree with the crash report. Seems that SoapHttpClientProtocol is also used in ServerRegistration. It seems related to another cyolito.com service. Didn't know it existed, but it seems that you could share your server with others. This means we will have to remove also anything to do with Public servers, while keeping the private server functionality untouched.
• Twitter: I already wanted to remove Twitter, even if it would work without the paying for an API, I never saw what it would do. Already pushed.
• BinaryFormatter: I knew that SoapHttpClientProtocol wasn't available for .NET 8, but I don't see any warning about this one. Is that also for .NET5+? I believe this is used as a backup to save the setting instead of XML. It looks that it isn't even hooked up and not used anymore. But, there are still used in the Cache (seems important) and in a Clone Utility (This could be redone without the BinaryFormatter).
• Windows 7 Multitouch References: I know that some people use a tablet PC to use ComicRack. Is there any alternative for this? I've not really check where this is hooked up and don't know how much of a difference it would make. But for these people it could be useful.
• Tao.OpenGl: Good to know that there is a drop in replacement. Performance is very bad on very high resolution images. I've already tried upping the max texture memory from 256mb to 4gb and it didn't affect performance. I didn't do any more exploration, but yes a new library is a must. Maybe in the short term changing to this new OpenGL library is the best idea. But do we necessarily need to be tied to OpenGL? I guess that if we want Linux Compatibility it might be the only option, but are there any alternatives for Hardware support?

[maforget]

For the plugin, I've already added an issue #3 that goes in the same direction. But I would like to keep compatibility with existing plugins. This seems very important. (I ain't gonna rewrite all my plugins, just for fun, when they work correctly). So I was hoping to keep the existing plugin system as a legacy.

Edit: Seems that newer version of IronPython do work with .NET 6, but existing plugins broke, when I tried to up the version too fast. Maybe we could at least make it work with at least the latest 2.7 version. If it works with .NET 6 here is hoping it will work with .NET 8.

[maforget]

• Update the News feed.
  • Can we do it via Github?

[Totengeist]

• CrashReport

  • Purpose: The code makes references to SoapHttpClientProtocol (deprecated), used for submitting crash reports to a PHP server.
  • Notes: It's likely that the community edition won't send crash reports. Therefore, I suggest considering the removal of this feature.

I opened a new thread on this before reading this one. Sorry about that.

I agree with removing crash reports at this stage, but I'm curious if the functionality might not be useful down the road. I've built some web services in the past and would be happy to start building a catch for these kinds of reports.

It's probably too early to discuss specifics, but maybe keep it in the back of your mind. I've worked in customer support roles, and getting good information out of people can be hard. Having a way for people to send these reports might help track down some of the more obscure bugs.

[maforget]

I agree that at one point it could be useful, we can always revert back the change at that point. But I personally don't want to have to check multiple crash logs daily. They can always make a post or an issue (I know it's hard for some).

[maforget]

Removed all references to SoapHttpClientProtocol in the server (just commented, for easier re-implementation) 849ff43

With that disabled the Share on the Internet function, most of the code is intact for re-implementation, but disabled the Checkbox so it's clear that this isn't used anymore. I don't really want or think it's a good idea to have a central server for sharing. We could create a simple ASP.NET Web API Server that people could host on their server of choice. Or should we just remove all of this in case we end up like tachiyomi? Or more like Kodi?

This all seemed to depends on the cyolito.com Server, unless I am mistaken. No one was using that function without that?

[saskir12]

As I never used this feature I can not be sure. Personally I go the route of having it on my NAS and using a VPN for otuside of my house.

But as far as I read from others (atleast on the old Forum) it is as you say.

[flaser]

• SQLite support? One (by today's standard's) unusual choice for CR was to use plain XML as the persistence layer. AFAIK, nowadays - for the quantities of data CR is expected to hold - SQLite is a pretty standard alternative (with robust C# support).

Depending on how CR uses it - either as a means to persist data between runs, or as an alternative to keeping everything in memory - there could be different trade-offs. I wonder how much (if any) benefit there'd be for the former.
However for the later, startup times could be reduced (as not everything would need to be in memory all the time), albeit with a potential hit in query times (precisely, as not everything would be in memory all the time).

I wonder, is this something worthwhile investigating, or not particularly, given that MS-, MySQL (& MariaDB) are already supported?
If so, would a less all-in-memory data handling be something worthwhile to pursue on its own, using existing DB support?

[maforget]

I really like SQLite instead of having the need to install 3rd party server like MariaDB and etc. I haven't looked over all the code, and don't know all of it yet. But from what I can see existing database like MySQL, just stores the XML file inside the database in a single column. Much of the criticism about the DB is that, it isn't optimized.

My intuition with the data with things like smart list, is that you pretty much need to have the whole data in memory to be able to filter the complete database. Of course you could have some optimization like only loading certain fields that are needed for the existing lists and have the other fields only fetched when needed, and probably these exists already. But with the current storage as XML, my guess is that it's pretty much all loaded in memory when loaded.

Maybe the XML format was an unusual choice, but when you get down to it, serialization is used a lot to save state of objects. It's pretty easy compared to complex system to save data. Be it binary, XML or JSON it all the same in the end, at least this is easily modifiable. It isn't a surprise that JSON is everywhere nowadays. I did do some test with an app I have, that used the MTGJson database to load all the Magic: The Gathering cards data. They have a SQLite version also, and with very minimal testing, the time to load the data was the same with Json and SQLite.

Of course SQL has great support with C# and things like the Entity Framework make it easy to create a DB based on existing code with all the DB piping done for you. But that would require some work to the program to be used differently. The more I work on this and try to figure out how it works, the more I feel like I surgeon that a builder. Carefully doing small changes, than the sweeping changes that people would like, that would require tearing down half the existing work.

[flaser]

But from what I can see existing database like MySQL, just stores the XML file inside the database in a single column. Much of the criticism about the DB is that, it isn't optimized.

UPDATE: I just checked, ComicStorageMySql.cs & ComicStorageMsSql.cs indeed just create tables with two hand-written commands:

ExecuteCommand("Create table changes (id varchar(40) not null primary key, update_counter bigint default 0, delete_counter bigint default 0)");
ExecuteCommand("Create table comics (id varchar(40) not null primary key, update_counter bigint default 0, data mediumtext)");

If this is true, then This is one area that may be easier to address today. C# has built-in ORM (Object Relation Mapping) that allows the auto-translation of C# classes/objects into DB entries. Each class is a table, class attributes are the table columns. Objects are persisted as rows in these tables with automated handling of (some) normalization: if an object attribute is not a primitive but class type, then it will be stored in its own table with a reference to it from the parent table.

The more I work on this and try to figure out how it works, the more I feel like I surgeon that a builder. Carefully doing small changes, than the sweeping changes that people would like, that would require tearing down half the existing work.

Fully endorsed. I've seen too many people shoot for big refactors without sufficient familiarity with the code-base. These inevitably peter out as the work is too big to handle in an agile fashion. (No chance for quick feedback, can't work in small increments).

My intuition with the data with things like smart list, is that you pretty much need to have the whole data in memory to be able to filter the complete database.

I'm not as sure, after all the whole point of a SQL query is to filter the whole DB ;)
E.g. by translating Smart List to a query we can move this to the SQL backend.

Now to get the proper benefit, we'd need indexing and the other myriad improvements one can make on the DB side.
So on one hand, CR would need DB programing expertise. On the other hand, all the DB "programming" benefits could be leveraged.

• DB side: Normalizing the DB, e.g. breaking the current data set into multiple tables could enable faster, cleaner lookups. (ORM can handle most of this automatically). Further improvements: Views, prepared statements, user-defined/custom functions, . ("DB Programming"). These speed up DB operations to differing degrees and allow moving some of the logic from the app to the DB.
• SW side: This is where a lot of project go wrong when their UI ends up mirroring the DB organization. This can be exceedingly painful for the user (e.g. you must first create a "Writer entry", then refer to that in the "Book entry" etc.). Here the challenge would be to initiate multiple DB transaction from a single UI and decide what makes sense to do as application logic vs db logic.

[DieselTech]

I really like SQLite instead of having the need to install 3rd party server like MariaDB and etc

Massively agree here. Especially considering the nature of ComicRack being a Windows only program for so long. (Not counting the recent efforts to dockerize it with Wine.) My experience in this space after so long showed me that a lot of the people that use it were just your normal run of the mill windows users, they won't be wanting to setup a full SQL server and be responsible for the upkeep of it.

pretty much need to have the whole data in memory to be able to filter the complete database.

I think the disclaimer here is: "within a reasonable time frame." Sure, it is pretty nice having the smart filters show results in near real-time, but I don't think a 100-350ms response time would be seen as too much by people.

In its current unoptimized state of just storing XML in the DB that might be true in order to keep response times down on queries. If it ever gets to the point of using proper queries than the database could be on disk without too much performance hit. In the right conditions SQLite can be faster than direct disk access. (Though obviously never as fast as in memory) My rough ballpark estimate would be around 10-100ms to return a full query if it would be done in sql natively.

Carefully doing small changes, than the sweeping changes that people would like, that would require tearing down half the existing work.

I also agree with flaser here. While these changes would be amazing and would add greatly to the modernization of the app, I think more time is needed to get familiar with the code. A change like this is too big to do at this point without knowing the data model in more detail. This can be put on the roadmap for down the line once everyone has had more time to spend with the code.

[Neurozone]

I'm a DBA, I can help with the database :)

[maforget]

What would be helpful is to have the SQL portion of the program be more streamlined. Not certain where the discussions were at, but the SQL portion just stores the XML inside a single column. A real DB schema where all the columns reflect the stored data.

I haven't looked into it, but when adding new information to the DB, there should be a mechanism to transparently upgrade the DB.

Also a way to export and import the DB to an XML using the Backup database.

Most of these should probably be done when upgrade to .NET so that we can use the new version of EntityFramework Core.

[Neurozone]

About database look at https://www.litedb.org/

[maforget]

What are the differences between this & SQLite? Even if these kind of database are easy to edit, I don't want to convert the DB to a new format. I kinda like that you can edit the file manually with a text editor. Like if moving the library to another drive or path, you can easily search & replace. That is way harder to do in SQL type DB's for a normal user.

[Neurozone]

Also to do a mass conversion batch to bootstrap the future database it can be useful : https://github.com/btxgit/crackid

[maforget]

There is no need for external tools, the program contains all the code to create and read info files. Would just save the ComicDatabase object to sql using Entity Framework.

[maforget]

I've updated the project files & added the PolySharp packages so some of the latest language feature are supported. Look at the documentation for PolySharp to see what it enables.

https://github.com/Sergio0694/PolySharp/?tab=readme-ov-file#documentation-

[maforget]

I've updated the projects files to support the newer SDK style, along with the above change to support the newer language functions. This is probably the limit of what we can do. I don't know if it will be possible to move to .NET 8+.

While trying to remove the need for BinaryFormatter, i've updated the Clone Utility to use a nuget package called DeepCloner, that seems to work but crashed the release build. So I've had to revert the changes for now.

The remote library is using a function called Windows Communication Foundation that is deprecated, that will need to be either scraped or redone using more modern alternative (that I've not looked what they are).

Edit: It seems that the replacement for WCF is CoreWCF.