Rethink current "incomplete folder" implementation

[FranciscoPombal]

Context:
#13013 (comment)

I'm not sure if we have the same definition of a "completed" torrent, and we should decide on that first. Does a torrent with only a few files selected (but each of these few files fully completed) count as "complete", or does it have to have all files selected and complete to count as "complete"?

This is a really good question (although not related to this topic, but rather to the current "incomplete folder" implementation) that worths a separate Issue. Do you mind creating it?
IMO, if "atomic unit" is torrent then only fully downloaded torrent should be considered as "complete".

This has implications on how moves should be handled for completed torrents, depending on whether or not any file was unselected for download, among possibly other things.

Related:
#7164

[glassez]

My general point is that torrent should never return to "incomplete" folder after it has once been moved to "complete" one (the current implementation allows this). So there are only two options left (i.e. under what conditions to move the torrent to the "complete" folder):

1. When all the selected files are downloaded (any additional files will be downloaded into "complete" folder unless the user move torrent into "incomplete" folder manually);
2. When exactly all files of torrent are downloaded (torrent with some "unselected" files will never be moved to "complete" folder).

Possible 3rd option is allow to specify desired behavior per torrent.

[glassez]

@FranciscoPombal
The Title of Issue and its description do not give a clear understanding of the problem. Some people think it's just a continuation of #13013.

[FranciscoPombal]

@glassez

I though about it some more.

From a UX perspective, a user adds "jobs" for qBittorrent to complete. A user cares about each "job" completing. Each job may or may not involve downloading all files in the torrent. So it is clear that complete job != complete torrent, and also that complete job == all of the _selected_ files in the torrent completely downloaded.

So, what this setting should provide the user is: as long as a job is incomplete after first being added, it will reside in the "incomplete folder"; the meaning of this folder is "it contains incomplete jobs". It is also clear, like you said, that a job should never go back to the "incomplete" folder. If the user selects new files in the torrent for an already completed "job", that's a new "additional job", so that when the new files are completed, they are also moved to the "completed" folder (but since the files are already in the "completed" folder, the move to that folder does nothing). Preventing things from flowing backwards simplifies the problem - at no point does qBittorrent have to check for files in 2 separate locations.

For example: suppose a torrent X has files A B C D E F. Initially, the user selects files A and B for download (so that the "job" is "download files A and B from torrent X"). The download finishes, and A and B are moved to the "completed" folder (so they end up as completed/X/A and completed/X/B). The fastresume is updated to reflect the fact that the torrent is in the "finished" state and thus its (selected) files reside under the "completed" folder. Then, the user selects files C and D for download. The fastresume is updated to mark the torrent as "partially finished" (this is different than "incomplete", see below); once the user actually resumes the torrent, qBittorrent starts downloading the new files to completed/X/, and once that's done, "moves" them to completed/X/ (so this "move" is effectively idempotent). The fastresume is updated to mark the torrent as 'finished" once again. Then the user pauses the torrent and possibly exits the client or whatever.

So the possible state transitions are:

incomplete -> finished
finished <-> partially finished

where qBittorrent makes the following assumptions:

incomplete state: torrent files always reside under /some/path/incomplete
finished and partially finished states: torrent files always reside under /some/path/completed

Is this a good idea? Are these states and transitions compatible with the rest of the states and associated logic (e.g. missing files, file size mismatch, errored state...).

Now: what should happen if a user gets files E and F (or partial pieces of them) from another source? Clearly, detection and use of those files must be somehow supported, because this is a somewhat common use case related to cross-seeding.

• If they put them in incomplete/X, and then select them in the UI, should qBittorrent look inside that folder and try to re-use whatever is there when a torrent is resumed, or just ignore anything in the incomplete/ directory if the state of the torrent itself is not incomplete? (I'm leaning towards the latter option).
• If they put them in completed/X, and then select them in the UI, can qBittorrent detect them and force a recheck of the torrent? (I would expect this).

Additionally, some precautions must be taken depending on whether the user enables or disables root folder creation, but that's almost a separate matter.

unless the user move torrent into "incomplete" folder manually);

Under this new system, if the user does this after the torrent has gone into "finished" or "partially finished" state, and then resumes the torrent, I would only expect qBittorrent to go look back in the "incomplete" folder if it can't find anything at all from said torrent under the completed folder.

Maybe later I'll try to formalize these ideas a bit better, perhaps with a flow chart or some pseud code block.

@FranciscoPombal
The Title of Issue and its description do not give a clear understanding of the problem. Some people think it's just a continuation of #13013.

What do you suggest?

[glassez]

@FranciscoPombal
Out of the fire and into the fire...
The current implementation is too complicated, which is why it has its drawbacks. The point is to simplify it as much as possible, getting rid of excesses that are not worth trying to maintain at the expense of reliability. But you want to complicate it again.
I'm going to get away from all these "states", etc. and make the location of the torrent explicit and completely user-controlled (as opposed to the current temporary folder, which works behind the scene).

1. When user adds torrent it sets its "save path" and torrent is really located there.
2. Optionally user sets "complete save path" where torrent will be moved once it's complete.
3. When user selects previously unselected files they download into current "save path".
4. When user sets torrent location it really affects current "save path" so there's no problem to point incomplete torrent to any folder with existing files (even to previously set "complete folder" of this torrent).

* Any "path" mentioned above can be set either directly or using some default value (e.g. "default save path").

So here we need only decide what torrent should be moved to "complete" folder: completely downloaded or partially (or allow user to set it).

[FranciscoPombal]

@glassez

I see, I think I was missing the point.

So here we need only decide what torrent should be moved to "complete" folder: completely downloaded or partially (or allow user to set it).

We could just add a another option: "Keep partially downloaded torrents in:".
That way, it's always clear where a torrent has to go.

• When it's first added, it's in the "incomplete" folder
• If the user had deselected a few files but all selected files have completed downloading, and the user has set a "partially downloaded save path", it is moved there
• If/when all files are downloaded and the user has set a "complete save path" the torrent is moved there once it's fully complete.

When user sets torrent location it really affects current "save path" so there's no problem to point incomplete torrent to any folder with existing files (even to previously set "complete folder" of this torrent).

Not so sure what you mean by this. Can you rephrase it?

Anyway, the problem then becomes: What should happen when a user deselects files? If we have agreed that torrents can't "move backwards", then it is possible that the user will end up with several incomplete/partially downloaded torrents in the "completed save path".

Also, suppose there is a torrent torrent X with files A, B and C. The user has selected all for download, and they are all complete except for C, which is partially complete. The user then deselects C; should the torrent immediately move to the "completed save path"?

[glassez]

We could just add a another option: "Keep partially downloaded torrents in:"...
If the user had deselected a few files but all selected files have completed downloading, and the user has set a "partially downloaded save path", it is moved there

👎
Sorry, this doesn't look like a simplification of the architecture. Manage another option, which seems to have done is not needed, subject to reasonable usage, this is unnecessary.
Besides the main idea is not "keep incomplete torrent" but "move torrent once it's complete".

Not so sure what you mean by this. Can you rephrase it?

"Set location" applies to current "save path".

[glassez]

Also, suppose there is a torrent torrent X with files A, B and C. The user has selected all for download, and they are all complete except for C, which is partially complete. The user then deselects C; should the torrent immediately move to the "completed save path"?

It just depends on which torrent (fully or partially completed) we eventually decide to move.

Anyway, the problem then becomes: What should happen when a user deselects files? If we have agreed that torrents can't "move backwards", then it is possible that the user will end up with several incomplete/partially downloaded torrents in the "completed save path".

So why not move only fully downloaded torrents?
This option is some kind of automation. When user "plays" with files inside torrent it's not so hard to manage "save path" manually as well.

[Seeker2]

"So here we need only decide what torrent should be moved to "complete" folder: completely downloaded or partially (or allow user to set it)."

For ease-of-use for average users, both temporary download folder and .qt! part files are best disabled...which means users will have to manually enable both.

If they are using a temp folder with move/copy on completion...
Doing a blanket move/copy of all the files of a torrent to its destination when the whole torrent completes may be a bit of a disk I/O and/or CPU show-stopper for extremely complex and large torrents.

[FranciscoPombal]

@glassez

So why not move only fully downloaded torrents?
This option is some kind of automation. When user "plays" with files inside torrent it's not so hard to manage "save path" manually as well.

Good point. I'm on board with this.

So we want:

1. Only torrents that complete with all files selected are moved to the "completed" folder.
2. Torrents never move backwards in the pipeline, i.e. the only possible direction to move is from "incomplete" -> "completed" folder. If a torrent is already in the "completed" folder, it stays there, even if the user deselects and deletes some files.

This is it, right?

@Seeker2

Doing a blanket move/copy of all the files of a torrent to its destination when the whole torrent completes may be a bit of a disk I/O and/or CPU show-stopper for extremely complex and large torrents.

When using such a feature you're always going to have to pay the price of moving the files around.