Automatically calculate piece length #55

codeclem · 2020-10-12T18:29:20Z

Ported from https://github.com/cmclark65/mktorrent

This would close issue #30

Thanks to @cmclark65 for the code and permission to port it!

Ported from https://github.com/cmclark65/mktorrent

cmclark65 · 2020-10-13T04:30:37Z

7 years ago I was happy with 24 bit as the largest piece size for a torrent target larger than 12.5 GiB. I don't know what size the largest common torrents are today, but maybe it would make sense to add a few more "maxes" to the array? Is 24 enough to keep the torrent size down for today's largest files/file collections being torrented? It is nearly trivial to add a few more sizes.

Obviously 24 is still a lot better than the old default of 18 for very large targets, and possibly good enough.

cmclark65 · 2020-10-13T05:06:08Z

Arguably it would be good to just get rid of the skanky array method I used, specify a maximum piece size such as 28 for the for loop, and specify a constant for the max number of pieces (2000 if the issue #30 is to be trusted; I seem to have targeted a max of more like 800) and then just have the for loop be "n == 1 << i; if ((m->size + n - 1) / n) <= PIECE_MAX m->piece_length = n;", the if following the loop would need to change i to 1 << i, ditch the later conversion from bit size to integer since it is already now done in the loop.

Does this make sense? I would do it this way now; they way i did it 7 years ago with the pre-defined maxes and the array kind of sucks.

codeclem · 2020-10-13T16:07:56Z

Common recommendations for piece size vary some. A 24 bit length would be a piece size of 16 MiB. A lot of people say you should never even go over 1 MiB, which I think is kind of unreasonably strict, and I have never heard anyone recommend a size larger than 16 MiB. Supposedly some clients don't handle 32 MiB+ pieces well, or even at all.

Looking through my currently seeded torrents, the largest torrent I have with a 1 MiB piece size is 49 GiB. I have many torrents in the 300-800 GiB range that all use 16 MiB pieces, and there are two torrents that are each 1.5 TiB and they use 64MiB pieces. I think that's the only time I have encountered pieces that large (qBitorrent doesn't even offer it as an option when creating torrents) and I can't find any torrents right now that use 32 MiB.

All that to say, I think 16 MiB is a very reasonable max default. This commit does not remove the ability for the user to specify their own piece length, and I think anyone who feels they need 32 or 64 MiB pieces would totally understand having to explictly say so.

As far as the implementation, I think it's fine as it is but I would be happy to refactor it if @Rudde has an opinion. For now I would wait to see if he's even interested in merging the feature at all.

FranciscoPombal · 2020-10-14T00:30:09Z

Piece sizes over 16 MiB can make sense for some very large torrents. Don't listen to the brain dead zealots claiming 16 MiB should be the max, that's just because their precious uT 2.2.1 doesn't support piece sizes over that.

cmclark65 · 2020-10-14T13:54:45Z

If a 24 bit piece size is the largest that a commonly used client supports, it seems like a good max for the auto-size code. Then this port of my code should be good to go; it is proven at least. I tested it well on 32 and 64 bit linux and freebsd at the time I first wrote it.

My suggestion for rewriting using a formula would require re-testing; I'd recommend doing it if you wanted to raise the max piece size from 2^24 but if not you probably have better things to do with your time. I hope the current maintainer buys these changes if only so that someone other than me gets some benefit out of my work. :)

init.c

uno20001 · 2020-10-14T14:55:10Z

init.c

+	uintmax_t piece_len_maxes[] = {
+		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+		(uintmax_t) BIT15MAX * ONEMEG, (uintmax_t) BIT16MAX * ONEMEG,
+		(uintmax_t) BIT17MAX * ONEMEG, (uintmax_t) BIT18MAX * ONEMEG,
+		(uintmax_t) BIT19MAX * ONEMEG, (uintmax_t) BIT20MAX * ONEMEG,
+		(uintmax_t) BIT21MAX * ONEMEG, (uintmax_t) BIT22MAX * ONEMEG,
+		(uintmax_t) BIT23MAX * ONEMEG
+	};


This should be const in my opinion.

Changed this and num_piece_len_maxes to const

uno20001 · 2020-10-14T14:55:13Z

init.c

+	int num_piece_len_maxes = sizeof(piece_len_maxes) /
+	    sizeof(piece_len_maxes[0]);


uno20001 · 2020-10-14T14:56:24Z

init.c

+		if (m->piece_length == 0)
+			m->piece_length = i;


i = num_piece_len_maxes in this case, so I think it'd be better to write m->piece_length = num_piece_len_maxes.

Makes sense, that is more clear. I've pushed a new commit with this change.

pobrn · 2020-10-22T22:25:21Z

init.c

+    /* determine the piece length based on the torrent size if
+	   it was not user specified. */


Please fix the indentation here. Use tabs.

pobrn · 2020-10-22T22:25:36Z

init.c

+        /* if user did specify a piece length, verify its validity */
+        FATAL_IF0(m->piece_length < 15 || m->piece_length > 28,
+            "the piece length must be a number between 15 and 28.\n");
+    }


Please fix the indentation here as well.

pobrn · 2020-10-23T12:09:34Z

mktorrent.h

+#define ONEMEG		1048576
+
+/* max torrent size in MB for a given piece length in bits */
+/* where an X bit piece length equals a 2^X byte piece size */
+#define	BIT23MAX	12800
+#define	BIT22MAX	6400
+#define	BIT21MAX	3200
+#define	BIT20MAX	1600
+#define	BIT19MAX	800
+#define	BIT18MAX	400
+#define	BIT17MAX	200
+#define	BIT16MAX	100
+#define	BIT15MAX	50


Please replace the tabs with single spaces.

pobrn · 2020-10-23T16:53:09Z

Thanks.

codeclem · 2020-10-23T19:23:56Z

Thank you!

* Add exclude command line option. Excludes files based on a glob string (#56) * Automatically calculate piece length (#55)

ghost · 2022-02-28T21:10:25Z

Is there a new release immediately planned with the merged pull requests?

edgeworn · 2022-05-22T14:43:29Z

The last release was over 5 years ago. Will there be a new release soon with this and related changes?

Automatically calculate piece length

699c6c8

Ported from https://github.com/cmclark65/mktorrent

uno20001 reviewed Oct 14, 2020

View reviewed changes

init.c Show resolved Hide resolved

uno20001 reviewed Oct 14, 2020

View reviewed changes

codeclem added 3 commits October 22, 2020 10:22

Make piece_len_maxes and num_piece_len_maxes const

e243eb8

Improve code clarity

3b32a75

Bring back validity check of user supplied piece length

3abc496

pobrn reviewed Oct 22, 2020

View reviewed changes

codeclem added 2 commits October 22, 2020 22:03

Fix indents

5c24277

Fix indents

470b649

pobrn reviewed Oct 23, 2020

View reviewed changes

Repace tabs with spaces

9e07199

pobrn merged commit ea1fbf2 into pobrn:master Oct 23, 2020

pobrn added a commit that referenced this pull request Jan 30, 2021

changelog: add pull requests #55 and #56

4c27913

* Add exclude command line option. Excludes files based on a glob string (#56) * Automatically calculate piece length (#55)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatically calculate piece length #55

Automatically calculate piece length #55

codeclem commented Oct 12, 2020

cmclark65 commented Oct 13, 2020

cmclark65 commented Oct 13, 2020

codeclem commented Oct 13, 2020 •

edited

Loading

FranciscoPombal commented Oct 14, 2020

cmclark65 commented Oct 14, 2020 •

edited

Loading

uno20001 Oct 14, 2020

codeclem Oct 22, 2020

uno20001 Oct 14, 2020

uno20001 Oct 14, 2020

cmclark65 Oct 14, 2020

codeclem Oct 22, 2020

pobrn Oct 22, 2020

pobrn Oct 22, 2020

pobrn Oct 23, 2020

pobrn commented Oct 23, 2020

codeclem commented Oct 23, 2020

ghost commented Feb 28, 2022

edgeworn commented May 22, 2022

		int num_piece_len_maxes = sizeof(piece_len_maxes) /
		sizeof(piece_len_maxes[0]);

		/* determine the piece length based on the torrent size if
		it was not user specified. */

Automatically calculate piece length #55

Automatically calculate piece length #55

Conversation

codeclem commented Oct 12, 2020

cmclark65 commented Oct 13, 2020

cmclark65 commented Oct 13, 2020

codeclem commented Oct 13, 2020 • edited Loading

FranciscoPombal commented Oct 14, 2020

cmclark65 commented Oct 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pobrn commented Oct 23, 2020

codeclem commented Oct 23, 2020

ghost commented Feb 28, 2022

edgeworn commented May 22, 2022

codeclem commented Oct 13, 2020 •

edited

Loading

cmclark65 commented Oct 14, 2020 •

edited

Loading