[e621] implement manual pagination

[ClosedPort22]

Sometimes the number of returned post objects is smaller than the per-page limit even though there are more posts. This causes gallery-dl to stop the pagination prematurely.

Example JSON
The API returns 318 posts.

This only seems to apply to deleted posts. Not sure if other Danbooru instances are affected as well.

[mikf]

I'd suggest introducing a new limit variable at the start of _pagination() and setting it to self.per_page or 1 depending on self.strategy and using it like if len(posts) < limit: return. Seems cleaner to me.

Maybe this option should better be called something like limit with "auto" (self.per_page or self.per_page // 2?) being the default and otherwise the number of returned posts when to stop?

[ClosedPort22]

Maybe this option should better be called something like limit with "auto" (self.per_page or self.per_page // 2?) being the default and otherwise the number of returned posts when to stop?

Currently gallery-dl does not actually check if the value is "auto". Should I only allow int and "auto" and raise an error if an invalid type is specified?

[mikf]

I'd check if the value is "auto" or None and use self.per_page in this case, and assume it is int otherwise and apply text.parse_int to it just to make sure. At least that is what I've usually done in such cases.

Your implementation from dd4a4a3 is fine as well.