Implement iteration API, in addition to pagination API

[Pimm]

Rationale

The Mollie API provides paginated endpoints, which are great if a developer using this library is implementing a paginated dashboard.

Some developers ‒ myself included ‒ (ab)use these paginated endpoints and the nextPage method to apply logic to a batch of values in a set which don't necessarily fit on one page.

mollieClient.payments.page()
.then(payments => {
	// Apply logic.
	return payments.nextPage();
})
.then(payments => {
	// Apply logic.
	return payments.nextPage();
});

This is inconvenient to the developer, because the nextPage pattern above requires a lot of boilerplate code; and depending on the exact code might be inefficient in terms of memory usage as well as load on the Mollie API.

Basic

This pull requests implements an API which lives alongside the paginated API, and is designed for this case:

for await (let payment of mollieClient.payments.iterate()) {
	// Apply logic. Use the break keyword when you're done.
}

Advanced

There is a (currently stage 2) proposal to add helpers to iterators. In anticipation of the inclusion of these helpers, some of them are polyfilled in our library.

This powers the following snippets:

// Find the 10 latest euro payments over €100.00.
const iterator = mollieClient.payments.iterate()
	.filter(({ amount }) => amount.currency == 'EUR' && parseFloat(amount.value) > 100)
	.take(10);
for await (let payment of iterator) {
	// Apply logic.
}

// Find a payment by description.
const payment = await mollieClient.payments.iterate()
	.find(({ description }) => description == 'Transaction #b0e547a7');

// List all Italian customers.
const iterator = mollieClient.customers.iterate()
	.filter(({ locale }) => locale == 'it_IT')
for await (let customer of iterator) {
	// Apply logic.
}

With a bit of luck, we can discard our polyfill if and when the proposal is accepted and lands in Node.js without any breaking changes.

Note that not the entire proposal is currently supported by the polyfill. Specifically (and deliberately), toArray is not included. toArray would allow developers to write mollieClient.payments.iterate().toArray(), which is super convenient, but also potentially puts a lot of stress on the Mollie API (which will return all payments ever made) as well as the server of the merchant (which will have to deserialise and store all of those payments).

Future

If the Mollie API starts to support some of the cases above natively ‒ without iteration, we might be able to expand the logic to use that to our advantage and improve efficiency.

Discussion

Mollie has experienced incidents where developers were making so many requests in rapid succession that the API was having performance and reliability issues.

Both before and after this PR was merged, we've discussed whether the introduction of the iteration API makes it too easy or too tempting to make an unreasonable amount of requests ‒ essentially performing an accidental DDoS attack on Mollie ‒ compared to the pagination API.

[edorivai]

Only comment I have is that it might be a good idea to write some unit tests for the HelpfulIterator, what do you think?

[Pimm]

Thanks, @edorivai. iteration/iteration.test.ts is essentially a test for HelpfulIterator. It doesn't test it directly like a unit test would, but it does cover all of the methods (forEach, filter, et cetera) except for map ‒ which TransformingNetworkClient uses and is therefore automatically covered.

What could be a good idea, though, is to add a test which iterates over several pages. iteration/iteration.test.ts only covers a handful of payments, so they're all on the same page.

[edorivai]

Oh yeah, that makes sense. These tests should cover it then