Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

Commit

Permalink
Rate limiting the concurrent export requests based on the `concurrent…
Browse files Browse the repository at this point in the history
…` option. Default is 8.
  • Loading branch information
btakita committed May 17, 2019
1 parent a164834 commit d3fc0c1
Showing 1 changed file with 5 additions and 3 deletions.
8 changes: 5 additions & 3 deletions src/api/export.ts
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ async function _export({
let match;
let pattern = /<a ([\s\S]+?)>/gm;

const promises = [];
const urls = [];

while (match = pattern.exec(cleaned)) {
const attrs = match[1];
Expand All @@ -183,12 +183,14 @@ async function _export({
const url = resolve(base.href, href);

if (url.protocol === protocol && url.host === host) {
promises.push(handle(url));
urls.push(url);
}
}
}

await Promise.all(promises);
for (let i = 0; i < urls.length; i += concurrent) {
await Promise.all(urls.slice(i, i + concurrent).map(handle));
}
}
}
}
Expand Down

0 comments on commit d3fc0c1

Please sign in to comment.