Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MaxMind] Improve performance of set generation #16

Closed
chr0mag opened this issue Dec 23, 2021 · 0 comments · Fixed by #24
Closed

[MaxMind] Improve performance of set generation #16

chr0mag opened this issue Dec 23, 2021 · 0 comments · Fixed by #24
Labels
enhancement New feature or request

Comments

@chr0mag
Copy link
Owner

chr0mag commented Dec 23, 2021

The MaxMind set generation logic is currently very simple. It just loops through the list of country IP blocks for the respective address families (ie. GeoLite2-Country-Blocks-IPv4.csv and GeoLite2-Country-Blocks-IPv6.csv) and writes to each CC.ipvX file as it goes.

This results in repeated appending to different files, causing lots of (slow) disk IO. A more performant solution would be to build a Map with the country codes as keys and the value a list of IP subnets belonging to that country. This would generate the country sets in memory (fast) rather than building them incrementally on disk. We can then write out the complete set files in one go at the end of processing, significantly reducing disk IO. This is what the DbIP provider does.

@chr0mag chr0mag added the enhancement New feature or request label Dec 23, 2021
chr0mag pushed a commit that referenced this issue Jan 9, 2022
MaxMind performance improvement.

Closes #16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant