Auto partition #644
Auto partition #644
Conversation
|
Can one of the admins verify this patch? |
|
ok to test |
|
Can you split the 503 work separately as that is PR #639. Thanks. |
This uses a pre-defined chunk size so that larger projects do not keep retrying access do the DA host until the request goes through. It should reduce the align time for some of the larger projects without the need to use tweak the max rest size manually.
|
Thanks. Do you have any test cases for this? |
|
Can you also suggest projects you verified the performance changes on - I'd like to look at them myself. |
|
Thanks for getting back about this patch.
Not yet, but I can certainly add a test for this. I have been busy this week (and possibly part of the next) with an ongoing release, so I couldn't dedicate time for this yet.
Sure, you can try on this project: https://github.com/jboss-fuse/camel. I'll send an email w/ the precise command line we use. We have internal details we cannot leak. |
|
Another project you can possibly try looking at is https://github.com/jboss-fuse/cxf. It's not as big as camel, but you should be able to notice an improvement as well. |
|
I have done some basic testing and I do see improvements as well. Nice! Thanks! I would prefer to see tests added and I think the docs (for https://release-engineering.github.io/pom-manipulation-ext/guide/dep-manip.html#rest-endpoint ) will also need updating. Once thats done, I'd be happy to merge. |
|
@orpiske Have you had chance to look at the final bits for this as it would be good to this into a release? |
|
I'm kind swamped, but I will try to send something to you today. |
|
Added both a test as well as a PR #645 with a small note about the auto-sizing. |
Auto partition the chunk size based on the project size. This changes the default behavior so that instead of sending all of the data in one chunk, it actually tries to find an optimal chunk size. In our tests, when analyzing a large project with more than 2200 sub-projects, the time to perform the REST call decreased from ~16 minutes to ~4 minutes. On a medium sized project, with more than 800 sub-projects, the time decreased from ~4 minutes to ~2 minutes.