Fixing Issue #499 Configuration with Large Number of Indexes #501
Conversation
…of Indexes With a large number of indexes the bootstrapping process was taking a significant amount of time, beyond the default Roxy HTTP configuration. After looking at setup.xqy it was found that some of the index removal calls were iterating over the items instead of calling the removal function with the entire set. Similarly, some of the index creation functions were recursively adding the items instead of calling the admin function with the entire set. After looking at docs for all supported ML versions it was found that calling the functions with all items was supported. Changes were made to remove and add indexes directly. This made a large impact in the bootstrapping time for scenarios with a large number of configuration items. As a reference, in a test setup with around 1,200 configuration items the bootstrap operation was reduced from around 20 minutes to under 2 minutes with the above being the only change.
|
Thanks for the PR! You took the effort eliminating recursion for all index types, which is helpful. Unfortunately, you also eliminated a number of xdmp:eval's, and version checks that allow backwards-compatibility. I'd like to keep existing compatibility in as much as possible. There are still quite some consultants having to work with MarkLogic 6, and sometimes even 5 as well. You also rewrote function calls to create index defs to copy them directly from the ml-config, but the ml-config is not validated, so there could be typos in there, and such. You have to be careful, as reported very recently by Joe: #492. Would you be willing to revise your PR? Just append more commits to it to make changes.. |
|
Thank you for your feedback, I will work to address the items you highlighted and commit back to the PR to make these changes more backwards compatible and also look at the issue of configuration validation before sending to the Admin API. |
Added back xdmp:evals due to backwards compatibility concerns for the creation of: path-namespaces, range-field-indexes, and range-path-indexes which had been changed to direct admin calls. Also added back version check and parameter validation for the creation of range-element-attribute-indexes and range-element-indexes.
|
Committed changes to PR to address where I had removed xdmp:evals, version checks, and validation in the original commit. |
|
I'll run a few tests before merging, but looks very good at first glance. Doesn't solve #492 yet, but we can do that separately.. |
|
@grtjn did you have a chance to test this? |
|
not yet, will try to make some time on short term.. |
|
Looks really promising, this would get rid of a lot of unnecessary complexity. Finally ready for running some tests with this. I'll leverage the self-test, and try to run against ML5 up to 8.. |
|
Discovered issue #520 when running self-test. Otherwise this runs clean against ML5 up to ML8, and my ML8 cluster! Merging now, sorry for the delay! |
Fixing Issue #499 Configuration with Large Number of Indexes
|
Awesome, thanks! |
#499 With a large number of indexes the bootstrapping process was taking
a significant amount of time, beyond the default Roxy HTTP configuration.
After looking at setup.xqy it was found that some of the index removal
calls were iterating over the items instead of calling the removal function
with the entire set. Similarly, some of the index creation functions were
recursively adding the items instead of calling the admin function with the
entire set. After looking at docs for all supported ML versions it was found
that calling the functions with all items was supported.
Changes were made to remove and add indexes directly. This made a large
impact in the bootstrapping time for scenarios with a large number of
configuration items.
As a reference, in a test setup with around 1,200 configuration items the
bootstrap operation was reduced from around 20 minutes to under
2 minutes with the above being the only change.