-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: Improve resmap performance with AppendAll and Transform functions #5427
base: master
Are you sure you want to change the base?
Commits on Apr 16, 2024
-
perf: resmap: Add Transform function
Introduce a new function, Transform, which can be used to safely modify entries an resmap. The callback function is called for each entry in the map and handles ID conflict detection, which was the most expensive operation causing significant performance issues. The ID cache is local to this function as we cannot guarantee that all mutations are performed using this function.
Configuration menu - View commit details
-
Copy full SHA for 7202cdb - Browse repository at this point
Copy the full SHA 7202cdbView commit details -
perf: NamespaceTransformer: Use Transform function for conflict detec…
…tion This avoids expensive calls to GetMatchingResourcesByCurrentId
Configuration menu - View commit details
-
Copy full SHA for 0990677 - Browse repository at this point
Copy the full SHA 0990677View commit details -
perf: resmap: Add AppendMany function
Introduce a new function, AppendMany, which can be used to safely add entries to a resmap. The function handles ID conflict detection, which was the most expensive operation causing significant performance issues. The ID cache is local to this function as we cannot guarantee that all mutations are performed using this function.
Configuration menu - View commit details
-
Copy full SHA for f5cdc84 - Browse repository at this point
Copy the full SHA f5cdc84View commit details -
perf: improve applyOrdering by avoid call to GetByCurrentId
This shaves of 14 seconds (one third) of the execution time for a kustomization tree with 4000 documents, from 40.68s to 27.41s 0 0% 5.44% 18.42s 40.56% sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).applySortOrder 0 0% 5.44% 18.40s 40.52% sigs.k8s.io/kustomize/api/internal/builtins.applyOrdering before (pprof) top20 -cum Showing nodes accounting for 5.85s, 12.88% of 45.41s total Dropped 622 nodes (cum <= 0.23s) Showing top 20 nodes out of 157 flat flat% sum% cum cum% 0 0% 0% 40.68s 89.58% github.com/spf13/cobra.(*Command).Execute 0 0% 0% 40.68s 89.58% github.com/spf13/cobra.(*Command).ExecuteC 0 0% 0% 40.68s 89.58% github.com/spf13/cobra.(*Command).execute 0 0% 0% 40.68s 89.58% main.main 0 0% 0% 40.68s 89.58% runtime.main 0 0% 0% 40.68s 89.58% sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1 0 0% 0% 40.12s 88.35% sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run 0.51s 1.12% 1.12% 33.20s 73.11% sigs.k8s.io/kustomize/api/resource.(*Resource).CurId 0 0% 1.12% 26.95s 59.35% sigs.k8s.io/kustomize/api/resmap.(*resWrangler).GetMatchingResourcesByCurrentId 0.35s 0.77% 1.89% 26.95s 59.35% sigs.k8s.io/kustomize/api/resmap.(*resWrangler).filteredById 0.07s 0.15% 2.05% 25.53s 56.22% sigs.k8s.io/kustomize/api/resmap.GetCurrentId 0 0% 2.05% 21.68s 47.74% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline) 0 0% 2.05% 21.68s 47.74% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap 0.54s 1.19% 3.24% 19.75s 43.49% sigs.k8s.io/kustomize/api/resource.(*Resource).GetGvk (inline) 1s 2.20% 5.44% 19.21s 42.30% sigs.k8s.io/kustomize/kyaml/resid.GvkFromNode 0 0% 5.44% 18.42s 40.56% sigs.k8s.io/kustomize/api/internal/builtins.(*SortOrderTransformerPlugin).Transform 0 0% 5.44% 18.42s 40.56% sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).applySortOrder 0 0% 5.44% 18.40s 40.52% sigs.k8s.io/kustomize/api/internal/builtins.applyOrdering 0.87s 1.92% 7.36% 16.55s 36.45% sigs.k8s.io/kustomize/kyaml/yaml.visitMappingNodeFields 2.51s 5.53% 12.88% 15.68s 34.53% sigs.k8s.io/kustomize/kyaml/yaml.visitFieldsWhileTrue after (pprof) top20 -cum Showing nodes accounting for 1.23s, 3.85% of 31.98s total Dropped 584 nodes (cum <= 0.16s) Showing top 20 nodes out of 184 flat flat% sum% cum cum% 0 0% 0% 27.41s 85.71% github.com/spf13/cobra.(*Command).Execute 0 0% 0% 27.41s 85.71% github.com/spf13/cobra.(*Command).ExecuteC 0 0% 0% 27.41s 85.71% github.com/spf13/cobra.(*Command).execute 0 0% 0% 27.41s 85.71% main.main 0 0% 0% 27.41s 85.71% runtime.main 0 0% 0% 27.41s 85.71% sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1 0 0% 0% 26.85s 83.96% sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run 0 0% 0% 22.07s 69.01% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline) 0 0% 0% 22.07s 69.01% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap 0.38s 1.19% 1.19% 20.69s 64.70% sigs.k8s.io/kustomize/api/resource.(*Resource).CurId 0 0% 1.19% 13.64s 42.65% sigs.k8s.io/kustomize/api/resmap.(*resWrangler).Append 0 0% 1.19% 13.55s 42.37% sigs.k8s.io/kustomize/api/resmap.(*resWrangler).GetMatchingResourcesByCurrentId (inline) 0.12s 0.38% 1.56% 13.55s 42.37% sigs.k8s.io/kustomize/api/resmap.(*resWrangler).filteredById 0.01s 0.031% 1.59% 12.67s 39.62% sigs.k8s.io/kustomize/api/resmap.GetCurrentId 0.21s 0.66% 2.25% 12.49s 39.06% sigs.k8s.io/kustomize/api/resource.(*Resource).GetGvk (inline) 0.51s 1.59% 3.85% 12.28s 38.40% sigs.k8s.io/kustomize/kyaml/resid.GvkFromNode 0 0% 3.85% 11.52s 36.02% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).IgnoreLocal 0 0% 3.85% 10.53s 32.93% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).AccumulateTarget 0 0% 3.85% 10.53s 32.93% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateResources 0 0% 3.85% 10.53s 32.93% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateTarget
Configuration menu - View commit details
-
Copy full SHA for e06a969 - Browse repository at this point
Copy the full SHA e06a969View commit details -
perf: Use AppendMany for SortOrderTransformer/newResMapFromResourceSl…
…ice/appendAll Append in a loop is very expensive, O(N*M). By using AppendMany we get closer to O(N+M) instead.
Configuration menu - View commit details
-
Copy full SHA for 0fe5421 - Browse repository at this point
Copy the full SHA 0fe5421View commit details -
perf: Improve SortOrderTransformer and use AppendMany
Reduce calls to CurId when sorting, and use AppendMany when collecting results.
Configuration menu - View commit details
-
Copy full SHA for f6ed5d5 - Browse repository at this point
Copy the full SHA f6ed5d5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4188920 - Browse repository at this point
Copy the full SHA 4188920View commit details -
Regenerate plugins make generate-kustomize-builtin-plugins gofmt -s -w -l plugin/builtin/sortordertransformer/SortOrderTransformer.go etc Possibly split in two, one for appendmany another for transform Run benchmark on both
Configuration menu - View commit details
-
Copy full SHA for 01e154a - Browse repository at this point
Copy the full SHA 01e154aView commit details