Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: Improve resmap performance with AppendAll and Transform functions #5427

Draft
wants to merge 8 commits into
base: master
Choose a base branch
from
Draft

perf: Improve resmap performance with AppendAll and Transform functions #5427

wants to merge 8 commits into from

Commits on Apr 16, 2024

  1. perf: resmap: Add Transform function 

    Introduce a new function, Transform, which can be used to safely modify
entries an resmap.

The callback function is called for each entry in the map and handles ID
conflict detection, which was the most expensive operation causing
significant performance issues.

The ID cache is local to this function as we cannot guarantee that all
mutations are performed using this function.
    @chlunde
    chlunde committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    7202cdb View commit details
    Browse the repository at this point in the history

  2. perf: NamespaceTransformer: Use Transform function for conflict detec… 

    …tion

This avoids expensive calls to GetMatchingResourcesByCurrentId
    @chlunde
    chlunde committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    0990677 View commit details
    Browse the repository at this point in the history

  3. perf: resmap: Add AppendMany function 

    Introduce a new function, AppendMany, which can be used to safely add
entries to a resmap.

The function handles ID conflict detection, which was the most expensive
operation causing significant performance issues.

The ID cache is local to this function as we cannot guarantee that all
mutations are performed using this function.
    @chlunde
    chlunde committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    f5cdc84 View commit details
    Browse the repository at this point in the history

  4. perf: improve applyOrdering by avoid call to GetByCurrentId 

    This shaves of 14 seconds (one third) of the execution time for a
kustomization tree with 4000 documents, from 40.68s to 27.41s

            0     0%  5.44%     18.42s 40.56%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).applySortOrder
            0     0%  5.44%     18.40s 40.52%  sigs.k8s.io/kustomize/api/internal/builtins.applyOrdering
before

    (pprof) top20 -cum
    Showing nodes accounting for 5.85s, 12.88% of 45.41s total
    Dropped 622 nodes (cum <= 0.23s)
    Showing top 20 nodes out of 157
        flat  flat%   sum%        cum   cum%
            0     0%     0%     40.68s 89.58%  github.com/spf13/cobra.(*Command).Execute
            0     0%     0%     40.68s 89.58%  github.com/spf13/cobra.(*Command).ExecuteC
            0     0%     0%     40.68s 89.58%  github.com/spf13/cobra.(*Command).execute
            0     0%     0%     40.68s 89.58%  main.main
            0     0%     0%     40.68s 89.58%  runtime.main
            0     0%     0%     40.68s 89.58%  sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1
            0     0%     0%     40.12s 88.35%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run
        0.51s  1.12%  1.12%     33.20s 73.11%  sigs.k8s.io/kustomize/api/resource.(*Resource).CurId
            0     0%  1.12%     26.95s 59.35%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).GetMatchingResourcesByCurrentId
        0.35s  0.77%  1.89%     26.95s 59.35%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).filteredById
        0.07s  0.15%  2.05%     25.53s 56.22%  sigs.k8s.io/kustomize/api/resmap.GetCurrentId
            0     0%  2.05%     21.68s 47.74%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline)
            0     0%  2.05%     21.68s 47.74%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap
        0.54s  1.19%  3.24%     19.75s 43.49%  sigs.k8s.io/kustomize/api/resource.(*Resource).GetGvk (inline)
            1s  2.20%  5.44%     19.21s 42.30%  sigs.k8s.io/kustomize/kyaml/resid.GvkFromNode
            0     0%  5.44%     18.42s 40.56%  sigs.k8s.io/kustomize/api/internal/builtins.(*SortOrderTransformerPlugin).Transform
            0     0%  5.44%     18.42s 40.56%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).applySortOrder
            0     0%  5.44%     18.40s 40.52%  sigs.k8s.io/kustomize/api/internal/builtins.applyOrdering
        0.87s  1.92%  7.36%     16.55s 36.45%  sigs.k8s.io/kustomize/kyaml/yaml.visitMappingNodeFields
        2.51s  5.53% 12.88%     15.68s 34.53%  sigs.k8s.io/kustomize/kyaml/yaml.visitFieldsWhileTrue

after

    (pprof) top20 -cum
    Showing nodes accounting for 1.23s, 3.85% of 31.98s total
    Dropped 584 nodes (cum <= 0.16s)
    Showing top 20 nodes out of 184
        flat  flat%   sum%        cum   cum%
            0     0%     0%     27.41s 85.71%  github.com/spf13/cobra.(*Command).Execute
            0     0%     0%     27.41s 85.71%  github.com/spf13/cobra.(*Command).ExecuteC
            0     0%     0%     27.41s 85.71%  github.com/spf13/cobra.(*Command).execute
            0     0%     0%     27.41s 85.71%  main.main
            0     0%     0%     27.41s 85.71%  runtime.main
            0     0%     0%     27.41s 85.71%  sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1
            0     0%     0%     26.85s 83.96%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run
            0     0%     0%     22.07s 69.01%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline)
            0     0%     0%     22.07s 69.01%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap
        0.38s  1.19%  1.19%     20.69s 64.70%  sigs.k8s.io/kustomize/api/resource.(*Resource).CurId
            0     0%  1.19%     13.64s 42.65%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).Append
            0     0%  1.19%     13.55s 42.37%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).GetMatchingResourcesByCurrentId (inline)
        0.12s  0.38%  1.56%     13.55s 42.37%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).filteredById
        0.01s 0.031%  1.59%     12.67s 39.62%  sigs.k8s.io/kustomize/api/resmap.GetCurrentId
        0.21s  0.66%  2.25%     12.49s 39.06%  sigs.k8s.io/kustomize/api/resource.(*Resource).GetGvk (inline)
        0.51s  1.59%  3.85%     12.28s 38.40%  sigs.k8s.io/kustomize/kyaml/resid.GvkFromNode
            0     0%  3.85%     11.52s 36.02%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).IgnoreLocal
            0     0%  3.85%     10.53s 32.93%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).AccumulateTarget
            0     0%  3.85%     10.53s 32.93%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateResources
            0     0%  3.85%     10.53s 32.93%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateTarget
    @chlunde
    chlunde committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    e06a969 View commit details
    Browse the repository at this point in the history

  5. perf: Use AppendMany for SortOrderTransformer/newResMapFromResourceSl… 

    …ice/appendAll

Append in a loop is very expensive, O(N*M). By using AppendMany we get closer to O(N+M)
instead.
    @chlunde
    chlunde committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    0fe5421 View commit details
    Browse the repository at this point in the history

  6. perf: Improve SortOrderTransformer and use AppendMany 

    Reduce calls to CurId when sorting, and use AppendMany when collecting
results.
    @chlunde
    chlunde committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    f6ed5d5 View commit details
    Browse the repository at this point in the history

  7. perf: ResAccumulator: Use map for resmap intersection

    @chlunde
    chlunde committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    4188920 View commit details
    Browse the repository at this point in the history

  8. NOTE: Draft, commiting 

    Regenerate plugins

	make generate-kustomize-builtin-plugins
	gofmt -s -w -l plugin/builtin/sortordertransformer/SortOrderTransformer.go etc

Possibly split in two, one for appendmany another for transform

Run benchmark on both
    @chlunde
    chlunde committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    01e154a View commit details
    Browse the repository at this point in the history