core: Stack with many resources combined with aspects results in StackOverflowErrors #26517

Saberos · 2023-07-26T12:30:57Z

Describe the bug

When using aspects in combination with stacks with a large number of resources we are running into StackOverflowError during synthesis in our Java CDK applications. The issue is somewhat system dependent as the default limits enforced by the JVM change depending on available system resources (and probably JVM vendor).

Expected Behavior

Stacks with many resources with aspects to synthesize successfully.

Current Behavior

Synthesis fails on a StackOverFlowError.

Reproduction Steps

I have managed to reproduce the issue in a minimal project, available here: https://github.com/Saberos/cdk-aspect-stackoverflowerror

cdk synth produces the error while the aspect is present. Once the aspect is removed synthesis completes as expected. Note that the stack size has been forcibly limited in the cdk.json to ensure the issue is reproducible independent of the system. Without the -Xss flag the issue may or may not appear on the system. Increasing the number of resources in the stack or the number of aspects should allow the issue to be reproduced on any system even without using this flag.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.86.0

Framework Version

2.86.0

Node.js Version

v18.12.1

OS

macOS Ventura 13.2.1

Language

Java

Language Version

Corretto-17.0.7.7.1

Other information

The StackOverflowError stacktrace:

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:java (default-cli) on project cdk-aspect-stackoverflowerror: An exception occured while executing the Java class. null
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:347)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:910)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: org.apache.maven.plugin.MojoExecutionException: An exception occured while executing the Java class. null
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:311)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:910)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: java.lang.StackOverflowError
    at com.fasterxml.jackson.databind.node.NodeCursor$ObjectCursor.currentNode (NodeCursor.java:224)
    at com.fasterxml.jackson.databind.node.TreeTraversingParser.currentNode (TreeTraversingParser.java:374)
    at com.fasterxml.jackson.databind.node.TreeTraversingParser.getText (TreeTraversingParser.java:205)
    at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeAnyScalar (JsonNodeDeserializer.java:564)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize (JsonNodeDeserializer.java:93)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize (JsonNodeDeserializer.java:20)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue (DefaultDeserializationContext.java:323)
    at com.fasterxml.jackson.databind.ObjectMapper._readValue (ObjectMapper.java:4650)
    at com.fasterxml.jackson.databind.ObjectMapper.readTree (ObjectMapper.java:2941)
    at com.fasterxml.jackson.core.JsonParser.readValueAsTree (JsonParser.java:2365)
    at software.amazon.jsii.JsiiObjectMapper$JsiiDeserializer.deserialize (JsiiObjectMapper.java:130)
    at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet (MethodProperty.java:129)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize (BeanDeserializer.java:314)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize (BeanDeserializer.java:177)
    at software.amazon.jsii.JsiiObjectMapper$JsiiDeserializer.deserialize (JsiiObjectMapper.java:149)
    at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet (MethodProperty.java:129)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize (BeanDeserializer.java:314)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize (BeanDeserializer.java:177)
    at software.amazon.jsii.JsiiObjectMapper$JsiiDeserializer.deserialize (JsiiObjectMapper.java:149)
    at com.fasterxml.jackson.databind.ObjectMapper._convert (ObjectMapper.java:4388)
    at com.fasterxml.jackson.databind.ObjectMapper.convertValue (ObjectMapper.java:4344)
    at software.amazon.jsii.JsiiObjectMapper.treeToValue (JsiiObjectMapper.java:46)
    at software.amazon.jsii.JsiiRuntime.processCallbackResponse (JsiiRuntime.java:165)
    at software.amazon.jsii.JsiiRuntime.requestResponse (JsiiRuntime.java:121)
    at software.amazon.jsii.JsiiRuntime.processCallbackResponse (JsiiRuntime.java:197)
... Above 2 lines repeating many times
    at software.amazon.jsii.JsiiRuntime.processCallbackResponse (JsiiRuntime.java:197)
    at software.amazon.jsii.JsiiRuntime.requestResponse (JsiiRuntime.java:121)
    at software.amazon.jsii.JsiiClient.callMethod (JsiiClient.java:184)
    at software.amazon.jsii.Kernel.call (Kernel.java:52)
    at software.amazon.awscdk.Stage.synth (Stage.java:109)
    at com.myorg.CdkAspectStackoverflowerrorApp.main (CdkAspectStackoverflowerrorApp.java:19)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:833)

The text was updated successfully, but these errors were encountered:

peterwoodworth · 2023-07-26T19:44:10Z

Thanks for reporting, might be best in the jsii repo, but we can certainly track it here.

mrgrain · 2023-08-02T16:13:32Z

Hi @Saberos can you confirm why you can't use -Xss to give the app sufficient memory?
Aspects are indeed memory expensive.

Saberos · 2023-08-03T08:06:17Z

@mrgrain just to ensure we're on the same page, the default stack size limit in the JVM seems to be commonly 1-2MB. It isn't so much that the program is incredibly memory expensive, just that aspects appear to be implemented using recursion which can exceed the stack size limit.

Increasing the limit is mostly inconvenient because we have to consider/manage the -Xss flag in all situations where the CDK app is ran, which is not always through cdk.json. Having to consider for example heap size for an application makes sense, but having to increase the stack size due to the implementation of a used library/framework seems a bit much to ask to me.

In my opinion the limit is also hit fairly easily, the scenarios do not seem that extreme. With -Xss1m, which appears to be a common default, I run into the issue at ~700 resources. It does not seem to matter if they are all in a single stack or spread our over multiple stacks, as long as it is a single CDK app. It is not a deeply nested structure in my example, just stacks with a completely flat list of resources. I fail to see the need for recursion instead of simply iterating over all these resources.

mrgrain · 2023-08-03T08:17:40Z

Thanks for the clarification @Saberos ! I agree having to consider -Xss can be a pain, as is managing the heap size.
I'm sure there could be a more stack size friendly implementation for Aspects, which would require a relatively large refactor.

Depending on your scenario, you can also apply Aspects more locally on specific parts of the Construct tree, not just the root. Assigned p2 since there is a workaround.

Saberos · 2023-08-03T11:08:41Z

A workaround for us for now was to refactor an aspect to set IAM permission boundaries to the CDK construct that has been introduced since we set that up. I do not know if that eliminated the recursion of just moves it to Node.js where the stack limit is higher, but for now it is sufficient for us to stay under the limit.

gefloh · 2023-09-18T12:37:39Z

We ran into the exact same problem when using an Aspect as a workaround for the issue described in #19114. Like in the scenario of the OP our stack is also not very big, so this issue came a little unexpected. Setting a higher -Xss in the cdk.json is fine for us and solved the problem but I agree that it is not ideal.

lucasam · 2023-12-21T22:19:13Z

I have a similar problem with a even more weird behavior.
With this aspect i have java.lang.StackOverflowError

public class EnvironmentStageAspect implements IAspect {

    private final StackStageType stackStageType;
    @Override
    public void visit(@NotNull IConstruct iConstruct) {
        if (iConstruct instanceof Function) {
            Function function = (Function) iConstruct;
            function.addEnvironment("BLA", stackStageType.toString());
        }
    }
}

IF i print the construct Id, the issue persists

    public void visit(@NotNull IConstruct iConstruct) {
        System.out.println(iConstruct.getNode().getId());
        if (iConstruct instanceof Function) {
            Function function = (Function) iConstruct;
            function.addEnvironment("BLA", stackStageType.toString());
        }
    }

But, if i print getAllContext the issue got fixed

    public void visit(@NotNull IConstruct iConstruct) {
    
    System.out.println(iConstruct.getNode().getAllContext());
     if (iConstruct instanceof Function) {
            Function function = (Function) iConstruct;
            function.addEnvironment("BLA", stackStageType.toString());
        }
    }
}

Saberos added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jul 26, 2023

github-actions bot added the @aws-cdk/core Related to core CDK functionality label Jul 26, 2023

peterwoodworth added p1 effort/large Large work item – several weeks of effort jsii This issue originates in jsii, or this feature must be implemented in jsii. and removed needs-triage This issue or PR still needs to be triaged. labels Jul 26, 2023

peterwoodworth added the needs-review label Jul 26, 2023

github-actions bot mentioned this issue Aug 1, 2023

Monthly PRs metrics report - July 2023 #26582

Closed

mrgrain added p2 and removed p1 labels Aug 3, 2023

mrgrain removed the needs-review label Aug 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core: Stack with many resources combined with aspects results in StackOverflowErrors #26517

core: Stack with many resources combined with aspects results in StackOverflowErrors #26517

Saberos commented Jul 26, 2023

peterwoodworth commented Jul 26, 2023

mrgrain commented Aug 2, 2023

Saberos commented Aug 3, 2023

mrgrain commented Aug 3, 2023

Saberos commented Aug 3, 2023

gefloh commented Sep 18, 2023

lucasam commented Dec 21, 2023

core: Stack with many resources combined with aspects results in StackOverflowErrors #26517

core: Stack with many resources combined with aspects results in StackOverflowErrors #26517

Comments

Saberos commented Jul 26, 2023

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

peterwoodworth commented Jul 26, 2023

mrgrain commented Aug 2, 2023

Saberos commented Aug 3, 2023

mrgrain commented Aug 3, 2023

Saberos commented Aug 3, 2023

gefloh commented Sep 18, 2023

lucasam commented Dec 21, 2023