Nested stack issue

[jamakase]

I am trying to make stack with nested stacks, but when i have Transform in nested stack, i get an error:
CreateStack cannot be used with templates containing Transforms. If this error is expected, this should be added somewhere to docs, that only parent stack should have Transforms.

In case i delete Transforms from nested stacks, i get Circular dependency between resources: [ApiGatewayApi, InviteUserFunction, LambdaInvokePermission], which i do not have if i create this stack separately.

Here is part of nested stack template

AWSTemplateFormatVersion: 2010-09-09
Resources:
  ApiGatewayApi:
    Type: AWS::Serverless::Api
    Properties:
      DefinitionUri: ./../target/swagger.yaml
      StageName: Prod
      Variables:
        InviteUserFunctionName: !Ref InviteUserFunction
  InviteUserFunction:
    Type: AWS::Serverless::Function
    Properties:
      Timeout: 30
      Role: !GetAtt LambdaFunctionIAMRole.Arn
      Handler: com.MyHandler
      Runtime: java8
      CodeUri: ./../target/some.jar
      Events:
        GetRequest:
          Type: Api
          Properties:
            RestApiId: !Ref ApiGatewayApi
            Path: /invite
            Method: POST
  LambdaInvokePermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName:
        !GetAtt InviteUserFunction.Arn
      Principal: apigateway.amazonaws.com
      SourceArn:
        Fn::Join:
        - ''
        - - 'arn:aws:execute-api:'
          - Ref: AWS::Region
          - ":"
          - Ref: AWS::AccountId
          - ":"
          - Ref: ApiGatewayApi
          - "/*"

[sanathkr]

You should use create-change-set and execute-change-set APIs to create a stack using with Transforms. create-stack API won't work

[ataeoalii]

@sanathkr how would you do that with a nested stack? There's no ChangeSet resource.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html

[schmitzcc]

+1

[windlessuser]

+1

[dejonghe]

Came here to mention something specifically about the Change Set requirement and Nested stacks. Currently Change Sets do not traverse Nested stacks, so I'm going to operate under the assumption that you can not nest with SAM which is a major pitfall.

[alanwill]

Only the reverse seems to work, i.e. you can have a SAM enabled template as the parent template just not the child, which is quite a bummer and limiting.

[alanwill]

This really should be tagged as a bug not a question @sanathkr

[sanathkr]

How about including Transform: 'AWS::Serverless-2016-10-31' to the parent template even if it doesn't use SAM resources. AFAIK, the nested templates should get transformed as well when they are processed.

If not, this is a bug that needs to be fixed.

[alanwill]

@sanathkr tried that before and just did it again to be sure, same error CreateStack cannot be used with templates containing Transforms. Thanks for marking this as a bug, hope it can get addressed soon.

[dinvlad]

Same here CreateStack cannot be used with templates containing Transforms. I hit the 200 resource limit in the SAM template, so it has to be split into nested stacks. Otherwise the development is stalled at the moment. Besides, that would be better for organizing these many resources.

EDIT: well, there are other resources that we can nest as @alanwill mentioned, which alleviates the problem in the short term but that's not the point. Organizing stacks by functionality would be good practice, but right now we're forced to put all of the functions into the same SAM template, and split off their supporting resources into nested stacks, even though logically we'd group functions by their resource domain, so to speak.

EDIT2: as mentioned by @dejonghe, this comes down to a bigger question of adding CloudFormation support to create change sets for nested stacks (which would be good by itself). Without that, I don't see how the current SAM implementation could solve this issue (i.e transforms require change sets).

EDIT3: the 200 limit above was actually reached after template "expansion" by transform. I have just 100 resources in total, including 54 SAM functions. Each function currently gets expanded into a "regular" CloudFormation function plus a function role resource, and if we bind them to an API (which is a crucial piece of the whole model), each trigger also gets 2 AWS::Lambda::Permission resources, one for the Test stage (which is created automatically by SAM for some reason), and one for the stage that we specify. So each function gets expanded into ~3 resources on average. That explains why I hit the limit: 54 functions x 3 + 46 other resources ~ 200.

So this places an upper limit on the total number of functions in SAM of ~ 70, if there are no other resources. A reasonable API can have more than that (and there are plenty of functions that can comprise the backend without directly serving the API).

One workaround for us besides making space for ~15 more functions (which we will hit pretty soon), is to use regular functions, which sort of defeats the purpose of SAM. Another alternative is creating multiple independent SAM stacks, and deploying them through changesets in a script, but that may be more cumbersome than simply using nested stacks.

Could we at least request that the 200 resource limit is increased globally by the CloudFormation team? I'm not sure what the proper channel for that is, as the official Support request doesn't allow to specify this limit (only the one for the total number of stacks). This would at least allow most teams to continue their work without interruption, until this is resolved (which may take a while, if I understood correctly).

(Sorry if that's beyond the scope of this issue)

[alanwill]

Any thoughts on this @sanathkr I now have a workaround to a workaround just to try to get around this. The workaround was to put all the SAM resources in the parent stack, but then that go too large in a single stack so I have another workaround to that. Any guidance you can provide here would be super helpful.

[dejonghe]

What I've done is create a CFN Template launcher in python with Boto that goes through all the steps. I have a Yaml file that defines stacks and can grab output from one stack and pass it as a parameter to another.

[Dunedan]

@sanathkr Any news regarding this issue? It'd be so great to have that fixed.

[rasika-jay]

Having the same issue, any update @sanathkr?

[dinvlad]

We moved away from SAM due to this problem. At the cost of more verbose templates and an extra packaging step, we also got more flexibility with 'native' resources.

[Widdershin]

Even further than that, this issue was a factor in my team dropping CloudFormation in favour of Terraform.

At least with Terraform, we can fork and fix issues ourselves, rather than just waiting.

[aehlke]

@dinvlad could you share what your template looks like?

[dinvlad]

@aehlke serverless functions can be replaced with 3 resources: AWS::Lambda::Function, AWS::IAM::Role and AWS::Lambda::Permission, then you just customize them as needed:

  Function:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.get
      Environment:
        Variables:
          VAR_NAME: !Ref ...
      Runtime: nodejs6.10
      Code:
        S3Bucket: !Ref LambdaBucket
        S3Key: !Ref LambdaPackageKey
      Role: !GetAtt FunctionRole.Arn

  FunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: FunctionRole
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - service:Action
                Resource:
                  - !GetAtt Resource.Arn

  FunctionRolePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref Function
      Action: lambda:InvokeFunction
      Principal: apigateway.amazonaws.com
      SourceArn:
        !Sub arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${ApiId}/${ApiStage}/GET/...

Before execution of the stack, you also need to upload lambda package as a zip file to s3://${LambdaBucket}/${LambdaPackageKey}.

[brettstack]

That's not what we have in mind. More like it will perform the transform on each individual template

[SamanehU]

Hope there will be a possibility in AWS SAM CLI to transform SAM template locally soon. Then I will be able to have the transformed stack as a nested stack in my master stack. @brettstack When this feature will be available to use?

[brettstack]

I can't comment on timeline, but we're actively talking about this internally and we'll create a thread in the SAM CLI repo for more discussion and details soon.

You could potentially hack this together yourself with sam validate --debug which logs the transformed template in latest SAM CLI version, but it won't be ideal. Alternatively, if you're really ambitious you could fork the CLI and have a crack at implementing Local Transform

[Dunedan]

@brettstack: Just out of curiosity: What's the motivation to solve that by applying the transformations locally? Couldn't that be done in the same way by CloudFormation as the transformation of the parent template?

[brettstack]

We want to do local transform for other reasons anyway. Two birds sort of deal.

[christianklotz]

I happened to run into the same nested SAM template issue a couple of days ago. Although it would be great if SAM templates would be expanded automatically by CloudFormation, for now I use sam-translate.py as workaround locally and for deployment.

1. Package SAM template to make sure all CodeUri point to S3 urls since aws-sam-translator doesn't support local uris.
2. Expand the packaged template using sam-translate.py
3. Upload expanded template to S3 bucket.
4. Deploy the root stack, providing the bucket the templates have been uploaded to.

Steps

I didn't have aws-sam-translator installed so the following worked for me.

pip install docopt PyYAML aws-sam-translator

As far as I can tell sam-translate.py itself doesn't get installed so create a copy of it on my machine.

sam package
  --s3-bucket "$BUCKET" \
  --template-file "$TEMPLATE_DIR/my-app.sam.yaml" \
  --output-template-file "$TEMPLATE_DIR/my-app.sam.pkg.yaml"

python sam-translate.py \
  --input-file="$TEMPLATE_DIR/my-app.sam.pkg.yaml" \
  --output-file="$TEMPLATE_DIR/my-app.cfn.json"

aws s3 cp --recursive "$TEMPLATE_DIR" "s3://$BUCKET/templates"

aws cloudformation deploy --template-file ./my-root-stack.yaml

Example of the root stack

Note: The translated template is a JSON file, not YAML.

AWSTemplateFormatVersion: 2010-09-09

Description: Nested stack featuring SAM

Parameters:
  TemplateBucket:
    Type: String
    Default: deployment-artifacts
    Description: |
      The bucket for the CloudFormation templates and packaged SAM application.

Resources:
  MyAppStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub https://s3.amazonaws.com/${TemplateBucket}/templates/my-app.cfn.json

[milancermak]

FWIW, I think local transformations are a grand idea. I have a CodePipeline pipeline set up, the CloudFormation CREATE_UPDATE deployment step doesn't work with SAM templates. I had to add a sam-translate step (as @christianklotz suggested 🔝 👏 ) to CodeBuild to make it all work.

[MathewLeung]

Thanks @christianklotz for sharing the local transformation workaround, however I still seem to be running into the same problem even with nested SAM.

Basically my SAM is now getting too big just getting over 200 resources. CloudWatch Log and DynamoDB are already separated out so what is in SAM are the Roles, API, Lambda, API Keys, Usage Plan.

I followed your instructions with the below approaches

1. With nested SAM, where root SAM has the API definition in its own sam file and lambda definitions are separated out in their own logical domains sam files, the local transformation of the lambda SAM will still complain about 'RestApiId must be a valid reference to an 'AWS::Serverless::Api' resource in same template', so no cloudformation template is generated

2. With the original sam, the generated cloudformation still contains over 200 resources and cannot be used to update the existing stack that was initially created via the sam changeset.

I must have missed something from your instructions, really appreciate for your help here.

[brettstack]

This is a top priority for us. Sorry for the headaches 🤕

[revolutionisme]

I understand that there is still no direct solution to use CreateStack with transforms. But is there a workaround where I can use REPLACE_ON_FAILURE action mode with a sam template? I would need this to reduce our testing time which for now is quite some time as any failure makes us undeploy the stack and redeploy again.

[Amanfavas]

@sanathkr, I am also getting the same issues
I will be creating the AWS lambda and API gateway creation through cloud formation file[FileName is template.yml]. I used the nested stack approach. Below I am giving my nested stack cloud formation code.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
deliforceserver:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: https://s3.ap-south-1.amazonaws.com/deliforcesck/template.template
TimeoutInMinutes: '60'
Outputs:
StackRef:
Value: !Ref deliforceserver
OutputFromNestedStack:
Value: !GetAtt deliforceserver.Outputs.BucketName

I will execute this file. I am getting an error like this "CreateStack cannot be used with templates containing Transforms."
I removed this line Transform: AWS::Serverless-2016-10-31 from template.yml and executed once again. That time getting an error as "Template format error: Unrecognized resource types: [AWS::Serverless::Function, AWS::Serverless::Api]".
How to fix this issues?
could you please provide the exact code for creating the lambda and apigateway through nested stack approach

[revolutionisme]

@Amanfavas your template.template files contains serverless functions. There are 2 ways you can fix it.

1. In your code pipeline, rather than using CREATE_UPDATE, use CREATE_CHANGESET, followed by EXECUTE_CHANGESET.

Or

2. Change your AWS::serverless::function to AWS::lambda::function (also change serverless API as well)

[Amanfavas]

@revolutionisme Thanks for replying my comment. I am used CREATE_CHANGESET for creating new API and lambda.But it getting more than 200 resources file is not possible. Like that. And also I changed AWS::Serverless::function to AWS::lambda::function. But that one also not working.If anybody know solution kindly provide the solution

[revolutionisme]

Maybe someone else can help you then, also posting the exact error you got after applying the changes helps as well.

[Amanfavas]

@revolutionisme my exact issue already I posted in above.

[keetonian]

@Amanfavas Cloudformation doesn't support transforms (like the SAM transform) in nested stacks. The template that you provide to the AWS::CloudFormation::Stack resource cannot require a transform (which means it can't have any AWS::Serverless resources inside of it). As @brettstack said earlier, this is a top priority for us to get this working.

Edit: I mistakenly wrote AWS::Serverless::Stack instead of AWS::Cloudformation::Stack

[MathewLeung]

@revolutionisme Thanks for replying my comment. I am used CREATE_CHANGESET for creating new API and lambda.But it getting more than 200 resources file is not possible. Like that. And also I changed AWS::Serverless::function to AWS::lambda::function. But that one also not working.If anybody know solution kindly provide the solution

I followed the example posted by @dinvlad on Sep 16, 2017 to get around the limit. You should give it a go.

[Amanfavas]

@keetonian In the template I provide AWS::Serverless::Stack as the resource type. But it getting an error like this
Template format error: Unrecognized resource types: [AWS::Serverless::Stack]
How to solve this?
Before You worked in a nested stack, Kindly provide the code for parent file[nested stack file] as well as chilld File[template.yml]

[keetonian]

@Amanfavas AWS::Serverless::Stack is not a valid resource type, sorry I mistyped the name of the resource above. It is currently not possible to nest a SAM template as a AWS::CloudFormation::Stack (nested stack) resource, you can only nest vanilla CloudFormation templates as this resource type.

[Amanfavas]

@keetonian Sorry,I didn't get the nest vanilla cloudformation templates

[keetonian]

Nested transforms have been released! Check out SAM v1.9.0 release notes for more information!