"sam package": Upload AWS::Include'd files to S3 and/or inline local includes

[0xdevalias]

(Copied from aws/serverless-application-model#567 as it seems to make more sense in this repo)

Originally described in aws/serverless-application-model#345 (comment)

I want to be able to break up my template file into 'logical sections', and use AWS::Include to combine it all back together, but I don't believe SAM will currently upload those included files to S3.

This would allow us to break large unwieldy SAM definitions into more manageable logical chunks.

Alternatively, supporting some form of 'local include' or 'local inline' as part of the package step would also solve this in a nice way (potentially even in a better way..). I would see this as reading a yaml file from a local path, and inlining it at the section of the AWS::Include in the 'packaged' yaml. This could then be deployed as normal.

Potentially relevant:

• Nested stack issue serverless-application-model#90 (comment) (RE: samtranslator)
• Nested stack issue serverless-application-model#90 (comment)
• https://github.com/awslabs/serverless-application-model/blob/master/bin/sam-translate.py

[sanathkr]

Absolutely. Marking this as a feature request for "sam package" command. We need to include these when we port package command from AWS CLI to SAM CLI

[ekcrisp]

I implemented this feature and submit a PR to aws-cli and am awaiting review. I wasn't aware package command was being moved to SAM cli

aws/aws-cli#3454

[georgealton]

@ekcrisp feature was merged into awscli recently, so now its possible to do have a template that has a local path to an Include, that will get automatically uploaded and stored in S3 when the package command is run.

This means we can create a AWS::Serverless::Api Resources that have an included swagger document. This needs to be inline as it's the only way to work with the new SAM Auth functionality, as well as the existing CORS functionality.

Here's an example resource

  ApiGatewayApi:
    Type: AWS::Serverless::Api
    Properties:
      Name: !Sub ${Name}API
      EndpointConfiguration: REGIONAL
      StageName: v1
      Auth:
        Authorizers:
          TokenAuth:
            FunctionPayloadType: TOKEN
            FunctionArn: !Ref AuthorizerLambda
            FunctionInvokeRole: !Ref AuthorizerRole
            Identity:
              Header: Authorization
              ValidationExpression: Bearer.*
              ReauthorizeEvery: 0 
      DefinitionBody:
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: swagger.yaml
      MethodSettings:
        - LoggingLevel: INFO
          MetricsEnabled: True
          # Trace-level Logging
          DataTraceEnabled: False
          # On all Paths & methods
          ResourcePath: "/*"
          HttpMethod: "*"

however with this template, attempting to run a sam validate raises an InvalidResourceException using sam validate

 george@Georges-MacBook-Pro  ~/evertz.io/tenant-management   master  sam validate
2018-10-30 10:36:53 Found credentials in shared credentials file: ~/.aws/credentials
Template provided at '/Users/george/evertz.io/tenant-management/template.yaml' was invalid SAM Template.
Error: [InvalidResourceException('ApiGatewayApi', "Unable to add Auth configuration because 'DefinitionBody' does not contain a valid Swagger")] ('ApiGatewayApi', "Unable to add Auth configuration because 'DefinitionBody' does not contain a valid Swagger")
 ✘ george@Georges-MacBook-Pro  ~/evertz.io/tenant-management   master  git rev-parse HEAD
86c6ffa44eaf608eeb0e81ef36c6276b4fd923f4

this commit deploys via cloudformation

This also affects cfn-lint, so It seems like this might be better implemented in the aws-sam-translator?

[ekcrisp]

@georgealton i did notice some things in sam local that were a little off but since my team is not blocked we are moving on. Also I'm not sure what the difference is between SAM validate and CF validate but we haven't had any issues with aws cloudformation validate command.

[dgc-ttam]

after running aws cloudformation package --template-file template.yaml --output-template-file sam-template.yaml --s3-bucket <mybucket> with a similar AWS::Include I get a sam-template.yaml with a clause that looks like:

      DefinitionBody:
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: oas-sub.yaml

and then aws cloudformation deploy --template-file sam-template.yaml gives me:
Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Transform AWS::Include failed with: The location parameter is not a valid S3 uri.
@georgealton how did you get your stack to deploy with a local include?
(fwiw: awscli: aws-cli/1.16.22 Python/3.7.0 Darwin/18.2.0 botocore/1.12.12
sam: 0.9.0)

[ekcrisp]

Try updating awscli, I think it was added in a newer version

[dgc-ttam]

it looks like it's an issue with my sam build:
sam build copies the template to .aws-sam/build/template.yaml and rewrites my include to be a relative path

        Location: ../../oas-sub.yaml

aws cloudformation package appears to ignore this relative path. if I change the sam built template to reference an absolute path or a path in the same directory then aws cloudformation package uploads the snippet and replaces Location with an s3 url.

[dgc-ttam]

actually, it looks like aws cloudformation package resolves local Location paths relative to the $PWD, but sam build is constructing paths relative to the template. aws cloudformation package doesn't modify the Location if it doesn't find the file.

[abbottdev]

Am I right in thinking that aws cli added support for this recently? Currently I believe I'm able to use the following to include a local import (and get the right substitutions):

  HelloWorldApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      DefinitionBody:
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: ./swagger.yaml

[deser]

Hi Guys,
has been the issue with SAM build resolved?

[ravsau]

any updates? I have this issue working with local swagger file.

Hi Guys,
has been the issue with SAM build resolved?

Update: For now I've solved this issue by uploading the swagger file to s3 before the sam package command in our deployment pipeline. And I reference the s3 file in my sam template.

 DefinitionBody:
          'Fn::Transform':
            Name: 'AWS::Include'
            Parameters:
              Location: s3://bucket-name/swagger/swagger.yaml 

[hugoduraes]

What's the status of this?

[sanathkr]

We have a whole new set of changes for sam package and sam deploy where you don't need AWS CLI anymore. I believe these issues should be resolved. Try running sam build followed by sam package and sam deploy.

Feel free to re-open the Issue if this is not the case.

[karthikvadla]

Am I right in thinking that aws cli added support for this recently? Currently I believe I'm able to use the following to include a local import (and get the right substitutions):

  HelloWorldApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      DefinitionBody:
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: ./swagger.yaml

I still get this error referencing to local path.

Creating ChangeSet-2020-08-05T16-33-11Z in sam-TestServiceLambda failed
    "StatusReason": "Transform AWS::Include failed with: The location parameter is not a valid S3 uri.",

Command failed.
Please see the logs above.

[ekcrisp]

@karthikvadla this was added to aws cli cloudformation package, not sam cli. You can try using https://docs.aws.amazon.com/it_it/cli/latest/reference/cloudformation/package.html

[PraveenSaga]

aws/aws-cli#3708 this problem exists with sam cli too. Files included by include transform are not uploading to s3 by "sam deploy" or "sam package"