-
Notifications
You must be signed in to change notification settings - Fork 2.4k
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Possibility to get AWS::Serverless::RestApi passed a template variable to Template files with Functions only #349
Comments
You can use CloudFormation export and import to pass the parameters from one stack to a different stack: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-stack-exports.html To deal with resource limit issue, CloudFormation recommends to use nested stacks: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-nested-stacks.html
|
@honglu thanks for your reply. We have been doing what you suggested above and now only serverless resources are left in the SAM when we hit the CF limit after transformation. There were 3 options we came up with and tried but all failed:
So at the end we are still blocked in that catch-22 situation due to variety of limits and limitations of SAM and CF. Is there any other way or ideas that can help us overcome it? PS: However I also came across another issue - #335 that indicates the option-1 was working until a few days back, which is really interesting and worth watching progressing on that issue. |
Hi Folks, #149 seems to imply one cannot split the api resource from the function resources. This issue seems to imply it is indeed possible, without giving template examples. I would be forever grateful if someone who's managed to do this would post an actual example that works. Many thanks in advance. |
We managed to split them before, but no longer. It suddenly stopped working so we had to redo all of our stack structure which meant a downtime in production. See #335 |
Does anyone know if this issue is dead? Or does anyone know if there is a way to use the aws lambda update-function-code CLI method to update the code and roll it out with checkpoints? |
It would also be great if |
Nested stacks are now supported in SAM, meaning you can nest resources in other stacks and use the outputs as values in the current stack. |
Even though they are supported, when I created the |
Can you post an example that creates this issue? Is it as simple as defining an API in one stack and referencing a Function in that API that is from a nested stack? |
Sure! These are two stacks that cause an error. template.yml
substack.yml
|
Thanks for the example. SAM currently requires the API and lambda functions invoked by the API Gateway to be in the same template in order to use the implicit API generation functionality of SAM. This is because the swagger is generated in the template and references the serverless functions using Are you saying you are hitting the CFN 200 resource limit when your template only contains the API Gateway resources and lambda function (and IAM roles) directly invoked by API Gateway and all other resources are in nested stacks? This would imply that you have maybe ~70 resource+methods all being serviced by separate Lambda functions. Is that correct? Some alternatives you could try:
Adding SAM support for the same intuitive API mapping within the |
@jlhood: Let med add some more feedback. I am composing serverless applications using the AWS::Serverless::Application resource. In one SAM template I need to nest two applications that both use an API gateway. In this scenario it's not possible to make the two nested applications share the api gateway because:
Any ideas for workarounds? |
@cshenrik Really appreciate the additional context and feedback. Agreed, adding triggers based on resources in your current template to functions defined within a nested application is not made easy by SAM today. Currently, you would have to manage your own API swagger document to add the nested app function as an integration. What do you think about the possibility of being able to specify event sources on the NestedApp:
Type: AWS::Serverless::Application
Properties:
Location:
ApplicationId: <app id>
SemanticVersion: 1.0.0
Events:
GetRoot:
Type: Api
Properties:
Path: /
Method: GET
OutputFunctionName: MyHandlerFunctionName
OutputFunctionArn: MyHandlerFunctionArn So here I have a nested SAR app called Need to flesh out the exact syntax and some more details but wanted to put it out there to get some feedback. |
I think adding a new property to Serverless::Api would solve this problem and more. Maybe we call this |
Just hit the CFN 200 limit today (yes, @jlhood, I am reusing roles) and looking for a workaround led me to this issue. I initially tried to pass the "AWS::Serverless::Api" as a parameter to "AWS::Serverless::Application" but then I got the dreaded "The REST API doesn't contain any methods" error as it seems CFN is trying to deploy the API before it builds the other nested stacks. Any workarounds? EDIT: Looking at my last created CFN stack the problem seems to be the Lambda Permissions is being created 2 times for each lambda function as pointed out in #285. I would love to move to a nested stack solution but #349 prevents me from this. I really don't want to create a different API endpoint for different resources but that's all I that comes to mind right now. |
@brettstack I'm thinking of your proposed solution and I'm not sure I follow. How can adding a property to the Serverless:Api make the nested stacks all share the same Serverless:Api reference? |
You would use GetAtt on the Lambda Functions Output from the nested stack. The API would then create the permissions and set up the route just like it does when you define API Events on a Function. |
@brettstack Yeah it would be interesting to also have a simplified syntax around adding function integrations on the Although again, I'm not opposed to adding a way to connect functions to |
Are you saying the S3 bucket is defined in the parent template? Can't you just pass through the S3 bucket ARN and SAM will create the permission for you? I suppose SAM also modifies the notification configuration on the bucket (not sure off the top of my head if something similar happens for SNS), so you would lose that (though you can set it yourself). |
@brettstack and @jlhood this sounds great. Are you going to put out an RFC, or get this prioritized. I think more people with hit the 200 CFN limit sooner than later due to #285. If #285 is fixed it may give you some breathing room to flesh out the RFC with more customer feedback. My current workaround is to have multiple API endpoints. It’s ugly but it works for now. BTW keep up the great work. |
@brettstack I got excited for a minute because I thought you were saying there is an existing workaround for shared Gateway - but I see you were thinking out loud. +1 for needing this, we're currently breaking up our API set into several separate gateways because of the limit. It's that permission per endpoint that's killing us - we have maybe a dozen lambdas that each handle maybe another dozen endpoints, and with 2 permissions per endpoint, we're blowing right through the max. My first thought was to break up into sub stacks, but the circular dependency/chicken&egg with the gateway and the functions had me running in circles. I'll be looking forward to enhancements in this area! |
@jlhood Has there been any traction on this? Not only is this great for applications with a large number of endpoints, but also for the sake of of modularity. In addition, I'm wondering if another, slightly similar, solution might be the possibility of being able to modularize our Swagger/OpenAPI documents for each function? I apologize if this is slightly out of scope, but I think this ticket illuminates the issue with modularity and AWS SAM. I also apologize if I overlook a reality that would prevent this solution from being implemented in CF. I'm relatively new to architecting serverless backends. Say I have a few lambda functions each acting as a separate endpoint (GET foo/bar/, POST foo/bar, etc...). For each I could write the specific paths and definitions for that endpoint. On deployment CF would add each APIG endpoint (resources, methods, models, etc..) based off the modular definition. In the subtemplate of the function, you could have a resource like In the master template you would have the A few caveats with this approach might be sacrificing some of the ability to combine definitions provided by OpenAPI. For example my I realize that this is a non-trivial solution 😄. Despite this issue, I really enjoy using SAM to deploy my backend. Thanks for all the awesome work! |
This is the approach I took: import {
Stack,
NestedStack,
aws_lambda as Lambda
} from "aws-cdk-lib";
import * as ApiGatewayV2 from "@aws-cdk/aws-apigatewayv2-alpha";
class HttpApi extends ApiGatewayV2.HttpApi {
public addRoute(handler: Lambda.IFunction, method: ApiGatewayV2.HttpMethod, path: string): void {
new ApiGatewayV2.HttpRoute(this, `${method}${path}`, {
httpApi: this,
routeKey: ApiGatewayV2.HttpRouteKey.with(path, method),
integration: new ApiGatewayV2Integrations.HttpLambdaIntegration(`${method}${path}LambdaIntegration`, handler),
});
}
}
class ChildStack extends NestedStack {
constructor(scope: Stack, id: string, httpApi: HttpApi) {
super(scope, id);
const handlerFunction = new Lambda.Function(this, "Handler", {
functionName: "Handler",
runtime: Lambda.Runtime.NODEJS_14_X,
handler: `function.handler`,
code: Lambda.Code.fromAsset('path/to/code'),
});
httpApi.addRoute(handlerFunction, ApiGatewayV2.HttpMethod.GET, "/mock/path")
}
}
class ParentStack extends Stack {
constructor(scope: Construct, id: string) {
super(scope, id);
const httpApi = new HttpApi(this, "HttpApi");
new ChildStack(this, "ChildStack", httpApi);
}
} This allowed me to create a single HttpApi in the parent stack, and eight child stacks, all using it. Not sure if this approach will work with a RestApi or not. |
I'm having this issue right now, no workarounds for now :/ |
surprised to see we are in 2022 and this issue still exists |
as for now I have to drop sam and deploy apigw + lambda via cli in Jenkinsfile, LOL. |
Same issue here, any update??? |
Also looking for a solution here |
Any updates on this? |
Seriously guys, we really need to pass down RestAPI to child stacks :D |
The number of people and orgs looking for this feature. Is it really that hard to AWS SAM peeps? |
So to explain, yes, this has technical limitations getting in the way. Right now, when you create an
Doing step 2 is the tricky part, because it has to happen right now where the API is defined. You could use One possible option we could explore if there was high demand for this would be to add some sort of route helpers to the |
This doesn't answer the original question exactly (using a single API Gateway), but if you're OK using an API Gateway per project (let's say each project is a microservice made up of multiple lambdas in each project/microservice), then this is how I did it. Posting in case it's useful to anyone. This enables some modularity so you can break your services down across multiple projects. The downside is that you create an API Gateway per project (which may OK for you if you're want to use the "Multiple API Gateway" pattern referenced here: https://docs.aws.amazon.com/prescriptive-guidance/latest/modernization-integrating-microservices/api-gateway-pattern.html#multiple-api-gateways). Details posted here: |
Adding a reproducible example for posterity. Save the following as Transform: AWS::Serverless-2016-10-31
Resources:
MyApi:
Type: AWS::Serverless::Api
Properties:
StageName: prod
MyFunction:
Type: AWS::Serverless::Function
Properties:
Runtime: python3.8
Handler: foo
InlineCode: bar
Events:
ApiEvent:
Type: Api
Properties:
Method: get
Path: /
RestApiId: !Ref MyApi Deploy:
It transforms into the following template: Transformed CloudFormation template
{
"Resources": {
"MyFunction": {
"Type": "AWS::Lambda::Function",
"Metadata": {
"SamResourceId": "MyFunction"
},
"Properties": {
"Code": {
"ZipFile": "bar"
},
"Handler": "foo",
"Role": {
"Fn::GetAtt": [
"MyFunctionRole",
"Arn"
]
},
"Runtime": "python3.8",
"Tags": [
{
"Key": "lambda:createdBy",
"Value": "SAM"
}
]
}
},
"MyFunctionApiEventPermissionprod": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"Action": "lambda:InvokeFunction",
"FunctionName": {
"Ref": "MyFunction"
},
"Principal": "apigateway.amazonaws.com",
"SourceArn": {
"Fn::Sub": [
"arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/GET/",
{
"__ApiId__": {
"Ref": "MyApi"
},
"__Stage__": "*"
}
]
}
}
},
"MyApiprodStage": {
"Type": "AWS::ApiGateway::Stage",
"Properties": {
"DeploymentId": {
"Ref": "MyApiDeployment1e98ccf811"
},
"RestApiId": {
"Ref": "MyApi"
},
"StageName": "prod"
}
},
"MyFunctionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"sts:AssumeRole"
],
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
}
}
]
},
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
],
"Tags": [
{
"Key": "lambda:createdBy",
"Value": "SAM"
}
]
}
},
"MyApiDeployment1e98ccf811": {
"Type": "AWS::ApiGateway::Deployment",
"Properties": {
"Description": "RestApi deployment id: 1e98ccf811ae5bf67840ade2be6857ddf75298a9",
"RestApiId": {
"Ref": "MyApi"
},
"StageName": "Stage"
}
},
"MyApi": {
"Type": "AWS::ApiGateway::RestApi",
"Metadata": {
"SamResourceId": "MyApi"
},
"Properties": {
"Body": {
"info": {
"version": "1.0",
"title": {
"Ref": "AWS::StackName"
}
},
"paths": {
"/": {
"get": {
"x-amazon-apigateway-integration": {
"httpMethod": "POST",
"type": "aws_proxy",
"uri": {
"Fn::Sub": "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyFunction.Arn}/invocations"
}
},
"responses": {}
}
}
},
"swagger": "2.0"
}
}
}
}
} The transformed template shows that "paths": {
"/": {
"get": {
"x-amazon-apigateway-integration": {
"httpMethod": "POST",
"type": "aws_proxy",
"uri": {
"Fn::Sub": "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyFunction.Arn}/invocations"
}
},
"responses": {}
}
}
}, The API definition of the
And here:
But since the API resource is not in the same template as where the event is defined, the SAM transform is unable to edit the API definition. Which is the current technical limitation described in the issue. |
Any plans to fix this? It will be 5 years soon |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
We have got a problem where the resource limit of CF is hit when transformed from our SAM Template file.
Given all our Lambdas are behind APIs and we want to maintain all APIs from Single API Gateway, and that we now have to split the SAM Template into multiples to make them independently deploy-able, is there a way that we define one template that has only
AWS::Serverless::Api
to build out API Gateway with given Swagger file, and then to pass its reference to other template files that will be deploying ourAWS::Serverless::Function
s.We simply want to pass the reference of
RestApiId
to the template files as a CFN Parameter.Is this possible at this moment? As the error on the attempt says otherwise - RestApiId must be a valid reference to an 'AWS::Serverless::Api' resource in same template, which is quite dis-heartening :-(
The text was updated successfully, but these errors were encountered: