-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
efs - ecs : Cannot re-mount an existing efs #26537
Comments
Have you tried researching the service errors? e.g. https://repost.aws/knowledge-center/fargate-unable-to-mount-efs This doesn't look like it's a CDK issue at first glance, but rather either a configuration issue or possibly a service bug. But we can't rule anything out yet, I'm just curious how much you've looked into + double checked the configuration |
@peterwoodworth I will try out the trobleshooting procedures indicated in the link. So far, I've tried to grant all IAM permissions to the task (just to see if the issue was there), I've also retained the Security Group, which (as expected honestly) don't make a difference and in terms of the task I am running is the same container I am using in the first creation, which is a redis:alpine instance. I doubt myself that this issue is CDK related. Where should I bring this up though? Is there a specific AWS Forum for each service? |
Well, it might be CDK related. I didn't look at this in-depth enough to rule out CDK. Though, I'm not super familiar with these services so I'm not sure without a deep dive. Another place to receive help is premium support, or repost Let me know if you are able to figure out where the error is coming from or if you've been able to unblock |
@peterwoodworth as of now, still stuck. Will let you know if I figure this out. Thank you Peter |
Sorry, what exactly is it that you mean by "remount" the file system? |
@peterwoodworth yeah sorry, it is vague. Basically I mean reusing, reattaching an existing EFS (that had a RetainPolicy.RETAIN set for instance) when launching a stack, so that the ECS tasks that were using said EFS, could mount it again on the same AccessPoint and retrieve the data. Does it make sense? |
Sorry for the delay @ETisREAL, I'm not exactly sure what you mean. If you provide clear repro steps, including the code deployed at each step it would be really helpful. Especially if it's a full reproduction that's as minimized as possible |
This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled. |
No worries @peterwoodworth Thanks for the help either way. In order to reproduce it:
const qmmTasksEfsSecurityGroup = new ec2.SecurityGroup(this, `qmmTasksEfsSecurityGroup`, {
vpc: props.vpc,
securityGroupName: `qmmTasksEfsSecurityGroup`
})
let qmmTasksEfs: efs.IFileSystem
let qmmRedisEfsAccessPoint: efs.IAccessPoint
if (true) {
qmmTasksEfs = new efs.FileSystem(this, `qmmTasksEfs`, {
fileSystemName: `qmmTasksEfs`,
vpc: props.vpc,
removalPolicy: cdk.RemovalPolicy.RETAIN,
securityGroup: qmmTasksEfsSecurityGroup,
encrypted: true,
lifecyclePolicy: efs.LifecyclePolicy.AFTER_30_DAYS,
enableAutomaticBackups: true
})
new cdk.CfnOutput(this, 'QlashMainClusterEFSID', {
exportName: 'QlashMainClusterEFSID',
value: qmmTasksEfs.fileSystemId
})
qmmRedisEfsAccessPoint = new efs.AccessPoint(this, `qmmRedisAccessPoint`, {
fileSystem: qmmTasksEfs,
path: '/redis',
createAcl: {
ownerGid: '1001',
ownerUid: '1001',
permissions: '750'
},
posixUser: {
uid: '1001',
gid: '1001'
}
})
qmmRedisEfsAccessPoint.applyRemovalPolicy(cdk.RemovalPolicy.RETAIN)
new cdk.CfnOutput(this, 'QlashMainClusterRedisAccessPointID', {
exportName: 'QlashMainClusterRedisAccessPointID',
value: qmmRedisEfsAccessPoint.accessPointId
})
} else {
qmmTasksEfs = efs.FileSystem.fromFileSystemAttributes(this, `qmmTasksEfs`, {
securityGroup: qmmTasksEfsSecurityGroup,
fileSystemId: config.QlashMainClusterEFSID
})
qmmRedisEfsAccessPoint = efs.AccessPoint.fromAccessPointId(this, `qmmRedisAccessPoint`, config.QlashMainClusterRedisAccessPointID)
// Redis
const qmmRedisServiceSecurityGroup = new ec2.SecurityGroup(this, `qmmRedisSecurityGroup`, {
vpc: props.vpc,
securityGroupName: `qmmRedisSecurityGroup`
})
qmmTasksEfsSecurityGroup.addIngressRule(
ec2.Peer.securityGroupId(qmmRedisServiceSecurityGroup.securityGroupId),
ec2.Port.tcp(2049),
'Allow inbound traffic from qmm_redis to qmmTasksEfs'
)
const qmmRedisTaskDefinition = new ecs.FargateTaskDefinition(this, `qmmRedisTask`, {
cpu: 2048,
memoryLimitMiB: 8192,
volumes: [
{
name: `qmm_redis_volume`,
efsVolumeConfiguration: {
fileSystemId: qmmTasksEfs.fileSystemId,
transitEncryption: 'ENABLED',
authorizationConfig: {
accessPointId: qmmRedisEfsAccessPoint.accessPointId,
iam: 'ENABLED'
}
}
}
]
})
qmmRedisTaskDefinition.addToTaskRolePolicy(
new iam.PolicyStatement({
actions: [
'elasticfilesystem:ClientWrite',
'elasticfilesystem:ClientMount',
'elasticfilesystem:ClientRootAccess',
'elasticfilesystem:DescribeMountTargets',
'elasticfilesystem:CreateAccessPoint',
'elasticfilesystem:DeleteAccessPoint'
],
resources: [qmmTasksEfs.fileSystemArn],
})
)
qmmRedisTaskDefinition.addToTaskRolePolicy(
new iam.PolicyStatement({
actions: [
'elasticfilesystem:DescribeAccessPoints',
'elasticfilesystem:DescribeFileSystems'
],
resources: ["*"],
})
)
qmmRedisTaskDefinition.addToTaskRolePolicy(
new iam.PolicyStatement({
actions: ['ec2:DescribeAvailabilityZones'],
resources: ['*']
})
)
const qmmRedisContainer = qmmRedisTaskDefinition.addContainer(`qmm_redis`, {
image: ecs.ContainerImage.fromAsset('redis'),
containerName: `qmm_redis`,
portMappings: [{ containerPort: 6379, name: `qmm_redis` }],
healthCheck: {
command: ["CMD", "redis-cli", "-h", "localhost", "-p", "6379", "ping"],
interval: cdk.Duration.seconds(20),
timeout: cdk.Duration.seconds(20),
retries: 5
},
logging: ecs.LogDriver.awsLogs({streamPrefix: `qmm_redis`, logRetention: RetentionDays.ONE_DAY}),
command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
})
qmmRedisContainer.addMountPoints({
sourceVolume: `qmm_redis_volume`,
containerPath: '/redis/data',
readOnly: false
})
const qmmRedisService = new ecs.FargateService(this, `qmmRedisService`, {
serviceName: `qmmRedisService`,
cluster: qlashMainCluster,
desiredCount: 1,
securityGroups: [qmmRedisServiceSecurityGroup],
taskDefinition: qmmRedisTaskDefinition
})
This will retain the EFS and Access Point because of the cdk.RetainPolic.RETAIN
|
Im having this same issue, when importing a efs file system |
Does it only happen when importing or re-using an existing EFS filesystem?
After you destroy the stack with removal policy as RETAIN, are you still able to see/list this filesystem ID in the EFS console? Not sure if this is a bug but looks like this filesystem ID is invalid when the resource is destroyed with retain removal policy? |
@pahud yes I can still see the filesystem from the console and list it with its id |
I'm having the same issue. When CDK destroy the current stack, it deletes the mount targets from the EFS |
Even when you explicitly set the lifecycle policy to retain ? qmmRedisEfsAccessPoint.applyRemovalPolicy(cdk.RemovalPolicy.RETAIN) Because I haven't noticed it if I set the policy to RETAIN |
Yes, I have set the removal policy for the AccessPoint, access point retains but entries in the "network" tab are get deleted. UPDATE: I have manually created the entries in the network tab. After that the service starts as expected |
I had same issue. In my code I don't retain VPC, so it makes sense, because mount targets have ENI and deletion of VPC will fail if it has ENI. So I solved this issue by creating new mount targets with CfnMountTarget when I reuse File System. Almost the same method as @vipulaSD wrote. |
Describe the bug
Hi, hope to find you well. I am trying to mount an existing EFS to a redis ECS Task. Everything works smoothly the first creation, but no luck when trying to remount the same FS which returns a puzzling error.
Expected Behavior
I should be able to remount the EFS, afterall what is the point of the RetainPolicy otherwise?
Current Behavior
This is my code:
Reproduction Steps
When running the following code trying to remount the EFS, you will get this error:
What realy sounds strange is this:
Possible Solution
I don't even know if this is something that is up to you guys or if it is an internal error from EFS itself
Additional Information/Context
I've tried giving the task permissions on everything, just to check if it was a permission issue, but to no good
CDK CLI Version
2.88
Framework Version
No response
Node.js Version
v18.15.0
OS
Linux - Ubuntu
Language
Typescript
Language Version
No response
Other information
No response
The text was updated successfully, but these errors were encountered: