Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't create FargateService #398

Open
polkx opened this issue Feb 21, 2024 · 12 comments
Open

Can't create FargateService #398

polkx opened this issue Feb 21, 2024 · 12 comments
Assignees
Labels
awaiting-release Fix was merged but not released kind/bug Some behavior is incorrect or out of spec P2 Bugs of moderate severity to be assigned to an engineer in the next iteration
Milestone

Comments

@polkx
Copy link
Collaborator

polkx commented Feb 21, 2024

When I try to create a FargateServiceArgs class the Error shows up and tells me that I must add taskDefinition or taskDefinitionArgs. Even when I add taskDefinition, error still shows up. Example:

Dependencies:

//> using scala "3.3.1"
//> using options -Werror -Wunused:all -Wvalue-discard -Wnonunit-statement
//> using plugin "org.virtuslab::besom-compiler-plugin:0.2.1"
//> using dep "org.virtuslab::besom-core:0.2.1"
//> using dep "org.virtuslab::besom-awsx:2.5.0-core.0.2"

Main class:

import besom.*
import besom.api.aws.ecs.Cluster
import besom.api.awsx.ecs.{FargateService, FargateServiceArgs}
import besom.api.awsx.ecs.inputs.{
  FargateServiceTaskDefinitionArgs,
  TaskDefinitionContainerDefinitionArgs,
  TaskDefinitionPortMappingArgs
}

@main def main = Pulumi.run {
  val cluster = Cluster("cluster")

  val taskDefinition =
    FargateServiceTaskDefinitionArgs(
      container = TaskDefinitionContainerDefinitionArgs(
        name = "my-task-def",
        image = "nginx",
        memory = 512,
        portMappings = List(TaskDefinitionPortMappingArgs(containerPort = 80))
      )
    )

  val service = FargateService(
    "my-service",
    FargateServiceArgs(
      cluster = cluster.arn,
      // taskDefinition = "my-task-def",
      taskDefinitionArgs = taskDefinition,
      desiredCount = 1
    )
  )

  Stack.exports(url = service.urn)
}

Stack trace:

Previewing update (dev):
     Type                        Name                   Plan       Info
 +   pulumi:pulumi:Stack         aws-hello-fargate-dev  create     2 errors; 37 messages
 +   └─ awsx:ecs:FargateService  my-service             create     

Diagnostics:
  pulumi:pulumi:Stack (aws-hello-fargate-dev):
    2024.02.21 11:44:24:850 scala-execution-context-global-20 [resource: my-service[awsx:ecs:FargateService]] ERROR besom.internal.ResourceDecoder.resolve:91
        Resolve resource: received error from gRPC call: 'UNKNOWN: Either `taskDefinition` or `taskDefinitionArgs` must be provided.', failing resolution
    2024.02.21 11:44:24:877 main  ERROR besom.internal.BesomModule.run:69
        io.grpc.StatusRuntimeException: UNKNOWN: Either `taskDefinition` or `taskDefinitionArgs` must be provided.
                at io.grpc.Status.asRuntimeException(Status.java:537)
                at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:538)
                at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
                at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
                at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
                at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
                at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
                at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
                at java.lang.Thread.run(Thread.java:1583)
        

    Exception in thread "main" io.grpc.StatusRuntimeException: UNKNOWN: Either `taskDefinition` or `taskDefinitionArgs` must be provided.
        at io.grpc.Status.asRuntimeException(Status.java:537)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:538)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

    Error: Either `taskDefinition` or `taskDefinitionArgs` must be provided.: Error: Either `taskDefinition` or `taskDefinitionArgs` must be provided.
        at new FargateService (/snapshot/awsx/bin/ecs/fargateService.js:47:19)
        at awsx:ecs:FargateService (/snapshot/awsx/bin/resources.js:25:45)
        at construct (/snapshot/awsx/bin/resources.js:43:12)
        at Provider.construct (/snapshot/awsx/bin/index.js:39:52)
        at Server.<anonymous> (/snapshot/awsx/node_modules/@pulumi/pulumi/provider/server.js:315:52)
        at Generator.next (<anonymous>)
        at fulfilled (/snapshot/awsx/node_modules/@pulumi/pulumi/provider/server.js:18:58)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)

    error: besom.internal.ResourceDecoder.resolve:91 [resource: my-service[awsx:ecs:FargateService]] Resolve resource: received error from gRPC call: 'UNKNOWN: Either `taskDefinition` or `taskDefinitionArgs` must be provided.', failing resolution
    error: an unhandled error occurred: '/opt/homebrew/bin/scala-cli /opt/homebrew/bin/scala-cli run .' exited with non-zero exit code: 1

Another example, the same result:

import besom.*

import besom.api.awsx.ecs.FargateService

@main def main = Pulumi.run {
  val service = FargateService("my-service")
  Stack.exports(url = service.urn)
}
@pawelprazak pawelprazak added kind/bug Some behavior is incorrect or out of spec needs-repro Needs repro steps before it can be triaged or fixed labels Feb 21, 2024
@pawelprazak
Copy link
Contributor

Thank you. We need to investigate why is this happening.

Note to self: the relevant code in the provider: https://github.com/pulumi/pulumi-awsx/blob/5586b5776d22c437044f2e20b23a1768d786fb97/awsx/ecs/ec2Service.ts#L53

@lbialy lbialy mentioned this issue Feb 22, 2024
@pawelprazak pawelprazak added the awaiting-release Fix was merged but not released label Feb 22, 2024
@pawelprazak pawelprazak added this to the 0.2.2 milestone Feb 22, 2024
@pawelprazak pawelprazak added P0 Bugs severe enough to interrupt existing work and removed needs-repro Needs repro steps before it can be triaged or fixed labels Feb 22, 2024
@keynmol
Copy link

keynmol commented Apr 2, 2024

Glad I found this issue!

I have a service definition like this:

//> using scala "3.3.3"
//> using options -Werror -Wunused:all -Wvalue-discard -Wnonunit-statement
//> using dep "org.virtuslab::besom-core:0.2.2"
//> using dep "org.virtuslab::besom-aws:6.23.0-core.0.2"
//> using dep "org.virtuslab::besom-awsx:2.5.0-core.0.2"
//> using dep "com.lihaoyi::upickle::3.2.0"

import besom.*
import besom.api.aws, besom.api.awsx, awsx.ecr, awsx.lb, awsx.ecs
import besom.api.awsx.ecs.inputs.*
import besom.api.awsx.lb.ApplicationLoadBalancerArgs
import besom.api.aws.ecs.inputs.ServiceNetworkConfigurationArgs
import besom.api.aws.cloudwatch.LogGroupArgs
import besom.json.*

@main def main = Pulumi.run {
  val repository =
    ecr.Repository(
      "sentiment-service-repo",
      ecr.RepositoryArgs(forceDelete = true)
    )

  val image = ecr.Image(
    "sentiment-service-image",
    ecr.ImageArgs(
      repositoryUrl = repository.url,
      context = p"../app",
      platform = "linux/amd64"
    )
  )

  val vpc = awsx.ec2.Vpc("sentiment-service-vpc")

  val loadBalancer = lb.ApplicationLoadBalancer(
    "sentiment-service-lb",
    ApplicationLoadBalancerArgs(subnetIds = vpc.publicSubnetIds)
  )
 
  val cluster = aws.ecs.Cluster("sentiment-service-cluster")

  val logGroup =
    aws.cloudwatch.LogGroup(
      "sentiment-service-log-group",
      LogGroupArgs(retentionInDays = 7, name = "sentiment-service-logs")
    )

  val service =
    image.imageUri.flatMap: image =>
      ecs.FargateService(
        "sentiment-service-fargate",
        ecs.FargateServiceArgs(
          networkConfiguration = ServiceNetworkConfigurationArgs(
            subnets = vpc.publicSubnetIds,
            assignPublicIp = true,
            securityGroups =
              loadBalancer.defaultSecurityGroup.map(_.map(_.id)).map(_.toList)
          ),
          cluster = cluster.arn,
          taskDefinitionArgs = FargateServiceTaskDefinitionArgs(
            containers = Map(
              "sentiment-service" -> TaskDefinitionContainerDefinitionArgs(
                image = image,
                name = "sentiment-service",
                cpu = 128,
                memory = 512,
                essential = true,
                logConfiguration = TaskDefinitionLogConfigurationArgs(
                  logDriver = "awslogs",
                  options = JsObject(
                    "awslogs-group"         -> JsString("log-group"),
                    "awslogs-region"        -> JsString("us-east-1"),
                    "awslogs-stream-prefix" -> JsString("ecs")
                  )
                ),
                /*portMappings = List(
                  TaskDefinitionPortMappingArgs(
                    targetGroup = loadBalancer.defaultTargetGroup
                  )
                )*/
              )
            )
          )
        )
      )

  Stack(logGroup).exports(
    image = image.imageUri,
    service = service,
    vpc = vpc.id,
    cluster = cluster.id,
    url = p"http://${loadBalancer.loadBalancer.dnsName}"
  )
}

And I get this exception IF I do any of the steps below:

Diagnostics:
  pulumi:pulumi:Stack (besom-smithy4s-kiss-dev):
    Error: Exactly one of [container] or [containers] must be provided: Error: Exactly one of [container] or [containers] must be provided
        at normalizeTaskDefinitionContainers (/snapshot/awsx/bin/ecs/containers.js:37:15)
        at new FargateTaskDefinition (/snapshot/awsx/bin/ecs/fargateTaskDefinition.js:34:79)
        at new FargateService (/snapshot/awsx/bin/ecs/fargateService.js:40:30)
        at awsx:ecs:FargateService (/snapshot/awsx/bin/resources.js:25:45)
        at construct (/snapshot/awsx/bin/resources.js:43:12)
        at Provider.construct (/snapshot/awsx/bin/index.js:39:52)
        at Server.<anonymous> (/snapshot/awsx/node_modules/@pulumi/pulumi/provider/server.js:315:52)
        at Generator.next (<anonymous>)
        at fulfilled (/snapshot/awsx/node_modules/@pulumi/pulumi/provider/server.js:18:58)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)

Here are the things that trigger this error:

  • Uncommenting the portMapping block
  • Not using flatMap with imageUri, and instead referencing it directly

Now, I've only been using Pulumi for like 2 days, so I don't know if it's a semantics problem or what, but the exception is very confusing.

Note that there's a pulumi-java issue: pulumi/pulumi-java#1325 and this awsx one pulumi/pulumi-awsx#820 aaaand I'm only now noticing that @pawelprazak is aware :D

@keynmol
Copy link

keynmol commented Apr 2, 2024

IIRC I've tried a couple of workarounds from both issues but neither of them worked with Besom.

@lbialy
Copy link
Collaborator

lbialy commented Apr 2, 2024

Hey, I think we have this fixed on current main (with some workarounds for upstream issues). We will cut a release soon. @pawelprazak can you take a look on this and advise if anything can be done to work around this issue on 0.2.2? Paweł has made a big refactoring in serde layer that brought us to be completely up to date with upstream Pulumi serde and this was done to fix this issue mostly. This change is however binary incompatible on core-providers edge so we have to release it as a part of 0.3.x. I think we should do this really fast, considering I found a nasty grpc memoization issue when writing post-Scalar blogpost (also already fixed).

@pawelprazak
Copy link
Contributor

pawelprazak commented Apr 2, 2024

Hi @keynmol, thank you for the reproduction and I'm sorry you've stumbled upon this cursed issue :)

What we know:

The key workaround was to use explicit val task = awsx.ecs.FargateTaskDefinition(...) instead of the one inlined in the arguments. Please let me know if the issue persists after this change, and I'll try to help to find more workarounds if necessary.

The workarounds might work on the current 0.2.2, but unfortunately there are known issues that required binary compatibility breaking changes and are awaiting for 0.3.0, that we plan to release soon.

P.s. I'm very curious if the flatMap will no longer be necessary with the workaround described above, because if not I would worry me ;)

@keynmol
Copy link

keynmol commented Apr 2, 2024

Good to hear 👍 I'm doing this for a blogpost as well so can easily wait.

I will try the workaround, and will excitedly wait for the 0.3.0 :)

@keynmol
Copy link

keynmol commented Apr 2, 2024

Actually it seems I was able to achieve what I wanted (I think, the architecture looks and works the way I wanted) by

  • removing target group from port mapping, only keeping containerPort
  • adding loadBalancers block to FargateService:
.. 60           loadBalancers = List(
.. 61             ServiceLoadBalancerArgs(
.. 62               containerName = "sentiment-service",
.. 63               containerPort = 80,
.. 64               targetGroupArn = loadBalancer.defaultTargetGroup.arn
.. 65             )
.. 66           ),

@lbialy
Copy link
Collaborator

lbialy commented Apr 2, 2024

Btw you should not need to flatMap on imageUri property to use its value as an argument for another Args. Args are built in a way that allows them to consume both raw values and values wrapped in Outputs (opaque type Input[A] = A | Output[A] essentially). We are working on adding a warning when you call a resource constructor in a body of lambda passed to flatMap on Output because, and this is a problem well understood in Scala community thanks to CT discourse, it is impossible for Pulumi to form a static plan of deployment when dynamic, monadic APIs are used (so, basically, applicative vs monadic from mill vs sbt discourse). In dry run computed properties (Outputs on resources) are unknown and behave like Option None - they short circuit and do not run flatMaps. Therefore they cut out subtrees of the deployment plan and that's exactly why we don't really flatMap anywhere beside computing derived properties that we later feed to args of another resource constructor that is called statically, not in a flatMap.

@lbialy
Copy link
Collaborator

lbialy commented Apr 2, 2024

That's a gotcha coming from the elasticity of Pulumi that Scala's ease of use of flatMap sort of exacerbates. Compile warn should help push people to use this power only when really necessary.

@pawelprazak pawelprazak reopened this Apr 2, 2024
@pawelprazak pawelprazak modified the milestones: 0.2.2, 0.3.0 Apr 2, 2024
@pawelprazak
Copy link
Contributor

Glad to hear you were able to workaround the issues. Feel free to reach out if any more problems crop up. I'll leave this open until release of 0.3.0, but I'm afraid that the underlying issue appears to be upstream unfortunately.

@pawelprazak pawelprazak self-assigned this Apr 2, 2024
@pawelprazak pawelprazak added P1 Bugs severe enough to be the next item assigned to an engineer and removed P0 Bugs severe enough to interrupt existing work labels Apr 4, 2024
@lbialy lbialy modified the milestones: 0.3.0, 0.4.0 Apr 9, 2024
@pawelprazak
Copy link
Contributor

@lbialy WDYT, can we close this now that the 0.3.x was released? I think we've done what we could on our side and the main problem in this thread is upstream, in the awsx provider.

@lbialy lbialy added P2 Bugs of moderate severity to be assigned to an engineer in the next iteration and removed P1 Bugs severe enough to be the next item assigned to an engineer labels May 25, 2024
@lbialy
Copy link
Collaborator

lbialy commented May 25, 2024

It is an open problem that is not fixed. I think we should leave this as a lower priority bug report and check after a new release of pulumi/pulumi-eks (there has been a new one few hours ago - 2.5.2 btw but I don't expect it to fix this problem).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting-release Fix was merged but not released kind/bug Some behavior is incorrect or out of spec P2 Bugs of moderate severity to be assigned to an engineer in the next iteration
Projects
None yet
Development

No branches or pull requests

4 participants