aws-stepfunctions: JsonPath.format incorrectly escapes newlines

[clareliguori]

Describe the bug

When I call JsonPath.format with a string that contains newlines, the resulting string in the CloudFormation template is escaped too many times and fails to deploy.

The example in the repro steps produces this template snippet. Note the two extra backslashes for the newline in States.Format('Hello\\\\n{}'.

"DefinitionString": "{\"StartAt\":\"MessageState\",\"States\":{\"MessageState\":{\"Type\":\"Pass\",\"Parameters\":{\"message.$\":\"States.Format('Hello\\\\n{}', $.name)\"},\"End\":true}}}",

Expected Behavior

Template can be successfully deployed.

If I manually edit the synthesized template to remove the two extra backslashes for the newline (States.Format('Hello\\n{}'), then the template can be successfully deployed and the state machine has the correct newline in its output:

"DefinitionString": "{\"StartAt\":\"MessageState\",\"States\":{\"MessageState\":{\"Type\":\"Pass\",\"Parameters\":{\"message.$\":\"States.Format('Hello\\n{}', $.name)\"},\"End\":true}}}",

Current Behavior

The template fails to deploy with:

Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the field 'message.$' must be a valid JSONPath or a valid intrinsic
function call at /States/MessageState/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: cc2099fa-e693-4e51-ab5e-4672cb9abfb6; P
roxy: null)" (RequestToken: 66bd7a2e-b18c-58a9-fe08-807ee96cc521, HandlerErrorCode: InvalidRequest)

Reproduction Steps

    const messageState = new sfn.Pass(this, 'MessageState', {
      parameters: {
        "message": sfn.JsonPath.format('Hello\n{}', sfn.JsonPath.stringAt('$.name')),
      },
    });

    new sfn.StateMachine(this, 'StateMachine', {
      definitionBody: sfn.DefinitionBody.fromChainable(messageState),
    });

Possible Solution

The escaping logic for newlines here for JsonPath.format seems to be right, so I'm wondering if there's another round of escaping that's happening when the template is synthesized:
https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-stepfunctions/lib/private/json-path.ts#L340-L356

Additional Information/Context

No response

CDK CLI Version

2.98.0

Framework Version

2.98.0

Node.js Version

v18.17.1

OS

Linux

Language

Typescript

Language Version

Typescript (5.2.2)

Other information

No response

[clareliguori]

If I use the return value of JsonPath.format as a stack output, then that string is not incorrectly escaped in the synthesized template:

const messageFormat = sfn.JsonPath.format('Hello\n{}', sfn.JsonPath.stringAt('$.name'));
new cdk.CfnOutput(this, 'MessageFormat', {
      value: messageFormat,
    });

Result:

  "Outputs": {
    "MessageFormat": {
      "Value": "States.Format('Hello\\n{}', $.name)"
    }
  },

So the double-escaping could be happening when the DefinitionString is created in the StateMachine construct

[clareliguori]

It looks like this is where the double-escaping could be happening - when toJsonString is called on the state machine graph JSON, after JsonPath.format has already escaped the newlines:
https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-stepfunctions/lib/state-machine.ts#L723

Example of doing toJsonString on my Pass state in a stack output vs what comes out of JsonPath.format:

  const messageFormat = sfn.JsonPath.format('Hello\n{}', sfn.JsonPath.stringAt('$.name'));
  const messageState = new sfn.Pass(this, 'MessageState', {
      parameters: {
        "message": messageFormat,
      },
    });

  new cdk.CfnOutput(this, 'MessageFormat', {
      value: messageFormat,
    });

  new cdk.CfnOutput(this, 'MessageStateDef', {
      value: cdk.Stack.of(this).toJsonString(messageState.toStateJson()),
    });

Output:

  "Outputs": {
    "MessageFormat": {
      "Value": "States.Format('Hello\\n{}', $.name)"
    },
    "MessageStateDef": {
      "Value": "{\"Type\":\"Pass\",\"Parameters\":{\"message.$\":\"States.Format('Hello\\\\n{}', $.name)\"},\"End\":true}"
    }
  },

[peterwoodworth]

Thanks for the investigation and PR 🙂 Much appreciated!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.