Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remote-exec does not work with ssh connection to ec2 Windows Server 2019 #25634

Closed
davidMcneil opened this issue Jul 21, 2020 · 10 comments · Fixed by hashicorp/nomad#9759
Closed
Labels
bug confirmed a Terraform Core team member has reproduced this issue explained a Terraform Core team member has described the root cause of this issue in code provisioner/remote-exec v0.12 Issues (primarily bugs) reported against v0.12 releases v0.13 Issues (primarily bugs) reported against v0.13 releases windows
Milestone

Comments

@davidMcneil
Copy link
Contributor

davidMcneil commented Jul 21, 2020

Terraform Version

Terraform v0.12.28

Terraform Configuration Files

resource "aws_instance" "windows-test" {
  ami           = "<Windows Server 2019 ami>"
  instance_type = "t2.micro"

  get_password_data = true
  key_name          = "<a key>"
  security_groups = [
    "<security group that allows ssh>"
  ]

  connection {
    type     = "ssh"
    host     = coalesce(self.public_ip, self.private_ip)
    user     = "Administrator"
    password = rsadecrypt(self.password_data, file("<path to the private key>"))
  }

  user_data =  <<EOF
    <powershell>
    # Install OpenSSH
    Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
    Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
    Start-Service sshd
    Set-Service -Name sshd -StartupType 'Automatic'
    </powershell>
EOF

  # This works, so the ssh connection is successfull
  provisioner "file" {
    source      = "<some local file>"
    destination = "/Windows/Temp"
  }

  # This does not work, it does not run at all
  provisioner "remote-exec" {
    inline = [
      "echo this is a test> test.txt"
    ]
  }
}

Expected Behavior

There should be a test.txt file on the ec2 instance. The remote-exec commands do not seem to be executed at all.

Actual Behavior

There is no test.txt file on the ec2 instance.

Steps to Reproduce

  1. terraform init
  2. terraform apply

Additional Context

References

@alisdair alisdair added provisioner/remote-exec windows v0.12 Issues (primarily bugs) reported against v0.12 releases labels Jul 22, 2020
@alisdair
Copy link
Contributor

Thanks for reporting this issue! I'm not sure what could be causing this problem. Could you try running TF_LOG=trace terraform apply and upload the log output (after redacting anything sensitive)?

@alisdair alisdair added the waiting-response An issue/pull request is waiting for a response from the community label Jul 22, 2020
@davidMcneil
Copy link
Contributor Author

Here are some of the logs. Looks like something is getting very messed up with the command passed to cmd.exe.

2020/07/23 21:28:44 [TRACE] EvalApplyProvisioners: provisioning aws_instance.windows-test with "remote-exec"
aws_instance.windows-test: Provisioning with 'remote-exec'...
2020-07-23T21:28:44.258-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:44 [DEBUG] Connecting to <ip-address>:22 for SSH
aws_instance.windows-test (remote-exec): Connecting to remote host via SSH...
aws_instance.windows-test (remote-exec):   Host: <ip-address>
aws_instance.windows-test (remote-exec):   User: Administrator
aws_instance.windows-test (remote-exec):   Password: true
aws_instance.windows-test (remote-exec):   Private key: false
aws_instance.windows-test (remote-exec):   Certificate: false
aws_instance.windows-test (remote-exec):   SSH Agent: true
aws_instance.windows-test (remote-exec):   Checking Host Key: false
2020-07-23T21:28:44.304-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:44 [DEBUG] Connection established. Handshaking for user Administrator
2020-07-23T21:28:44.742-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:44 [DEBUG] Telling SSH config to forward to agent
2020-07-23T21:28:44.742-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:44 [DEBUG] Setting up a session to request agent forwarding
2020-07-23T21:28:45.010-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [WARN] error forwarding agent: forwarding request denied
2020-07-23T21:28:45.011-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] starting ssh KeepAlives
2020-07-23T21:28:45.011-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] opening new ssh session
aws_instance.windows-test (remote-exec): Connected!
2020-07-23T21:28:45.146-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] Starting remote scp process:  scp -vt /tmp
2020-07-23T21:28:45.210-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] Started SCP session, beginning transfers...
2020-07-23T21:28:45.210-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] Beginning file upload...
2020-07-23T21:28:45.256-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] SCP session complete, closing stdin pipe.
2020-07-23T21:28:45.256-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] Waiting for SSH session to complete.
2020-07-23T21:28:45.306-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [ERROR] scp stderr: "Sink: C0644 40 terraform_2095348772.sh\r\n"
2020-07-23T21:28:45.306-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] opening new ssh session
2020-07-23T21:28:45.399-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] starting remote command: chmod 0777 /tmp/terraform_2095348772.sh
2020-07-23T21:28:45.541-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] remote command exited with '0': chmod 0777 /tmp/terraform_2095348772.sh
2020-07-23T21:28:45.541-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] opening new ssh session
2020-07-23T21:28:45.643-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] starting remote command: /tmp/terraform_2095348772.sh
aws_instance.windows-test (remote-exec): �[2J�[?25l�[H
aws_instance.windows-test (remote-exec): �[H�]0;c:\windows\system32\cmd.exe��[?25h�[?25l�[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C
aws_instance.windows-test (remote-exec): �[40X�[40C�[H�[?25h�[?25lThe directory name is invalid.
2020-07-23T21:28:45.770-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] remote command exited with '0': /tmp/terraform_2095348772.sh
2020-07-23T21:28:45.770-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] opening new ssh session
2020-07-23T21:28:45.819-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] Starting remote scp process:  scp -vt /tmp
aws_instance.windows-test (remote-exec): �[?25h
2020-07-23T21:28:45.949-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] Started SCP session, beginning transfers...
2020-07-23T21:28:45.950-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] Copying input data into temporary file so we can read the length
2020-07-23T21:28:45.961-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:45 [DEBUG] Beginning file upload...
2020-07-23T21:28:46.264-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:46 [DEBUG] SCP session complete, closing stdin pipe.
2020-07-23T21:28:46.264-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:46 [DEBUG] Waiting for SSH session to complete.
2020/07/23 21:28:46 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2020/07/23 21:28:46 [TRACE] <root>: eval: *terraform.EvalWriteState
2020/07/23 21:28:46 [TRACE] EvalWriteState: recording 2 dependencies for aws_instance.windows-test
2020/07/23 21:28:46 [TRACE] EvalWriteState: writing current state object for aws_instance.windows-test
2020-07-23T21:28:46.329-0500 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/07/23 21:28:46 [ERROR] scp stderr: "Sink: C0644 0 terraform_2095348772.sh\r\n"
2020/07/23 21:28:46 [TRACE] <root>: eval: *terraform.EvalIf
2020/07/23 21:28:46 [TRACE] <root>: eval: *terraform.EvalIf
2020/07/23 21:28:46 [TRACE] <root>: eval: *terraform.EvalWriteDiff
2020/07/23 21:28:46 [TRACE] <root>: eval: *terraform.EvalApplyPost
2020/07/23 21:28:46 [TRACE] <root>: eval: *terraform.EvalUpdateStateHook
2020/07/23 21:28:46 [TRACE] statemgr.Filesystem: have already backed up original terraform.tfstate to terraform.tfstate.backup on a previous write
aws_instance.windows-test: Creation complete after 6m36s [id=i-0a242325420dc75ee]
2020/07/23 21:28:46 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 262
2020/07/23 21:28:46 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2020/07/23 21:28:46 [TRACE] [walkApply] Exiting eval tree: aws_instance.windows-test
2020/07/23 21:28:46 [TRACE] vertex "aws_instance.windows-test": visit complete
2020/07/23 21:28:46 [TRACE] dag/walk: visiting "provisioner.file (close)"
2020/07/23 21:28:46 [TRACE] vertex "provisioner.file (close)": starting visit (*terraform.graphNodeCloseProvisioner)
2020/07/23 21:28:46 [TRACE] vertex "provisioner.file (close)": evaluating
2020/07/23 21:28:46 [TRACE] [walkApply] Entering eval tree: provisioner.file (close)
2020/07/23 21:28:46 [TRACE] <root>: eval: *terraform.EvalCloseProvisioner
2020/07/23 21:28:46 [TRACE] dag/walk: visiting "provider.aws (close)"
2020/07/23 21:28:46 [TRACE] dag/walk: visiting "meta.count-boundary (EachMode fixup)"
2020/07/23 21:28:46 [TRACE] vertex "provider.aws (close)": starting visit (*terraform.graphNodeCloseProvider)
2020/07/23 21:28:46 [TRACE] vertex "meta.count-boundary (EachMode fixup)": starting visit (*terraform.NodeCountBoundary)
2020/07/23 21:28:46 [TRACE] vertex "provider.aws (close)": evaluating
2020/07/23 21:28:46 [TRACE] vertex "meta.count-boundary (EachMode fixup)": evaluating
2020/07/23 21:28:46 [TRACE] dag/walk: visiting "provisioner.remote-exec (close)"
2020/07/23 21:28:46 [TRACE] [walkApply] Entering eval tree: meta.count-boundary (EachMode fixup)
2020/07/23 21:28:46 [TRACE] vertex "provisioner.remote-exec (close)": starting visit (*terraform.graphNodeCloseProvisioner)
2020/07/23 21:28:46 [TRACE] vertex "provisioner.remote-exec (close)": evaluating
2020/07/23 21:28:46 [TRACE] [walkApply] Entering eval tree: provisioner.remote-exec (close)
2020/07/23 21:28:46 [TRACE] <root>: eval: *terraform.EvalCloseProvisioner
2020/07/23 21:28:46 [TRACE] <root>: eval: *terraform.EvalCountFixZeroOneBoundaryGlobal
2020/07/23 21:28:46 [TRACE] [walkApply] Entering eval tree: provider.aws (close)
2020/07/23 21:28:46 [TRACE] [walkApply] Exiting eval tree: meta.count-boundary (EachMode fixup)
2020/07/23 21:28:46 [TRACE] <root>: eval: *terraform.EvalCloseProvider
2020/07/23 21:28:46 [TRACE] vertex "meta.count-boundary (EachMode fixup)": visit complete
2020/07/23 21:28:46 [TRACE] GRPCProvider: Close
2020-07-23T21:28:46.343-0500 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-07-23T21:28:46.344-0500 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-07-23T21:28:46.349-0500 [DEBUG] plugin: plugin process exited: path=<path>.terraform/plugins/linux_amd64/terraform-provider-aws_v2.70.0_x4 pid=17482
2020-07-23T21:28:46.349-0500 [DEBUG] plugin: plugin exited
2020/07/23 21:28:46 [TRACE] [walkApply] Exiting eval tree: provider.aws (close)
2020/07/23 21:28:46 [TRACE] vertex "provider.aws (close)": visit complete
2020-07-23T21:28:46.351-0500 [DEBUG] plugin: plugin process exited: path=<path>bin/terraform pid=17437
2020-07-23T21:28:46.351-0500 [DEBUG] plugin: plugin exited
2020/07/23 21:28:46 [TRACE] [walkApply] Exiting eval tree: provisioner.file (close)
2020/07/23 21:28:46 [TRACE] vertex "provisioner.file (close)": visit complete
2020-07-23T21:28:46.351-0500 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-07-23T21:28:46.354-0500 [DEBUG] plugin: plugin process exited: path=<path>bin/terraform pid=17407
2020-07-23T21:28:46.354-0500 [DEBUG] plugin: plugin exited
2020/07/23 21:28:46 [TRACE] [walkApply] Exiting eval tree: provisioner.remote-exec (close)
2020/07/23 21:28:46 [TRACE] vertex "provisioner.remote-exec (close)": visit complete
2020/07/23 21:28:46 [TRACE] dag/walk: visiting "root"
2020/07/23 21:28:46 [TRACE] vertex "root": starting visit (terraform.graphNodeRoot)
2020/07/23 21:28:46 [TRACE] vertex "root": visit complete
2020/07/23 21:28:46 [TRACE] statemgr.Filesystem: have already backed up original terraform.tfstate to terraform.tfstate.backup on a previous write
2020/07/23 21:28:46 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 263
2020/07/23 21:28:46 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2020/07/23 21:28:46 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info

2020/07/23 21:28:46 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

@ghost ghost removed the waiting-response An issue/pull request is waiting for a response from the community label Jul 24, 2020
@alisdair
Copy link
Contributor

Thanks! I was able to reproduce this issue with the steps below:

  1. Generate a keypair:

    $ ssh-keygen -f id_rsa -t rsa -N ""
  2. Use this configuration:

    provider "aws" {
      region = "us-east-1"
    }
    
    resource "aws_key_pair" "deployer" {
      key_name   = "deployer-key"
      public_key = file("${path.module}/id_rsa.pub")
    }
    
    resource "aws_security_group" "windows" {
      name_prefix = "windows"
      description = "allow SSH and RDP ingress, and http egress"
    
      ingress {
        from_port   = 22
        to_port     = 22
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
      }
      ingress {
        from_port   = 3389
        to_port     = 3389
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
      }
      egress {
        from_port   = 80
        to_port     = 80
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
      }
      egress {
        from_port   = 443
        to_port     = 443
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
      }
    }
    
    resource "aws_instance" "windows-test" {
      ami           = "ami-0f38562b9d4de0dfe"
      instance_type = "t2.micro"
    
      get_password_data = true
      key_name          = aws_key_pair.deployer.key_name
      security_groups   = [aws_security_group.windows.name]
    
      connection {
        type     = "ssh"
        host     = coalesce(self.public_ip, self.private_ip)
        user     = "Administrator"
        password = rsadecrypt(self.password_data, file("${path.module}/id_rsa"))
      }
    
      user_data = <<EOF
        <powershell>
        Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
        Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
        Start-Service sshd
        Set-Service -Name sshd -StartupType 'Automatic'
        </powershell>
    EOF
    
      provisioner "file" {
        source      = "file-test.txt"
        destination = "/Windows/Temp/file-test.txt"
      }
    
      provisioner "remote-exec" {
        inline = [
          "echo this is a test > test.txt"
        ]
      }
    }
    
    output "admin_password" {
      value = rsadecrypt(aws_instance.windows-test.password_data, file("${path.module}/id_rsa"))
    }
    
    output "address" {
      value = aws_instance.windows-test.public_dns
    }
  3. terraform apply

Using Remote Desktop to connect to the instance, I can see that there's an empty file at C:\tmp. This makes me suspect that there's something going awry because we assume that all SSH connections are to hosts with a temporary directory at /tmp.

@alisdair
Copy link
Contributor

alisdair commented Jul 24, 2020

I tried updating the script path in the connection block:

  connection {
    type        = "ssh"
    host        = coalesce(self.public_ip, self.private_ip)
    user        = "Administrator"
    password    = rsadecrypt(self.password_data, file("${path.module}/id_rsa"))
    script_path = "/Windows/Temp/terraform_%RAND%.bat"
  }

This leads to a new and more difficult to fix error, which is that remote-exec inline scripts are uploaded with a shebang line:

const (
// DefaultShebang is added at the top of a SSH script file
DefaultShebang = "#!/bin/sh\n"
)

This results in a new error which can't be fixed with configuration:

'#!' is not recognized as an internal or external command, operable program or batch file.

The underlying issue here is that Terraform expects an SSH connection to result in a UNIX-like environment, whereas the OpenSSH server on Windows 10 results in a cmd.exe shell instead. I think to address this, we would need to at least allow the shebang prefix to be disabled for inline scripts.

As an alternative, we might be able to alter the remote-exec provisioner to execute single commands directly rather than generating a shell script and using scp to upload it.

@alisdair alisdair added confirmed a Terraform Core team member has reproduced this issue v0.13 Issues (primarily bugs) reported against v0.13 releases labels Jul 24, 2020
@alisdair
Copy link
Contributor

Ah! One final update: despite the shebang-induced error, the rest of the batch script actually does execute. While there's some ugly and confusing output from the SSH implementation, I think this might mean that you actually can use remote-exec with OpenSSH on Windows 10, so long as you set a script_path as described above.

If you could confirm the workaround is effective for you, that would be helpful!

@davidMcneil
Copy link
Contributor Author

@alisdair Awesome! The work around worked thanks! I appreciate your time.

@danieldreier danieldreier added new new issue not yet triaged explained a Terraform Core team member has described the root cause of this issue in code bug and removed new new issue not yet triaged labels Aug 5, 2020
@tgross
Copy link
Member

tgross commented Sep 3, 2020

I've run into this over on the Nomad team (ref hashicorp/nomad#8748) and having trouble with this workaround. (Unfortunately because we need to be able to push custom binaries from the tester's workstation, we execute this provisioning via userdata.)

locals {
  # workaround for https://github.com/hashicorp/terraform/issues/25634
  script_path = var.platform == "windows_amd64" ? "C:/Windows/Temp/terraform_%RAND%.bat" : ""
}

resource "null_resource" "provision_nomad" {
  connection {
    type        = "ssh"
    user        = var.connection.user
    host        = var.connection.host
    port        = var.connection.port
    private_key = var.connection.private_key
    timeout     = "15m"
    script_path = local.script_path
  }

  provisioner "remote-exec" {
    inline = [
     # this is actually templated, but I've verified the arguments themselves are ok
      "C:\opt\provision.ps1 -nomad_version 0.12.3"
    ]
  }
}

The symptoms are as described above: the remote-exec doesn't seem to be executed at all (there's a large gap of whitespace on the apply output) and there's a zero-byte file where we'd expect the temporary script to appear:


PS C:\Users\Administrator> ls C:\Windows\Temp


    Directory: C:\Windows\Temp


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
...
-a----         9/3/2020   6:00 PM              0 terraform_1230590878.bat

Running the script directly via ssh (ex. ssh -i keys/example.pem Administrator@1.2.3.4 C:/opt/provision.sh -nomad_version 0.12.3) works just fine.

@hhofs
Copy link
Contributor

hhofs commented Nov 9, 2020

EDIT: opened PR #26865 which contains a fix

image

@alisdair
Copy link
Contributor

Thanks to @hhofs, we now have a fix for this issue—see #26865. This is now merged into the main branch, and is scheduled to be released shortly with 0.14.1.

@ghost
Copy link

ghost commented Dec 13, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked as resolved and limited conversation to collaborators Dec 13, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug confirmed a Terraform Core team member has reproduced this issue explained a Terraform Core team member has described the root cause of this issue in code provisioner/remote-exec v0.12 Issues (primarily bugs) reported against v0.12 releases v0.13 Issues (primarily bugs) reported against v0.13 releases windows
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants