salt-masterless is not bootstrapping

[bambash]

Hi,

I'm running into an issue with the salt-masterless provisioner. Salt is not getting properly bootstrapped, and states are not executing. I'm using an AWS EC2 instance and Ubuntu 16.04.

Terraform Version

Terraform v0.10.8

Terraform Configuration Files

resource "aws_instance" "ec2_instance" {
  lifecycle = {
    prevent_destroy = "false"
  }

  connection {
    type     = "ssh"
    user     = "ubuntu"
    host     = "${aws_instance.ec2_instance.private_ip}"
  }

  ami = "${var.ami}"
  instance_type = "t2.medium"
  key_name = "mykey"

  vpc_security_group_ids = ["my_sg_id"]
  subnet_id = "my_subnet_id"

  root_block_device {
      volume_type = "gp2"
      volume_size = "25"
      delete_on_termination = "true"
  }

  provisioner "salt-masterless" {
      "local_state_tree" = "/path/to/salt"
      "log_level" = "debug"
  }
}

Debug Output

2017-11-16T14:22:56.055-0600 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2017/11/16 14:22:56 scp stderr (length 37): Sink: C0644 0 terraform_778679725.sh
module.instance.aws_instance.ec2_instance: Provisioning with 'salt-masterless'...
module.instance.aws_instance.ec2_instance (salt-masterless): Provisioning with Salt...
2017-11-16T14:22:56.057-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 opening new ssh session
2017-11-16T14:22:56.057-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 ssh session open error: 'client not available', attempting reconnect
2017-11-16T14:22:56.057-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 connecting to TCP connection for SSH
module.instance.aws_instance.ec2_instance (salt-masterless): Downloading saltstack bootstrap to /tmp/install_salt.sh
2017-11-16T14:22:56.102-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 handshaking with SSH
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "output.instance_public_ip"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "provisioner.salt-masterless (close)", waiting for: "module.instance.aws_instance.ec2_instance"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "output.instance_private_dns", waiting for: "module.instance.output.instance_private_dns"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "output.instance_private_ip", waiting for: "module.instance.output.instance_private_ip"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "output.instance_public_dns", waiting for: "module.instance.output.instance_public_dns"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "module.instance.output.instance_public_dns", waiting for: "module.instance.aws_instance.ec2_instance"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "module.instance.output.instance_public_ip", waiting for: "module.instance.aws_instance.ec2_instance"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "module.instance.output.instance_id", waiting for: "module.instance.aws_instance.ec2_instance"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "module.instance.output.instance_private_dns", waiting for: "module.instance.aws_instance.ec2_instance"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "module.instance.output.instance_private_ip", waiting for: "module.instance.aws_instance.ec2_instance"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "provisioner.remote-exec (close)", waiting for: "module.instance.aws_instance.ec2_instance"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "output.instance_id", waiting for: "module.instance.output.instance_id"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "output.instance_public_ip", waiting for: "module.instance.output.instance_public_ip"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "root", waiting for: "meta.count-boundary (count boundary fixup)"
2017/11/16 14:22:56 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "module.instance.aws_instance.ec2_instance"
2017-11-16T14:22:56.441-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 [DEBUG] Telling SSH config to forward to agent
2017-11-16T14:22:56.441-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 [DEBUG] Setting up a session to request agent forwarding
2017-11-16T14:22:56.441-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 opening new ssh session
2017-11-16T14:22:56.600-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 [INFO] agent forwarding enabled
2017-11-16T14:22:56.692-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 starting remote command: curl -L https://bootstrap.saltstack.com -o /tmp/install_salt.sh || wget -O /tmp/install_salt.sh https://bootstrap.saltstack.com
module.instance.aws_instance.ec2_instance (salt-masterless): Installing Salt with command sudo sh /tmp/install_salt.sh
2017-11-16T14:22:56.734-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 opening new ssh session
2017-11-16T14:22:56.822-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 starting remote command: sudo sh /tmp/install_salt.sh
module.instance.aws_instance.ec2_instance (salt-masterless): Creating remote temporary directory: /tmp/salt
2017-11-16T14:22:56.862-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 opening new ssh session
module.instance.aws_instance.ec2_instance (salt-masterless): Creating directory: /tmp/salt
2017-11-16T14:22:56.872-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 remote command exited with '127': sudo sh /tmp/install_salt.sh
2017-11-16T14:22:56.958-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:56 starting remote command: mkdir -p '/tmp/salt'
module.instance.aws_instance.ec2_instance (salt-masterless): Uploading local state tree: /path/to/salt
2017-11-16T14:22:57.002-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 opening new ssh session
module.instance.aws_instance.ec2_instance (salt-masterless): Creating directory: /tmp/salt/states
2017-11-16T14:22:57.042-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 remote command exited with '0': mkdir -p '/tmp/salt'
2017-11-16T14:22:57.074-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 remote command exited with '0': curl -L https://bootstrap.saltstack.com -o /tmp/install_salt.sh || wget -O /tmp/install_salt.sh https://bootstrap.saltstack.com
2017-11-16T14:22:57.092-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 starting remote command: mkdir -p '/tmp/salt/states'
2017-11-16T14:22:57.137-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 Uploading dir '/path/to/salt/states' to '/tmp/salt/states'
2017-11-16T14:22:57.137-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 opening new ssh session
2017-11-16T14:22:57.137-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 remote command exited with '0': mkdir -p '/tmp/salt/states'
2017-11-16T14:22:57.182-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 Starting remote scp process:  scp -rvt /tmp/salt/states
2017-11-16T14:22:57.222-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 Started SCP session, beginning transfers...
2017-11-16T14:22:57.222-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 Beginning file upload...
2017-11-16T14:22:57.271-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 Beginning file upload...
2017-11-16T14:22:57.313-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 Beginning file upload...
2017-11-16T14:22:57.361-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 SCP session complete, closing stdin pipe.
2017-11-16T14:22:57.361-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 Waiting for SSH session to complete.
2017-11-16T14:22:57.402-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 scp stderr (length 74): Sink: C0644 873 packages.sls
2017-11-16T14:22:57.402-0600 [DEBUG] plugin.terraform: Sink: C0644 28 top.sls
2017-11-16T14:22:57.402-0600 [DEBUG] plugin.terraform: Sink: C0644 19 minion
module.instance.aws_instance.ec2_instance (salt-masterless): Removing directory: /srv/salt
2017-11-16T14:22:57.402-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 opening new ssh session
2017-11-16T14:22:57.501-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 starting remote command: rm -rf '/srv/salt'
2017-11-16T14:22:57.554-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 remote command exited with '0': rm -rf '/srv/salt'
module.instance.aws_instance.ec2_instance (salt-masterless): Moving /tmp/salt/states to /srv/salt
2017-11-16T14:22:57.554-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 opening new ssh session
2017-11-16T14:22:57.652-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 starting remote command: sudo mv /tmp/salt/states /srv/salt
2017-11-16T14:22:57.702-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 remote command exited with '0': sudo mv /tmp/salt/states /srv/salt
2017-11-16T14:22:57.703-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 opening new ssh session
module.instance.aws_instance.ec2_instance (salt-masterless): Running: salt-call --local  state.highstate --file-root=/srv/salt --pillar-root=/srv/pillar --retcode-passthrough -l debug
2017-11-16T14:22:57.792-0600 [DEBUG] plugin.terraform: salt-masterless-provisioner (internal) 2017/11/16 14:22:57 starting remote command: sudo salt-call --local  state.highstate --file-root=/srv/salt --pillar-root=/srv/pillar --retcode-passthrough -l debug
2017/11/16 14:22:57 [TRACE] root.instance: eval: *terraform.EvalIf
2017/11/16 14:22:57 [TRACE] root.instance: eval: *terraform.EvalWriteState
2017/11/16 14:22:57 [TRACE] root.instance: eval: *terraform.EvalWriteDiff
2017/11/16 14:22:57 [TRACE] root.instance: eval: *terraform.EvalApplyPost
2017/11/16 14:22:57 [TRACE] root.instance: eval: *terraform.EvalUpdateStateHook
module.instance.aws_instance.ec2_instance: Creation complete after 1m2s (ID: i-05ad77b6033455eec)

Expected Behavior

salt should be installed, and states should be executed

Actual Behavior

salt was not installed, states were not executed

ubuntu@host:$ sudo salt-call
sudo: salt-call: command not found

Steps to Reproduce

1. terraform init
2. terraform apply

[bambash]

remote command exited with '127': sudo sh /tmp/install_salt.sh

[subbarao]

This fix #16704 will give complete remote command output, useful for debugging.

[gapotts]

Thanks @subbarao #16704 fixes my salt provision issues using google provider.

[bambash]

I'm also running into an issue where the provisioner does not wait to get an SSH connection. It tries to establish a connection immediately after ec2 is created, before port 22 is open. Provisioning then exits with a failure:

module.instance.aws_instance.ec2_instance (salt-masterless): Provisioning with Salt...
module.instance.aws_instance.ec2_instance (salt-masterless): Downloading saltstack bootstrap to /tmp/install_salt.sh
module.instance.aws_instance.ec2_instance: Still creating... (20s elapsed)

Error: Error applying plan:

1 error(s) occurred:

* module.instance.aws_instance.ec2_instance: 1 error(s) occurred:

* Unable to download Salt: dial tcp 10.0.0.9:22: getsockopt: connection refused

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.```

[gapotts]

@bambash, if you want the fix now follow the link to the PR then go to the authors source repo, download his terraform branch and re-build it from that. That is how I got it to work in the mean time.

[apparentlymart]

#16704 is now merged, so I think this is fixed in master. It'll be included in the next release. If you need the fix sooner, you may wish to try building Terraform from master to get the fix before it's in an official release.

Sorry for the weirdness here!

[bambash]

thanks guys!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.