Unable to upload cloudinit snippets via ssh

[parliament718]

I'm having trouble uploading snippets. First time doing this, please help.

On my Proxmox PVE I created a user & API key:

sudo pveum user add terraform@pve
sudo pveum role add Terraform -privs "Datastore.Allocate Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify SDN.Use VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Migrate VM.Monitor VM.PowerMgmt User.Modify"
sudo pveum aclmod / -user terraform@pve -role Terraform
sudo pveum user token add terraform@pve provider --privsep=0

┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ terraform@pve!provider               │
├──────────────┼──────────────────────────────────────┤
│ info         │ {"privsep":"0"}                      │
├───

Now I'm trying to upload cloud-init yamls for each environment (with replacements from a template file)

provider "proxmox" {
   endpoint  = "https://78.100.100.10:8006"
   api_token = "terraform@pve!provider=<my-api-key>"
   insecure  = true
   
   ssh {
      agent    = true 
      username = "terraform"
   }
}

resource "proxmox_virtual_environment_file" "cloud_init" {
   for_each          = var.VM_CONFIG
   node_name     = "pve"
   content_type  = "snippets"
   datastore_id   = "snippets"

   source_raw {
      data = templatefile("${path.module}/cloud-init-template.yaml", {
         ENVIRONMENT = each.value.ENVIRONMENT
      })
      file_name = "/snippets/snippets/${each.value.ENVIRONMENT}-init.yaml"
   }
}

But when I apply I get the ssh error:

failed to open SSH client: unable to authenticate user "terraform" over SSH to "192.168.100.10:22". Please verify that ssh-agent is correctly loaded with an authorized key via 'ssh-add -L' (NOTE: configurations in ~/.ssh/config are not considered by the provider): failed to dial 192.168.100.10:22: dial tcp 192.168.100.10:22:

First of all, not sure why the error prints my PVE's internal IP address (192.168.100.10). My provider is set to the external address (78.100.100.10). The only place the internal address is mentioned is at my router level where I forward the external port 22 to the internal address shown in the error. I was surprised to see the internal IP surfaced, but that's probably unrelated.

The thing is, I've already added my jumphost's sshkey to the pve. That is, ssh-add -L prints the same key as i have in my pve's ~/.ssh/authorized_keys.

My pve users and realms are as follows:

Then I noticed the documentation says this:

Snippets cannot be uploaded by non-PAM accounts
Due to limitations in the Proxmox VE API, certain files (snippets, backups) need to be uploaded using SFTP. This requires the use of a PAM account (standard Linux account).

Is that what's affecting me? I tried using an alternative provider like:

provider "proxmox" {
   endpoint  = "https://78.100.100.10:8006"
   password = "<my-pass>"
   username = "root@pam"
   insecure  = true
   
   ssh {
      agent    = true
   }
}

But it gives the same error.

I can succesfully upload files using

sfto root@78.100.100.10 << EOF
    put cloud-init-template.yaml  /root   
EOF

Why can't I upload my snippets?

[bpg]

Hi @parliament718 👋🏼

First of all, I'm not sure why the error prints my PVE's internal IP address (192.168.100.10). My provider is set to the external address (78.100.100.10).

The provider needs to know the node's IP address to establish an SSH connection, so it uses a few different techniques to obtain this address, as documented here. However, depending on your particular PVE setup, those methods may not work. You might need to manually configure which IP addresses to use for SSH connections for each PVE node in the cluster.

From the provided examples, I think this provider configuration might work for you:

provider "proxmox" {
   endpoint  = "https://78.100.100.10:8006"
   api_token = "terraform@pve!provider=<my-api-key>"
   insecure  = true
   
   ssh {
      agent    = true 
      username = "terraform"
      node {
         name    = "pve"
         address = "78.100.100.10"
      }
   }
}

However, per your comment:

The thing is, I've already added my jumphost's sshkey to the pve. That is, ssh-add -L prints the same key as I have in my pve's ~/.ssh/authorized_keys.

If your PVE setup requires SSH connections to be made via a jumphost/bastion host, this might not work with the provider. There is an open ticket to add support for this.

[parliament718]

Hi, thanks a lot for the quick response.

The above code using API access didn't work. However, it did work by adding it to the password-based authentication.

provider "proxmox" {
   endpoint  = "https://${var.PROXMOX_PVE_IP}:8006"
   password = var.PROXMOX_PASSWORD
   username = "root@pam"
   insecure  = true
   
   ssh {
      agent    = true 
      username = "root"

      node {
         name    = "pve"
         address = var.PROXMOX_PVE_IP
      }
   }
}

Not ideal but good enough for me! Thanks again.

[bpg]

You should be able to use a non-root user account for SSH, if the user has required sudo permissions.
Along with the API Token auth, it is possible to have fully password-less auth for the provider without exposing the root user.