Terraform deploy Fundamentals enclave

[uwardlaw]

• new vms from iso (DC, 2019, 10 domain, 10 workgroup) with basic configs for testing
• WINDOWS_10_DOMAIN_TEMPLATE
• WINDOWS_10_WORKGROUP_TEMPLATE
• WINDOWS_2019_TEMPLATE
• WINDOWS_2019_DC_TEMPLATE

(Removed tasks after deciding that Terraform will not customize)
Write terraform file to clone VMS and join them to domain
Test terraform clones to ensure networking connectivity/configs is preserved
Document command line configs for input into packer templates

[uwardlaw]

Maybe this is useful. You can turn off terraform waiting for IPs with wait_for_guest_net = false

[marissaeinhorn]

Solved no operating system found errors when deploying windows 10 VM- firmware must be set to check templates specified firmware, bios is the default. github issue

[marissaeinhorn]

Logs from VM customization are stored on the VM in C:/Windows/TEMP/vmware-imc/guestcust.log

[marissaeinhorn]

Check if winrm is disabled on win 10 template

[marissaeinhorn]

Configured winrm on windows 10 template, install still failing during customization but the clone is further along in the install process.

Next step is to add connection block to terraform file explicitly specifying how to connect to winrm article

[uwardlaw]

We suspect that WinRM is not configured properly and causing the customization block to fail. Event Viewer says:

The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests.

and

WSMan operation Enumeration failed, error code 2150858770

We suspect this might be due to misconfiguration of WinRM. Windows documentation says:

- By default, no WinRM listener is configured. Even if the WinRM service is running, WS-Management protocol messages that request data can't be received or sent.
- Internet Connection Firewall (ICF) blocks access to ports.

We can confirm that the listener is not started with netstat -aon | select-string 5985.

The documentation further suggests using winrm quickconfig but this fails with the same The client cannot connect... message, after stating the service started.

When trying to start the Windows Remote Management service alone via Services we receive the error:

Windows could not start the Windows Remote Management (WS-Management) service on Local Computer.

Error 1079: The account specified for this service is different from the account specified for other services running in the same process.

[marissaeinhorn]

Changed winrm service account to run as Local Service
In GPO enabled winrm client settings, cloned template has errors when connecting to the service locally.
Added run_once_command_list option to terraform config. Will execute winrm quickconfig -force on machine when setting ip, gw, hostname

[marissaeinhorn]

Added auto_logon and auto_logon_count to main.tf file. Allows terraform to finish customization without errors.

[marissaeinhorn]

Writing terraform file to clone DC server, debugging error:

│ Error: error sending customization spec: A specified parameter was not correct: spec.identity.userData.computerName
│
│   with vsphere_virtual_machine.cloned_virtual_machine1,
│   on main.tf line 86, in resource "vsphere_virtual_machine" "cloned_virtual_machine1":
│   86: resource "vsphere_virtual_machine" "cloned_virtual_machine1" {
│

Resolved do not include special characters in the computerName hostname terraform variable article

[marissaeinhorn]

Error during deployment of DC template windows setup could not configure windows to run on this computer's hardware

Appears to be a known terraform problem when cloning VM with active directory running article. Workaround is clone a windows server 2019 vm and run dcpromo in post deployment actions

[marissaeinhorn]

Article on AD provisioner with description on winrm requirements #91

[marissaeinhorn]

To promote domain controller added remote-exec block and connection block specifying winrm settings

Apply failed with error

 Error: remote-exec provisioner error
│
│   with vsphere_virtual_machine.cloned_virtual_machine1,
│   on main.tf line 128, in resource "vsphere_virtual_machine" "cloned_virtual_machine1":
│  128:   provisioner "remote-exec" {
│
│ timeout - last error: unknown error Post "http://10.0.0.26:5985/wsman": dial tcp 10.0.0.26:5985: i/o timeout
╵

Added additional arguments to connection block

[marissaeinhorn]

Winrm executing but connection fails with error

null_resource.dcpromo_command (remote-exec): Connecting to remote host via WinRM...
null_resource.dcpromo_command (remote-exec):   Host: 10.0.0.26
null_resource.dcpromo_command (remote-exec):   Port: 5985
null_resource.dcpromo_command (remote-exec):   User: Administrator
null_resource.dcpromo_command (remote-exec):   Password: true
null_resource.dcpromo_command (remote-exec):   HTTPS: false
null_resource.dcpromo_command (remote-exec):   Insecure: false
null_resource.dcpromo_command (remote-exec):   NTLM: false
null_resource.dcpromo_command (remote-exec):   CACert: false

added null_resource to address potential race condition terraform issue

[marissaeinhorn]

Created null_resource and ran apply . Fails with error

│ Error: remote-exec provisioner error
│
│   with null_resource.dcpromo_command,
│   on main.tf line 144, in resource "null_resource" "dcpromo_command":
│  144:     provisioner "remote-exec" {
│
│ timeout - last error: unknown error Post "http://10.0.0.26:5985/wsman": dial tcp 10.0.0.26:5985: i/o timeout
╵

[marissaeinhorn]

terraform local-exec executes commands on the machine running terraform not the resource article

[marissaeinhorn]

Moved connection block into remote-exec block. Terraform no longer attempts to connect before VM is done creating. Connection still fails

vsphere_virtual_machine.cloned_virtual_machine1 (remote-exec): Connecting to remote host via WinRM...
vsphere_virtual_machine.cloned_virtual_machine1 (remote-exec):   Host: 10.0.0.26
vsphere_virtual_machine.cloned_virtual_machine1 (remote-exec):   Port: 5985
vsphere_virtual_machine.cloned_virtual_machine1 (remote-exec):   User: Administrator
vsphere_virtual_machine.cloned_virtual_machine1 (remote-exec):   Password: true
vsphere_virtual_machine.cloned_virtual_machine1 (remote-exec):   HTTPS: false
vsphere_virtual_machine.cloned_virtual_machine1 (remote-exec):   Insecure: true
vsphere_virtual_machine.cloned_virtual_machine1 (remote-exec):   NTLM: false
vsphere_virtual_machine.cloned_virtual_machine1 (remote-exec):   CACert: false

[marissaeinhorn]

From windows command line winrm get winrm/config gets current winrm settings.

terraform run_once_command_list executed winrm quickconfig -force but not subsequent commands

Added closing } and all winrm config commands run

[marissaeinhorn]

winrm functioning on DC. Previous errors due to routing problem from control to VM. Moved VM onto control's network adapter and successfully ran test command.

Next incorporate dcpromo command

[marissaeinhorn]

Added /unattend to the dcpromo command, previous install hung after command was issued by remote-exec reference

[marissaeinhorn]

Successfully promoted domain controller, but when system reboots terraform exits with an error. Added a delayed reboot and winrm disconnect to prevent the problem

[marissaeinhorn]

In provisioner block added on_failure="continue" will ignore errors generated by this block. reference

[uwardlaw]

Terraform cannot configure multiple Windows 2019 DCs because the IP addresses are all the same. We need a way to distinctly route between different student VMs.

The first step is removing hardcoded IP addresses from the Terraform file so they can be set by Ansible.

Then we need to add the student##-core routers. The connections between the student##-core router and control need to be unique while the connections behind student##-core router can be static set. We will reach things behind the student##-core with SSH bastion.

Blocked by #71 and #26

[marissaeinhorn]

Attempting to join windows 10 machine to the domain. terraform apply exits with the following error

 Error: error sending customization spec: Cannot complete customization.
│
│   with vsphere_virtual_machine.cloned_virtual_machine2,
│   on main.tf line 227, in resource "vsphere_virtual_machine" "cloned_virtual_machine2":
│  227: resource "vsphere_virtual_machine" "cloned_virtual_machine2" {
│
╵

Must include local admin credentials as well as domain admin credentials iot join a computer to a domain example

Domain controller must be members preferred dns server

[marissaeinhorn]

Adding core router to tf file. Vyos hostnames must not include special characters. reference

[marissaeinhorn]

vyos requires a dhcp server to be running on the network so a remote-exec provisioner can connect over ssh.

[marissaeinhorn]

Addded file provisioner block to copy vyos config from control to the core-rtr. Config contains static ip address for internal interface

Config is copied over, router reboots and shows up with eth2 and eth3 no ip addresses for either

[marissaeinhorn]

remote-exec shell $PATH= /usr/local/bin:/usr/bin:/bin:/usr/games

[uwardlaw]

Decision was made to stop using Terraform for customization.

[uwardlaw]

@marissaeinhorn As the fundamentals enclave is no longer a priority, the milestone marker is being removed and the issue is left over. Please convert any residual problems that affect VATER core into new issues for consideration in milestone v0.2.

[marissaeinhorn]

Closing outstanding issues defined in #289 and #290