-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Windows 2016 windows-restart
provisioner not waiting for reboot after patching
#6733
Comments
@nwesoccer does this happen also on the first reboot? You might want to try setting the WinRM service to delayed start. |
@gusztavvargadr It happens after windows patches have been applied no matter which reboot it is. It's not about WinRM coming up to fast, the problem is that the server takes a while to go down because it's in the "Getting Windows ready. Don't turn off your computer". For some reason this state allows packer to pass the |
As soon as I think I've finally nailed this provisioner to the wall, someone walks in with another bug 😭 I'll see if I can add whatever error code is generated by "Getting Windows ready. Don't turn off your computer" and make sure it doesn't pass the check. |
I'm seeing this issue too. As a temporary/dirty hack, I've put a {
"type": "shell-local",
"command": "echo \"Sleeping for 2 minutes...\"; sleep 120"
}, edit: The built-in pause_before provisioner configuration is a much better implementation of this hack. Inserting a {
"type": "windows-restart"
},
{
"type": "powershell",
"pause_before": "120s",
"inline": ["dir c:\\"]
} |
Not sure if it helps, but this blog has documented the four places you can check for to see if there is a pending reboot in progress: http://ilovepowershell.com/2015/09/10/how-to-check-if-a-server-needs-a-reboot/ |
@SwampDragons - I was able to pair down my template that has this same issue to a fairly simple repro: https://gist.github.cerner.com/ks8252/b75f146ab5a62e0337b9c4418c6aa015 In that gist I've included both the Packer template and my Packer log (obtained via PACKER_LOG=1) which shows the issue. The relevant bit in the log is:
If you put a |
Thanks, Kevin! |
I would love to use the pause_before, however sometimes it takes 5-10 minutes to finally shut down after the patches are applied so it's a pretty big waste of time. It's also prone to issues, if we pause for 10 minutes and it takes 11 mins we've found that the packer run could be "successful" even though the full reboot to finish applying patches did not yet occur. For now stepping through with |
I'm also experiencing this problem when trying to use PSWindowsUpdate 2.0.0.4 to perform windows updates on Windows Server 2016 (or even just showing the available updates). This problem ONLY happens on Hyper-V; the exact same stuff works just fine under Virtual Box. This occurs with both packer 1.3.1 and 1.2.5. The symptoms are that soon after my script issues an Whatever PSWindowsUpdate is doing under Hyper-V, seems to trick Packer into thinking that the step is completed (successfully, too!), when in fact it still has a long way to go. |
@nwesoccer @brettjacobson what is your detailed setup if I might ask? I'm building using a quite similar setup and it works fine for me. I have Windows 10 Pro latest for the Hyper-V and VirtualBox host, with Packer 1.2.5. Updates and windows-restart work fine with Windows 10 Enterprise, Windows 2016 Standard and Standard Core guests. For the guests I'm using the publicly available evaluation ISOs. The only Windows Update customization I make is I turn off automatic Windows Updates as the first thing after the first boot to make sure it kicks off only when I request it. |
@gusztavvargadr For my case, I'm stuck using Windows 10, 1703 (thanks, Corp IT) as my host. so I fear this may be a situation where packer is no longer supporting Hyper-V as it was in 1703. It seems vagrant requires Hyper-V from 1709, now, so would not surprise me. Tomorrow I'm spinning up a Server 1709 box to try it out on (which will probably answer my questions). slightly off-topic; how are you running both Hyper-V and VirtualBox on your host? I thought they were mutually exclusive? |
It works just fine on packer 1.2.4:
The same when I upgrade to packer 1.2.5. But then after I upgraded to 1.3.0, it does not detect ongoing installation properly:
And the same on 1.3.1 |
@brettjacobson yes, you can't run both Hyper-V and VirtualBox at the same time, but you can turn off Hyper-V and then VirtualBox works after a reboot: https://stackoverflow.com/questions/30496116/how-to-disable-hyper-v-in-command-line. |
@azr I'm not certain this is a problem with the windows-restart provisioner per se. It appears to me that its a problem where the regular powershell provisioner is thinking a step is completed (and thus, moving on to a windows-restart step which is always next with windows updates) prematurely. its like whatever Windows update processes are doing during the step are tricking packer+hyperv+powershell into thinking the step is complete when it is still active. I personally am only seeing this behavior with HyperV; with virtual box it behaves as expected (ie, waits for the long windows update process to finish before moving on to the next provisioner step; which in these scenarios are always windows-restart). |
I think this issue is getting a little confusing. OP, @nwesoccer (and others) can you confirm if you believe this to be true: |
The tests that I brought above is from VirtualBox, on Windows 10 host machine. |
@SwampDragons packer/provisioner/windows-restart/provisioner.go Lines 137 to 142 in 4e14710
|
Also having this problem. Using "pause_before" seems to be the best workaround, but it is impossible to know the best value to use. Sometimes you only need a few minutes, others it could be 30 depending on system specs and which updates are installing. |
Also having this problem. @Occams Great find, I compared the code to the commit prior to the latest one and it seems to make sense as to why this is not functioning correctly. It will move onto the next step after a hard 10s, no matter what state the reboot is in, due to the break escaping the for loop. |
For what it's worth, I commented out the break mentioned above and recompiled the binary, tested against my previously failing code where the windows-restart provisioner now properly loops when receiving the 1115 return code, until the VM has actually actually restarted.
|
Tested and working in my environment also. |
Sorry for not responding promptly. @brettjacobson I'm not exactly sure how to verify which it is. |
I linked binaries in my comment up there so you don't have to do this. But if you want to build packer manually, follow these steps
to run
|
@azr Whoops, my brain didn't put two and two together to map darwin to mac. Will give it a try. |
No worries, I should have been clearer 🙂 ( I edited it in case ) |
1.3.2-dev works. no more forced pauses. |
Great news, thanks everyone; merge incoming. 🎉 |
I think the problems I am having were due to my Host Windows 10 being build 1703 (with an old Hyper-V). Currently everything is working on another machine with 1803, and I'll be trying 1709 next week. This is with packer 1.3.1. (the problem is IT will only let me have 1703 at the moment) |
https://github.com/hashicorp/packer/files/2441352/packer_windows_amd64.zip worked for me. It failed before on my 1703 host but with this version verything works just fine. I also had the same issues on another host, 1803, but I did not try the fix there. @brettjacobson, it can be some timing difference between slower and faster hosts maybe? How exactly is it going on your 1803, is packer really waiting for the updates installation and the output is something like this?
|
@shurick81 with the dev version of packer linked above, I'm seeing the output exactly as you describe on both 1803 and 1709 hosts. But at least it is detecting the restart is finished, and goes on to the next step. This fix seems to work for my scenarios. |
I could add some code to clean up the output, but that's a cosmetic fix and can probably wait; thanks everyone for all of your input on this issue. |
@shurick81 i've tried your release but i get this error packer version packer validate C:\sre\packer\master.json
|
I think I've nailed down my issue to something about my WinRM config, but also the fact that I noticed the VM does not get an IP from DHCP after the automatic reboot performed by windows update. Still have no idea how to fix that though. |
@fdaforno I can't reproduce your validation error. :-/ |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
We start with a base image for windows 2016 and windows 2016 core directly from iso. This packer template clones that vm and applies windows patches, reboots, applies windows patches again (because windows tends to require this multiple times), then reboots. The reboot following each of the windows 2016 patches cause the vm to go into the "Getting Windows ready. Don't turn off your computer" state. The windows-restart provisioner completes prior to the machine even shutting down. We have confirmed this behavior by adding -debug and stepping slowly through and watching the vm.
Install-WindowsUpdate -MicrosoftUpdate -AcceptAll -IgnoreReboot -Verbose
In the following screenshot you can see the machine is still on even though the windows-restart provisioner claims the machine successfully restarted. If we wait there for 5 or so minutes, the vm starts a progress counter, once it gets to 100% it reboots.
The text was updated successfully, but these errors were encountered: