-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Technical Report on Nomad 0.4.1-dev compatibility to Windows TP5 environment #1488
Comments
Thanks for putting this together! This will be a huge help in getting docker/windows support to where it should be! |
@sitano Great report!
I can provide PR for setting nat as default network mode on windows (but maybe after #1475 gets merged)
I think it will be fixed in WinServ2016 RTM, so we have to just wait (MS says: september 2016)
I think it can be fixed in Nomad. Maybe I'll take a look next week For me the main problem is syslog, and how to replace it for windows containers. I assume #1469 (plugin system for logging) won't be anytime soon. |
I can write log plugin if you would give me a spec. |
This is really good! Thanks @sitano |
Another option could be to add support of syslog logging driver into dockerd it self, which is even better from my pov. I did not research reasons they do not provide it out of the box. And I think that should not be something big. I can research that. /and ask docker guys about this /and /or help them with getting syslog for windows. What do you think? In this case Nomad can keep syslog. |
Another option: etwlogs driver as default for windows. Both options need research... |
Update on how to run latest docker on latest Windows moby/moby#25336.
|
I did small research. It seems like syslog/journald logging support just cut out off the docker since May 14, 2015 mark (moby/moby@655a58e). I can look into adding specific windows support if you think syslog will do fine for Nomad? Current situation is they use custom syslog driver with tls support (moby/moby@4b98193). Not sure it can work for win - easy to check though. Maybe its nothing preventing us from adding it. |
@sitano |
@mwieczorek no yet. But it seems like it should work. Guys from irc #docker-dev said they don't mind adding syslog if we prove it works ;) I don't have time now doing that. You can try if you want. I think it will be good having some automated tests there as a prove of concept for the docker guys coming along the PR. |
What do you think about #stats support missing on Windows? |
@sitano about syslog - maybe I can try. Do you know any good implementation of syslog server for windows? |
@mwieczorek It's nice news about microsoft/hcsshim#59 - didn't see that. About windows logging. I surprisingly have found Nomad client doesn't support Windows syslog server mode https://github.com/hashicorp/nomad/blob/master/client/driver/logging/syslog_server_windows.go?? Why is that? About working server - don't know. Didn't touch any. Googling don't show much about compiling directly native unix syslog to mingw i.e. But, for windows there are maybe good options using C# servers: Nice C++ version: https://github.com/MaxBelkov/visualsyslog maybe we could also use just Go version of SysLog server for automated tests which is more native to the language of the choice. I don't know whether we need to provide working syslog TLS tests for getting syslog support into Docker. |
@sitano @dadgar @diptanu |
@mwieczorek Nice work. Yeah that would be acceptable |
I created issue moby/moby#25689 to enable syslog logging driver for windows. |
Nice, seems like we are getting closer. I am little bit off fixing some stuff in consul cross dc serf interconnect (docker / windows / azure). Glad we are moving forward with that. We are totally thinking of running big cluster based on this. I think its worth updating a status of the subtasks in the description... |
about stats: moby/moby#25737 |
Update:
@sitano You can check (Windows server 2016 RTM, nomad built from master, docker built from master) and let us know if you get any new issues |
@mwieczorek At this point is Docker containers launched by Nomad working on Windows Server 2016? Does any of this work also apply to Docker on Mac? Curious if you all have tested that, as I am assuming there is a decent cross-over with the work you all have done. |
@dadgar Yes, with Windows Server 2016 RTM, and unreleased Nomad/Docker versions (built from master). Also: I tested only running/stopping simple containers, on one host. (but will do some more advanced scenarios) About Docker on Mac...I don't have Mac so I cannot say anything. |
@mwieczorek Can you also please test if nomad is reporting the correct stats with the docker driver? |
@diptanu Sure, I'll let you know here. |
Windows + Docker: Consul TCP/HTTP Health checks will not function due to limitations of the Windows NAT configuration.Published Ports on Windows Containers do not loopback. This is due to a limitation in default If this is to work seamlessly, nomad needs to discover the Docker IP address that was assigned to the container and use that to perform health checks instead of the Host IP. As a work-around, we're trying to see if it would be possible to use a powershell script to check health, and use the The reason why we suspect this could work, is that we can use Update 1 We seem to be able to use check {
name = "Args Test"
type = "script"
command = "C:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe"
args = ["-Command", "If ((Invoke-WebRequest -UseBasicParsing -Uri http://localhost:${Env:IISPort}${Env:HealthPath}).StatusCode -eq 200) {Exit 0} else {Exit 1}"]
interval = "10s"
timeout = "60s"
} |
I'm going to close this issue in lieu of the remaining test items in #2633 |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
This is a report on investigation of nomad 0.4.1-dev to Windows TP5 docker compatibility in https://github.com/StefanScherer/docker-windows-box environment. Case shows there are critical incompatibilities to windows/docker platform, which are required to be fixed in order nomad to be able to run jobs.
Some of them were already fixed by @mwieczorek .
Issues
[+] Nomad does not support Windows volumes bind path: Volume binds for windows containers #1321 patch (51149e4) in 0.4.1-dev (by @mwieczorek).
[-] Windows docker does not support Nomad's default networking type: bridging. It supports only NAT, and overlay. Port mapping - nat only according to docs.
[?] Bridging is bad practice anyway. It should not be default.
[-] Windows Docker NAT networking does not support local ip address binding in port binding(#1475) (
-p ip:port:port
form)[-] Windows Docker does not have SysLog logging driver plugin. Default MUST be support which is json-file. (#688)
[-] Windows 2016 TP 5 has a bug in virtual nat switch port binding, so binded ports are not accessable from host machine
[-] UDP ports binding do not work due: Failed to create endpoint on network nat: HNS failed with error MicrosoftDocs/Virtualization-Documentation#273 (moby/moby#22084)
[-] Calling GET /containers/4f2e29f66db28b629009c21dda19317bab7a28e9a916983907add24133862a01/stats?stream=true: returned error: Windows does not support stats
[-] Nomad panics after container start with Null Ptr Deref in SyncServices
[-] Can't pull and build image if its missing in cache (just not works, yet trying)
[-] Can't delete image on failure, there is stopped container
Nomad version
Output from
nomad version
Nomad v0.4.1-dev. at least 5018972
Operating system and Environment details
docker info
:Reproduction steps
Simple container used for tests:
docker run -it -p 80:80 microsoft/iis:windowsservercore cmd
Run attached
iis.nomad
example withnomad run iis.nomad
.Nomad logs
nomad.log
Docker logs
dockerd.log
Job file
iis.nomad
Links
The text was updated successfully, but these errors were encountered: