Question: Can installer be used with xhyve/virtualbox hypervisors?

[cmoulliard]

Question : Can openshift installer (tectonic) be used to install a all-in-one cluster using with xhyve/virtualbox hypervisor ?

[wking]

At the moment we're just AWS and libvirt. Libvirt can handle a number of hypervisors, including VirtualBox. I'm not sure how portable we are between those hypervisors. Maybe we work on all of them, maybe not. I'm only aware of folks testing with the QEMU backend (which we hope to cover with CI soon). To test this out, you'll want to toggle the libvirt URI when working through the libvirt procedure.

[cmoulliard]

Many thanks. I will have a look. Will openshift installer be used to perform oc cluster up ?

[wking]

Will openshift installer be used to perform oc cluster up?

Maybe eventually? For now, the next-gen command is openshift-install (still a work in progress). @smarterclayton or @aaronlevy may be able to weigh in on oc integration plans. But if there's much to say on this point, it's probably worth spinning off into its own issue.

[smarterclayton]

@jwforres as well, this isn’t our primary focus but we will come back to it soon and try to find a path that minimizes the wrong sorts of install duplication.

[jwforres]

Last week I tried going down the path @wking suggests above and hit an issue with libvirt hanging trying to connect to the vbox uri, if someone gets it to connect successfully let me know.

I don't recommend trying the alternate path of making libvirt + qemu work on a Mac, libvirt is not set up to work with qemu in the mac package managers (like homebrew). I wound up re-compiling from source. It was a rabbit hole and in the end only got me a few steps further in the installer before it broke again on another piece around networking that wasn't getting compiled in to the homebrew version of libvirt.

[jwforres]

As far as long term about how the installer and oc relate, these are discussions we are absolutely having, and we don't want to end up with 3 or 4 installers. We need to find the right way to tie oc cluster up into what the new operator-based installation model can do for us.

[eparis]

@rabugopsl also tried getting it to work inside a libvirt container on mac. i don't think he succeeded either...

[rabugopsl]

I’m working on a CentOS7 container which might do the trick. I’m having issues with some libvirt resources, but I hope I can have those sorted out to have a fully functional environment for everyone.

…

On Sep 6, 2018, at 1:22 PM, Eric Paris ***@***.***> wrote: @rabugopsl <https://github.com/rabugopsl> also tried getting it to work inside a libvirt container on mac. i don't think he succeeded either... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#201 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AgV5f55so8UtIPlZ8ZD29VaD18mrTSWEks5uYWfggaJpZM4WZRvv>.

[crawford]

VirtualBox is not supported by the installer. In order for a hypervisor to be supported, libvirt and Ignition must support it. Right now, Ignition technically supports VirtualBox, but the implementation is subpar and was rushed due to product demands.

VirtualBox needs support for user-provided data (similar to VMware's Guest Properties mechanism). Until it is added, I'm going to push back pretty hard on supporting that hypervisor.

As for xhyve, does it have the mechanism I've described above? If so, support for it will need to be added to Ignition and then we can try to enable it in the installer. It's worth noting that libvirt on OS X doesn't support networking, which is the bigger roadblock in my mind. We need to be able to do basic DNS and DHCP manipulation in order to properly bring up a cluster.

[ironcladlou]

I got the installer working on OSX using VT-x nested virtualization from a Fedora 28 VM using Fusion 11 Pro. I put my experiment notes in this gist in case anybody's curious.

[jianzhangbjz]

/cc

[sspeiche]

/cc @praveenkumar @LalatenduMohanty @gbraad

[gbraad]

oc cluster up

This has officially been removed to focus efforts on installer to become the new way to standup a possible all in one

[gbraad]

Great to see, but I have to be honest, nested is not the way forward. We need to be able to use native hypervisors as the overhead gives a noticeable performance degradation. It works for development of the installer, but not for use of the cluster

[gbraad]

@crawford the most likely target that meets the requirements would be HyperV. For macOS it would seem VirtualBox could be a short term solution as xhyve/hyperkit needs a whole lot more work. Anything we can help with? Pointers would be appreciated.

[LalatenduMohanty]

Nested virtualization is not something Minishift use case can use as it has the performance drawback which impacts the user experience. We need to find a way to boot the VM directly on the hypervisors.

[crawford]

@gbraad As I mentioned before, Ignition needs to support the hypervisors you intend on targeting. Adding support to Ignition tends to be pretty straightforward for hypervisors which have a decent hypervisor-to-guest bridge. The current support for VirtualBox is built on a terrible hack that was rushed through due to product demands. I would much prefer it if we used a better mechanism or committed the necessary functionality to VirtualBox to support this use case. The alternative is VMware, hich works out of the box today.

Additionally, the installer will need to support the creation of the resources for the cluster. On Linux, we use libvirt with a libvirt provider for Terraform. Unfortunately, libvirt for OSX is built without support for networking, so either libvirt will need to be updated to support networking on OSX or an alternate Terraform provider will need to be written (or found) that can manage the resources on OSX.

[DanyC97]

@crawford any reason why VMware was removed ? in tectonic you used you have a module and i suspect it was working with ignition...isn't that the case anymore ?

[crawford]

@DanyC97 the installer has been completely rewritten and shares almost nothing with the previous incarnation used by Tectonic. The installation flow is quite different from what Tectonic was doing. We will circle back and add support for new platforms eventually.

[DanyC97]

@crawford thx for taking the time to respond so quickly. Will be good if you have time to put together a lowl level flow from bootkube all the way to the operators and how each of them contribute to form the cluster

[wking]

Will be good if you have time to put together a lowl level flow from bootkube all the way to the operators...

This is the description of the install flow. After that, see the docs for the cluster-version operator and individual operators for details about how they are managed and what they do.

[eparis]

At this time we do not intend to address this issue. We apologize for the inconvenience.

[durandom]

https://github.com/code-ready/osp4 is the current work on getting ocp4 installed locally

[gbraad]

Currently in PoC phase, but we will be adding lifecycle management tooling soon, plus support for more hypervisors. Actual support for installer-based deployments is being investigated on discussed.

[gbraad]

I suggest we update

Support for libvirt on Mac OS is currently broken and being worked on.
Ref

to reflect the changed status.