Hi all, the last year I’ve been running a lot of tests in my OpenShift lab environment, and I was quite curious about how OpenShift Virtualization works.   OpenShift Virtualization is a feature of Red Hat OpenShift Container Platform (OCP) that allows you to run and manage virtual machines alongside containers in the same OCP Cluster.

As most of virtualization technologies like VMware, Hyper-V, KVM and others, OpenShift Virtualization requires hardware virtualization, which usually means that you need to use bare-metal servers for your OCP cluster.  By default, OpenShift Virtualization uses the Linux Kernel Virtual Machine (KVM), which is a full-virtualization solution for Linux on x86 hardware. KVM uses the virtualization extensions Intel VT or AMD-V to enable device emulation.  Of course, in a lab environment isn’t always possible to have physical servers available for this kind of tests, so we usually leverage nested virtualization, like when we want to run a virtual ESXi host on top of a physical ESXi host.

One of the option we have, is to run our OCP cluster as VMs on top of VMware or KVM, and enable nested virtualization:

 

But what if we are trying to run an OCP cluster in a Cloud provider like AWS EC2, Azure VMs or Google Cloud Engine?  In this case the situation is a bit more complicated, as we can’t just expose the Hardware Virtualization extensions, like Intel VT or AMD-V,  to enable device emulation

And well, that’s my case as I don’t have enough resources in my VMware-based homelab to run an OCP cluster, not even mention OCP Virtualization, so I’ve deployed an OpenShift Cluster on Google Cloud using GCE VM instances, which works perfectly fine until I tried to use OpenShift Virtualization of course :)

In this kind of scenario you need to switch OpenShift Virtualization from its default of hardware virtualization to QEMU-based software emulation.  Just then you will be able to run a virtual machine using OpenShift Virtualization, even in a non-bare metal instance such as GCE VM Instance.

In this post I’ll explain how to enable Software Emulation in OpenShift Virtualization, to be able to run VMs when the default Hardware Virtualization isn’t available.

 

NOTE: The following workaround is not supported by Red Hat, even for dev/test/sandbox environments.   For performance reasons, hardware virtualization, where the virtual machine uses the hardware directly, is  always preferable over software virtualization in real-world implementations. Modern CPUs, memory, and hardware components are designed with hardware virtualization in mind.   Running VMs through OpenShift Virtualization’s software emulation method can result in performance degradation.

 

The Error

I’ve been trying Red Hat OpenShift on Google Cloud, using Google Cloud VM instances, and of course it’s not possible to enable nested virtualization.   So, whenever I try to create and run a virtual machine on OpenShift Virtualization on top these Google Cloud VMs, OpenShift Virtualization will try to run this VM using the default hardware emulation, which of course will fail as this is a non-bare-metal instance of OpenShift Virtualization, and nested virtualization isn’t enabled.

In that casse you will the following warning messages in the Virtualization > Overview section:

If you check the details of the Virtual Machine status, you would see the Virtual Machine isn’t schedulable as the pod can’t find any suitable host, as we can see in the following image:

Pre-Requisites:

This workaround has been tested on Red Hat OpenShift 4.15 installed on non-bare-metal instances like GCE VM instances, and it should works also for other Cloud providers like AWS or Azure.  For older versions you can try the following options:

In addition, OpenShift Virtualization has to be enabled and available in the default namespace, openshift-cnv.  You can enable OpenShift Virtualization by using the Operator as you can see in the following link: https://docs.openshift.com/container-platform/4.15/virt/install/installing-virt.html#virt-installing-virt-operator_installing-virt

 

Procedure

For older versions of OpenShift Virtualization (as described in Pre-requisites), it was possible to enable software emulation by changing the KubeVirt configmap, or the HyperConverged cluster in the OpenShift Virtualization operator, but these options aren’t available anymore in the last release (4.15).

So, after some hours of research, and burning my brain, I’ve found this GitHub repository for the HyperConverged Cluster Operator and how to customize it: https://github.com/kubevirt/hyperconverged-cluster-operator/tree/main/deploy/kustomize

From here, the workaround is quite simple by modifying the Subscription artifact just like we can see in the following link about KVM Emulation: https://github.com/kubevirt/hyperconverged-cluster-operator/blob/main/deploy/kustomize/kvm_emulation/subscription.patch.yaml

  • Edit the HyperConverged subscription with the following command:
Copy to Clipboard
  • At this point, all we need to do, is to add the following environment variable in the spec section:
Copy to Clipboard

 

After saving the changes, these are applied immediately and there is no need of any kind of reboot or restart.

 

So of course, if now I try to create and run a new Virtual Machine, you can see there isn’t any kind of error and the VM is up and running, and the guest operating system booting without issues:

 

As you can see, it’s quite simple procedure so you can test OpenShift Virtualization without bare-metal instances :)