Daemon Security Blog RSS Feed

[01/10/2018] Running CentOS with Bhyve

With the addition of UEFI in FreeBSD (since version 11), users of bhyve can use the UEFI boot loader instead of the grub2-bhyve port for booting operating systems such as Microsoft Windows, Linux and OpenBSD. The following page provides information necessary for setting up bhyve with UEFI boot loader support:

https://wiki.freebsd.org/bhyve/UEFI

Features have been added to vmrun.sh to make it easier to setup the UEFI boot loader, but the following is required to install the UEFI firmware pkg:# pkg install -y uefi-edk2-bhyveWith graphical support, you can use a vnc client like tigervnc, which can be installed with the following command:# pkg install -y tigervncIn the case of most corporate or government environments, the Linux of choice is RHEL, or CentOS. Utilizing bhyve, you can test and install CentOS in a bhyve VM the same way you would deploy a Linux VM in production. The first step is to download the CentOS iso (for this tutorial I used the CentOS minimal ISO):

http://isoredirect.centos.org/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1708.iso

I normally use a ZFS Volume (zvol) when running bhyve VMs. Run the following commands to create a zvol (ensure you have enough disk space to perform these operations):# zfs create -V20G -o volmode=dev zroot/centos0
(zroot in this case is the zpool I am using)
Similar to my previous post about vmrun.sh, you need certain items to be configured on FreeBSD in order to use bhyve. The following commands are necessary to get things running:# echo "vfs.zfs.vol.mode=2" >> /boot/loader.conf
# kldload vmm
# ifconfig tap0 create
# sysctl net.link.tap.up_on_open=1
net.link.tap.up_on_open: 0 -> 1
# ifconfig bridge0 create
# ifconfig bridge0 addm em0 addm tap0
# ifconfig bridge0 up
(replace em0 with whatever your physical interface is).
There are a number of utilities that can be used to manage bhyve VMs, and I am sure there is a way to use vmrun.sh to run Linux VMs, but since all of the HowTos for running Linux use the bhyve command line, the following script is what I use for running CentOS with bhyve.#!/bin/sh
#
# General bhyve install/run script for CentOS
# Based on scripts from pr1ntf and lattera

HOST="127.0.0.1"
PORT="5901"
ISO="/tmp/centos.iso"
VMNAME="centos"
ZVOL="centos0"
SERIAL="nmda0A"
TAP="tap1"
CPU="1"
RAM="1024M"
HEIGHT="800"
WIDTH="600"

if [ "$1" == "install" ];
then

#Kill it before starting it
bhyvectl --destroy --vm=$VMNAME

bhyve -c $CPU -m $RAM -H -P -A \
-s 0,hostbridge \
-s 2,virtio-net,$TAP \
-s 3,ahci-cd,$ISO \
-s 4,virtio-blk,/dev/zvol/zroot/$ZVOL \
-s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
-s 30,xhci,tablet \
-s 31,lpc -l com1,/dev/$SERIAL \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
$VMNAME

#kill it after
bhyvectl --destroy --vm=$VMNAME

elif [ "$1" == "run" ];
then

#Kill it before starting it
bhyvectl --destroy --vm=centos

bhyve -c $CPU -m $RAM -w -H \
-s 0,hostbridge \
-s 2,virtio-net,$TAP \
-s 4,virtio-blk,/dev/zvol/zroot/$ZVOL \
-s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
-s 30,xhci,tablet \
-s 31,lpc -l com1,/dev/$SERIAL \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
$VMNAME &

else
echo "Please type install or run";
fi
The variables at the top of the script can be adjusted to fit your own needs. With the addition of the graphics output protocol in UEFI (or UEFI-GOP), a VNC console is launched and hosted with the HOST and PORT setting. There is a password option available for the VNC service, but the connection should be treated as insecure. It is advised to only listen on localhost with the VNC console and tunnel into the host of the bhyve VM. Now with the ISO copied to /tmp/centos.iso, and the script saved as centos.sh you can run the following command to start the install:# ./centos.sh installAt this point, using vncviewer (on the local machine, or over an SSH tunnel), you should be able to bring up the console and run the CentOS installer as normal. The absolutely most critical item is to resolve an issue with the booting of UEFI after the installation has completed. Because of the path used in bhyve, you need to run the following to be able to boot CentOS after the installation:# cp -f /mnt/sysimage/boot/efi/EFI/centos/grubx64.efi /mnt/sysimage/boot/efi/EFI/BOOTWith this setting changed, the same script can be used to launch your CentOS VM as needed:# ./centos.sh runIf you are interested in a better solution for managing your Linux VM, take a look at the various bhyve management ports in the FreeBSD ports tree.

Author: Michael Shirk