bhyve logo

TLDR;

bhyve (“bee-hive”) is BSD’s builtin hypervisor, on Intel and AMD processors that allows the users to create virtual machines.

bhyve supports any version of FreeBSD i386/amd64 with VirtIO support. OpenBSD, NetBSD and GNU/Linux are supported using the sysutils/grub2-bhyve port. UEFI support is available for FreeBSD, OpenBSD, GNU/Linux, SmartOS and Windows:

  • FreeBSD 8.4-RELEASE and newer, including CURRENT
  • OpenBSD amd64/i386 5.2 and newer
  • GNU/Linux amd64/i386 (Many distributions)
  • NetBSD amd64 6.1 and newer
  • SmartOS 20151001 and newer
  • Windows x64 Vista, 7, 8, 10
  • Windows Server 2008r2, 2012r2, 2016

In the office we encourage our staff to rotate different Operating Systems periodically. We find this encouragement helps expand and enhance their knowledge, familiarity and system adminsitration skills of the available and different OS’s to date. It also helps them produce more compatiable code when writing/developing open-source code in support of public projects or small inhouse projects that help improve their day-2-day pentesting or incident response roles.

This post is a quick walkthrough, of preparing and installing virtual machines natively on FreeBSD utilising bhyve.

Setup

Networking - Manual

Setup tap0

 echo "net.link.tap.up_on_open=1" >> /etc/sysctl.conf
 sysctl net.inet.ipforwarding=1

For users utlising a wired connection, you can create a bridged interface:

 ifconfig bridge create
 ifconfig bridge0 name em0bridge
 ifconfig em0bridge up
 ifconfig tap0 create
 ifconfig em0bridge addm tap0

For users utlising a wireless connection, bridge will not work as wireless interfaces can only have 1x MAC address. Instead we can use a NAT’ed interface:

ifconfig tap0 create
ifconfig tap0 inet 172.16.1.1/24

Networking - Automated

For auto-bridge setup add these to /etc/rc.conf

 cloned_interfaces="bridge0 tap0"
 ifconfig_bridge0_name="em0bridge"
 ifconfig_em0bridge="addm em0 addm tap0 up"
 gateway_enable="yes"

For our wireless and NAT’ed interface, add the following to /etc/rc.conf

 cloned_interfaces="lo1 tap0"
 ifconfig_tap0="inet 172.16.1.1/24"
 pf_enable="YES"
 natd_enable="YES"
 natd_interface="wlan0"
 natd_flags="-u -m -dynamic"
 gateway_enable="yes"

NAT and PF firewall

You will need to add the following lines into pf.conf, for the pf firewall to correctly route your NAT’ed interfaces traffic

 virt_net="172.16.1.1/24"
 
 nat on $ext_if from $virt_net to any -> ($ext_if)
 
 block in all
 pass in on tap0

Loading Kernel Modules - Manual

load kernel modules

 kldload vmm
 kldload nmdm

Loading Kernel Modules - Automatic

Run the following in a shell, to add these to your boot.conf:

 echo 'vmm_load="YES"' >> /boot/loader.conf
 echo 'nmdm_load="YES"' >> /boot/loader.conf
 echo 'if_tap_load="YES"' >> /boot/loader.conf
 echo 'if_bridge_load="YES"' >> /boot/loader.conf

Installation

Assuming your using pkg manager, installing bhyve is as simple as:

 pkg install bhyve grub2-bhyve uefi-edk2-bhyve tigervnc-viewer

Creating our first Virtual Machine

Preparing a Virtual Disk

Zfs volume creation; the following command creates a 20GB ZFS volume for our virtual machine:

 zfs create -V20G -o volmode=dev zroot/kalivm

Booting a Linux Distro

Booting a Debian/Kali iso (we assume you know how to download an appropriate install ISO, we temporarily download ours to /tmp)

 bhyve -c 2 -m 2G -w -H \
 -s 0,hostbridge \
 -s 3,ahci-cd,/tmp/kali.iso \
 -s 4,virtio-blk,/dev/zvol/zroot/kalivm \
 -s 5,virtio-net,tap0 \
 -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \
 -s 30,xhci,tablet \
 -s 31,lpc -l com1,stdio \
 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
 kalivm

Next use tigervnc viewer to view the virtual machine’s console and complete the installtion:

 vncviewer :5900

IMPORTANT: AT END OF DEBIAN INSTALL DROP TO SHELL

 mkdir /target/boot/efi/EFI/BOOT/
 # copy file - workaround for bhyve grub package #
 # Pay attention to destination file bootx64.efi #
 cp /target/boot/efi/EFI/debian/grubx64.efi /target/boot/efi/EFI/BOOT/bootx64.efi

Freeing resources

After powering down the VM we need to dispose of its resources before we can reboot, this is as simple as:

 bhyvectl --destroy --vm=kalivm

Booting our Linux VM

 bhyve -c 2 -m 2G -w -H \
 -s 0,hostbridge \
 -s 4,virtio-blk,/dev/zvol/zroot/kalivm \
 -s 5,virtio-net,tap0 \
 -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait \
 -s 30,xhci,tablet \
 -s 31,lpc -l com1,stdio \
 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
 kalivm

tigervnc bhyve vm

Other

Converting ESX/VirtualBox virtualised disks to bhyve

bhyve uses the same disk format as Qemu, converting a disk image is as simple as following the next few steps:

Don’t forget to install the qemu tools first:

 pkg install qemu-devel

then:

 qemu-img convert -f vmdk -O raw virtual_box_image.vmdk bhyve_raw_image.img

Conclusion

Creating virtual machines natively in BSDs hypervisor is relativly straightforward. We have brought networking differences to attention, NAT vs bridging for those that utilise wireless networks over desk-wired ethernet. However, the creation of virtual machines can still be quite techncial; There are a few differnet packages that can make this process easier, they tend to use of pre-built templates, and wizards to guide the user through the setup and installation processes. If the above walkthrough is too daunting we suggest you try cbsd instead of manual creation.

References

Share on: