Running pfSense as a VM guest on FreeNAS 9.10 host using Bhyve

Update 31/03/2016: FreeNAS 9.10 is now marked STABLE and 9.3 is in maintenance mode until the version 10 release. There are issues to be aware of though, I have updated the post below, new information is in italics.

Update 15/06/2016: I have tested and started using the vmrun.sh script that allows guest VMs to be successfully rebooted and shutdown from inside the guest OS.

IX Systems recently announced the FreeNAS 9.10-nightly train, an unsupported a version of FreeNAS 9 using a FreeBSD 10 base. This allows users to make use of FreeBSD 10 features including Bhyve, the FreeBSD virtualisation module. I have tested this out to run pfSense firewall as a guest on my HP Microserver running FreeNAS. It was a very experimental process but I managed to get a working configuration which I’ve documented below for my own reference and for others trying to do a similar thing.

The information here is not a step-by-step guide. It involves diving under the hood of theĀ  FreeNAS Web UI and there is no official support available from IX Systems, the FreeNAS and FreeBSD communities may also struggle to help you, so I am assuming you have a decent enough understanding of computer networking and FreeBSD’s networking stack to muddle your way through problems you will likely encounter.

If you follow the information here, there is no warranty, I am not liable if it deletes your data, gets you hacked, burns your house down or anything else. If you follow the information contained here you do so entirely at your own risk.

Hardware

My home server is a HP Microserver N36L, with 8GB ECC RAM, 2xTB WD Red hard disks in ZFS RAID1 (mirrored).

I am using an Intel PRO/1000 PCI-e network interface card instead of the on-board HP/Broadcom NIC, mainly because of VLAN issues I had with the on-board NIC. Most people won’t use VLANs, just replace the VLAN interfaces with the appropriate physical interface name in your own config.

To do this, you must have the following:

  • A CPU with support for virtualisation.
  • Two network cards, or a switch with 802.1q VLAN support.
  • A ZFS zpool, UFS probably will work but will involve some significant differences.
  • At least 8GB RAM, preferably more, that is the minimum for FreeNAS and some of it will be dedicated to pfSense’s use.
  • A 16GB+ USB flash drive for FreeNAS.

Preparing the Images

FreeNAS

Download FreeNAS 9.3 from their website, follow the official instructions to install it to your USB flash drive.

Boot it on your system and then use the System Update menu to switch to the 9.10-Nightlies 9.10-STABLE train, upgrade the system and reboot into FreeNAS 9.10.

pfSense

Download the 4GB embedded image of pfSense from their website, this is not the normal installation image, the file name is:

pfSense-CE-2.3.1-RELEASE-4g-amd64-nanobsd.img.gz

Download this onto your freshly installed FreeNAS system, unzip the file using gunzip.

On FreeNAS Web UI create a 4GB zvol, call it pfSense.

Copy the embedded image to the zvol using ‘dd’, my zpool is called “tank”, change this in the command below to the name of your zpool.

dd if=pfSense-CE-2.3.1-RELEASE-4g-amd64-nanobsd.img of=/dev/zvol/tank/pfSense bs=1M

Tunables

Go to system > tunables in the Web UI, these are the settings you will need, some are not applicable in all situations so read the descriptions here.

Variable cloned_interfaces
Value bridge0 bridge1 tap0 tap1
Type rc

This creates the virtual interfaces we need at boot time.

Variable ifconfig_bridge0
Value addm tap0 addm vlan1 up
Type rc

This configures and brings up bridge0, adding the two interfaces tap0 and vlan1 to it, replace “vlan1” with your LAN interface name.

Variable ifconfig_bridge1
Value addm tap1 addm vlan100 up
Type rc

As above, this configures the WAN bridge. Replace “vlan100” with your WAN interface name.

Variable net.link.bridge.pfil_onlyip
Value 0
Type Sysctl

If you are not using PPPoE you may exclude this. If you are using PPPoE, this is required to prevent the packet filter from blocking anything that’s not IP leaving the bridge.

Variable net.link.tap.up_on_open
Value 1
Type Sysctl

This tells FreeBSD to enable (bring up) the tap interfaces when they are opened by pfSense.

Variable nmdm_load
Value YES
Type loader

This loads the kernel module for nmdm, which is used for gaining serial console access to pfSense locally, this is necessary because Bhyve does not support VGA console access.

Variable vmm_load
Value YES
Type loader

This loads the VMM module for virtualisation support in the kernel.

You can test this first by running ‘kldload vmm’ from the CLI, then look at the output from dmesg, if you have errors then your CPU might not support VM extensions, or they may be disabled in the BIOS.

Network

I have configured my VLAN interfaces through FreeNAS’s network interface menu. You should configure your LAN/WAN interfaces through the networks menu too.

While it’s possible to configure interfaces in the tunables section, if you do so FreeNAS will try to DHCP configure all available interfaces, including bridges, which is undesirable. It must have at least one interface manually configured to prevent this, even if that is just to bring up an interface and nothing more.

The WAN-side interface, VLAN or physical, should be configured to come up (options=up), but should not have an IP address on it. You don’t need it, you’re more secure without it.

My LAN-side interface has the FreeNAS IP address, this goes against FreeBSD best practice advice which is to put the IP address on the bridge. By the time I got this into a working state I didn’t want to break things again by making this change, it probably would work by only bringing up the interface in FreeNAS network configuration and then using tunables to add the IP to the ifconfig_bridge0 entry. edit: When doing so using tunables however the IP address is not present when the jails come up, this causes jails to join an isolated bridge network disconnected from the LAN, this needs further testing.

Boot scripts

On the FreeNAS Web UI, go to Tasks then Init/Shutdown Scripts.

Create a new entry and put the following:

type command
command sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 384 -C /dev/nmdm0A -d /dev/zvol/tank/pfSense -t tap0 -t tap1 pfsense &
when postinit

This command starts pfSense with 384MB RAM and two network interfaces, it can be adjusted to suit your own requirements.

384MB is about the minimum amount of RAM I’ve found necessary on a minimal pfSense 2.3 installation, this was increased from 256MB on version 2.2, obviously the more services you run the more RAM you will need to allocate but keep it as small as possible so not to take too much RAM away from the host.

A serial console is installed on nmdm0, the number can be adjusted if this is in use.

You must end the command with an ampersand (&) so that the command is put into the background and does not hang the VGA console.

Accessing pfSense

Reboot FreeNAS to cause it to boot pfSense as it should do.

To access pfSense, as root on FreeNAS run the command:

cu -l /dev/nmdm0B

As it is a serial console, you will need to tap return to see the menu. You can then configure pfSense’s interfaces in order to access it’s Web UI.

Be careful to check that you have your pfSense vtnet interfaces matched to the correct tap/bridge interface on FreeNAS and so aren’t exposing your unconfigured pfSense box to the Internet, that would be very bad.

Try a reboot, you should see it shutdown and come back up if your loop is properly configured.

To exit the serial console type tilde+fullstop (~.) and you should drop to FreeNAS. If accessing FreeNAS remotely this may also drop you to the local terminal.

In most cases you will just use SSH to manage pfSense but this provides a fail-safe.

How well does it work?

Quite well, pfSense is happy and performs well under Bhyve. I haven’t experienced any crashes or weird behaviour from pfSense.

The boot loop means pfSense cannot be totally shut down without rebooting itself, I’m not aware of a way to force this either, I’ll probably look into this in the near future but at the moment having it automatically reboot is a good thing. Edit: Reboot and shutdown from inside the guest now works successfully when using the vmrun.sh script which responds appropriately to the bhyve exit code.

The network configuration is still a bit unstable, I enabled IPv6 on the LAN interface only to find once it restarted the interface configuration it managed to drop it out of bridge0. However, this only happens if you make configuration changes on a running system, which should almost never need to happen, just be careful when you do. The configuration will come up again properly on reboot.

FreeNAS 9.10 shows tap devices as configurable in the FreeNAS Network menu, however they are not usable by FreeNAS, so this is probably a bug owing to the fact FreeNAS doesn’t normally support tap interfaces, expect the unexpected when making modifications in tunables. Unfortunately bridge interfaces which are usable by FreeNAS, are not recognised as configurable and so can only be configured using tunables.

Bhyve is a new virtualisation system, it seems to work well but it isn’t as mature as other VM systems such as KVM or VMWare. It lacks support for features common in alternatives such as USB support or VGA consoles, it can run various BSD systems and Linux but it’s only really useful for servers, not graphical desktops.

FreeNAS 9.10 is a nightly train, I will periodically update it but I haven’t yet updated it since making the changes, I don’t expect it will cause any breakages but I cannot be sure yet. Edit: FreeNAS 9.10 is now out as STABLE and upgrade is recommended by iX Systems. I have now upgraded to the stable branch, this was successful and the configuration continued to function through the upgrade (woot!), there are several other issues with the stable branch that hit me such as HTTPS being broken, pkg is broken in newly created jails, iohyve does not work, but that’s another story… I personally would recommend waiting until at least a patch release in April before upgrading to STABLE for these reasons.

Iohyve is a Bhyve management tool, based on iocage which in-turn is a more advanced replacement for Warden. I haven’t tested iohyve to see how well it interacts with the manually configured bhyve jail for pfSense. I have some experience with iohyve and I know it would not be suitable for managing a pfSense guest due to it supporting only a single network bridge.

In hindsight

Doing this has been an interesting experience, I learned a lot from it and encountered a number of issues both limitations in my hardware, bugs in FreeNAS and quirks of FreeBSD’s network stack. I look forward to proper Bhyve support in FreeNAS 10 and I hope it comes with flexible support for bridged interfaces.