Running pfSense as a VM guest on FreeNAS 9.10 host using Bhyve

Update 31/03/2016: FreeNAS 9.10 is now marked STABLE and 9.3 is in maintenance mode until the version 10 release. There are issues to be aware of though, I have updated the post below, new information is in italics.

Update 15/06/2016: I have tested and started using the vmrun.sh script that allows guest VMs to be successfully rebooted and shutdown from inside the guest OS.

IX Systems recently announced the FreeNAS 9.10-nightly train, an unsupported a version of FreeNAS 9 using a FreeBSD 10 base. This allows users to make use of FreeBSD 10 features including Bhyve, the FreeBSD virtualisation module. I have tested this out to run pfSense firewall as a guest on my HP Microserver running FreeNAS. It was a very experimental process but I managed to get a working configuration which I’ve documented below for my own reference and for others trying to do a similar thing.

The information here is not a step-by-step guide. It involves diving under the hood of theΒ  FreeNAS Web UI and there is no official support available from IX Systems, the FreeNAS and FreeBSD communities may also struggle to help you, so I am assuming you have a decent enough understanding of computer networking and FreeBSD’s networking stack to muddle your way through problems you will likely encounter.

If you follow the information here, there is no warranty, I am not liable if it deletes your data, gets you hacked, burns your house down or anything else. If you follow the information contained here you do so entirely at your own risk.

Hardware

My home server is a HP Microserver N36L, with 8GB ECC RAM, 2xTB WD Red hard disks in ZFS RAID1 (mirrored).

I am using an Intel PRO/1000 PCI-e network interface card instead of the on-board HP/Broadcom NIC, mainly because of VLAN issues I had with the on-board NIC. Most people won’t use VLANs, just replace the VLAN interfaces with the appropriate physical interface name in your own config.

To do this, you must have the following:

  • A CPU with support for virtualisation.
  • Two network cards, or a switch with 802.1q VLAN support.
  • A ZFS zpool, UFS probably will work but will involve some significant differences.
  • At least 8GB RAM, preferably more, that is the minimum for FreeNAS and some of it will be dedicated to pfSense’s use.
  • A 16GB+ USB flash drive for FreeNAS.

Preparing the Images

FreeNAS

Download FreeNAS 9.3 from their website, follow the official instructions to install it to your USB flash drive.

Boot it on your system and then use the System Update menu to switch to the 9.10-Nightlies 9.10-STABLE train, upgrade the system and reboot into FreeNAS 9.10.

pfSense

Download the 4GB embedded image of pfSense from their website, this is not the normal installation image, the file name is:

pfSense-CE-2.3.1-RELEASE-4g-amd64-nanobsd.img.gz

Download this onto your freshly installed FreeNAS system, unzip the file using gunzip.

On FreeNAS Web UI create a 4GB zvol, call it pfSense.

Copy the embedded image to the zvol using ‘dd’, my zpool is called “tank”, change this in the command below to the name of your zpool.

dd if=pfSense-CE-2.3.1-RELEASE-4g-amd64-nanobsd.img of=/dev/zvol/tank/pfSense bs=1M

Tunables

Go to system > tunables in the Web UI, these are the settings you will need, some are not applicable in all situations so read the descriptions here.

Variable cloned_interfaces
Value bridge0 bridge1 tap0 tap1
Type rc

This creates the virtual interfaces we need at boot time.

Variable ifconfig_bridge0
Value addm tap0 addm vlan1 up
Type rc

This configures and brings up bridge0, adding the two interfaces tap0 and vlan1 to it, replace “vlan1” with your LAN interface name.

Variable ifconfig_bridge1
Value addm tap1 addm vlan100 up
Type rc

As above, this configures the WAN bridge. Replace “vlan100” with your WAN interface name.

Variable net.link.bridge.pfil_onlyip
Value 0
Type Sysctl

If you are not using PPPoE you may exclude this. If you are using PPPoE, this is required to prevent the packet filter from blocking anything that’s not IP leaving the bridge.

Variable net.link.tap.up_on_open
Value 1
Type Sysctl

This tells FreeBSD to enable (bring up) the tap interfaces when they are opened by pfSense.

Variable nmdm_load
Value YES
Type loader

This loads the kernel module for nmdm, which is used for gaining serial console access to pfSense locally, this is necessary because Bhyve does not support VGA console access.

Variable vmm_load
Value YES
Type loader

This loads the VMM module for virtualisation support in the kernel.

You can test this first by running ‘kldload vmm’ from the CLI, then look at the output from dmesg, if you have errors then your CPU might not support VM extensions, or they may be disabled in the BIOS.

Network

I have configured my VLAN interfaces through FreeNAS’s network interface menu. You should configure your LAN/WAN interfaces through the networks menu too.

While it’s possible to configure interfaces in the tunables section, if you do so FreeNAS will try to DHCP configure all available interfaces, including bridges, which is undesirable. It must have at least one interface manually configured to prevent this, even if that is just to bring up an interface and nothing more.

The WAN-side interface, VLAN or physical, should be configured to come up (options=up), but should not have an IP address on it. You don’t need it, you’re more secure without it.

My LAN-side interface has the FreeNAS IP address, this goes against FreeBSD best practice advice which is to put the IP address on the bridge. By the time I got this into a working state I didn’t want to break things again by making this change, it probably would work by only bringing up the interface in FreeNAS network configuration and then using tunables to add the IP to the ifconfig_bridge0 entry. edit: When doing so using tunables however the IP address is not present when the jails come up, this causes jails to join an isolated bridge network disconnected from the LAN, this needs further testing.

Boot scripts

On the FreeNAS Web UI, go to Tasks then Init/Shutdown Scripts.

Create a new entry and put the following:

type command
command sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 384 -C /dev/nmdm0A -d /dev/zvol/tank/pfSense -t tap0 -t tap1 pfsense &
when postinit

This command starts pfSense with 384MB RAM and two network interfaces, it can be adjusted to suit your own requirements.

384MB is about the minimum amount of RAM I’ve found necessary on a minimal pfSense 2.3 installation, this was increased from 256MB on version 2.2, obviously the more services you run the more RAM you will need to allocate but keep it as small as possible so not to take too much RAM away from the host.

A serial console is installed on nmdm0, the number can be adjusted if this is in use.

You must end the command with an ampersand (&) so that the command is put into the background and does not hang the VGA console.

Accessing pfSense

Reboot FreeNAS to cause it to boot pfSense as it should do.

To access pfSense, as root on FreeNAS run the command:

cu -l /dev/nmdm0B

As it is a serial console, you will need to tap return to see the menu. You can then configure pfSense’s interfaces in order to access it’s Web UI.

Be careful to check that you have your pfSense vtnet interfaces matched to the correct tap/bridge interface on FreeNAS and so aren’t exposing your unconfigured pfSense box to the Internet, that would be very bad.

Try a reboot, you should see it shutdown and come back up if your loop is properly configured.

To exit the serial console type tilde+fullstop (~.) and you should drop to FreeNAS. If accessing FreeNAS remotely this may also drop you to the local terminal.

In most cases you will just use SSH to manage pfSense but this provides a fail-safe.

How well does it work?

Quite well, pfSense is happy and performs well under Bhyve. I haven’t experienced any crashes or weird behaviour from pfSense.

The boot loop means pfSense cannot be totally shut down without rebooting itself, I’m not aware of a way to force this either, I’ll probably look into this in the near future but at the moment having it automatically reboot is a good thing. Edit: Reboot and shutdown from inside the guest now works successfully when using the vmrun.sh script which responds appropriately to the bhyve exit code.

The network configuration is still a bit unstable, I enabled IPv6 on the LAN interface only to find once it restarted the interface configuration it managed to drop it out of bridge0. However, this only happens if you make configuration changes on a running system, which should almost never need to happen, just be careful when you do. The configuration will come up again properly on reboot.

FreeNAS 9.10 shows tap devices as configurable in the FreeNAS Network menu, however they are not usable by FreeNAS, so this is probably a bug owing to the fact FreeNAS doesn’t normally support tap interfaces, expect the unexpected when making modifications in tunables. Unfortunately bridge interfaces which are usable by FreeNAS, are not recognised as configurable and so can only be configured using tunables.

Bhyve is a new virtualisation system, it seems to work well but it isn’t as mature as other VM systems such as KVM or VMWare. It lacks support for features common in alternatives such as USB support or VGA consoles, it can run various BSD systems and Linux but it’s only really useful for servers, not graphical desktops.

FreeNAS 9.10 is a nightly train, I will periodically update it but I haven’t yet updated it since making the changes, I don’t expect it will cause any breakages but I cannot be sure yet. Edit: FreeNAS 9.10 is now out as STABLE and upgrade is recommended by iX Systems. I have now upgraded to the stable branch, this was successful and the configuration continued to function through the upgrade (woot!), there are several other issues with the stable branch that hit me such as HTTPS being broken, pkg is broken in newly created jails, iohyve does not work, but that’s another story… I personally would recommend waiting until at least a patch release in April before upgrading to STABLE for these reasons.

Iohyve is a Bhyve management tool, based on iocage which in-turn is a more advanced replacement for Warden. I haven’t tested iohyve to see how well it interacts with the manually configured bhyve jail for pfSense. I have some experience with iohyve and I know it would not be suitable for managing a pfSense guest due to it supporting only a single network bridge.

In hindsight

Doing this has been an interesting experience, I learned a lot from it and encountered a number of issues both limitations in my hardware, bugs in FreeNAS and quirks of FreeBSD’s network stack. I look forward to proper Bhyve support in FreeNAS 10 and I hope it comes with flexible support for bridged interfaces.

44 Replies to “Running pfSense as a VM guest on FreeNAS 9.10 host using Bhyve”

  1. Hello David,

    Thanks for the nice blog entry. This was interesting and I wanted to try it out for myself. I have only started using FreeNAS since March so it was a bit of a challenge for me to get started (had to read a lot of documentation to understand all the things you are talking about).

    I am running my setup at home on a Dell PowerEdge T20 with 12GB of RAM, 4 x 1TB disks (various makes and models) and two NICs in addition to the built-in one.

    Initially I tried to do the network interfaces different though. I tried doing passthru as I had 2 NICs to spare. Never worked because they seem to be cheap ones that have a MSI-X compatibility problem with byhyve (not sure whether this description is accurate). You have to read to the bottom of the bhyve passthru wiki to find out. So I ended up using the same configuration with bridged adapters as you did.

    My next problem was that the command I entered was too long for the Web UI to accept. So I had to convert it to a script and make pfsense start that way.

    If you get a chance to experiment some more please share your results. Thanks again.

    Cheers,
    Wolfgang

    1. Hi Wolfgang,
      Yes the command can easily go over the maximum length permitted, the solution is to save it into a script and call the script instead, as you have done, just be aware you must add an ampersand “&” at the end of the script call so that the script is executed as a background process, this avoids it hanging your VGA console because it will wait for it to exit, which will never happen!

      I have improved on the execution script but I have not had time to update the blog post, I am now successfully using the vmrun.sh script included with bhyve which is more intelligent and can interpret bhyve’s exit codes, this means you can reboot the guest as well as shut it down permanently from inside the guest OS and the vmrun script will respond appropriately. The command I am now using is as follows, you can also call this from a custom script if your version is too long:

      sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 384 -C /dev/nmdm0A -d /dev/zvol/tank/pfSense -t tap0 -t tap1 -t tap2 pfsense &

      cheers,
      David

  2. Hi David,

    after several years of using a Fritz!Box (popular DSL router in Germany) I decided to take the plunge and set up my IP infrastructure by myself. Your blog was very helpful, pfSense on bhyve is now up&running quite stable since several weeks. I’m using FreeNAS 9.10 on a supermicro atom board. DSL connection is done via a Draytek Vigor 130 running as PPPoE modem triggered by that bhyve’ pfSense.

    To make a long story short: You can also use the full install of pfSense, since it also opens a serial connection which can be captured by cu -l. I checked this with pfSense 2.3.1. This gives one advantage if you use pfBlockerNG, since it still has an annoying bug which prevents unbound+pfBlockerNG from starting up properly from the nano edition. And you can install some more stuff into that VM if you like to do so πŸ˜‰

    Some more insights from my learning curve (I guess you know that already, so this is more for the other beloved readers of this blog):

    If pfSense should be used as DHCP server, use static IP addresses for all interfaces in FreeNAS, for FreeNAS itself, any bhyve VMs (pfSense!) and for the jails. Sounds trivial, but it took me some time to find out πŸ™

    Setting those static IPs for the FreeNAS jails was somewhat strange using the Web GUI of FreeNAS, i.e. it did not really work. I used the ssh login to FreeNAS, employing directly the warden script, this worked. `warden help set’ is your friend.

    Since I use pfBlockerNG within pfSense with a lot of black lists, I had to upgrade the memory option to 768 MB, 256 MB was too few.

    Cheerio,
    Heiko

    1. Hi Heiko,
      Glad to hear my blog post helped you!

      I am not a user of pfBlockerNG so I haven’t come across that problem. There is an advantage to running the embedded image in this instance in that during host shutdown FreeNAS gives the guests only a couple of seconds to shutdown which may not be enough, the embedded image is designed to be more resilient against sudden loss of power.

      You are right that 256MB is not enough, the post is written for pfSense 2.2 which was fine with 256MB under my light use but after upgrading to pfSense 2.3 I found that memory utilisation increased significantly and so I have allocated more RAM on my server to pfSense. Unless you have plenty of memory though it is best to keep the allocation as small as possible, this memory will be taken away from FreeNAS’s use for ZFS caching which is quite important!

      IP addresses in FreeNAS should be statically configured, as I mentioned in the post it will attempt to automatically configure all interfaces by DHCP if no manual configuration is set, this is not desirable for security and also as you say when pfSense is the DHCP server FreeNAS will attempt to configure its network interfaces before it even starts booting pfSense so the DHCP server will not be available and this will fail.

      Thanks for your input!

      cheers,
      David

  3. Hi,

    Excellent setup and project!

    Quick questions:
    1. Would not vm-to-metal-extensions (eg iomimo/vt-d/amd-vi) be required to get your pfSense to work with your NICs at full speed? Or is the drawback of not having it negligible?
    2. If it (#1 above) is a problem; could there be a way to get around it by somehow running pfSense in a container using FreeNas’ kernel (docker-style)? Or would perhaps the security issues be to great?
    3. Why is 8Gb mem a minimum? I thought the ZFS cache hunger had been debunked as a myth…?

    1. Thanks Fred, in answer to your questions, all IMHO as I’m not any sort of authority on the subject…

      1. Bhyve only works with hardware CPU virtualisation support but it is not necessary to pass-through peripherals directly to the guest OS.

      That said, yes there will be some performance impact using virtual network devices etc however even with my modest hardware I am able to throughput the 40Mbit/s I need for my Internet connection.

      With faster hardware or throughput requirements you might benefit from assigning hardware directly if you’re unable to get the performance you need.

      2. My understanding is that pfSense must use it’s own customised kernel, it will not function on the kernel FreeNAS uses so cannot be jailed. I’m unsure if jails can even be assigned multiple network interfaces. I would also expect a less secure environment using jails which is important for a firewall.

      3. 8GB is the minimum specified for FreeNAS, this seems a reasonable requirement in my experience. Like any minimum requirements, you can often get away with providing it with less, but you will be compromising your experience of FreeNAS through your decision to do so.

      There are many myths around ZFS, mostly coming from people obsessive about performance and data integrity, but people have different expectations, if it works for you then go for it.

      The issue for me here is that unlike jails, any allocation of resources to bHyve guests will prevent FreeNAS using that resource. This will compromise the ability of FreeNAS to fulfill its primary function, which is to be a NAS.

  4. Thank you for introducing a very nice project. I happen to have exactly the same (good old) HP microservers, so I have tried setting up pfsense-2.3.2 and FreeNAS-9.10 with a two-port INTEL NIC (PRO/1000PT). Everything works fine, but the throughput of SCP data transfer to FreeNAS I am getting is not as good as 40Mbps as you report, but rather only 2.5Mbps or so. I wonder if you can share any performance tuning you may have done. The only difference from your environment is VLANs vs. physical LAN/WAN ports, but I do not think that would make a huge difference.

    1. Hi Rob,
      I have made no further performance tuning on this server however I am careful not to over-load it as the CPU on this server is weak and is easily overwhelmed, so I remove unused jails and plugins on a regular basis.

      The 40Mbit figure is routing performance from the LAN to the WAN, this is not traffic directed at the NAS itself which introduces a range of other potential performance bottlenecks you might be experiencing.

      If you still get slow performance when bhyve is not running then you can discount that as the cause. If it is only occurring when bhyve is running then check your load averages are below 2.0, look at the process table in the GUI or “top” from the CLI and look if anything is consuming too much CPU or memory resources.

      My CPU is typically between 1-1.5 load average, which is high enough but this is largely due to other software I have running and not pfSense.

  5. Hi David
    Thanks for this blog entry. I had noticed that there was a pfsense template for freenas 10 so, decided to follow this guide. I have everything working perfectly, however for securitys sake, I was wondering if its possible to run the physical network card for the LAN without assigning an IP to the freenas host. IE, if I create bridge0, and join igb0 and tap0 to it, can I avoid having to set an IP within the freenas GUI? I already have a 10GBe card that runs the SMB share and webgui, and its on a different subnet… I’d just like to make sure the link between the LAN and the physical port can work without assigning an ip directly to igb0.
    Mobo is a supermicro x11, and since the smb is running off the 10gbe card, the 2 onboard nics aren’t being used by FreeNas – can the jail access those cards directly without having to create a bridge, or is that a necessary requirement to enable the jail to talk to the host?
    Many thanks

    1. Hi Ben, I don’t think I understand your question or set up well enough to give a proper answer, but a few tips might help. Feel free to clarify or follow up after if I’m getting the wrong end of the stick…

      You can bring up any interface without assigning an IP address, to do so add the interface in the web GUI menu, leave all the boxes empty except for the “options” box, here enter the word “up”. This will bring up an otherwise unconfigured interface, this is how my WAN vlan is configured but physical interfaces work all the same.

      Jails must have a bridge, all jails use the same bridge so effectively must all be on the same Ethernet layer 2 network. FreeNAS will use bridge0 for jails, this usually includes the FreeNAS host physical interface. If you don’t want that you can put the pfSense LAN into a different bridge such as bridge2 and give it one of those other physical ports but the jails won’t be able to make use of pfsense that way.

  6. Hi David
    Thanks for the clarification. I have a chelsio S320e dual port SFP+ which is the main NIC that FreeNAS uses. The onboard motherboard NIC’s are not being used which is what I want to assign to pfSense.
    If FreeNAS uses that as the main interface, can I bring up the Lan interface for pfsense without an IP specified in the config panel? I understand how to do this now that you’ve told me how to do it with the WAN, but I was rather hoping to avoid setting an IP for the pfSense dedicated LAN port on the FreeNAS side. Its working quite well at the moment, I’d just prefer to remove the IP layer from the FreeNas side and only specify it within the pfSense Jail (ie the igb0 adaptor has 192.168.0.30, bridge0 to tap0, tap0 is set to 192.168.0.31 within the pfsense jail – I’d like to simply bring igb0 up without specifying the 0.30 address
    Thanks again for the write up. I’m staggered by the performance compared to hyperV

    1. Yes, the method I described should work fine. FreeNAS does not see any difference between the LAN and WAN bridges, only difference here is you FreeNAS management IP is on a separate LAN altogether. I would suggest using bridge1 and bridge2 for pfSense and leaving bridge0 for FreeNAS jails.

  7. Thanks David – I did just that – I assigned igb0 and tap0 to bridge1 and igb1 and tap1 to bridge2, left bridge0 intact and connected to the SFP+ card. Working perfectly, and as far as the ethernet ports are concerned, they’re both only bridged to the pfSense instance, not the jails nor the FreeNas distro. Perfect!
    Thanks again for your help!

  8. Hi David – Just an update for anyone interested, the full version of pfsense can easily be installed. Setup 2 zvols, one for the IMG to be written to via the DD command above, the other one a bigger partition (I put mine as 20gb as Im going to run squid off it and I want it to be large). If you use the initial startup command, add another -C entry to mount the second zvol as another visible media;

    sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 512 -C /dev/nmdm0A -d /dev/zvol/Jails/pfsense -d /dev/zvol/Jails/Firewall -t tap0 -t tap1 pfsense &

    Upon reboot, this will launch the pfsense zvol as the boot drive, you can then do the basic/automated install and it will then copy the files to the Firewall zvol. When the install is finished, go back to the init script and remove ‘-d /dev/zvol/Jails/pfsense’ and then reboot – it will then boot using the Firewall zvol and you can then run through the initial configuration via the console….

    FreeNAS best practice suggests keeping Jails off the boot device, I recommend either a ssd but a fast usb stick also works. Both options work just fine.

    1. Hi Ben,
      Thanks for your update.

      I’ve not tried it myself but instead of making two zvols, it might be easier just using the installation ISO and one zvol to install to, you can launch this from vmrun.sh for the first boot using the -I parametre e.g.:

      /usr/share/examples/bhyve/vmrun.sh -I /path/to/pfSense-CE-2.3.2-RELEASE-amd64.iso

      After install is complete you can remove the -I option from the boot command.

      In my case the boot device is a USB flash disk but I use my standard HDD array to store the zvols with the rest of the data, the speed is quite adequate as pfSense does not make much use of disk access.

      cheers,
      David

  9. Hello David,

    thank you very much for writing this post.
    Ever since I read it, I wanted to try it out myself. I purchased an additional NIC (NC360T), installed it and followed your instructions on my N54L with FreeNAS-9.10.1-U1.
    Setting up the pfSense VM worked like a charm with your guidance, however the network configuration was a complete mess to the point, that I couldn’t reach my box via SSH and had to manually roll back the FreeNAS settings DB by connecting the N54L to a monitor+keyboard πŸ™

    Now I’m wondering, if you could give me some insights of what might have went wrong, or point me in the right direction.

    The onboard NIC (bge0) on my N54L gets its IP from a DHCP server (my SOHO router) and this IP is also the one I use to reach the FreeNAS web interface. Then there are other Jails/Plugins behind bridge0, each with their own IP (default setup by FreeNAS).
    My Plan is to use the new dual NIC (em0, em1) exclusively for pfSense. So em0 should be the WAN interface that gets its IP from my ISP (no PPPoE) and em1 should be the LAN interface that forwards everything to my router (router is DHCP server and switch) and from there to all other devices, including the N54L onboard NIC, etc.

    I tried to adapt your setup to my plan, but as already mentioned, after a reboot the FreeNAS web interface (onboard NIC) wasn’t reachable anymore. This was before I even adjusted the cabling (em0/em1 had no cables yet).
    Since I already had a bridge0 in my setup, I replaced bridge0 and bridge1 from your guide with bridge1 and bridge2 respectively.
    Also what is the third tap (tap2) you are giving to vmrun.sh? There is no mention of tap2 before.

    Do you have an idea of what I could try next?

    1. Hi Jan,
      Did FreeNAS’s VGA console report having an IP address at all? pfSense will run a DHCP server by default on the LAN interface so FreeNAS might have received an IP address via bridge0 on em1 and left bge0 unconfigured. The key to this is the ensure that em0 and em1 are both configured manually but without an IP address, this is done by adding the interfaces in the Web UI and entering only the word “up” in the options box. This way FreeNAS will not attempt to DHCP configure these interfaces.

      That said I would recommend you always statically configure the IP address on FreeNAS to avoid problems, you can always patch a cable in and get access that way.

      Regarding tap2 you can skip that out, it is an optional interface that I use as a DMZ for servers that I don’t want to give full access to my LAN.

      Once this is up and working for you, you might be best connecting pfSense’s LAN to the internal ports on your SOHO router instead of the WAN and then disable DHCP on your SOHO router allowing pfSense to provide DHCP for your network, this would allow you to make better use of pfSense’s features.

      Good luck!

  10. Thank you for your quick reply!
    I was finally able to set everything up as planned πŸ™‚
    I guess the problem was, that I had only one NIC (bge0) configured in the FreeNAS web-interface. Adding the others (em0/em1) and just setting them to “up” as you suggested solved my problems.

    1. Hi Rob,
      I haven’t used FreeNAS 10 yet and don’t intend on upgrading my NAS system for a while after it is released. If I get time I’ll spin it up in a VM to take a look, but I doubt that there are significant differences that would prevent this guide working. Give it a try if you like and feel free to comment with your experiences.

    2. I’ve had a bit of a play with FreeNAS 10 nightlies and it looks like the method will be very different, all the required functionality is now supported in the Web UI so there should be no more fiddling with tunables. There’s even a pfSense VM image for easy installation. I wouldn’t recommend moving up to version 10 yet though, I tried it out after RC1 was announced and I’ve encountered a lot of problems with it already but hopefully once they’ve been ironed out the process should be much simpler.

  11. This was really useful. Do you know of any other sites or resources that can explain a little more about the whole concepts of what you’ve done above. I too have a microserver running ESXI and was using VMware experimenting with machines. I want to switch it over to freenas to use as a traditional NAS, but also run other machines on it as well. I know it’s capable of a lot more than what am doing with it at the moment. Ideally i’d like to get it set up with VPN access/a firewall, and maybe a media server (like plex etc). Just i get a bit lost sometimes following these articles. Naturally they’re aimed at the more technically advanced given the concepts discussed, but never anything that bridges the gap between keen learner and linux overlord πŸ™‚

    In any event, thanks for the post above!
    Andy

    1. Hi Andy,
      The main resources I referenced for this project were:

      https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/
      https://wiki.freebsd.org/action/show/bhyve?action=show&redirect=BHyVe

      The FreeBSD handbook is an excellent resource and has helped me learn a lot about FreeBSD. I would strongly recommend installing a FreeBSD server as unfortunately FreeNAS does a very good job at abstracting away the nuts and bolt of FreeBSD, once you remove that you’ll be in an environment whre you are challenged to learn much faster. FreeNAS is essentially nanoBSD with a custom web UI and set of scripts, nanoBSD is a liveCD-like image of FreeBSD designed for embedded systems, there’s more information on that here:

      https://www.freebsd.org/doc/en/articles/nanobsd/

      There is good networking stuff in the handbook but it helps to have the fundamental concept right, best place for that is the Cisco CCENT/ICND material, that will cover all of the concepts used in this projects and more so building from the ground up.

      HTH
      David

  12. Great, Thanks for that will have a look at it. I must admit, when looking at linux distro’s i can’t help but be seduced by the more modern looking polished interfaces. I need to try and get back to basics.

    thx

  13. Big thanks for your tutorial. Things work like a champ for over 3 months. No problem at all. Additionally, I’ve managed to passthrough 3 ethernet ports from mainboard to vm machine, probably overkill as far as my network traffic requriments, but I can sleep well πŸ™‚

    1. Hi Jan,
      Funnily, I had just left a comment regarding FreeNAS 10 only a few hours before you posted:

      http://davidnelson.me/?p=439#comment-469

      Basically the answer is no this guide is no longer relevant for FreeNAS 10. The setup is very different but much simpler due to GUI support for VM guests and bridges, in fact the bridges are set up automatically when you configure a VM guest network interface in bridge mode.

      I expect that the kind of person who might be setting up FreeNAS will be able to figure it out without needing much networking expertise. However I haven’t had time to examine how FreeNAS 10 configures the networking so I can’t say whether or not it is done in a sensible way for security by default so I may discuss that in another post soon.

      No, I would not recommend using FreeNAS 10 for anything besides testing on a separate box with test data only. I encountered many bugs and I came away quite surprised they saw it fit to put the Release Candidate label on it as it’s barely BETA quality but it does show a lot of promise, I think it will be a good release when it arrives.

  14. Hi David
    With Freenas 11 having reverted to the 9.10 style of doing things, would you say that the above guide is still relevant? I’m interested in loading both pfSense and Centos under bhyve but the old style gui doesn’t seem to be feature complete as yet.

    1. Hi Ben, I haven’t tested it specifically but as far as I’m aware FreeNAS 11 has all the necessary features available and also uses the same UI as 9.10 so I expect it will work the same as described.

  15. I have followed everything you have and I’m having issues with the network interfaces. When in pfsense I get vtnet0 and vtnet1. Neither of them are working, as if nothing is passing through. In freenas, ifconfig tap0, tap1 status says “no courier” which i’m assuming should say “active”. ifconfig_bridge0 value is “addm tap0 addm em2 up”. I can verify em2 is the interface I want to use and in freenas em2 is showing “active” and has “up” in options.

    1. Hi Travis,
      If your tap devices say “no carrier” then the bhyve process is not attaching to them, you should check your vmrun.sh parametres to check these are correct, you should have option “-t tap0 -t tap1” somewhere in the command line.

      The physical interfaces you put into your bridges depend on what hardware is in your server, if em2 is the network card attached to your LAN them yes this is the one you want in bridge0. Conversely the interface attached to the WAN should be in bridge1.

      1. I have -tap0 and tap1 in the command line. I am getting vtnet0 and vtnet1 in pfsense. My mobo has two onboard nic’s, em0 and em1. I put in another 2 port Intel nic giving me em2 and em3. Freenas is using em0 which I have set to static. em2 is showing active but tap0 is not. i have, ifconfing_bridge0 addm em2 addm tap0 up

        I’m just testing it at the moment, so I have em2 plugged into my current network (same network as em0). I only have “up” in options and dhcp off in freenas. Could it still be a problem that it’s plugged into the same network? I was trying it this way to see if I could get a dhcp address assigned to me in pfsense.

        Do I need to just plug em2 into the modem right away? I was hoping it would work this way so I wouldn’t have to disconnect my interrupt my current internet connection.

        1. So long as em0 and em2 are not also in the same bridge then you should not have any probles with it being on the same LAN, however if you have jails then you may get a conflict with using bridge0 which FreeNAS will attempt to use for connecting jail epair interfaces to its primary NIC which is em0 in your case, so you might be best changing this to use bridge1 and bridge2 instead. If you don’t have a process ID showing against the tap0 interface however this still won’t work and resolving this is your primary issue, are there any additional tap interfaces listed in ‘ifconfig’ such as tap3/4/5?

          1. I do have other jails so I will skip bridge0. I do have tap0/1/2/3/4/5 interfaces. Not sure why? Also, in freenas/network/interfaces I have configured em0/1/2/3. I can also configure tap1/2/3/4/5 but they aren’t. Do the tap interfaces also have to be configured in there?

            Thanks for your help.

          2. it may be that those tap interfaces were already present on boot due to earlier testing and it was unable to use it so it can create new tap interfaces but because the new interface is not in the bridge then it won’t get attached to the networks. This isn’t usually an issue once everything is configure and pfSense launches at boot but right now you should shutdown pfSense and delete those unused tap interfaces before restarting pfSense then it should be able to bind to tap0 and tap1 ok.

  16. I finally just deleted all extra interfaces and started over from scratch. Of course it’s working now!!!

    Thanks so much for your help!
    Travis

  17. Thanks for the writeup! I managed to get this running in FreeNAS 11, however pfSense doesn’t seem to be starting on boot. Running the same command manually that I have in the post init script seems to work to get it started though. Any thoughts?

    1. Is your init script set to “post init”? I’m not sure why it wouldn’t while the script itself works manually triggered.

      1. Hi David –

        Thanks for the reply. Here’s a screenshot from freenas: https://www.dropbox.com/s/lklxy0neresl8rp/Screen%20Shot%202017-10-06%20at%2010.58.28%20AM.png?dl=0

        I use tap2,3 and bridge2,3 because of other things running on my freenas box.

        So immediately after a reboot, I ssh’ed into my freenas box to check if the VM had started, and it wasn’t running, so I ran the command manually. Terminal output is copied below. As you can see, the first time I got an error “bhyveload: Could not open ‘/dev/nmdm0A'”, but running it a couple seconds later, it worked fine… thoughts?

        root@freenas:~ # sh /usr/share/examples/bhyve/vmrun.sh -c 2 -C /dev/nmdm0A -d /dev/zvol/ssd/pfSense -t tap3 -t tap2 pfsense &
        [1] 12875
        root@freenas:~ # Launching virtual machine “pfsense” …
        bhyveload: Could not open ‘/dev/nmdm0A’

        [1] Exit 64 sh /usr/share/examples/bhyve/vmrun.sh -c 2 -C /dev/nmdm0A -d /dev/zvol/ssd/pfSense -t tap3 -t tap2 pfsense
        root@freenas:~ # sh /usr/share/examples/bhyve/vmrun.sh -c 2 -C /dev/nmdm0A -d /dev/zvol/ssd/pfSense -t tap3 -t tap2 pfsense &
        [1] 12883
        root@freenas:~ # Launching virtual machine “pfsense” …

        1. It’s hard to tell if that error is a cause or symptom of the problem. You should look in /var/log/messages and see if there are any clues there.

Leave a Reply

Your email address will not be published. Required fields are marked *