iohyve / bhyve USB controller passthrough goodness

I used to have a Windows machine that was dedicated to running software for our printer since the open-source software wouldn't scan and print correctly. Thanks to iohyve and a $20 PCI USB card, I was able to eliminate this machine and replace it with a slimmed down virtual machine.

Here are some tips and steps to get it working:

• Only PCI devices that support MSI (message signaled interrupts) will work. You can figure all of this out with the command:
  
  pciconf -lvc
  
  In my case, I get:
  
  xhci1@pci0:3:0:0:        class=0x0c0330 card=0x12421b21 chip=0x12421b21 rev=0x00 hdr=0x00
      vendor     = 'ASMedia Technology Inc.'
      device     = 'ASM1142 USB 3.1 Host Controller'
      class      = serial bus
      subclass   = USB
      cap 05[50] = MSI supports 8 messages, 64 bit
      cap 11[68] = MSI-X supports 8 messages
                   Table in map 0x10[0x2000], PBA in map 0x10[0x2080]
      cap 01[78] = powerspec 3  supports D0 D3  current D0
      cap 10[80] = PCI-Express 2 endpoint max data 128(512) NS
                   link x2(x2) speed 5.0(5.0) ASPM disabled(L0s/L1)
      ecap 0002[100] = VC 1 max VC0
      ecap 0001[200] = AER 1 0 fatal 0 non-fatal 0 corrected
      ecap 0019[280] = PCIe Sec 1 lane errors 0
      ecap 0018[300] = LTR 1
• bhyve expects that PCI devices that will be passed through must be associated with the ppt driver. Rebooting your machine is not required. In the above example, the device is associated with the xhci driver. To associate it with the ppt driver:
  
  devctl detach pci3:0:0
  devctl set driver pci3:0:0 ppt
  If you execute pciconf -lvc again, you'll now notice:
  
  ppt1@pci0:3:0:0:        class=0x0c0330 card=0x12421b21 chip=0x12421b21 rev=0x00 hdr=0x00
• In modern bhyve, you must supply the -S argument to bhyve in order to get PCI passthrough working. Note that -u is provided to make Windows like my UTC clock:
  
  iohyve set vm_name_here bargs = "-H -w -S -u"
• Create the mapping for the PCI device in question:
  
  iohyve set vm_name_here pcidev:1=passthru,3/0/0
• Limit or completely disable network access to your virtual server!

Installing Windows 10 in TrueOS (FreeBSD) in a bhyve container using iohyve

Here's an abbreviated HOWTO install Windows 10 using iohyve on TrueOS.
iohyve is a handy script that helps manage virtual machines in bhyve. More information is available here.

1. Download installation ISOs
   You can use the Windows Media Creation Tool to download .iso files for Windows 10.
   Download the Red Hat VirtualIO network adapter ISO.
2. Install the prerequisite packages for iohyve and bhyve
   pkg install tigervnc bhyve-firmware uefi-edk2-bhyve iohyve
3. Initialize iohyve if you have never used iohyve before
   iohyve setup pool=zfs_poolname_here net=network_nic_here kmod=1
4. Initialize components
   iohyve cpfw /usr/local/share/uefi-firmware/BHYVE_UEFI.fd
   iohyve cpiso /path/to/Windows_10_installer.iso
   iohyve cpiso /path/to/virtio-win.iso
5. Create and configure device (example uses 32 gigabytes for C:)
   iohyve create win10 32g
   iohyve set win10 loader=uefi ram=2G cpu=2 vnc=YES vnc_tablet=YES vnc_port=6901 vnc_wait=YES bargs="-H -w" fw=BHYVE_UEFI.fd
6. Install Windows 10
   iohyve install win10 Windows_10_installer.iso
   vncviewer localhost:6901
   ... stuff happens.
7. Once install is completed and the VM shuts down, install the virtio network driver
   iohyve install win10 virtio-win.iso
   
   Navigate the explorer to the "d:" drive, open the NetKVM directory for your architecture, find the .inf file and right-click it to "Install." Congratulations, your VM can now download teh viruz!
8. Do cool things to your Windows using zfs commands.

Some caveats about bhyve:

1. You can't run bhyve and VirtualBox at the same time.
2. If you try to run bhyve, and then want to run VirtualBox, it seems that it is not possible to kldunload the byhve module vmm that occupies the hypervisor (it is possible to unload nmdm).