from Hacker News

Why libvirt supports only 14 PCIe hotplugged devices on x86-64

by andreyvit on 8/6/23, 4:49 PM with 68 comments

• by josephcsible on 8/6/23, 6:01 PM

  > It already supports a number of obscure options (you can make QEMU claim to support a CPU feature regardless of whether the host CPU supports it, really?), so adding one more woild fit in just fine.

  > Nope. “there are no plans to address it further or fix it in an upcoming release”.

  <https://bugzilla.redhat.com/show_bug.cgi?id=1408810>

  I could see that being the response of an individual open-source developer working for free. But that was IBM saying that, and people pay big bucks to IBM to fix things like this.

• by formerly_proven on 8/6/23, 6:15 PM

  > So if you wish to have more than 14 PCIe slots in your VM, you’ll have to use QEMU directly.

  No need, libvirt can pass arbitrary options to QEMU.

  https://libvirt.org/kbase/qemu-passthrough-security.html

• by tedunangst on 8/6/23, 6:33 PM

  I'm curious to know more about the VM host machine that they plugged 15 e1000 cards into to test this limitation. And even more curious about the non-test environment in which somebody ran into this limitation.

  I can only imagine trying to passthrough 20 nvme devices to a guest, but it seems like a very weird configuration.

• by EmilioPeJu on 8/6/23, 7:01 PM

  If I'm not wrong, the pre-allocation of I/O ranges in PCIe bridges is needed only if you intend to hot-plug devices that were not present in the first enumeration.. but in VMs the hardware is known from the start and the PCIe enumeration can assign I/O ranges only if devices underneath actually needs them... is there a reason why hot-plugging is needed in VMs?

• by magicalhippo on 8/6/23, 8:37 PM

  I ran into this on FreeNAS which uses Bhyve. Not sure if it's FreeNAS' way of doing things, but adding a virtual disk using VirtIO creates a separate SATA controller.

  I tried forwarding quad NVMe's and couldn't get it working until I discovered I was hitting this limitation between the existing disks and VirtIO network card.

• by mixmastamyk on 8/6/23, 6:42 PM

  Would like to hear more about why i/o ports stayed fixed and "usage decreased over time." USB/TB devices must not use them, right?

• by joelhaasnoot on 8/7/23, 10:08 AM

  But this doesn't answer the question why 14 and not 16. There's a diff of two there...