PCIem

What is PCIem?

PCIem is a framework that creates virtual PCIe devices in the Linux kernel by leveraging a few novel techniques to populate synthetic cards as legitimate PCI devices to the host OS.

To brief what PCIem is: a framework for (Albeit not limited to) developing and testing PCIe device drivers without requiring actual hardware on the host.

Comparison with libfvio-user

PCIem and libfvio-user are two different solutions for different needs, there may be confusion when comparing both so herein the differences (See figure 1 for more).

The main one is that libvfio-user usually relies on a client (That implements the vfio-user protocol), usually QEMU (Through KVM, using VM exits) to expose the emulated PCIe device to the guest. You write your vfio server (With a callback mechanism usually) which then interacts with the client.

What PCIem does instead is, expose the device directly on the host; no KVM, no guests, no virtual machines, nothing. The device appears on the host's PCIe bus as if it was physically connected.

Architecture

graph LR
    subgraph Kernel ["Host Linux Kernel"]
        direction TB

        RealDriver["Real PCIe Driver"]

        subgraph Framework ["PCIem Framework"]
            direction TB
            Config["PCI Config Space"]
            BARs["BARs"]
            IRQ["Interrupts"]
            DMA["DMA / IOMMU"]
        end

    end

    Interface(("/dev/pciem"))

    subgraph User ["Linux Userspace"]
        direction TB
        Shim["Device Emulation"]
    end

    Framework <==> Interface
    Interface <==> Shim

Current Features

• BAR creation: Programmatically create and monitor BAR regions.
• BAR MMIO tracing: MMIO read/write detection and notification to avoid polling for accesses.
• Interrupts support: Legacy/MSI/MSI-X interrupt injection.
• PCI capability framework: PCI capabilities system (Linked-list underneath).
• DMA system: IOMMU-aware DMA accesses from/to userspace device.
  • P2P support (Preliminar!): Peer-to-peer DMA between devices with whitelist-based access control.
• Userspace: Implement your PCIe devices entirely in userspace.

Minimum supported environment

• Linux Kernel Version: 6.6
• C Compiler: gcc-12
• Architecture: amd64/i386, aarch64, riscv

aarch64 note: Tested w/Raspberry Pi 4b, device-tree (No ACPI)

riscv note: Tested w/VisionFive 2 (Device-tree) and Muse Pi Pro (Device-tree and ACPI)

CI Status

Examples

Bochs BGA (drm) Card

A Bochs BGA-compatible (bochs-drm) card that can be driven by userspace utilities such as Weston; uses SDL3.

https://github.com/user-attachments/assets/d2f7abe4-aef4-4d3a-a30b-ab7ec793faed

NVME Controller

NVME controller with 1GB of storage attached to. User can freely format, mount, create and remove files from the memory.

https://github.com/user-attachments/assets/29ce370a-7317-4fd4-9d1d-c3e868e1083d

Intel HDA Card

ICH6-compatible emulation model that can play samples in conjunction with pipewire.

NOTE: Slight audio crackling only heard on recording, works fine otherwise

https://github.com/user-attachments/assets/7264a18c-6fcb-46c4-acde-9531b0be7ff6

Intel E1000 Card

A pretty naive implementation of the E1000 card. Supports communications redirection to/from your real NIC. Can also use Wi-Fi but it's less stable.

https://github.com/user-attachments/assets/b6b9862c-349c-49da-8a79-4769dfa31915

License

• PCIem kernel components: GPLv2
• Examples: dual MIT/GPLv2.

References

• Blog post: https://cakehonolulu.github.io/introducing-pciem/
• Hackernews post: https://news.ycombinator.com/item?id=46689065
• PCI Express specification: https://pcisig.com/specifications