Welcome to p2pmem-test, a utility for testing PCI Peer-2-Peer (P2P) communication between p2pmem and NVMe devices. This utility becomes very interesting as NVMe devices become p2pmem devices via the Controller Memory Buffer (CMB) and Persistent Memory Region (PMR) features.
Should be as simple as:
git submodule update --init
make
sudo make install
p2pmem-test is an active project. We will happily consider Pull Requests (PRs) submitted against the official repo.
-
In order to run any P2P traffic, you'll need to have a (Linux based) OS that supports p2pdma. This framework is available in all Linux kernels 4.20 or newer, however it is NOT compiled in by default. You will almost certainly have to compile a kernel from source and install that. The instructions on how to do this are beyond the scope of this document but there is a tool that can help in my kernel-tools repo. UPDATE: Mainline userspace support was added in v6.2. This allows userspace applications (like p2pmem-test) to directly map p2pmem via a sysfs file. While this implies no changes to p2pmem-test it does change the invocation (see later).
-
If you are not using a NVMe SSD as your p2pmem device or you are working on a pre-v6,2 kernel You probably want to install the separate p2pmem-pci modul installed to expose the device to userspace. Once installed, you should see the device exposed as a /dev/p2pmemX. Note howeever this step is no longer mandatory as this driver is included in the kernel package files generated by kernel-tools. UPDATE: This approach is no longer needed for NVMe CMB-based p2pmem regions as they are now exposed in mainline via a sysfs entry.
-
Make sure your system has at least one p2pdma capable device. Examples include an Eideticom IOMEM device, a Microsemi NVRAM card or a CMB enabled NVMe SSD that supports the WDS and RDS features (e.g. The Everspin NVNitro card or the Eideticom NoLoadTM device). Basically this is any PCI EP capable of exposing a BAR with a driver that ties into the p2pdma framework (so things like FPGAs and GPUs can be made to work too).
-
In addition to the p2pmem capable device you need at least one other NVMe SSD. This does not have to be CMB enabled (but it is OK if it does support CMB), any standard NVMe SSD will do. Preferably you will have more than one NVMe SSD but its not a requirement. Ideally place the two (or more) devices noted in the two previous points behind a PCIe switch (for example the Microsemi Switchtec or a PLX switch). If you do not have a switch you can connect both devices to the Root Complex (RC) on the CPU but two things may happen:
- Performance may drop. Many RCs are inefficient at routing P2P traffic.
- It might not work at all. Many RCs block P2P traffic. The Linux kernel now maintains a white-list of working root ports.
-
You'll almost certainly want to disable the IOMMU in your system via either the BIOS or the kernel. There are many cases where you can enable the IOMMU but things certainly get a bit trickier so I recommend you start with IOMMU off and then go from there.
-
You may also need to disable the PCIe Access Control Services (ACS) by either the BIOS and/or the kernel because TLP redirection (activated as part of ACS) kills P2P traffic.
-
Finally, p2pmem-test requires the libargconfig submodule. You'll need to either clone recursively or via
git submodule update --init.
When one of the NVMe devices and the p2pmem device are the same PCI EP then CMB should be automatically used. This means the NVMe device should detect the data is in its CMB and do an internal data movement rather than an external DMA. If the p2pmem device is neither of the NVMe devices then two external DMAs will occur. We refer to this second option as a bounce buffer for obvious reasons.
./p2pmem-test /dev/nvme0n1 /dev/nvme1n1 /dev/p2pmem0 -c 1 -s 4k --check
./p2pmem-test /dev/nvme0n1 /dev/nvme1n1 /sys/bus/pci/devices/0000\:03\:00.0/p2pmem/allocate -c 1 -s 4k --check
(Note the PCIe device number may be different on your system).
Copy one 4KB chunk from /dev/nvme0n1 to /dev/nvme0n1 via the memory exposed by /dev/p2pmem0. Perform a check on the data (i.e. write know data to /dev/nvme0n1 and validate that by reading /dev/nvme1n1 after the p2pmem based transfer).
p2pmem-test -h