Napatech SmartNIC family support 1/10/25/40/50/100 Gb/s Ethernet speeds on various cards. The NT200A01 card can potentially support all speeds within the one card. The Napatech SmartNICs operate using a bifurcated driver, the main driver resides in userspace and DPDK run on top of that driver. The Napatech SmartNIC driver suite provide it’s own kernel driver and does therefore not need igb_uio. In order to run TRex on Napatech SmartNICs the Napatech driver suite must be install and running.
-
Download latest driver suite from https://www.napatech.com/downloads/
As of writing this is the following release: "Huntington Beach 4.1½" - ntanl_package_3gd-11.0.2-linux.tar.gz
[bash]>tar xf ntanl_package_3gd-11.0.2-linux.tar.gz
[bash]>[bash]>cd ntanl_package_3gd_linux_11.0.2
[bash]>sh ./package_install_3gd-11.0.2.sh --noprompt
[ Install documentation ]
[ Extract and install tools ]
Verifying archive integrity... All good.
Uncompressing Napatech A/S tools package v 3.8.2.16-4e54d....................................................................................................................................................................................................................................
######################################################################
######################################################################
######################################################################
###### _ _ _____ _______ ______ _____ _ _ ######
###### | \ | | /\ | __ \ /\|__ __| ____/ ____| | | | ######
###### | \| | / \ | |__) / \ | | | |__ | | | |__| | ######
###### | | / /\ \ | ___/ /\ \ | | | __|| | | __ | ######
###### | |\ |/ ____ \| | / ____ \| | | |___| |____| | | | ######
###### |_| \_/_/ \_\_| /_/ \_\_| |______\_____|_| |_| ######
###### ######
###### N T T O O L S ######
###### ( T o o l s P a c k a g e ) ######
######################################################################
######################################################################
######################################################################
Commencing installation procedure..
Creating installation directory at /opt/napatech3 [Done]
Installing tools [Done]
Installing documentation [Done]
######################################################################
# Installation completed successfully #
# See /tmp/nt_tools_3gd.log for a full install log #
######################################################################
[ Extract and build driver ]
Verifying archive integrity... All good.
Uncompressing Napatech A/S driver package v 3.8.2.16-4e54d.............................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
..............................................................................................................................................................................................................................
######################################################################
######################################################################
######################################################################
###### _ _ _____ _______ ______ _____ _ _ ######
###### | \ | | /\ | __ \ /\|__ __| ____/ ____| | | | ######
###### | \| | / \ | |__) / \ | | | |__ | | | |__| | ######
###### | | / /\ \ | ___/ /\ \ | | | __|| | | __ | ######
###### | |\ |/ ____ \| | / ____ \| | | |___| |____| | | | ######
###### |_| \_/_/ \_\_| /_/ \_\_| |______\_____|_| |_| ######
###### ######
###### N T D R I V E R S ######
###### ( D r i v e r P a c k a g e ) ######
######################################################################
######################################################################
######################################################################
Commencing installation procedure..
Creating installation directory at /opt/napatech3 [Done]
Checking build and install disk space [Done]
Installing driver source [Done]
Compiling the driver [Done]
Installing driver [Done]
Compiling the netdev driver [Done]
Installing netdev driver [Done]
Installing libraries [Done]
Installing header files [Done]
Installing binaries [Done]
Installing config files [Done]
Installing examples [Done]
Installing documentation [Done]
Checking dynamic linker bindings:
Regenerate dynamic linker bindings [Done]
Installing symbolic links [Done]
######################################################################
# Installation completed successfully #
# See /tmp/nt_driver_3gd.log for a full install log #
######################################################################
[ Extract and install the Image control applications ]
[ Install Images ]
[ Package installation done... ]
[bash]>Please also see https://docs.napatech.com/reader/DJq6hS577FbL1RHaPnWnaQ/h19IOLPuF4gKSxOYrtod2w for detailed instructions
[bash]>cd /opt/napatech3/images/
[bash]>/opt/napatech3/bin/imgctrl -p --ini trex --ntimg NT200A01-2x100/200-8002-10-02-0000-180227-1015.ntimg
imgctrl v3.6.2.1-b49d9 (Mar 12 2018 06:21:51) (Logfile: /tmp/imgctrl.log)
Scanning for Napatech adapters..
Adapter dual upgrade commencing..
Writing FPGA bank 0
Erasing................... 100% done
Writing................... 100% done
Verifying................. 100% done
Done writing FPGA bank 0
Note: You can inspect the upload result through the Bank0 Contents
section using the --query mode. To try the new image, remember
to switch bank using the --switch mode. See --help for more information.
Writing FPGA bank 1
Erasing................... 100% done
Writing................... 100% done
Verifying................. 100% done
Done writing FPGA bank 1
Note: You can inspect the upload result through the Bank1 Contents
section using the --query mode. To try the new image, remember
to switch bank using the --switch mode. See --help for more information.
FLASH UPGRADE completed
BANK0 : 200-9516-10-07-00-180226-0843.ntimg
BANK1 : 200-9515-10-07-00-180221-1628.ntimg
PRIMARY BANK : 0
Upgrade process completeStarting the Napatech NTService daemon will create a default configuration file if no configuration file is present.
[bash]>/opt/napatech3/bin/ntstart.sh
Loading nt3gd driver [Done]
Creating driver device file [Done]
Loading nt3gd_netdev driver [Done]
Creating driver device file [Done]
Starting NTService (this may take a while) [Done]
[bash]>/opt/napatech3/bin/ntstop.sh
Stopping NTService (this may take a while)
NTService stopped [Done]
[bash]>The /opt/napatech3/config/ntservice.ini contains the following settings that control the number and size of hostbuffers. To enable TRex to simultaneous operate with may CPU cores/threads towards the SmartNIC, we need to increase the number of hostbuffers:
...
HostBuffersRx = [4,16,-1] # [x1, x2, x3], ...
HostBuffersTx = [4,16,-1] # [x1, x2, x3], ...
...
----
They need to be changed to:
----
...
HostBuffersRx = [32,16,-1] # [x1, x2, x3], ...
HostBuffersTx = [32,16,-1] # [x1, x2, x3], ...
... [bash]>sed -i -- 's/x = \[4/x = \[32/g' /opt/napatech3/config/ntservice.ini
[bash> [bash]>/opt/napatech3/bin/ntstart.sh
Loading nt3gd driver [Done]
Creating driver device file [Done]
Loading nt3gd_netdev driver [Done]
Creating driver device file [Done]
Starting NTService (this may take a while) [Done]
[bash]>Make sure to use a version of TRex downloaded from https://github.com/cisco-system-traffic-generator/trex-core (https://github.com/cisco-system-traffic-generator/trex-core/releases). TRex versions downloaded from here are in source form and contains a built-in DPDK
[bash]>cd linux_dpdk
[bash]>./b configure --with-ntacc
Setting top to : /usr/builds/trex-core
Setting out to : /usr/builds/trex-core/linux_dpdk/build_dpdk
Checking for program 'g++, c++' : /usr/bin/g++
Checking for program 'ar' : /usr/bin/ar
Checking for program 'gcc, cc' : /usr/bin/gcc
Checking for program 'ar' : /usr/bin/ar
Checking for program 'ldd' : /usr/bin/ldd
Checking for library z : yes
Checking for OFED : Found needed version 4.0
Checking for library ibverbs : yes
Checking for NTAPI : Found needed NTAPI library
'configure' finished successfully (14.827s)
[bash]>./b
Waf: Entering directory `/usr/builds/trex-core.github.rsync/linux_dpdk/build_dpdk'
Info: Using external libverbs.
update version files
...
[1206/1210] Compiling version.c
[1207/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-debug-o
[1208/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-debug
[1209/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-o
[1210/1210] Linking build_dpdk/linux_dpdk/_t-rex-64
Waf: Leaving directory `/usr/builds/trex-core.github.rsync/linux_dpdk/build_dpdk'
*** generating softlinks ***
'build' finished successfully (15.621s) [bash]>/opt/napatech3/bin/adapterinfo
adapterinfo (v. 3.8.2.16-4e54d)
==============================================================================
System: 3.8.2.16
------------------------------------------------------------------------------
Adapters: 1
Ports: 2
==============================================================================
Adapter: 2 - NT200A01-01-SCC-2x100-E3-FF-ANL Analysis Network Adapter
------------------------------------------------------------------------------
Bus ID: 0000:82:00.0 #(1)
PCI ID: 18F4:01A5
FPGA ID: 200-9516-10-07-00
Feature level: N-ANL11
Profile: Capture
Supported profiles: Capture
AVR version: 3.5b
Ports: 2
Port Range: 0 - 1
Descriptor: Standard
Timestamp: Native Unix
TimestampMethod: EOF (End of Frame)
RX host buffers: 32
TX host buffers: 32
Port #0 MAC : 00.0D.E9.11.22.33
Port #1 MAC : 00.0D.E9.11.22.34
SDRAM size : 12GB
Bus type : PCI Express 3
Link width supported : x16
Link width negotiated : x16
Link speed : 8.0 GT/s
Max payload supported : 512
Max payload negotiated : 256
Max expected throughput : 107209 Mbps
==============================================================================
[bash]>-
Napatech bus-id. Omit the domain 0000: in the config file
To address the individual ports on the Napatech SmartNIC, you need to use a syntax of <pci-device>/<port-no>, see below for examples:
### Config file generated by dpdk_setup_ports.py ###
- port_limit: 2
version: 2
interfaces: ['82:00.0/0', '82:00.0/1']
port_info:
- ip: 1.1.1.1
default_gw: 2.2.2.2
- ip: 2.2.2.2
default_gw: 1.1.1.1
platform:
master_thread_id: 0
latency_thread_id: 1
dual_if:
- socket: 1
threads: [8,9,10,11,12,13,14]In lieu of the ip and default_gw settings, you can alternatively specify a MAC-based configuration. In such case the port_info settings could look as follows:
port_info:
- dest_mac: 00:0d:e9:11:22:34 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
src_mac: 00:0d:e9:11:22:33
- dest_mac: 00:0d:e9:11:22:33 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
src_mac: 00:0d:e9:11:22:34You can obtain the MAC addresses from the output of the adapterinfo tool
It is possible to use dpdk_setup_ports.py script to produce the example config file:
[bash]> cd trex-core
[bash]> cd scripts/
[bash]> sudo ./dpdk_setup_ports.py -c 82:00.0/0 82:00.0/1 --force-macs
### Dumping config to screen, use -o flag to save to file
### Config file generated by dpdk_setup_ports.py ###
- port_limit: 2
version: 2
interfaces: ['82:00.0/0', '82:00.0/1']
port_info:
- dest_mac: 00:0d:e9:11:22:34 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
src_mac: 00:0d:e9:11:22:33
- dest_mac: 00:0d:e9:11:22:33 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
src_mac: 00:0d:e9:11:22:34
platform:
master_thread_id: 0
latency_thread_id: 2
dual_if:
- socket: 1
threads: [1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63]
[source,bash]Note that when using a NT100E3 SmartNIC (with a single 100G port) - for optimal performance, typically you would specify a dummy interface as the second port in a pair (see below as well)
Server: |
UCSC-C240-M4SX |
CPU: |
2 x Intel® Xeon® CPU E5-2650 v4 @ 2.20GHz |
RAM: |
64GB @ 2400 MHz |
NICs: |
2 x NT100E3-1-PTP Network Adapter 1x100Gb |
QSFP: |
2 x CFP4 |
OS: |
CentOS 7.3 |
TRex: |
pre-v2.38 using 10 cores per dual interface. |
Loopback with optic cable.
We got two NICs, each one has single interface and belong to different NUMA.
In order to get optimal performance we have used 2 dummy interfaces as pair to real ones.
trex_cfg.yaml:
...
interfaces: ['07:00.0', 'dummy', 'dummy', '8a:00.0']
...Ran bidirectional traffic between the two ports with following trex-console command:
start -p 0 3 -m 99.9% -f stl/imix.py
start -p 0 3 -m 99.9% -f stl/bench.py -t size=1514
start -p 0 3 -m 99.9% -f stl/bench.py -t size=590
start -p 0 3 -m 99.9% -f stl/bench.py -t size=128
start -p 0 3 -m 99.9% -f stl/bench.py -t size=64Results summary:
