-
Notifications
You must be signed in to change notification settings - Fork 71
PFqLoad
Nicola Bonelli edited this page Jun 8, 2017
·
15 revisions
pfq-load
is a Haskell script designed to load PFQ kernel module and related pfq-omatized device drivers. The script allows:
- to choose the algorithm used to distribute IRQ (available algorithms are: naive, round-robin, even, odd, all-in:core, comb:core).
- to specify the first core used by IRQ affinity.
- to exclude some cores from IRQ affinity.
- to specify the number of hardware queues in use.
- to set ethtool parameters for each device.
-> pfq-load --help
pfq-load (PFQ 6.0.0)
pfq-load [OPTIONS] [ITEM]
Common flags:
-c --config=FILE Specify config file (default ~/.pfq.conf)
-k --kmodule=ITEM Override the kmodule specified in config file
-a --algorithm=ITEM Irq affinity algorithm: naive, round-robin, even,
odd, all-in:id, comb:id
-g --governor=ITEM Set cpufreq governor
--first-core=NUM First core used for irq affinity
-e --exclude=CORE Exclude core from irq affinity
-q --queues=INT Specify hardware channels (i.e. Intel RSS)
--force Force PFQ reload!
-? --help Display help message
-V --version Print version information
--numeric-version Print just the version number
pfq-load
reads a configuration file, usually stored in ~/pfq.conf.
Such a file specifies the paths where binaries are stored, as well as, the options required to load the kernel modules and setup the optimal configuration.
The following example shows a typical configuration for the Intel ixgbe device driver.
#
# PFQ configuration file
#
pfq_module : "/opt/PFQ/kernel/pfq.ko"
pfq_options : [ "capt_batch_len=127"
, "xmit_batch_len=127"
, "skb_tx_pool_size=1024"
, "skb_rx_pool_size=1024"
, "tx_retry=256"
]
exclude_core: []
irq_affinity: ["multiple/1" ]
cpu_governor: "performance"
drivers:
- drvmod: "/opt/ixgbe-5.0.4/src/ixgbe.ko"
drvopt: [
"LRO=0,0,0,0"
, "MDD=0,0,0,0"
, "FCoE=0,0,0,0"
, "DCA=1,1,1,1"
, "AtrSampleRate=0,0,0,0"
, "InterruptThrottleRate=1,1,1,1"
, "VEPA=0,0,0,0"
]
pci : [
["8086:10fb", "e6.b=2e"]
, ["8086:151c", "e6.b=2e"]
]
instances: 2
devices:
- devname : "enp2s0f0"
devspeed : 10000
channels : null
flowctrl : no
ethopt : [
["-G", "rx 256"]
, ["-G", "tx 768"]
, ["-C", "tx-frames-irq 512"]
, ["-C", "rx-usecs 30"]
, ["-K", "rx off"]
, ["-X", "hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da" ]
]
- devname : "enp2s0f1"
devspeed : 10000
channels : null
flowctrl : no
ethopt : [
["-G", "rx 256"]
, ["-G", "tx 768"]
, ["-C", "tx-frames-irq 512"]
, ["-C", "rx-usecs 30"]
, ["-K", "rx off"]
, ["-X", "hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da" ]
]
- devname : "ens1f0"
devspeed : 10000
channels : 1
flowctrl : no
ethopt : [
["-G", "rx 256"]
, ["-G", "tx 768"]
, ["-C", "tx-frames-irq 512"]
, ["-C", "rx-usecs 30"]
, ["-K", "rx off"]
, ["-X", "hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da" ]
]
- devname : "ens1f1"
devspeed : 10000
channels : 1
flowctrl : no
ethopt : [
["-G", "rx 256"]
, ["-G", "tx 768"]
, ["-C", "tx-frames-irq 512"]
, ["-C", "rx-usecs 30"]
, ["-K", "rx off"]
, ["-X", "hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da" ]
]
Load PFQ and configure the system to use 3 hardware queues:
-> pfq-load -q2 --force
Using /root/.pfq.conf.yaml config...
Unloading ixgbe...
-> /sbin/rmmod ixgbe
Unloading pfq...
-> /sbin/rmmod pfq
-> /usr/bin/cpufreq-set -g performance -c 0
-> /usr/bin/cpufreq-set -g performance -c 1
-> /usr/bin/cpufreq-set -g performance -c 2
-> /usr/bin/cpufreq-set -g performance -c 3
-> /usr/bin/cpufreq-set -g performance -c 4
-> /usr/bin/cpufreq-set -g performance -c 5
-> /usr/bin/cpufreq-set -g performance -c 6
-> /usr/bin/cpufreq-set -g performance -c 7
Loading pfq [/sbin/insmod /opt/PFQ/kernel/pfq.ko capt_batch_len=127 xmit_batch_len=127 skb_tx_pool_size=1024 skb_rx_pool_size=1024 tx_retry=256]
-> /sbin/insmod /opt/PFQ/kernel/pfq.ko capt_batch_len=127 xmit_batch_len=127 skb_tx_pool_size=1024 skb_rx_pool_size=1024 tx_retry=256
Unloading vanilla/standard drivers...
Loading /opt/ixgbe-5.0.4/src/ixgbe.ko ["LRO=0,0,0,0","MDD=0,0,0,0","FCoE=0,0,0,0","DCA=1,1,1,1","AtrSampleRate=0,0,0,0","InterruptThrottleRate=1,1,1,1","VEPA=0,0,0,0","RSS=2,2"]
-> /sbin/insmod /opt/ixgbe-5.0.4/src/ixgbe.ko LRO=0,0,0,0 MDD=0,0,0,0 FCoE=0,0,0,0 DCA=1,1,1,1 AtrSampleRate=0,0,0,0 InterruptThrottleRate=1,1,1,1 VEPA=0,0,0,0 RSS=2,2
Activating enp2s0f0...
-> /sbin/ifconfig enp2s0f0 up
Setting speed (10000) for enp2s0f0...
-> /sbin/ethtool -s enp2s0f0 speed 10000 duplex full autoneg off
Cannot set new settings: Invalid argument
not setting speed
not setting duplex
not setting autoneg
Enabling flow control for enp2s0f0...
-> /sbin/ethtool -A enp2s0f0 autoneg off rx off tx off
autoneg unmodified, ignoring
Setting channels to 2...
-> /sbin/ethtool -L enp2s0f0 combined 2
combined unmodified, ignoring
no channel parameters changed, aborting
current values: tx 0 rx 0 other 1 combined 2
-> /sbin/ethtool -G enp2s0f0 rx 256
-> /sbin/ethtool -G enp2s0f0 tx 768
-> /sbin/ethtool -C enp2s0f0 tx-frames-irq 512
-> /sbin/ethtool -C enp2s0f0 rx-usecs 30
-> /sbin/ethtool -K enp2s0f0 rx off
-> /sbin/ethtool -X enp2s0f0 hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da
Activating enp2s0f1...
-> /sbin/ifconfig enp2s0f1 up
Setting speed (10000) for enp2s0f1...
-> /sbin/ethtool -s enp2s0f1 speed 10000 duplex full autoneg off
Cannot set new settings: Invalid argument
not setting speed
not setting duplex
not setting autoneg
Enabling flow control for enp2s0f1...
-> /sbin/ethtool -A enp2s0f1 autoneg off rx off tx off
Setting channels to 2...
-> /sbin/ethtool -L enp2s0f1 combined 2
combined unmodified, ignoring
no channel parameters changed, aborting
current values: tx 0 rx 0 other 1 combined 2
-> /sbin/ethtool -G enp2s0f1 rx 256
-> /sbin/ethtool -G enp2s0f1 tx 768
-> /sbin/ethtool -C enp2s0f1 tx-frames-irq 512
-> /sbin/ethtool -C enp2s0f1 rx-usecs 30
-> /sbin/ethtool -K enp2s0f1 rx off
-> /sbin/ethtool -X enp2s0f1 hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da
Activating ens1f0...
-> /sbin/ifconfig ens1f0 up
Setting speed (10000) for ens1f0...
-> /sbin/ethtool -s ens1f0 speed 10000 duplex full autoneg off
Enabling flow control for ens1f0...
-> /sbin/ethtool -A ens1f0 autoneg off rx off tx off
Setting channels to 1...
-> /sbin/ethtool -L ens1f0 combined 1
-> /sbin/ethtool -G ens1f0 rx 256
-> /sbin/ethtool -G ens1f0 tx 768
-> /sbin/ethtool -C ens1f0 tx-frames-irq 512
-> /sbin/ethtool -C ens1f0 rx-usecs 30
-> /sbin/ethtool -K ens1f0 rx off
-> /sbin/ethtool -X ens1f0 hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da
Activating ens1f1...
-> /sbin/ifconfig ens1f1 up
Setting speed (10000) for ens1f1...
-> /sbin/ethtool -s ens1f1 speed 10000 duplex full autoneg off
Enabling flow control for ens1f1...
-> /sbin/ethtool -A ens1f1 autoneg off rx off tx off
Setting channels to 1...
-> /sbin/ethtool -L ens1f1 combined 1
-> /sbin/ethtool -G ens1f1 rx 256
-> /sbin/ethtool -G ens1f1 tx 768
-> /sbin/ethtool -C ens1f1 tx-frames-irq 512
-> /sbin/ethtool -C ens1f1 rx-usecs 30
-> /sbin/ethtool -K ens1f1 rx off
-> /sbin/ethtool -X ens1f1 hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da
-> /usr/bin/setpci -v -d "8086:10fb" "e6.b=2e"
0000:02:00.0 @e6 2e
0000:02:00.1 @e6 2e
-> /usr/bin/setpci -v -d "8086:151c" "e6.b=2e"
0000:04:00.1 @e6 2e
0000:04:00.0 @e6 2e
Setting irq affinity...
-> pfq-affinity -f 0 -a multiple/1 -m TxRx enp2s0f0 enp2s0f1 ens1f0 ens1f1
Setting binding for device enp2s0f0 (Just TxRx):
irq 48 -> core 0 {mask = 1}
irq 49 -> core 1 {mask = 2}
Setting binding for device enp2s0f1 (Just TxRx):
irq 52 -> core 2 {mask = 4}
irq 53 -> core 3 {mask = 8}
Setting binding for device ens1f0 (Just TxRx):
irq 56 -> core 4 {mask = 10}
Setting binding for device ens1f1 (Just TxRx):
irq 58 -> core 5 {mask = 20}
PFQ ready.