Skip to content

PFqLoad

Jump to bottom Edit New page
Nicola Bonelli edited this page Jun 8, 2017 · 15 revisions

pfq-load script

pfq-load is a Haskell script designed to load PFQ kernel module and related pfq-omatized device drivers. The script allows:

  • to choose the algorithm used to distribute IRQ (available algorithms are: naive, round-robin, even, odd, all-in:core, comb:core).
  • to specify the first core used by IRQ affinity.
  • to exclude some cores from IRQ affinity.
  • to specify the number of hardware queues in use.
  • to set ethtool parameters for each device.

Command line options

-> pfq-load --help
pfq-load (PFQ 6.0.0)

pfq-load [OPTIONS] [ITEM]

Common flags:
  -c --config=FILE      Specify config file (default ~/.pfq.conf)
  -k --kmodule=ITEM     Override the kmodule specified in config file
  -a --algorithm=ITEM   Irq affinity algorithm: naive, round-robin, even,
                        odd, all-in:id, comb:id
  -g --governor=ITEM    Set cpufreq governor
     --first-core=NUM   First core used for irq affinity
  -e --exclude=CORE     Exclude core from irq affinity
  -q --queues=INT       Specify hardware channels (i.e. Intel RSS)
     --force            Force PFQ reload!
  -? --help             Display help message
  -V --version          Print version information
     --numeric-version  Print just the version number

Config file

pfq-load reads a configuration file, usually stored in ~/pfq.conf. Such a file specifies the paths where binaries are stored, as well as, the options required to load the kernel modules and setup the optimal configuration.

The following example shows a typical configuration for the Intel ixgbe device driver.

#
# PFQ configuration file
#


pfq_module  :   "/opt/PFQ/kernel/pfq.ko"                                                  

pfq_options :   [ "capt_batch_len=127"
                , "xmit_batch_len=127"
                , "skb_tx_pool_size=1024"
                , "skb_rx_pool_size=1024"
                , "tx_retry=256"
                ]

exclude_core:   []
irq_affinity:   ["multiple/1" ]
cpu_governor:   "performance"


drivers:
  - drvmod:         "/opt/ixgbe-5.0.4/src/ixgbe.ko"
    drvopt:         [
                      "LRO=0,0,0,0"
                    , "MDD=0,0,0,0"
                    , "FCoE=0,0,0,0"
                    , "DCA=1,1,1,1"
                    , "AtrSampleRate=0,0,0,0"
                    , "InterruptThrottleRate=1,1,1,1"
                    , "VEPA=0,0,0,0"
                    ] 
    pci : [ 
            ["8086:10fb", "e6.b=2e"]
          , ["8086:151c", "e6.b=2e"]
          ] 

    instances:      2
    devices: 
        - devname  : "enp2s0f0"
          devspeed : 10000
          channels : null
          flowctrl : no
          ethopt   : [ 
                        ["-G", "rx 256"] 
                      , ["-G", "tx 768"] 
                      , ["-C", "tx-frames-irq 512"]
                      , ["-C", "rx-usecs 30"]
                      , ["-K", "rx off"]
                      , ["-X", "hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da" ]
                     ] 
        - devname  : "enp2s0f1"
          devspeed : 10000
          channels : null
          flowctrl : no
          ethopt   : [ 
                        ["-G", "rx 256"] 
                      , ["-G", "tx 768"] 
                      , ["-C", "tx-frames-irq 512"]
                      , ["-C", "rx-usecs 30"]
                      , ["-K", "rx off"]
                      , ["-X", "hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da" ]
                     ]

        - devname  : "ens1f0"
          devspeed : 10000
          channels : 1
          flowctrl : no
          ethopt   : [ 
                        ["-G", "rx 256"] 
                      , ["-G", "tx 768"] 
                      , ["-C", "tx-frames-irq 512"]
                      , ["-C", "rx-usecs 30"]
                      , ["-K", "rx off"]
                      , ["-X", "hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da" ]
                     ] 
        - devname  : "ens1f1"
          devspeed : 10000
          channels : 1
          flowctrl : no
          ethopt   : [ 
                        ["-G", "rx 256"] 
                      , ["-G", "tx 768"] 
                      , ["-C", "tx-frames-irq 512"]
                      , ["-C", "rx-usecs 30"]
                      , ["-K", "rx off"]
                      , ["-X", "hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da" ]
                     ]

Example

Load PFQ and configure the system to use 3 hardware queues:

-> pfq-load -q2 --force
Using /root/.pfq.conf.yaml config...
Unloading ixgbe...
-> /sbin/rmmod ixgbe
Unloading pfq...
-> /sbin/rmmod pfq
-> /usr/bin/cpufreq-set -g performance -c 0
-> /usr/bin/cpufreq-set -g performance -c 1
-> /usr/bin/cpufreq-set -g performance -c 2
-> /usr/bin/cpufreq-set -g performance -c 3
-> /usr/bin/cpufreq-set -g performance -c 4
-> /usr/bin/cpufreq-set -g performance -c 5
-> /usr/bin/cpufreq-set -g performance -c 6
-> /usr/bin/cpufreq-set -g performance -c 7
Loading pfq [/sbin/insmod /opt/PFQ/kernel/pfq.ko capt_batch_len=127 xmit_batch_len=127 skb_tx_pool_size=1024 skb_rx_pool_size=1024 tx_retry=256]
-> /sbin/insmod /opt/PFQ/kernel/pfq.ko capt_batch_len=127 xmit_batch_len=127 skb_tx_pool_size=1024 skb_rx_pool_size=1024 tx_retry=256
Unloading vanilla/standard drivers...
Loading /opt/ixgbe-5.0.4/src/ixgbe.ko ["LRO=0,0,0,0","MDD=0,0,0,0","FCoE=0,0,0,0","DCA=1,1,1,1","AtrSampleRate=0,0,0,0","InterruptThrottleRate=1,1,1,1","VEPA=0,0,0,0","RSS=2,2"]
-> /sbin/insmod /opt/ixgbe-5.0.4/src/ixgbe.ko LRO=0,0,0,0 MDD=0,0,0,0 FCoE=0,0,0,0 DCA=1,1,1,1 AtrSampleRate=0,0,0,0 InterruptThrottleRate=1,1,1,1 VEPA=0,0,0,0 RSS=2,2
Activating enp2s0f0...
-> /sbin/ifconfig enp2s0f0 up
Setting speed (10000) for enp2s0f0...
-> /sbin/ethtool -s enp2s0f0 speed 10000 duplex full autoneg off
Cannot set new settings: Invalid argument
  not setting speed
  not setting duplex
  not setting autoneg
Enabling flow control for enp2s0f0...
-> /sbin/ethtool -A enp2s0f0 autoneg off rx off tx off
autoneg unmodified, ignoring
Setting channels to 2...
-> /sbin/ethtool -L enp2s0f0 combined 2
combined unmodified, ignoring
no channel parameters changed, aborting
current values: tx 0 rx 0 other 1 combined 2
-> /sbin/ethtool -G enp2s0f0 rx 256
-> /sbin/ethtool -G enp2s0f0 tx 768
-> /sbin/ethtool -C enp2s0f0 tx-frames-irq 512
-> /sbin/ethtool -C enp2s0f0 rx-usecs 30
-> /sbin/ethtool -K enp2s0f0 rx off
-> /sbin/ethtool -X enp2s0f0 hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da
Activating enp2s0f1...
-> /sbin/ifconfig enp2s0f1 up
Setting speed (10000) for enp2s0f1...
-> /sbin/ethtool -s enp2s0f1 speed 10000 duplex full autoneg off
Cannot set new settings: Invalid argument
  not setting speed
  not setting duplex
  not setting autoneg
Enabling flow control for enp2s0f1...
-> /sbin/ethtool -A enp2s0f1 autoneg off rx off tx off
Setting channels to 2...
-> /sbin/ethtool -L enp2s0f1 combined 2
combined unmodified, ignoring
no channel parameters changed, aborting
current values: tx 0 rx 0 other 1 combined 2
-> /sbin/ethtool -G enp2s0f1 rx 256
-> /sbin/ethtool -G enp2s0f1 tx 768
-> /sbin/ethtool -C enp2s0f1 tx-frames-irq 512
-> /sbin/ethtool -C enp2s0f1 rx-usecs 30
-> /sbin/ethtool -K enp2s0f1 rx off
-> /sbin/ethtool -X enp2s0f1 hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da
Activating ens1f0...
-> /sbin/ifconfig ens1f0 up
Setting speed (10000) for ens1f0...
-> /sbin/ethtool -s ens1f0 speed 10000 duplex full autoneg off
Enabling flow control for ens1f0...
-> /sbin/ethtool -A ens1f0 autoneg off rx off tx off
Setting channels to 1...
-> /sbin/ethtool -L ens1f0 combined 1
-> /sbin/ethtool -G ens1f0 rx 256
-> /sbin/ethtool -G ens1f0 tx 768
-> /sbin/ethtool -C ens1f0 tx-frames-irq 512
-> /sbin/ethtool -C ens1f0 rx-usecs 30
-> /sbin/ethtool -K ens1f0 rx off
-> /sbin/ethtool -X ens1f0 hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da
Activating ens1f1...
-> /sbin/ifconfig ens1f1 up
Setting speed (10000) for ens1f1...
-> /sbin/ethtool -s ens1f1 speed 10000 duplex full autoneg off
Enabling flow control for ens1f1...
-> /sbin/ethtool -A ens1f1 autoneg off rx off tx off
Setting channels to 1...
-> /sbin/ethtool -L ens1f1 combined 1
-> /sbin/ethtool -G ens1f1 rx 256
-> /sbin/ethtool -G ens1f1 tx 768
-> /sbin/ethtool -C ens1f1 tx-frames-irq 512
-> /sbin/ethtool -C ens1f1 rx-usecs 30
-> /sbin/ethtool -K ens1f1 rx off
-> /sbin/ethtool -X ens1f1 hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da
-> /usr/bin/setpci -v -d "8086:10fb" "e6.b=2e"
0000:02:00.0 @e6 2e
0000:02:00.1 @e6 2e
-> /usr/bin/setpci -v -d "8086:151c" "e6.b=2e"
0000:04:00.1 @e6 2e
0000:04:00.0 @e6 2e
Setting irq affinity...
-> pfq-affinity -f 0  -a multiple/1 -m TxRx enp2s0f0 enp2s0f1 ens1f0 ens1f1
Setting binding for device enp2s0f0 (Just TxRx):
   irq 48 -> core 0 {mask = 1}
   irq 49 -> core 1 {mask = 2}
Setting binding for device enp2s0f1 (Just TxRx):
   irq 52 -> core 2 {mask = 4}
   irq 53 -> core 3 {mask = 8}
Setting binding for device ens1f0 (Just TxRx):
   irq 56 -> core 4 {mask = 10}
Setting binding for device ens1f1 (Just TxRx):
   irq 58 -> core 5 {mask = 20}
PFQ ready.
Add a custom footer
Add a custom sidebar
Clone this wiki locally