iRDMA Flow Control Verification

6.0         Priority Flow Control - Verification

6.1           Priority Counters

Priority flow control counters for each interface are available in ethtool. They measure the number of Xon and Xoff (transmit on and off) frames sent and received by that interface.

To view them:

# ethtool -S | grep prio

Counters are named with tx/rx, priority number, and either xon or xoff.

For example:

# ethtool -S enp175s0f0 | grep prio tx_priority_0_xon.nic: 1 tx_priority_0_xoff.nic: 6434 tx_priority_1_xon.nic: 1 tx_priority_1_xoff.nic: 6434 tx_priority_2_xon.nic: 2 tx_priority_2_xoff.nic: 14864 tx_priority_3_xon.nic: 1 tx_priority_3_xoff.nic: 6434 tx_priority_4_xon.nic: 0 tx_priority_4_xoff.nic: 0 tx_priority_5_xon.nic: 1 tx_priority_5_xoff.nic: 6434 tx_priority_6_xon.nic: 1 tx_priority_6_xoff.nic: 6434 tx_priority_7_xon.nic: 1 tx_priority_7_xoff.nic: 6434 rx_priority_0_xon.nic: 0 rx_priority_0_xoff.nic: 0 rx_priority_1_xon.nic: 0 rx_priority_1_xoff.nic: 0 rx_priority_2_xon.nic: 0 rx_priority_2_xoff.nic: 0 rx_priority_3_xon.nic: 0 rx_priority_3_xoff.nic: 0 rx_priority_4_xon.nic: 0 rx_priority_4_xoff.nic: 0 rx_priority_5_xon.nic: 0 rx_priority_5_xoff.nic: 0 rx_priority_6_xon.nic: 0 rx_priority_6_xoff.nic: 0 rx_priority_7_xon.nic: 0 rx_priority_7_xoff.nic: 0

Note that the Rx counters all 0.

When adapters are connected through a switch, the rx_priority_* counters might be 0, indicating that the adapter has not received any pause frames from the switch. Depending on the level of stress in the network, this is acceptable if the switch has

enough buffering to keep up with the host demand. However, for high stress traffic such as HPC applications at larger scale, often the switch sends pause frames to the host. In general, it is expected to see both tx and rx_priority counters.

Note that some of the Tx counters have the same value.

In the 800 Series QoS implementation, if PFC is enabled for any priority in a traffic class, all priorities in that traffic class get pause frames. This means that the counters for all priorities in the same TC are incremented in unison, regardless of the particular single priority that is causing PFC to trigger. If all priorities are mapped to the same TC, they all increment in unison.

This implementation is in line with 802.1Q recommendations.

• 802.1Q Section 37.3: NOTE 2 — All priorities within a traffic class typically share similar traffic handling requirements (e.g., loss and bandwidth).
• 802.1Q Section 8.6.8: NOTE 1 — Two or more priorities can be combined in a single queue. In this case if one or more of the priorities in the queue are paused, it is possible for frames in that queue not belonging to the paused priority to not