两台IB交换机使用server side方式做高可用的记录
两台IB交换机使用server side方式做高可用的记录
额外注意：IB交换机也分可不可以管理，比如SX6036就是可以管理的IB交换机，
两台可以管理的IB交换机可以在IB交换机上配置高可用来实现其中一台IB交换机意外离线而导致的IB网络失效的问题。
若是不可管理的IB交换机，就只能在服务器端使用opensm来实现IB网络的高可用性。
介绍：
三台pcserver，安装RHEL7.2，每台pcserver上安装有两块HCA卡，每块HCA卡2个端口（port），
只不过每块HCA卡的第二个端口（port）没有插入IB线缆，并且每块HCA卡的第二个端口（port）在nmtui中被disable掉了。
三台主机的主机名为
rac1
rac2
node1
两台IB交换机，SX6036。
在三台pcserver上，安装mellanox的ofed驱动，ofed驱动使用opensmd服务来管理IB的子网。如下所示：
三台主机的/etc/opensm/opensm.conf文件的内容如下（只贴自行修改的部分）：
rac1的ip地址如下：
rac2的ip地址如下：
node1（第三台pcsever）的ip地址如下：
rac1:
rac2:
node1:
额外注意：IB交换机也分可不可以管理，比如SX6036就是可以管理的IB交换机，
两台可以管理的IB交换机可以在IB交换机上配置高可用来实现其中一台IB交换机意外离线而导致的IB网络失效的问题。
若是不可管理的IB交换机，就只能在服务器端使用opensm来实现IB网络的高可用性。
介绍：
三台pcserver，安装RHEL7.2，每台pcserver上安装有两块HCA卡，每块HCA卡2个端口（port），
只不过每块HCA卡的第二个端口（port）没有插入IB线缆，并且每块HCA卡的第二个端口（port）在nmtui中被disable掉了。
三台主机的主机名为
rac1
rac2
node1
两台IB交换机，SX6036。
在三台pcserver上，安装mellanox的ofed驱动，ofed驱动使用opensmd服务来管理IB的子网。如下所示：
[root@rac1 ~]# systemctl status opensmd
● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager
Loaded: loaded (/etc/rc.d/init.d/opensmd)
Active: active (running) since Mon 2017-06-05 18:39:07 CST; 14h ago
Docs: man:systemd-sysv-generator(8)
Process: 2762 ExecStop=/etc/rc.d/init.d/opensmd stop (code=exited, status=0/SUCCESS)
Process: 2786 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS)
Main PID: 2793 (opensm)
CGroup: /system.slice/opensmd.service
└─2793 /usr/sbin/opensm --daemon
Jun 05 18:39:06 rac1 OpenSM[2791]: Loading Cached Option:guid = 0x7cfe900300917f11
Jun 05 18:39:06 rac1 OpenSM[2791]: Loading Cached Option:guid = 0x7cfe900300917ed1
Jun 05 18:39:06 rac1 OpenSM[2793]: /var/log/opensm.log log file opened
Jun 05 18:39:06 rac1 OpenSM[2793]: OpenSM 4.8.1.MLNX20170118.1a8ad26
Jun 05 18:39:06 rac1 OpenSM[2793]: Entering DISCOVERING state
Jun 05 18:39:06 rac1 OpenSM[2793]: Entering STANDBY state
Jun 05 18:39:07 rac1 opensmd[2786]: Starting IB Subnet Manager.[ OK ]
Jun 05 18:39:07 rac1 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager.
Jun 05 18:43:06 rac1 OpenSM[2793]: Entering DISCOVERING state
Jun 05 18:43:06 rac1 OpenSM[2793]: Entering STANDBY state
[root@rac1 ~]#
[root@rac2 ~]# systemctl status opensmd
● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager
Loaded: loaded (/etc/rc.d/init.d/opensmd)
Active: active (running) since Mon 2017-06-05 18:48:07 CST; 14h ago
Docs: man:systemd-sysv-generator(8)
Process: 1679 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS)
Main PID: 1711 (opensm)
CGroup: /system.slice/opensmd.service
└─1711 /usr/sbin/opensm --daemon
Jun 05 18:48:06 rac2 OpenSM[1695]: Loading Cached Option:guid = 0x248a0703005d07f1
Jun 05 18:48:06 rac2 OpenSM[1695]: Loading Cached Option:guid = 0x248a0703005d0841
Jun 05 18:48:06 rac2 OpenSM[1711]: /var/log/opensm.log log file opened
Jun 05 18:48:06 rac2 OpenSM[1711]: OpenSM 4.8.1.MLNX20170118.1a8ad26
Jun 05 18:48:06 rac2 OpenSM[1711]: Entering DISCOVERING state
Jun 05 18:48:06 rac2 OpenSM[1711]: SM port is down
Jun 05 18:48:07 rac2 opensmd[1679]: Starting IB Subnet Manager.[ OK ]
Jun 05 18:48:07 rac2 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager.
Jun 05 18:48:46 rac2 OpenSM[1711]: SM port is up
Jun 05 18:48:46 rac2 OpenSM[1711]: Entering STANDBY state
[root@rac2 ~]# [root@host1 ~]# systemctl status opensmd
● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager
Loaded: loaded (/etc/rc.d/init.d/opensmd)
Active: active (running) since Mon 2017-06-05 18:33:50 CST; 14h ago
Docs: man:systemd-sysv-generator(8)
Process: 2935 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS)
Main PID: 2941 (opensm)
CGroup: /system.slice/opensmd.service
└─2941 /usr/sbin/opensm --daemon
Jun 05 18:33:49 host1 systemd[1]: Starting LSB: Activates/Deactivates InfiniBand Subnet Manager...
Jun 05 18:33:49 host1 OpenSM[2941]: /var/log/opensm.log log file opened
Jun 05 18:33:49 host1 OpenSM[2941]: OpenSM 4.8.1.MLNX20170118.1a8ad26
Jun 05 18:33:49 host1 OpenSM[2941]: Entering DISCOVERING state
Jun 05 18:33:49 host1 OpenSM[2941]: Entering STANDBY state
Jun 05 18:33:50 host1 opensmd[2935]: Starting IB Subnet Manager.[ OK ]
Jun 05 18:33:50 host1 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager.
Jun 05 18:43:09 host1 OpenSM[2941]: Entering DISCOVERING state
Jun 05 18:43:09 host1 OpenSM[2941]: Entering MASTER state
[root@host1 ~]#
三台主机的/etc/opensm/opensm.conf文件的内容如下（只贴自行修改的部分）：
rac1上的/etc/opensm/opensm.conf文件的内容：
# The port GUID on which the OpenSM is running
guid 0x7cfe900300917f11
guid 0x7cfe900300917ed1rac2上的/etc/opensm/opensm.conf文件的内容：
# The port GUID on which the OpenSM is running
guid 0x248a0703005d07f1
guid 0x248a0703005d0841rac2上的/etc/opensm/opensm.conf文件的内容：
# The port GUID on which the OpenSM is running
guid 0x248a0703005d07f1
guid 0x248a0703005d0841rac1的ip地址如下：
[root@rac1 ~]# ip addr show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp5s0f0: mtu 1500 qdisc mq state UP qlen 1000
link/ether 0c:c4:7a:ac:3c:c8 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.101/24 brd 192.168.100.255 scope global enp5s0f0
valid_lft forever preferred_lft forever
inet6 fe80::ec4:7aff:feac:3cc8/64 scope link
valid_lft forever preferred_lft forever
3: enp5s0f1: mtu 1500 qdisc mq state UP qlen 1000
link/ether 0c:c4:7a:ac:3c:c9 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/24 brd 172.17.0.255 scope global enp5s0f1
valid_lft forever preferred_lft forever
inet6 fe80::ec4:7aff:feac:3cc9/64 scope link
valid_lft forever preferred_lft forever
4: ib0: mtu 2044 qdisc mq state UP qlen 1024
link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:11 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 10.10.10.101/24 brd 10.10.10.255 scope global ib0
valid_lft forever preferred_lft forever
inet6 fe80::7efe:9003:91:7f11/64 scope link
valid_lft forever preferred_lft forever
5: ib1: mtu 4092 qdisc mq state DOWN qlen 1024
link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:12 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
6: ib2: mtu 2044 qdisc mq state UP qlen 1024
link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 10.10.11.102/24 brd 10.10.11.255 scope global ib2
valid_lft forever preferred_lft forever
inet6 fe80::7efe:9003:91:7ed1/64 scope link
valid_lft forever preferred_lft forever
7: ib3: mtu 4092 qdisc mq state DOWN qlen 1024
link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
8: virbr0: mtu 1500 qdisc noqueue state DOWN
link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
9: virbr0-nic: mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff
[root@rac1 ~]#
rac2的ip地址如下：
[root@rac2 ~]# ip addr show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp5s0f0: mtu 1500 qdisc mq state UP qlen 1000
link/ether 0c:c4:7a:ab:d3:4c brd ff:ff:ff:ff:ff:ff
inet 192.168.100.102/24 brd 192.168.100.255 scope global enp5s0f0
valid_lft forever preferred_lft forever
inet6 fe80::ec4:7aff:feab:d34c/64 scope link
valid_lft forever preferred_lft forever
3: enp5s0f1: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 0c:c4:7a:ab:d3:4d brd ff:ff:ff:ff:ff:ff
4: ib0: mtu 4092 qdisc mq state DOWN qlen 1024
link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:07:f1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 10.10.10.105/24 brd 10.10.10.255 scope global ib0
valid_lft forever preferred_lft forever
5: ib1: mtu 4092 qdisc mq state DOWN qlen 1024
link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:07:f2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
6: ib2: mtu 2044 qdisc mq state UP qlen 1024
link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:08:41 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 10.10.11.107/24 brd 10.10.11.255 scope global ib2
valid_lft forever preferred_lft forever
inet6 fe80::268a:703:5d:841/64 scope link
valid_lft forever preferred_lft forever
7: ib3: mtu 4092 qdisc mq state DOWN qlen 1024
link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:08:42 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
8: virbr0: mtu 1500 qdisc noqueue state DOWN
link/ether 52:54:00:74:ff:7b brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
9: virbr0-nic: mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
link/ether 52:54:00:74:ff:7b brd ff:ff:ff:ff:ff:ff
[root@rac2 ~]# node1（第三台pcsever）的ip地址如下：
[root@host1 ~]# ip addr show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp5s0f0: mtu 1500 qdisc mq state UP qlen 1000
link/ether 0c:c4:7a:ab:e5:ce brd ff:ff:ff:ff:ff:ff
inet 192.168.100.103/24 brd 192.168.100.255 scope global enp5s0f0
valid_lft forever preferred_lft forever
inet6 fe80::ec4:7aff:feab:e5ce/64 scope link
valid_lft forever preferred_lft forever
3: enp5s0f1: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 0c:c4:7a:ab:e5:cf brd ff:ff:ff:ff:ff:ff
4: ib0: mtu 2044 qdisc mq state UP qlen 1024
link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:16:c2:31 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 10.10.10.109/24 brd 10.10.10.255 scope global ib0
valid_lft forever preferred_lft forever
inet6 fe80::7efe:9003:16:c231/64 scope link
valid_lft forever preferred_lft forever
5: ib1: mtu 4092 qdisc mq state DOWN qlen 1024
link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:16:c2:32 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
6: ib2: mtu 2044 qdisc mq state UP qlen 1024
link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7d:a1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 10.10.11.111/24 brd 10.10.11.255 scope global ib2
valid_lft forever preferred_lft forever
inet6 fe80::7efe:9003:91:7da1/64 scope link
valid_lft forever preferred_lft forever
7: ib3: mtu 4092 qdisc mq state DOWN qlen 1024
link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7d:a2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
8: virbr0: mtu 1500 qdisc noqueue state DOWN
link/ether 52:54:00:53:65:c8 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
9: virbr0-nic: mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
link/ether 52:54:00:53:65:c8 brd ff:ff:ff:ff:ff:ff
[root@host1 ~]# rac1:
[root@rac1 ~]# ibhosts
Ca : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"
Ca : 0x7cfe90030016c230 ports 2 "host1 HCA-1"
Ca : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"
[root@rac1 ~]# ibaddr
GID fe80::7cfe:9003:91:7f11 LID start 0x2 end 0x2
[root@rac1 ~]# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.40.7000
Hardware version: 1
Node GUID: 0x7cfe900300917f10
System image GUID: 0x7cfe900300917f13
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 2
LMC: 0
SM lid: 4
Capability mask: 0x02514868
Port GUID: 0x7cfe900300917f11
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02514868
Port GUID: 0x7cfe900300917f12
Link layer: InfiniBand
CA 'mlx4_1'
CA type: MT4099
Number of ports: 2
Firmware version: 2.40.7000
Hardware version: 1
Node GUID: 0x7cfe900300917ed0
System image GUID: 0x7cfe900300917ed3
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 5
LMC: 0
SM lid: 8
Capability mask: 0x0251486a
Port GUID: 0x7cfe900300917ed1
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02514868
Port GUID: 0x7cfe900300917ed2
Link layer: InfiniBand
[root@rac1 ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:7cfe:9003:0091:7f11
base lid: 0x2
sm lid: 0x4
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 56 Gb/sec (4X FDR)
link_layer: InfiniBand
Infiniband device 'mlx4_0' port 2 status:
default gid: fe80:0000:0000:0000:7cfe:9003:0091:7f12
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X)
link_layer: InfiniBand
Infiniband device 'mlx4_1' port 1 status:
default gid: fe80:0000:0000:0000:7cfe:9003:0091:7ed1
base lid: 0x5
sm lid: 0x8
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 56 Gb/sec (4X FDR)
link_layer: InfiniBand
Infiniband device 'mlx4_1' port 2 status:
default gid: fe80:0000:0000:0000:7cfe:9003:0091:7ed2
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X)
link_layer: InfiniBand
[root@rac1 ~]# ibv_
ibv_asyncwatch ibv_dcini ibv_devices ibv_intf ibv_rc_pingpong ibv_srq_pingpong ibv_uc_pingpong ibv_umr
ibv_cc_pingpong ibv_dctgt ibv_devinfo ibv_polldcinfo ibv_shared_mr ibv_task_pingpong ibv_ud_pingpong ibv_xsrq_pingpong
[root@rac1 ~]# ibv_devinfo
hca_id: mlx4_1
transport: InfiniBand (0)
fw_ver: 2.40.7000
node_guid: 7cfe:9003:0091:7ed0
sys_image_guid: 7cfe:9003:0091:7ed3
vendor_id: 0x02c9
vendor_part_id: 4099
hw_ver: 0x1
board_id: MT_1090120019
phys_port_cnt: 2
Device ports:
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 8
port_lid: 5
port_lmc: 0x00
link_layer: InfiniBand
port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: InfiniBand
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 2.40.7000
node_guid: 7cfe:9003:0091:7f10
sys_image_guid: 7cfe:9003:0091:7f13
vendor_id: 0x02c9
vendor_part_id: 4099
hw_ver: 0x1
board_id: MT_1090120019
phys_port_cnt: 2
Device ports:
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 4
port_lid: 2
port_lmc: 0x00
link_layer: InfiniBand
port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: InfiniBand
[root@rac1 ~]# ibnetdiscover
#
# Topology file: generated on Tue Jun 6 08:37:33 2017
#
# Initiated from node 7cfe900300917f10 port 7cfe900300917f11
vendid=0x2c9
devid=0xc738
sysimgguid=0x7cfe900300f17940
switchguid=0x7cfe900300f17940(7cfe900300f17940)
Switch 36 "S-7cfe900300f17940" # "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[31] "H-7cfe90030016c230"[1](7cfe90030016c231) # "host1 HCA-1" lid 3 4xFDR
[33] "H-248a0703005d07f0"[1](248a0703005d07f1) # "rac2 HCA-1" lid 0 4xFDR
[35] "H-7cfe900300917f10"[1](7cfe900300917f11) # "rac1 HCA-1" lid 2 4xFDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x248a0703005d07f3
caguid=0x248a0703005d07f0
Ca 2 "H-248a0703005d07f0" # "rac2 HCA-1"
[1](248a0703005d07f1) "S-7cfe900300f17940"[33] # lid 0 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe90030016c233
caguid=0x7cfe90030016c230
Ca 2 "H-7cfe90030016c230" # "host1 HCA-1"
[1](7cfe90030016c231) "S-7cfe900300f17940"[31] # lid 3 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe900300917f13
caguid=0x7cfe900300917f10
Ca 2 "H-7cfe900300917f10" # "rac1 HCA-1"
[1](7cfe900300917f11) "S-7cfe900300f17940"[35] # lid 2 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR
[root@rac1 ~]# iblinkinfo
CA: rac2 HCA-1:
0x248a0703005d07f1 0 1[ ] ==( 4X 14.0625 Gbps Initialize/ LinkUp)==> 1 33[ ] "MF0;switch-138276:SX6036/U1" ( )
CA: host1 HCA-1:
0x7cfe90030016c231 3 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 31[ ] "MF0;switch-138276:SX6036/U1" ( )
Switch: 0x7cfe900300f17940 MF0;switch-138276:SX6036/U1:
1 1[ ] ==( Down/ Polling)==> [ ] "" ( )
1 2[ ] ==( Down/ Polling)==> [ ] "" ( )
1 3[ ] ==( Down/ Polling)==> [ ] "" ( )
1 4[ ] ==( Down/ Polling)==> [ ] "" ( )
1 5[ ] ==( Down/ Polling)==> [ ] "" ( )
1 6[ ] ==( Down/ Polling)==> [ ] "" ( )
1 7[ ] ==( Down/ Polling)==> [ ] "" ( )
1 8[ ] ==( Down/ Polling)==> [ ] "" ( )
1 9[ ] ==( Down/ Polling)==> [ ] "" ( )
1 10[ ] ==( Down/ Polling)==> [ ] "" ( )
1 11[ ] ==( Down/ Polling)==> [ ] "" ( )
1 12[ ] ==( Down/ Polling)==> [ ] "" ( )
1 13[ ] ==( Down/ Polling)==> [ ] "" ( )
1 14[ ] ==( Down/ Polling)==> [ ] "" ( )
1 15[ ] ==( Down/ Polling)==> [ ] "" ( )
1 16[ ] ==( Down/ Polling)==> [ ] "" ( )
1 17[ ] ==( Down/ Polling)==> [ ] "" ( )
1 18[ ] ==( Down/ Polling)==> [ ] "" ( )
1 19[ ] ==( Down/ Polling)==> [ ] "" ( )
1 20[ ] ==( Down/ Polling)==> [ ] "" ( )
1 21[ ] ==( Down/ Polling)==> [ ] "" ( )
1 22[ ] ==( Down/ Polling)==> [ ] "" ( )
1 23[ ] ==( Down/ Polling)==> [ ] "" ( )
1 24[ ] ==( Down/ Polling)==> [ ] "" ( )
1 25[ ] ==( Down/ Polling)==> [ ] "" ( )
1 26[ ] ==( Down/ Polling)==> [ ] "" ( )
1 27[ ] ==( Down/ Polling)==> [ ] "" ( )
1 28[ ] ==( Down/ Polling)==> [ ] "" ( )
1 29[ ] ==( Down/ Polling)==> [ ] "" ( )
1 30[ ] ==( Down/ Polling)==> [ ] "" ( )
1 31[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 3 1[ ] "host1 HCA-1" ( )
1 32[ ] ==( Down/ Polling)==> [ ] "" ( )
1 33[ ] ==( 4X 14.0625 Gbps Initialize/ LinkUp)==> 0 1[ ] "rac2 HCA-1" ( )
1 34[ ] ==( Down/ Polling)==> [ ] "" ( )
1 35[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 2 1[ ] "rac1 HCA-1" ( )
1 36[ ] ==( Down/ Polling)==> [ ] "" ( )
CA: rac1 HCA-1:
0x7cfe900300917f11 2 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 35[ ] "MF0;switch-138276:SX6036/U1" ( )
[root@rac1 ~]# ibswitches
Switch : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[root@rac1 ~]# ibnodes
Ca : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"
Ca : 0x7cfe90030016c230 ports 2 "host1 HCA-1"
Ca : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"
Switch : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[root@rac1 ~]# rac2:
[root@rac2 ~]# ibhosts
Ca : 0x7cfe900300917ed0 ports 2 "rac1 HCA-2"
Ca : 0x7cfe900300917da0 ports 2 "host1 HCA-2"
Ca : 0x248a0703005d0840 ports 2 "rac2 HCA-2"
[root@rac2 ~]# ibaddr
GID fe80::248a:703:5d:841 LID start 0x1 end 0x1
[root@rac2 ~]# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.40.7000
Hardware version: 1
Node GUID: 0x248a0703005d07f0
System image GUID: 0x248a0703005d07f3
Port 1:
State: Initializing
Physical state: LinkUp
Rate: 56
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02514868
Port GUID: 0x248a0703005d07f1
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02514868
Port GUID: 0x248a0703005d07f2
Link layer: InfiniBand
CA 'mlx4_1'
CA type: MT4099
Number of ports: 2
Firmware version: 2.40.7000
Hardware version: 1
Node GUID: 0x248a0703005d0840
System image GUID: 0x248a0703005d0843
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 1
LMC: 0
SM lid: 8
Capability mask: 0x0251486a
Port GUID: 0x248a0703005d0841
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02514868
Port GUID: 0x248a0703005d0842
Link layer: InfiniBand
[root@rac2 ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:248a:0703:005d:07f1
base lid: 0x0
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 56 Gb/sec (4X FDR)
link_layer: InfiniBand
Infiniband device 'mlx4_0' port 2 status:
default gid: fe80:0000:0000:0000:248a:0703:005d:07f2
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X)
link_layer: InfiniBand
Infiniband device 'mlx4_1' port 1 status:
default gid: fe80:0000:0000:0000:248a:0703:005d:0841
base lid: 0x1
sm lid: 0x8
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 56 Gb/sec (4X FDR)
link_layer: InfiniBand
Infiniband device 'mlx4_1' port 2 status:
default gid: fe80:0000:0000:0000:248a:0703:005d:0842
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X)
link_layer: InfiniBand
[root@rac2 ~]# ibv_devinfo
hca_id: mlx4_1
transport: InfiniBand (0)
fw_ver: 2.40.7000
node_guid: 248a:0703:005d:0840
sys_image_guid: 248a:0703:005d:0843
vendor_id: 0x02c9
vendor_part_id: 4099
hw_ver: 0x1
board_id: MT_1090120019
phys_port_cnt: 2
Device ports:
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 8
port_lid: 1
port_lmc: 0x00
link_layer: InfiniBand
port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: InfiniBand
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 2.40.7000
node_guid: 248a:0703:005d:07f0
sys_image_guid: 248a:0703:005d:07f3
vendor_id: 0x02c9
vendor_part_id: 4099
hw_ver: 0x1
board_id: MT_1090120019
phys_port_cnt: 2
Device ports:
port: 1
state: PORT_INIT (2)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: InfiniBand
port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: InfiniBand
[root@rac2 ~]# ibnetdiscover
#
# Topology file: generated on Tue Jun 6 08:36:34 2017
#
# Initiated from node 248a0703005d0840 port 248a0703005d0841
vendid=0x2c9
devid=0xc738
sysimgguid=0x7cfe900300ef40c0
switchguid=0x7cfe900300ef40c0(7cfe900300ef40c0)
Switch 36 "S-7cfe900300ef40c0" # "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0
[31] "H-7cfe900300917da0"[1](7cfe900300917da1) # "host1 HCA-2" lid 8 4xFDR
[33] "H-248a0703005d0840"[1](248a0703005d0841) # "rac2 HCA-2" lid 1 4xFDR
[35] "H-7cfe900300917ed0"[1](7cfe900300917ed1) # "rac1 HCA-2" lid 5 4xFDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe900300917ed3
caguid=0x7cfe900300917ed0
Ca 2 "H-7cfe900300917ed0" # "rac1 HCA-2"
[1](7cfe900300917ed1) "S-7cfe900300ef40c0"[35] # lid 5 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe900300917da3
caguid=0x7cfe900300917da0
Ca 2 "H-7cfe900300917da0" # "host1 HCA-2"
[1](7cfe900300917da1) "S-7cfe900300ef40c0"[31] # lid 8 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x248a0703005d0843
caguid=0x248a0703005d0840
Ca 2 "H-248a0703005d0840" # "rac2 HCA-2"
[1](248a0703005d0841) "S-7cfe900300ef40c0"[33] # lid 1 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDR
[root@rac2 ~]# iblinkinfo
CA: rac1 HCA-2:
0x7cfe900300917ed1 5 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 7 35[ ] "MF0;switch-cad412:SX6036/U1" ( )
CA: host1 HCA-2:
0x7cfe900300917da1 8 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 7 31[ ] "MF0;switch-cad412:SX6036/U1" ( )
Switch: 0x7cfe900300ef40c0 MF0;switch-cad412:SX6036/U1:
7 1[ ] ==( Down/ Polling)==> [ ] "" ( )
7 2[ ] ==( Down/ Polling)==> [ ] "" ( )
7 3[ ] ==( Down/ Polling)==> [ ] "" ( )
7 4[ ] ==( Down/ Polling)==> [ ] "" ( )
7 5[ ] ==( Down/ Polling)==> [ ] "" ( )
7 6[ ] ==( Down/ Polling)==> [ ] "" ( )
7 7[ ] ==( Down/ Polling)==> [ ] "" ( )
7 8[ ] ==( Down/ Polling)==> [ ] "" ( )
7 9[ ] ==( Down/ Polling)==> [ ] "" ( )
7 10[ ] ==( Down/ Polling)==> [ ] "" ( )
7 11[ ] ==( Down/ Polling)==> [ ] "" ( )
7 12[ ] ==( Down/ Polling)==> [ ] "" ( )
7 13[ ] ==( Down/ Polling)==> [ ] "" ( )
7 14[ ] ==( Down/ Polling)==> [ ] "" ( )
7 15[ ] ==( Down/ Polling)==> [ ] "" ( )
7 16[ ] ==( Down/ Polling)==> [ ] "" ( )
7 17[ ] ==( Down/ Polling)==> [ ] "" ( )
7 18[ ] ==( Down/ Polling)==> [ ] "" ( )
7 19[ ] ==( Down/ Polling)==> [ ] "" ( )
7 20[ ] ==( Down/ Polling)==> [ ] "" ( )
7 21[ ] ==( Down/ Polling)==> [ ] "" ( )
7 22[ ] ==( Down/ Polling)==> [ ] "" ( )
7 23[ ] ==( Down/ Polling)==> [ ] "" ( )
7 24[ ] ==( Down/ Polling)==> [ ] "" ( )
7 25[ ] ==( Down/ Polling)==> [ ] "" ( )
7 26[ ] ==( Down/ Polling)==> [ ] "" ( )
7 27[ ] ==( Down/ Polling)==> [ ] "" ( )
7 28[ ] ==( Down/ Polling)==> [ ] "" ( )
7 29[ ] ==( Down/ Polling)==> [ ] "" ( )
7 30[ ] ==( Down/ Polling)==> [ ] "" ( )
7 31[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 8 1[ ] "host1 HCA-2" ( )
7 32[ ] ==( Down/ Polling)==> [ ] "" ( )
7 33[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 1[ ] "rac2 HCA-2" ( )
7 34[ ] ==( Down/ Polling)==> [ ] "" ( )
7 35[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 5 1[ ] "rac1 HCA-2" ( )
7 36[ ] ==( Down/ Polling)==> [ ] "" ( )
CA: rac2 HCA-2:
0x248a0703005d0841 1 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 7 33[ ] "MF0;switch-cad412:SX6036/U1" ( )
[root@rac2 ~]# ibswitches
Switch : 0x7cfe900300ef40c0 ports 36 "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0
[root@rac2 ~]# ibnodes
Ca : 0x7cfe900300917ed0 ports 2 "rac1 HCA-2"
Ca : 0x7cfe900300917da0 ports 2 "host1 HCA-2"
Ca : 0x248a0703005d0840 ports 2 "rac2 HCA-2"
Switch : 0x7cfe900300ef40c0 ports 36 "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0
[root@rac2 ~]# node1:
[root@host1 ~]# ibhosts
Ca : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"
Ca : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"
Ca : 0x7cfe90030016c230 ports 2 "host1 HCA-1"
[root@host1 ~]# ibaddr
GID fe80::7cfe:9003:16:c231 LID start 0x3 end 0x3
[root@host1 ~]# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.40.7000
Hardware version: 1
Node GUID: 0x7cfe90030016c230
System image GUID: 0x7cfe90030016c233
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 3
LMC: 0
SM lid: 4
Capability mask: 0x02514868
Port GUID: 0x7cfe90030016c231
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02514868
Port GUID: 0x7cfe90030016c232
Link layer: InfiniBand
CA 'mlx4_1'
CA type: MT4099
Number of ports: 2
Firmware version: 2.40.7000
Hardware version: 1
Node GUID: 0x7cfe900300917da0
System image GUID: 0x7cfe900300917da3
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 8
LMC: 0
SM lid: 8
Capability mask: 0x0251486a
Port GUID: 0x7cfe900300917da1
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02514868
Port GUID: 0x7cfe900300917da2
Link layer: InfiniBand
[root@host1 ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:7cfe:9003:0016:c231
base lid: 0x3
sm lid: 0x4
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 56 Gb/sec (4X FDR)
link_layer: InfiniBand
Infiniband device 'mlx4_0' port 2 status:
default gid: fe80:0000:0000:0000:7cfe:9003:0016:c232
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X)
link_layer: InfiniBand
Infiniband device 'mlx4_1' port 1 status:
default gid: fe80:0000:0000:0000:7cfe:9003:0091:7da1
base lid: 0x8
sm lid: 0x8
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 56 Gb/sec (4X FDR)
link_layer: InfiniBand
Infiniband device 'mlx4_1' port 2 status:
default gid: fe80:0000:0000:0000:7cfe:9003:0091:7da2
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X)
link_layer: InfiniBand
[root@host1 ~]# ibv_devinfo
hca_id: mlx4_1
transport: InfiniBand (0)
fw_ver: 2.40.7000
node_guid: 7cfe:9003:0091:7da0
sys_image_guid: 7cfe:9003:0091:7da3
vendor_id: 0x02c9
vendor_part_id: 4099
hw_ver: 0x1
board_id: MT_1090120019
phys_port_cnt: 2
Device ports:
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 8
port_lid: 8
port_lmc: 0x00
link_layer: InfiniBand
port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: InfiniBand
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 2.40.7000
node_guid: 7cfe:9003:0016:c230
sys_image_guid: 7cfe:9003:0016:c233
vendor_id: 0x02c9
vendor_part_id: 4099
hw_ver: 0x1
board_id: MT_1090120019
phys_port_cnt: 2
Device ports:
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 4
port_lid: 3
port_lmc: 0x00
link_layer: InfiniBand
port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: InfiniBand
[root@host1 ~]# ibnetdiscover
#
# Topology file: generated on Tue Jun 6 08:38:13 2017
#
# Initiated from node 7cfe90030016c230 port 7cfe90030016c231
vendid=0x2c9
devid=0xc738
sysimgguid=0x7cfe900300f17940
switchguid=0x7cfe900300f17940(7cfe900300f17940)
Switch 36 "S-7cfe900300f17940" # "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[31] "H-7cfe90030016c230"[1](7cfe90030016c231) # "host1 HCA-1" lid 3 4xFDR
[33] "H-248a0703005d07f0"[1](248a0703005d07f1) # "rac2 HCA-1" lid 0 4xFDR
[35] "H-7cfe900300917f10"[1](7cfe900300917f11) # "rac1 HCA-1" lid 2 4xFDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe900300917f13
caguid=0x7cfe900300917f10
Ca 2 "H-7cfe900300917f10" # "rac1 HCA-1"
[1](7cfe900300917f11) "S-7cfe900300f17940"[35] # lid 2 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x248a0703005d07f3
caguid=0x248a0703005d07f0
Ca 2 "H-248a0703005d07f0" # "rac2 HCA-1"
[1](248a0703005d07f1) "S-7cfe900300f17940"[33] # lid 0 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe90030016c233
caguid=0x7cfe90030016c230
Ca 2 "H-7cfe90030016c230" # "host1 HCA-1"
[1](7cfe90030016c231) "S-7cfe900300f17940"[31] # lid 3 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR
[root@host1 ~]# iblinkinfo
CA: rac1 HCA-1:
0x7cfe900300917f11 2 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 35[ ] "MF0;switch-138276:SX6036/U1" ( )
CA: rac2 HCA-1:
0x248a0703005d07f1 0 1[ ] ==( 4X 14.0625 Gbps Initialize/ LinkUp)==> 1 33[ ] "MF0;switch-138276:SX6036/U1" ( )
Switch: 0x7cfe900300f17940 MF0;switch-138276:SX6036/U1:
1 1[ ] ==( Down/ Polling)==> [ ] "" ( )
1 2[ ] ==( Down/ Polling)==> [ ] "" ( )
1 3[ ] ==( Down/ Polling)==> [ ] "" ( )
1 4[ ] ==( Down/ Polling)==> [ ] "" ( )
1 5[ ] ==( Down/ Polling)==> [ ] "" ( )
1 6[ ] ==( Down/ Polling)==> [ ] "" ( )
1 7[ ] ==( Down/ Polling)==> [ ] "" ( )
1 8[ ] ==( Down/ Polling)==> [ ] "" ( )
1 9[ ] ==( Down/ Polling)==> [ ] "" ( )
1 10[ ] ==( Down/ Polling)==> [ ] "" ( )
1 11[ ] ==( Down/ Polling)==> [ ] "" ( )
1 12[ ] ==( Down/ Polling)==> [ ] "" ( )
1 13[ ] ==( Down/ Polling)==> [ ] "" ( )
1 14[ ] ==( Down/ Polling)==> [ ] "" ( )
1 15[ ] ==( Down/ Polling)==> [ ] "" ( )
1 16[ ] ==( Down/ Polling)==> [ ] "" ( )
1 17[ ] ==( Down/ Polling)==> [ ] "" ( )
1 18[ ] ==( Down/ Polling)==> [ ] "" ( )
1 19[ ] ==( Down/ Polling)==> [ ] "" ( )
1 20[ ] ==( Down/ Polling)==> [ ] "" ( )
1 21[ ] ==( Down/ Polling)==> [ ] "" ( )
1 22[ ] ==( Down/ Polling)==> [ ] "" ( )
1 23[ ] ==( Down/ Polling)==> [ ] "" ( )
1 24[ ] ==( Down/ Polling)==> [ ] "" ( )
1 25[ ] ==( Down/ Polling)==> [ ] "" ( )
1 26[ ] ==( Down/ Polling)==> [ ] "" ( )
1 27[ ] ==( Down/ Polling)==> [ ] "" ( )
1 28[ ] ==( Down/ Polling)==> [ ] "" ( )
1 29[ ] ==( Down/ Polling)==> [ ] "" ( )
1 30[ ] ==( Down/ Polling)==> [ ] "" ( )
1 31[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 3 1[ ] "host1 HCA-1" ( )
1 32[ ] ==( Down/ Polling)==> [ ] "" ( )
1 33[ ] ==( 4X 14.0625 Gbps Initialize/ LinkUp)==> 0 1[ ] "rac2 HCA-1" ( )
1 34[ ] ==( Down/ Polling)==> [ ] "" ( )
1 35[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 2 1[ ] "rac1 HCA-1" ( )
1 36[ ] ==( Down/ Polling)==> [ ] "" ( )
CA: host1 HCA-1:
0x7cfe90030016c231 3 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 31[ ] "MF0;switch-138276:SX6036/U1" ( )
[root@host1 ~]# ibswitches
Switch : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[root@host1 ~]# ibnodes
Ca : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"
Ca : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"
Ca : 0x7cfe90030016c230 ports 2 "host1 HCA-1"
Switch : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[root@host1 ~]#