How to monitor and control the InfiniBand fabric:

Infiniband commands:

ibswitches :-Identify all switches in the fabric

ibhosts :-Identify all HCAs ( Host channel adapters) in a subnet
ibnetdiscover :- Displays infinibad topology.
Ibtracert slid dlid : - display routes between two nodes
Ibportstate lid port ( ibportstate 15 23) : -Display state and speed of the node
Prefquery lid port (prefquery 15 23) : - Displays node health in the fabric.
Ibdatacounts lid port (ibdatacounts) :- Displays data counters for a node
smpquery switchinfo lid :- Displays very detailed information for a node – intensive troubleshooting
smpquery portinfo lid port (smpquery portinfo 15 23) : - displays information about a port.
ibdiagnet –v –r :- Creates log file with comprehensive diagnostics for the entire fabric.
Ibdiagpath –v –l slid dlid : - Creates a log file with path information between source and destination nodes
Iblinkinfo :- Gives you details about node connections to switch ports.

To map LIDs to GUIDs

1.osmtest –f c –i inventory.txt
2. cat inventory.txt |grep -e '^lid' -e 'port_guid' -e 'desc' |sed 's/^lid/\nlid/'> mapping.txt
3. Edit the latter half of the mapping.txt file to remove the nonessential information.

Single Node Source & Destination Pair Network

Ibv_devinfo ibdiagpath Ibdiagnet Run performance tests
Ibstat Ibtracert Ibnetdiscover /usr/bin/ib_write_bw
Ibportstate Ibv_rc_pingpong Ibhosts /usr/bin/ib_write_lat
Ibroute Ibv_srq_pinpong Ibswitches /usr/bin/ib_read_bw
smpquery Ibv_ud_pingpong saquery /usr/bin/ib_read_lat
prefquery Ib_send_bw sminfo /usr/bin/ib_send_bw
Ib_write_bw smpdump /usr/bin/ib_send_lat
iblinkinfo

- /etc/init.d/openibd status –to check if HCA driver is loaded or not

IPoIB runs in two modes “Datagram” & “Connected” (Default is Connected)

To edit default mode – edit /etc/inifiniband/openib.conf > ‘SET_IPOIB_CM=no’ , restart service openibd
Make sure /sys/class/net/ib0/mode has “datagram” in it.
Network configuration: edit /etc/sysconfig/network/ifcfg-ib0
DEVICE=ib0
BOOTPROTO=static
IPADDR=10.10.0.1
NETMASK=255.255.255.0
NETWORK=10.10.0.0
BROADCAST=10.10.0.255
ONBOOT=yes