Installing and Configuring BYNET Software for InfiniBand - Data Stream Architecture - Data Stream Extensions

Data Stream Extensions Installation, Configuration, and Upgrade Guide for Customers

Product
Data Stream Architecture
Data Stream Extensions
Release Number
16.00
Published
November 2016
Language
English (United States)
Last Update
2018-04-17
dita:mapPath
rte1474305422984.ditamap
dita:ditavalPath
3151_DSE_ICUCustomer_1600.ditaval.ditaval
dita:id
B035-3151
lifecycle
previous
Product Category
Software
Teradata Tools and Utilities
The SM3G node software must be installed before the BYNET software to ensure proper PMA ID and node type assignment. Server Management Web Services software, version 11.01.00 or later, is required on the system.
The following procedure contains steps for downloading the BYNET driver and the other software used for BYNET InfiniBand adapters. All nodes, managed servers, and managed storage servers in a system must use BYNET version 3.6.4.1 or later. The BYNET version on the nodes and servers must be compatible. To be compatible, the first three numbers must match, but the patch number can be different. For example, 3.6.4.1 and 3.6.4.2 are compatible.
Make sure you have downloaded the latest software packages from the Teradata Software Server at http://tssprod.teradata.com:8080/TSFS/. Refer to the TSS User Manual on the website for instructions.
The software packages used for BYNET InfiniBand adapters on managed servers and managed storage servers are:
  • BYNET (latest compatible version)
  • ibutils2
  • libibumad
  • libibmad
  • opensm-libs
  • opensm
  • infiniband-diags
  • mstflint
  • teradata-connectx-3-fw
  1. Verify that an /etc/opt/teradata/sm3g/smainfo exists on the server. Sample contents of a server smainfo file:
    PMA 72
    BYNETIP 39.64.16.8
    BYNETMASK 255.240.0.0
    BYNET1IP 39.0.16.8
    BYNET1MASK 255.240.0.0
    The smainfo file is generated for the server by server management and must exist before installing the BYNET driver. The file should not contain a SYSTEM entry, but must contain the following:
    • A unique PMAID (PMA)
    • Unique BYNET IP addresses for BYNET Domain 0 (BYNETIP and BYNETMASK entries)
    • Unique BYNET IP addresses for BYNET Domain 1 (BYNET1IP and BYNET1MASK entries)
  2. Type the following command to create the /etc/opt/teradata/bynet directory: mkdir -p /etc/opt/teradata/bynet
  3. Create an /etc/opt/teradata/bynet/netdev file.
  4. In the netdev file, specify local loopback for BYNET Domain 0 and the network interfaces to be used for BYNET Domain 1. For example:
    lo domain 0
    mlx4_0_1 domain 1
    mlx4_0_2 domain 1			
    
  5. Install the InfiniBand software packages which consist of the BYNET driver, InfiniBand utilities, and adapter firmware:
    1. Use PUT's Install/Upgrade Software operation to install the packages.
    2. Choose the directory containing the Teradata Software Server downloads as the Available Package Source(s) during PUT's Media Source(s) step. If PUT does not auto-select the latest package for installation, manually select it.
  6. Reboot the server.
  7. Type the following command to verify both InfiniBand adapter ports are functioning properly: ibinfo -d local_status Sample command response:
    channel                            logical physical link
    name    guid              port lid state   state    speed
    =====   ============      ===  === =====   =====    =====
    mlx4_0  0002c90300a11c41  1    0x6 ACTIVE  LinkUp   4XQDR
    mlx4_0  0002c90300a11c42  2    0x7 ACTIVE  LinkUp   4XQDR
    
    In the example above, link speed may also show 4xFDR10, depending on the model of InfiniBand switch.
  8. Type bam -s in the command line and verify the following:
    • Domain 0 row: The Net name column contains lo, which identifies a server using local loopback for BYNET Domain 0.
    • Domain 1 rows:
      • The Nodes column shows the correct number of nodes and servers on each BYNET network interface of BYNET Domain 1.
      • The number of nodes and servers in the Nodes column equals the number of routes in the Routes column for each BYNET network interface of BYNET Domain 1.
    Sample command response.
    Version information:
    03.06.04.00 BLM commands
    03.06.04.00 BLM driver
    13.09.18.10 BLM protocol
    
    Node is running in protocol emulation mode.
    
    Domain  System  State
    0       1       attached
    1       -       attached
    
    Domain  Nodes  Routes  Net name
    0       1      1       lo
    1       8      8       mlx4_0_1
    1       8      8       mlx4_0_2
    
  9. Type the following commands to verify that the BYNET Domain 0 (byn0) and BYNET Domain 1 (byn1) interfaces on the server are configured with the IP address and netmask defined in the /etc/opt/teradata/sm3g/smainfo file. # ifconfig byn0 # ifconfig byn1 Sample system response:
    # ifconfig byn0
    byn0       Link encap:Ethernet HWaddr 00:00:27:40:10:c1
               inet addr:39.64.16.8  Bcast:39.79.255.255 Mask:255.240.0.0
               inet6 addr: fe80::200:27ff:fe40:10c1/64 Scope:Link
               UP BROADCAST RUNNING MTU:65520 Metric:1
               RX packets:0 errors:0 dropped:0 overruns:0 frame:0
               TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
               collisions:0 txqueuelen:1000
               RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
    
    # ifconfig byn1
    byn1       Link encap:Ethernet HWaddr 00:00:27:20:08:01
               inet addr:39.0.16.8  Bcast:39.63.255.255 Mask:255.240.0.0
               inet6 addr: fe80::200:27ff:fe20:801/64 Scope:Link
               UP BROADCAST RUNNING MTU:65520 Metric:1
               RX packets:0 errors:0 dropped:0 overruns:0 frame:0
               TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
               collisions:0 txqueuelen:1000
               RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
    
  10. Use the following command to ping every node and server on the BYNET Domain 1 (byn1) interface: ping -c 1 IP address For example, if you ping 39.0.16.9, the system response should be similar to the following:
    # ping -c 1 39.0.16.9
    PING 39.0.16.9 (39.0.16.9) 56(84) bytes of data.
    64 bytes from 39.0.16.9: icmp_seq=1 ttl=64 time=0.011 ms
    
    --- 39.0.16.9  ping statistics---
    1 packets transmitted, 1 received, 0% packet loss, time 1 ms
    rtt min/avg/max/mdev = 0.029/3.590/7.151/3.561 ms
    
  11. As appropriate, add BYNET entries to the /etc/hosts file: Sample BYNET host file entries:
    39.0.8.8   uda001-8
    39.0.8.9   uda001-9
    39.0.8.10  uda001-10
    39.0.16.8  uda002-8
    39.0.16.9  uda002-9
    39.0.16.10 uda002-10
    /etc/hosts changes take effect immediately.