The IntelliFlex system can accommodate up to 36 groups using four pairs of second-tier 36-port switches. This allows for up to 468 nodes.
You can replace a Mellanox SB7800 36-port switch with an Nvidia/Mellanox QM8700 40-port switch.
System Network Topology for a 36-Group IntelliFlex System
- The system consists of up to 36 groups.
- The first cabinet contains IntelliFlex processing nodes.
- The other cabinets may also contain IntelliFlex processing nodes.
- The first cabinet contains a system VMS for Server Management, and may contain additional TMSs.
- A cabinet VMS for Server Management is added for every 50 servers.
- Each clique forms a group of up to 13 nodes. The nodes in the group share the first-tier switches in the first cabinet of the pair.
- Four ports on each of the first-tier InfiniBand switches in each group are used to provide the uplink connections to the second-tier switches.
- The system uses four sets of 36-port second-tier switches. These second-tier switches are located in cabinet 2, cabinet 4, cabinet 6, and cabinet 8.
- Of the four uplink connections to the second-tier switches, one connects to each second-tier switch (for each fabric).
- When connected to a UDA network, the second-tier switches can be used for UDA connections. Distribute the UDA connections evenly across the pairs of second-tier switches for best performance.