Hyperplane-16 InfiniBand Cluster Total Cost of Ownership Analysis

April 7, 2020• 8 min read

In this post we'll walk through using our Total Cost of Ownership (TCO) calculator to examine the cost of a variety of Lambda Hyperplane-16 clusters. We have the option to include 100 Gb/s EDR InfiniBand networking, storage servers, and complete rack-stack-label-cable service. The purpose of this post is to give you a clearer picture of the costs involved when building a cluster, both fixed and variable.

A TCO Calculator

All results in this post are generated using our Hyperplane-16 TCO Calculator. A Lambda engineer can walk you through how to use the TCO Calculator. Just send us an email: sales@lambdalabs.com to request a walkthrough.

Trying out different configurations

Using the above calculator and the guidance of a Lambda engineer, we're able to quickly create a table showing the amortized cost / year / node of an InfiniBand networked Hyperplane-16 cluster. As you can see, the prices have automatically adjusted based on the costs and system specifications that we input into the calculator: in our case:

• InfiniBand networking set to TRUE
• system administration cost of $10,000 / year / four servers
• management node count to 0
• storage node count to 0

A quick introduction to the Hyperplane-16

The Hyperplane-16 is a massive 10kW Deep Learning training appliance from Lambda. It includes:

• 16x NVIDIA Tesla V100 SXM3 GPUs
• NVLink & NVSwitch for fast GPU-to-GPU communication within the server
• 8x 100 Gb/s InfiniBand cards for fast GPU-to-GPU communication across multiple servers (using GPU-Direct RDMA) during distributed training

The default system price used in the TCO calculator is $240,000. This is the price for a Lambda Hyperplane-16 Premium, an HGX-2 platform configuration that exactly matches the specifications of the DGX-2.

For a more comprehensive hardware overview, you can check out the Hyperplane-16 landing page or see our Hyperplane-16 announcement blog post. We calculate the total cost of ownership for a complete Hyperplane-16 cluster including:

• Lambda Hyperplane-16s
• Sufficient 100 Gb/s EDR InfiniBand network (Mellanox MSB7800) + 10GBase-T Network (Arista 7050Tx-48)
• Some number of Lambda NVMe storage servers
• Some number of Lambda management nodes
• Sufficient Racks, PDUs, rack crates
• Complete cluster-level rack, stack, label, and cable services from Lambda
• Complete co-location costs & ongoing system administration costs

3x Hyperplane-16 Cluster Summary

Here's a cluster and TCO summary for a 3x Hyperplane-16 cluster:

Cluster TCO Summary

How we conducted the TCO analysis

1. Our first step is to design the cluster. We've done that step for you.
2. We then list out all components of the designed cluster: nodes, InfiniBand HCAs, switches, cables, PDUs, racks, rack crates, etc. We then assign a price to all components. This document is known as our "priced bill of materials" or "priced BoM".
3. We then assign a power draw to all IT equipment in the cluster, this is our "Cluster kW draw".
4. We dynamically modify the rack, switch, and cable counts to accommodate the number of nodes.
5. We assign a price to the annual salary for a data center system administrator.
6. We assign a $ / kW / mo figure to the co-location service. In this TCO, we use $260 / kW / month. We multiply this with our Cluster kW draw to arrive at our monthly co-location bill. We assume that you will be at near 100% utilization.

We can now calculate the TCO as:

Cluster kW draw (power) = Sum of all component TDPs in the cluster

Annual Co-location cost ($) = $/kW/mo * Cluster kW draw * 12 months

TCO for N Years ($) = N * (Annual Co-location Cost + Annual  Administration Cost) + Total Upfront Cost

Generating a Network Topology

Within the TCO calculator, we dynamically generate a 100% port-to-port bandwidth spine & leaf network topology based on the components in the cluster. An example spine & leaf network topology looks like this:

A non-blocking 100% port-to-port bandwidth InfiniBand spine and leaf network topology with MSB7800 switches. As the number of nodes in the cluster model is increased, this topology is expanded to accommodate for additional usable ports. Note: as the number of nodes approaches 20, we would recommend switching to a CS7520 director switch instead of a manual spine & leaf topology using multiple MSB7800s. The CS7520 is a director switch that can provide full port-to-port bandwidth for up to 216-ports. It has an internal backplane that allows it to create a similar spine and leaf topology as seen above but without any additional cables.

Generated Cluster Priced BOM