Running a PyTorch®-based benchmark on an NVIDIA GH200 instance - Lambda Docs

Running a PyTorch®-based benchmark on an NVIDIA GH200 instance

This tutorial describes how to run an NGC-based benchmark on an On-Demand Cloud (ODC) instance backed with the NVIDIA GH200 Grace Hopper Superchip. The tutorial also outlines how to run the benchmark on other ODC instance types to compare performance. The benchmark uses a variety of PyTorch® examples from NVIDIA's Deep Learning Examples repository.

Prerequisites

To run this tutorial successfully, you'll need the following:

Setting up your environment

Launch your GH200 instance

Begin by launching a GH200 instance:

  1. In the Lambda Cloud console, navigate to the SSH keys page, click Add SSH Key, and then add or generate a SSH key.
  2. Navigate to the Instances page and click Launch Instance.
  3. Follow the steps in the instance launch wizard.
    • Instance type: Select 1x GH200 (96 GB).
    • Region: Select an available region.
    • Filesystem: Don't attach a filesystem.
    • SSH key: Use the key you created in step 1.
  4. Click Launch instance.
  5. Review the EULAs. If you agree to them, click I agree to the above to start launching your new instance. Instances can take up to five minutes to fully launch.

Set the required environment variables

Next, set the environment variables you need to run the benchmark:

  1. In the Lambda Cloud console, navigate to the Instances page, find the row for your instance, and then click Launch in the Cloud IDE column. JupyterLab opens in a new window.
  2. In JupyterLab's Launcher tab, under Other, click Terminal to open a new terminal.
  3. Open your .bashrc file for editing:
   nano ~/.bashrc
  1. At the bottom of the file, set the following environment variables. Replace <GIT-USERNAME> with your GitHub username and <GIT-EMAIL> with the email address associated with your GitHub account:
   export NAME_NGC=pytorch:24.10-py3
   export NAME_TYPE=ODC
   export NAME_DATASET=all
   export NAME_TASKS=all
   export NUM_GPU=1
   export NAME_GPU=GH200_96GB
   export NAME_RESULTS=gh200_benchmark_results
   export GIT_USERNAME=<GIT-USERNAME>
   export GIT_EMAIL=<GIT-EMAIL>

  1. Save and exit.

  2. Update your environment with your new environment variables:

    source ~/.bashrc

Running the GH200 benchmark

Run the benchmark

Now that you've set up your environment, you can run the benchmark on your GH200 instance:

  1. In your web browser, navigate to the lambdal/deeplearning-benchmark repository on GitHub and then fork the repository. By using your own fork instead of the original repository, you'll be able to push your benchmark results to a single location.

  2. In your ODC instance's JupyterLab terminal, pull the NGC PyTorch® Docker image:

    sudo docker pull nvcr.io/nvidia/${NAME_NGC} 2>&1

  3. Clone the LambdaLabsML/DeepLearningExamples repository, check out its lambda/benchmark branch, and then clone your forked repository:

    git clone https://github.com/LambdaLabsML/DeepLearningExamples.git &&
cd DeepLearningExamples &&
git checkout lambda/benchmark &&
cd .. &&
git clone https://github.com/${GIT_USERNAME}/deeplearning-benchmark.git

  4. Navigate to the pytorch directory, and then download and prepare the dataset that the benchmark will use. This step might take up to 20 minutes to complete:

    cd deeplearning-benchmark/pytorch &&
mkdir ~/data &&
sudo docker run --gpus all --rm --shm-size=256g \
   -v ~/DeepLearningExamples/PyTorch:/workspace/benchmark \
   -v ~/data:/data \
   -v $(pwd)"/scripts":/scripts \
nvcr.io/nvidia/${NAME_NGC} \
/bin/bash -c "cp -r /scripts/* /workspace; ./run_prepare.sh ${NAME_DATASET}"

  5. Create a PyTorch® configuration file for the benchmark:

    cp scripts/config_v2/config_pytorch_${NUM_GPU}x${NAME_GPU}_v2.sh scripts/config_v2/config_pytorch_${NAME_TYPE}_${NUM_GPU}x${NAME_GPU}_$(hostname)_v2.sh

  6. Create a new directory named gh200_benchmark_results to store the benchmark results in:

    mkdir -p ${NAME_RESULTS}

  7. Run the benchmark:

    sudo docker run --rm --shm-size=1024g \
   --gpus all \
   -v ~/DeepLearningExamples/PyTorch:/workspace/benchmark \
   -v ~/data:/data \
   -v $(pwd)"/scripts":/scripts \
   -v $(pwd)/${NAME_RESULTS}:/results \
nvcr.io/nvidia/${NAME_NGC} \
/bin/bash -c "cp -r /scripts/* /workspace; ./run_benchmark.sh ${NAME_TYPE}_${NUM_GPU}x${NAME_GPU}_$(hostname)_v2 ${NAME_TASKS} 3000"

Compile the results to CSV

When the benchmark completes, it publishes the results to a subdirectory of your results_v2 directory. You can compile a summary of these results to CSV by running the following scripts from the pytorch folder:

   python scripts/compile_results_pytorch_v2.py --path ${NAME_RESULTS} --precision fp32 &&
   python scripts/compile_results_pytorch_v2.py --path ${NAME_RESULTS} --precision fp16

The resulting CSV files appear in the pytorch directory:

   pytorch-train-throughput-v2-fp16.csv
   pytorch-train-throughput-v2-fp32.csv

Push the results to GitHub

Finally, push the results to your GitHub repository:

  1. In your web browser, log into GitHub and create a new fine-grained personal access token with the following configuration. Make sure to copy the token and paste it somewhere safe for future use:

    • Token name: GH200 benchmarking
    • Respository access: Select Only select repositories, and then select your deeplearning-benchmark fork from the dropdown.
    • Permissions: Under Repository permissions, set Contents to Read and write.

  2. In your terminal in JupyterLab, set an environment variable for your GitHub token. Replace <GIT-TOKEN> with the personal access token you created in step 1:

    export GIT_TOKEN=<GIT-TOKEN>

  3. Configure your Git credentials:

    git config --global user.name "${GIT_USERNAME}" &&
git config --global user.email "${GIT_EMAIL}"

  4. Navigate to the pytorch directory, and then fetch the latest changes from the repo's master branch and merge them into your current branch:

    cd ~/deeplearning-benchmark/pytorch &&
git pull origin master

  5. Navigate to the pytorch directory and then commit your results:

    git add scripts/config_v2 &&
git add ${NAME_RESULTS} &&
git add pytorch-train-throughput-v2-fp16.csv &&
git add pytorch-train-throughput-v2-fp32.csv &&
git commit -m "Update from $(hostname)"

  6. Set origin to your forked repository:

    git remote set-url origin https://${GIT_USERNAME}:${GIT_TOKEN}@github.com/${GIT_USERNAME}/deeplearning-benchmark.git

  7. Push the results to your forked repository:

    git push origin master

That's it! To see your benchmarks, navigate to the pytorch folder in your forked repo on GitHub, and then click on one of the two CSV files. GitHub renders your CSV files in an easy-to-scan tabular format by default.

Running the benchmark on other instance types

Without other data to compare it to, your GH200 benchmark data is of limited use. You can run the benchmark on other ODC instance types by modifying the instructions in the Setting up your environment section above. Make the following changes:

   <GPU-TYPE>_<GPU-VRAM>GB_<GPU-CONNECTION-TYPE>

For example, if you're planning to benchmark an 8xH100 80GB SXM instance, you should set NAME_GPU and NUM_GPU as follows:

   export NAME_GPU=H100_80GB_SXM
   export NUM_GPU=8

Important

If the instance type doesn't explicitly state a GPU connection type, omit _<GPU-CONNECTION-TYPE> from the naming pattern.

After you've launched and set up your instance, you can run the instructions in the Running the GH200 benchmark section as normal. As before, you can compare results by clicking on each of the CSV file you generated.

Cleaning up

When you're done with your instances, terminate them to avoid incurring unnecessary costs:

  1. In the Lambda Cloud console, navigate to the Instances page.
  2. Select the checkboxes of the instances you want to delete.
  3. Click Terminate. A dialog appears.
  4. Follow the instructions and then click Terminate instances to terminate your instances.

Next steps