# NVLink seems to prevent PyTorch training loop from starting

## post by mnazemi on Jun 10, 2022

I have a Lambda server with eight A6000 GPUs and NVLink.

When I utilize PyTorch’s distributed data parallel (DDP) to train my models with two GPUs, NVLink is used successfully based on the performance counters. However, as soon as I increase the number of GPUs to three or more (up to eight), the training loop gets stuck at the very beginning (or possibly the first backward pass).

If I set the environment variable `NCCL_P2P_DISABLE=1`, I can use as many GPUs as I like, but I obviously don’t get the benefits of NVLink.

If I set `NCCL_DEBUG=INFO`, I get the following output:

```
...NCCL output...
```

I was wondering if anyone has had a similar issue and knows how to resolve it.

Thanks!

## post by markd on Jun 14, 2022

If it is getting ‘stuck’ or ‘hanging’ do you have AMD CPUs?

And do you have the kernel command line set to iommu=soft?

Another issue I have seen was cross hosts and there were processes already running on the other GPU. Sometimes it took a little while for the previous job cleanup. (nvidia-smi to see what is running on the GPUs).

I do not use the NCCL_P2P_DISABLE=1

I normally run the DDP, NCCL test with:

`NCCL_DEBUG=INFO NCCL_ALGO=Ring NCCL_NET_GDR_LEVEL=4 python …`

To see your nvlink topology:

```bash
$ nvidia-smi topo -m
```

And to see the nvlink status:

```bash
$ nvidia-smi nvlink -s
```

You can use ‘CUDA_VISIBLE_DEVICES=0,1’ to run on the first two GPUs or ‘CUDA_VISIBLE_DEVICES=1,2’ to run on the second and third devices (if you are trying to compare).

You can see that with:

```bash
$ cat /proc/cmdline
```

To fix it you can:

```bash
$ sudo sed -iE ‘/^GRUB_CMDLINE.*DEFAULT/ s/"$/ iommu=soft"/’ /etc/default/grub
$ sudo update-grub
$ sudo reboot
```

## post by mnazemi on Jun 15, 2022

Thank you for your detailed explanation. I should have mentioned that I use AMD CPUs and I don’t see this issue when using two GPUs.

The problem was indeed resolved by setting `iommu=soft`.

Given your explanation, I found a similar solution that relied on setting `pci=noats`, but it didn’t work for me.

It appears that there is currently no way to use the hardware MMU to get better performance.

Ironically, I don’t get a noticeable speedup when setting `iommu=soft` and `NCCL_DEBUG=INFO NCCL_ALGO=Ring NCCL_NET_GDR_LEVEL=4`. Here is the output of `nvidia-smi topo -m`:

```
...nvidia-smi output...
```