# Getting unhandled cuda error when trying to use TF distributed strategy for training on 4 GPUs

## post by ivogeorg on Jun 11, 2020

**ivogeorg**  
Jun 2020

## Brief

Attempted to run distributed training on all 4 GPUs for the first time, using TF’s simple MirroredStrategy (which uses NCCL all-reduce), and immediately got an unhandled cuda error in nccl_ops.cc. (see brief error log below) Due to another error described [here](https://deeptalk.lambda.ai/t/started-getting-tensorflow-profiler-error-after-cuda-update/1676), which causes the notebook kernel to restart, I cannot at this time reproduce the error to provide a longer log.

## Configuration

- Lambda QUAD Titan V (2018)  
- Lambda Stack  
- Ubuntu 18.04  
- Anaconda (conda 4.8.3) environment with TensorFlow 2.2.0  
- Running [this notebook](https://github.com/tensorflow/docs/blob/master/site/en/tutorials/distribute/keras.ipynb) in jupyter notebook

## Error log (short)

```swift

2020-06-10 13:36:56.319859: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-06-10 13:36:57.698341: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-10 13:37:04.630742: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at nccl_ops.cc:104 : Unknown: Error invoking NCCL: unhandled cuda error
```

### Note

I can’t reproduce the error to provide a longer log, because of the newly occurring error described in [Started getting TensorFlow profiler error after CUDA update](https://deeptalk.lambda.ai/t/started-getting-tensorflow-profiler-error-after-cuda-update/1676), which causes the kernel to be restarted.

## Possibly solved

After fixing [this other error](https://deeptalk.lambda.ai/t/started-getting-tensorflow-profiler-error-after-cuda-update/1676) which was blocking replication of this one, this error has also gone away. It is possible that the software updates shown in the other error description were the solution.