Problem with tensorflow in virtual environment

I have an issue with running tensorflow 2.4 in virtual env on workstation with ubuntu 18.04.

The base lambda stack installation is with tensorflow-gpu 1.15. But I need a 2.x tensorflow for testing. So I created a virtual env and installed tensorflow manually with commands:

python3 -m venv lambda-stack-without-tensorflow
source lambda-stack-without-tensorflow/bin/activate
pip install tensorflow-gpu

I have tested it with a minimal example and getting the error:

tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library ‘libcudart.so.11.0’; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory

the nvidia setup is:

• driver version: 460.39
• cuda version: 11.2

What to do? How to get it work?

I also have this issue, using pyenv. Specifically, the system tensorflow does this:

import tensorflow as tf

2021-03-04 02:09:28.476244: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0

2021-03-04 02:09:28.478779: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0

tf.__version__

‘2.4.1’

tf.test.is_gpu_available()

WARNING:tensorflow:From /tensorflow/python/framework/test_util.py:1: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.

Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.

2021-03-04 02:09:40.285870: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set

I can see that it’s trying to load libcusolver.so.10 instead of 11, but why does libcudnn.so.8 fail?

Moreover, where is libcudnn?

ldconfig -p | grep libcudnn

returns nothing.

Breaking news:

It’s here:

/usr/lib/python3/dist-packages/tensorflow

Yes CuDNN is with tensorflow and pytorch, and it is the specific version it is built with.

Pytorch: /usr/lib/python3/dist-packages/torch/lib/libcudnn.so.8

Tensorflow: /usr/lib/python3/dist-packages/tensorflow/libcudnn.so.8

And yes you can install a version from NVIDIA after registration and the EULA:

CUDA Deep Neural Network (cuDNN) | NVIDIA Developer

See: Lambda Stack: an AI software stack that's always up-to-date

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/libcudnn8_8.1.1.33-1+cuda11.2_amd64.deb
sudo dpkg -i libcudnn8_8.1.1.33-1+cuda11.2_amd64.deb