Potential Lambda Stack bugs

post by md27182 on Mar 13, 2021

Wasn’t sure where to post bugs for the Lambda Stack, so I thought I’d post here.

After installing Lambda Stack using the commands from this blog post on Ubuntu 20.04 LTS, you end up with a virtualenv installation that is broken. Typing the virtualenv command into the terminal gives

ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'

I figured out that this was coming from the package python3-virtualenv. After removing it with

sudo apt-get purge python3-virtualenv

things started working again.

post by markd on Mar 15, 2021

Thank you! I will take a look and make sure it gets to the correct people.

I just tried it on a machine I installed this morning and did not see a problem.

virtualenv only comes back with the normal usage output. So I tried some other commands.

mark@lamda-dual:~$ sudo apt-get install python3-venv

...

mark@lamda-dual:~/python-virtual-environments$ dpkg --list | grep virtualenv

ii python3-virtualenv 20.0.17-1 all Python virtual environment creator

ii virtualenv 20.0.17-1 all Python virtual environment creator

mark@lamda-dual:~/python-virtual-environments$ virtualenv --version

virtualenv 20.0.17 from /usr/lib/python3/dist-packages/virtualenv/__init__.py

mark@lamda-dual:~/python-virtual-environments$ python3 -m venv env

mark@lamda-dual:~/python-virtual-environments$ source env/bin/activate

(env) mark@lamda-dual:~/python-virtual-environments$ deactivate

mark@lamda-dual:~/python-virtual-environments$

Mark

post by md27182 on Mar 16, 2021

There’s always a chance that something other than lambda stack was causing the issue. I also installed pipenv so maybe that caused some kind of conflict?

Actually, I ended up having another issue with my lambda stack installation. Normally when I run a python script that uses tensorflow, I see an output like this:

2021-03-16 02:56:41.367083: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-03-16 02:56:41.367103: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-03-16 02:56:41.367111: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-03-16 02:56:41.367120: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-03-16 02:56:41.367128: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-03-16 02:56:41.367134: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-03-16 02:56:41.367141: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-03-16 02:56:41.367149: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8

However, when I ran it using the lambda stack, it was unable to find one of these files, and so GPU usage was disabled. I don’t remember which one was missing. Same thing happened when I tried to use tensorflow with a couple of the official NVIDIA docker images. Finally, I tried the google docker images and I haven’t had any issues since.

I saw a lot of people online saying that certain tensorflow versions are incompatible with certain CUDA versions and that this was often the cause of these kinds of errors. I was surprised to see lambda stack include tensorflow 2.3 and CUDA 11.1, which from what I’ve seen online, seem to be incompatible.

I’m sure your team has thought this all through, but I just wanted to share my experience in case it’s helpful.

post by markd on Mar 17, 2021

Yes, thank you for the update. I will test that also.