Dist-upgrade does not upgrade to Cuda 11.1

post by dkkim93 on Dec 10, 2020

Hello. First of all, thank you to the Lambda team for providing the Lambda stack package!

I would like to install the Cuda 11.1 on my Ubuntu 18.04 with RTX 3090 to run a PyTorch program with GPU.

I followed the instruction and first installed by:

LAMBDA_REPO=$(mktemp) && 
wget -O${LAMBDA_REPO} https://lambdalabs.com/static/misc/lambda-stack-repo.deb && 
sudo dpkg -i ${LAMBDA_REPO} && rm -f ${LAMBDA_REPO} && 
sudo apt-get update && sudo apt-get install -y lambda-stack-cuda
sudo reboot

and then

sudo apt-get update && sudo apt-get dist-upgrade.

However, unfortunately, the installed Cuda version is still 10.0, and I would like to ask for help.

I appreciate your time.

Additional information:

python3                                                                  6s
Python 3.6.9 (default, Oct  8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.4.0'

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

nvidia-smi
Thu Dec 10 09:33:53 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01    Driver Version: 455.45.01    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    On   | 00000000:01:00.0  On |                  N/A |
| 32%   34C    P8    35W / 350W |    360MiB / 24265MiB |     31%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1260      G   /usr/lib/xorg/Xorg                164MiB |
|    0   N/A  N/A      1405      G   /usr/bin/gnome-shell               41MiB |
|    0   N/A  N/A      1902      G   ...AAAAAAAA== --shared-files      150MiB |
+-----------------------------------------------------------------------------+

post by jeremy on Dec 14, 2020

The latest version of the CUDA toolkit we support on Ubuntu 18.04 is version 10.0.130.

If you want to use version 11.1, you’ll need to upgrade to Ubuntu 20.04.

You can do that like so:

sudo sed -i '/^AutomaticLoginEnable/ s/true/false/' /etc/gdm3/custom.conf &&\
sudo apt -y update && sudo apt -y dist-upgrade && sudo do-release-upgrade -d -f DistUpgradeViewNonInteractive

Then, once the release upgrade completes, and the machine reboots, run the following to reinstall Lambda Stack:

LAMBDA_REPO=$(mktemp) && 
    wget -O${LAMBDA_REPO} https://lambdalabs.com/static/misc/lambda-stack-repo.deb && 
    sudo dpkg -i ${LAMBDA_REPO} && rm -f ${LAMBDA_REPO} && 
    sudo apt-get update && sudo apt-get install -y lambda-stack-cuda

post by Karun on May 3, 2022

After I upgraded (with several hoops and tries) to Ubuntu 20.04 Desktop (from 18.04) via

sudo sed -i ‘/^AutomaticLoginEnable/ s/true/false/’ /etc/gdm3/custom.conf &&
sudo apt -y update && sudo apt -y dist-upgrade && sudo do-release-upgrade -d -f DistUpgradeViewNonInteractive

And then rebooted and installed lambda stack (again I had to try a few times) via

I have two issues:

1. I am unable to run the command ‘nvidia-smi’ as I am getting communication error as shown below.

“NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.”

So when I want to install

sudo apt install nvidia-prime

But it is failing with problem in not able to install lambda-desktop : Depends: python-is-python3 but it is not going to be installed

You might want to run ‘apt --fix-broken install’ to correct these.

The following packages have unmet dependencies:

lambda-desktop : Depends: python-is-python3 but it is not going to be installed

E: Unmet dependencies. Try ‘apt --fix-broken install’ with no packages (or specify a solution).

post by Karun on May 4, 2022

Just a hunch, but I’m thinking there still might be some python*-*dev packages installed that are blocking this.

Can you send me the output of:

dpkg -l | grep python

dpkg -l | grep python | grep minimal

Hi Matt,

I removed the offending python-minimal 2.7.15~rc1-1 amd64 minimal subset of the Python language (default version in /var/lib/dpkg/status as per this ubuntu - Unable to install python-dev - Stack Overflow

and did sudo apt-get install -f python-is-python3.

Now I cleaned config via sudo dpkg --configure -a and sudo apt --fix-broken install and all went well.

And now it seems straightened. Is it safe to follow the clean install that you recommended again?

Now nvidia-smi is working as expected.

Thanks