Pytorch sometimes fails to recognize GPU

post by sgh42 on Sep 1, 2020

I’m using PyTorch on a Tensorbook, and sometimes it fails to see the GPU. That is, it becomes true that torch.cuda.is_available() == False. So far, I’ve found that rebooting reliably fixes the problem, but that’s of course not a very satisfactory long-term approach. Some googling indicates that the usual way of fixing the problem is to change the versions of CUDA libraries or other components (by reinstalling them). I’d rather not do this as I’m using the lambda stack and I don’t want to then have a non-standard installation. Is anyone else seeing this? I can post versions of various components if that’s helpful.

• Lambda Tensorbook - Unable to recognize GPU with PyTorch

post by sabalaba on Sep 10, 2020

Have you done any other installations alongside your Lambda Stack install? You can always email support@lambdalabs.com to report a Lambda Stack bug. Not sure if it’s a bug on our end. Is it a clean Ubuntu 20.04 LTS + Lambda Stack install?

post by sgh42 on Sep 14, 2020

Thanks. Yes, it’s a brand-new Tensorbook. I installed my project using conda so it has local (to the project) versions of python (3.7.7) and pytorch (1.6.0). My conda environment doesn’t do anything with CUDA (I rely on the pytorch official Docker images when I deploy to the cloud).

post by sabalaba on Sep 18, 2020

Do you have this problem when you’re outside of conda and use the Lambda Stack versions of pytorch?

post by sgh42 on Sep 18, 2020

I do see the same with /usr/bin/python, is that the correct python? I’m not sure where Lamba Stack installs stuff.

post by sabalaba on Sep 19, 2020

Can you try updating to the latest version of Lambda Stack with this command:

sudo apt-get update && sudo apt-get upgrade -y

post by sgh42 on Sep 23, 2020

Thanks. I’ve done that, it didn’t immediately fix the problem (CUDA was still unavailable) so I rebooted. I’ll let you know if the problem reoccurs post-reboot.

post by sabalaba on Sep 27, 2020

Did this fix the problem for you? Would be great to know!

post by sgh42 on Sep 28, 2020

Unfortunately, I did the updates and it didn’t fix the problem :(, I still have to reboot sometimes to “reactivate” CUDA wrt PyTorch.