GPU not used, rebooted, nvidia-smi has failed

post by kelvin on Apr 7, 2021

For some reason, tensorflow is not using GPU even it’s showing in the device list, and we rebooted the Lambda. Then:

nvidia-smi has failed because it couldn’t communicate with the NVIDIA driver.

Anyway to fix it?

output of dpkg -l |grep nvidia

ii libnvidia-cfg1-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA binary OpenGL/GLX configuration library ii libnvidia-common-450 450.102.04-0ubuntu0.18.04.1 all Shared files used by the NVIDIA libraries ii libnvidia-compute-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA libcompute package ii libnvidia-compute-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVIDIA libcompute package ii libnvidia-decode-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA Video Decoding runtime libraries ii libnvidia-decode-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVIDIA Video Decoding runtime libraries ii libnvidia-encode-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVENC Video Encoding runtime library ii libnvidia-encode-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVENC Video Encoding runtime library ii libnvidia-extra-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 Extra libraries for the NVIDIA driver ii libnvidia-fbc1-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library ii libnvidia-fbc1-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVIDIA OpenGL-based Framebuffer Capture runtime library ii libnvidia-gl-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD ii libnvidia-gl-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD ii libnvidia-ifr1-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA OpenGL-based Inband Frame Readback runtime library ii libnvidia-ifr1-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVIDIA OpenGL-based Inband Frame Readback runtime library ii nvidia-compute-utils-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA compute utilities ii nvidia-cuda-dev:amd64 10.0.130-0lambda3 amd64 CUDA development files ii nvidia-cuda-doc 10.0.130-0lambda3 all CUDA toolkit documentation ii nvidia-cuda-gdb 10.0.130-0lambda3 amd64 CUDA Debugger ii nvidia-cuda-toolkit 10.0.130-0lambda3 amd64 CUDA development toolkit ii nvidia-dkms-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA DKMS package ii nvidia-driver-440 450.66-0lambda0~18.04.2 amd64 Transitional package for nvidia-driver-450 ii nvidia-driver-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA driver metapackage ii nvidia-kernel-common-450 450.102.04-0ubuntu0.18.04.1 amd64 Shared files used with the kernel module ii nvidia-kernel-source-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA kernel source package ii nvidia-modprobe 455.45.01-0lambda1 amd64 utility to load NVIDIA kernel modules and create device nodes ii nvidia-profiler 10.0.130-0lambda3 amd64 NVIDIA CUDA profiler ii nvidia-settings 455.45.01-0lambda1 amd64 Tool for configuring the NVIDIA graphics driver ii nvidia-utils-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA driver support binaries ii xserver-xorg-video-nvidia-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA binary Xorg driver