# Lambda Vector 4x Quadro RTX 6000 Only See 3 GPUs ## post by jwic on Aug 24, 2021 ``` Looks like we may be seeing an IRQ conflict? How would we go about resolving this? ``` Thanks! ``` from syslog: Aug 24 17:23:03 dc3dsgpu-app001 kernel: [ 926.881151] genirq: Flags mismatch irq 263. 00000080 (nvidia) vs. 00000000 (nvidia) Aug 24 17:23:03 dc3dsgpu-app001 kernel: [ 926.881186] NVRM: GPU 0000:61:00.0: Tried to get IRQ 263, but another driver Aug 24 17:23:03 dc3dsgpu-app001 kernel: [ 926.881188] NVRM: has it and is not sharing it. Aug 24 17:23:03 dc3dsgpu-app001 kernel: [ 926.881189] NVRM: You may want to verify that no audio driver is using the IRQ. Aug 24 17:23:03 dc3dsgpu-app001 kernel: [ 926.881191] NVRM: GPU 0000:61:00.0: request_irq() failed (-16) Aug 24 17:23:03 dc3dsgpu-app001 kernel: [ 926.881253] genirq: Flags mismatch irq 263. 00000080 (nvidia) vs. 00000000 (nvidia) Aug 24 17:23:03 dc3dsgpu-app001 kernel: [ 926.881267] NVRM: GPU 0000:61:00.0: Tried to get IRQ 263, but another driver Aug 24 17:23:03 dc3dsgpu-app001 kernel: [ 926.881269] NVRM: has it and is not sharing it. Aug 24 17:23:03 dc3dsgpu-app001 kernel: [ 926.881269] NVRM: You may want to verify that no audio driver is using the IRQ. Aug 24 17:23:03 dc3dsgpu-app001 kernel: [ 926.881271] NVRM: GPU 0000:61:00.0: request_irq() failed (-16) ``` +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.63.01 Driver Version: 470.63.01 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro RTX 6000 On | 00000000:01:00.0 Off | Off | | 33% 28C P8 13W / 260W | 5MiB / 24220MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 Quadro RTX 6000 On | 00000000:2E:00.0 Off | Off | | 33% 35C P8 13W / 260W | 5MiB / 24220MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 2 Quadro RTX 6000 On | 00000000:41:00.0 Off | Off | | 34% 38C P8 11W / 260W | 73MiB / 24217MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 2375 G /usr/lib/xorg/Xorg 4MiB | | 1 N/A N/A 2375 G /usr/lib/xorg/Xorg 4MiB | | 2 N/A N/A 2375 G /usr/lib/xorg/Xorg 64MiB | | 2 N/A N/A 2494 G /usr/bin/gnome-shell 6MiB | +-----------------------------------------------------------------------------+ lspci : 01:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1) 2e:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1) 41:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1) 61:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev ff) ## post by jwic on Aug 30, 2021 ``` Rebooted and disabled the Bluetooth controller, wifi controller, and audio controller which we don’t really need. Looks like the fourth CPU is still not coming online. While in the bios I did also check and saw the all the PCI slots were set to x16 speed. ``` ``` Aug 25 16:38:06 dc3dsgpu-app001 kernel: [ 6.671532] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 470.63.01 Tue Aug 3 20:44:16 UTC 2021 Aug 25 16:38:07 dc3dsgpu-app001 kernel: [ 8.004153] NVRM: GPU at PCI:0000:61:00: GPU-f14244ba-2a85-4182-2410-4f8477f57d89 Aug 25 16:38:07 dc3dsgpu-app001 kernel: [ 8.004156] NVRM: Xid (PCI:0000:61:00): 79, pid=2166, GPU has fallen off the bus. Aug 25 16:38:07 dc3dsgpu-app001 kernel: [ 8.004158] NVRM: GPU 0000:61:00.0: GPU has fallen off the bus. Aug 25 16:38:07 dc3dsgpu-app001 kernel: [ 8.004174] NVRM: GPU 0000:61:00.0: GPU serial number is 1322820062618. Aug 25 16:38:27 dc3dsgpu-app001 kernel: [ 28.283638] NVRM: GPU 0000:61:00.0: Tried to get IRQ 263, but another driver Aug 25 16:38:27 dc3dsgpu-app001 kernel: [ 28.283639] NVRM: has it and is not sharing it. Aug 25 16:38:27 dc3dsgpu-app001 kernel: [ 28.283640] NVRM: You may want to verify that no audio driver is using the IRQ. Aug 25 16:38:27 dc3dsgpu-app001 kernel: [ 28.283640] NVRM: GPU 0000:61:00.0: request_irq() failed (-16) Aug 25 16:38:27 dc3dsgpu-app001 kernel: [ 28.283741] NVRM: GPU 0000:61:00.0: Tried to get IRQ 263, but another driver Aug 25 16:38:27 dc3dsgpu-app001 kernel: [ 28.283742] NVRM: has it and is not sharing it. Aug 25 16:38:27 dc3dsgpu-app001 kernel: [ 28.283743] NVRM: You may want to verify that no audio driver is using the IRQ. Aug 25 16:38:27 dc3dsgpu-app001 kernel: [ 28.283743] NVRM: GPU 0000:61:00.0: request_irq() failed (-16) Aug 25 16:38:28 dc3dsgpu-app001 kernel: [ 29.305870] NVRM: GPU 0000:61:00.0: Tried to get IRQ 263, but another driver Aug 25 16:38:28 dc3dsgpu-app001 kernel: [ 29.305873] NVRM: has it and is not sharing it. Aug 25 16:38:28 dc3dsgpu-app001 kernel: [ 29.305874] NVRM: You may want to verify that no audio driver is using the IRQ. Aug 25 16:38:28 dc3dsgpu-app001 kernel: [ 29.305876] NVRM: GPU 0000:61:00.0: request_irq() failed (-16) Aug 25 16:38:28 dc3dsgpu-app001 kernel: [ 29.306009] NVRM: GPU 0000:61:00.0: Tried to get IRQ 263, but another driver Aug 25 16:38:28 dc3dsgpu-app001 kernel: [ 29.306011] NVRM: has it and is not sharing it. Aug 25 16:38:28 dc3dsgpu-app001 kernel: [ 29.306012] NVRM: You may want to verify that no audio driver is using the IRQ. Aug 25 16:38:28 dc3dsgpu-app001 kernel: [ 29.306014] NVRM: GPU 0000:61:00.0: request_irq() failed (-16) ```