Lambda Quad - How to Unbond NVLINK from P2P/SLI mode

post by mrwig on Dec 5, 2021

My lambda quad 4x a6000 workstation has the GPUs setup to use NV4 and SLI/P2P options. This is reducing memory bandwidth between all 4 cards to 14GB/S. I need the cards to be connected solely via the nvlink bridge, so two pairs of linked cards at the highest shared memory speed possible.

How can I reconfigure the nvlinked cards so that they are no longer bonded 4way but operating as two separate nvlinked pairs?

Some current outputs from terminal can be viewed here.

post by Karun on Dec 1, 2022

Any update?

I have a blade system with 4 A6000.

What does the below output mean?

$ nvidia-smi nvlink -gt d -i 0,1,2,3

GPU 0: NVIDIA RTX A6000 (UUID: GPU-938d731b-…)

Link 0: Data Tx: 63,654,019,510 KiB

Link 0: Data Rx: 7,631,152,229 KiB

Link 1: Data Tx: 63,650,865,559 KiB

Link 1: Data Rx: 7,632,899,051 KiB

Link 2: Data Tx: 63,652,014,108 KiB

Link 2: Data Rx: 7,631,403,371 KiB

Link 3: Data Tx: 63,652,312,551 KiB

Link 3: Data Rx: 7,629,171,248 KiB