Warnings on H100 instance running PyTorch

post by keithlostracco on Jun 26, 2023

I’m getting the printout below when running a PyTorch training loop. Is there anything that I should do to configure the instance? It states I may get lower performance, I’m not sure if that is the case though when using PyTorch.

2023-06-26 21:25:08.743143: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-06-26 21:25:08.904188: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.

WARNING: No preset parameters were found for the device that Open MPI detected:

Local host: 209-20-157-229

Device name: mlx5_0

Device vendor ID: 0x02c9

Device vendor part ID: 4122

Default device parameters will be used, which may result in lower performance. You can edit any of the files specified by the btl_openib_device_param_files MCA parameter to set values for your device.

NOTE: You can turn off this warning by setting the MCA parameter btl_openib_warn_no_device_params_found to 0.

No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port.

Local host: 209-20-157-229 Local device: mlx5_0 Local port: 1 CPCs attempted: udcm

post by yanos on Jul 4, 2023

Hello @keithlostracco!

This warning messages can be safely ignored.

See also here.

Best, Yanos