Vulkan enumeratePhysicalDevices: ErrorInitializationFailed on Headless A100 Server

[hamondyan]

Describe the bug：

I am unable to initialize a ManiSkill environment on a headless server equipped with an NVIDIA A100 GPU. The process fails with a Vulkan error: RuntimeError: vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed.

This appears to be related to the NVIDIA driver and Vulkan ICD (Installable Client Driver) configuration, as system-level checks for NVIDIA's graphics libraries are also failing.

Troubleshooting Steps Taken：

Followed Headless Rendering Guide: I have followed the official documentation for Vulkan setup on headless servers: https://maniskill.readthedocs.io/en/latest/user_guide/getting_started/installation.html#vulkan
I created three JSON files, but it still does not work.

[knightnemo]

I am encountering the same bug with H200 cards, with the following error message: RuntimeError: vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed. However I suspect this is a problem with ManiSkill & Vulkan, so scrapping through the issues section at that repo may be a better place to find answers. If you have solved the problem, can you provide the solutions here? Would be very helpful.

[hamondyan]

I did not solve it yet....

[xuanlinli17]

As in #68 (comment), try

DISPLAY="" CUDA_VISIBLE_DEVICES=0 VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json python xxx
(use other icd filenames path if different)

[knightnemo]

Thanks for the great work @xuanlinli17 , as well as the prompt reply. The solution I got working on a 8-card headless H200 server was reinstalling the CUDA driver to a newer version (in my case, I used CUDA Version: 12.7, Driver Version: 565.57.01), accompanied by following the instructions of https://maniskill.readthedocs.io/en/latest/user_guide/getting_started/installation.html#vulkan. Using numpy==1.24.4, even without any display (either virtual or physical), I am able to run the environment.

I believe such errors, as well as errors of Segmentation Fault during initialization of scene's cameras (which I also encountered), is mostly due to the inability of vulkan to find or utilize GPU devices. One can run vulkaninfo to check whether vulkan can find the GPUs. A successful snippet of running the command should contain sth similiar to this:

VK_LAYER_INTEL_nullhw (INTEL NULL HW) Vulkan version 1.1.73, layer version 1:
	Layer Extensions: count = 0
	Devices: count = 9
		GPU id = 0 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 1 (llvmpipe (LLVM 15.0.7, 256 bits))
		Layer-Device Extensions: count = 0

		GPU id = 2 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 3 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 4 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 5 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 6 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 7 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 8 (NVIDIA H200)
		Layer-Device Extensions: count = 0