Graphics card cluster

A research report released by Meta shows that its 16,384 NVIDIA H100 graphics card cluster used to train the 405 billion parameter model Llama 3 experienced 419 unexpected failures in 54 days, an average of one every three hours. More than half of the failures were caused by the graphics card or its high-bandwidth memory (HBM3). Due to the huge scale of the system and the high synchronization of tasks, a single graphics card failure may cause the entire training task to be interrupted and need to be restarted. Despite this, the Meta team has maintained more than 90% of effective...