{"id":16647,"date":"2024-07-29T09:19:40","date_gmt":"2024-07-29T01:19:40","guid":{"rendered":"https:\/\/www.1ai.net\/?p=16647"},"modified":"2024-07-29T09:19:40","modified_gmt":"2024-07-29T01:19:40","slug":"meta-%e8%ae%ad%e7%bb%83-llama-3-%e9%81%ad%e9%81%87%e9%a2%91%e7%b9%81%e6%95%85%e9%9a%9c%ef%bc%8c16384-%e5%9d%97-h100-gpu-%e8%ae%ad%e7%bb%83%e9%9b%86%e7%be%a4%e6%af%8f-3-%e5%b0%8f%e6%97%b6","status":"publish","type":"post","link":"https:\/\/www.1ai.net\/en\/16647.html","title":{"rendered":"Meta training Llama 3 encountered frequent failures, and the 16,384 H100 GPU training cluster "struck" every 3 hours"},"content":{"rendered":"

Meta<\/a> \u53d1\u5e03\u7684\u4e00\u4efd\u7814\u7a76\u62a5\u544a\u663e\u793a\uff0c\u5176\u7528\u4e8e\u8bad\u7ec3 4050 \u4ebf\u53c2\u6570\u6a21\u578b Llama 3 \u7684 16384 \u4e2a\u82f1\u4f1f\u8fbe<\/a> H100 \u663e\u5361\u96c6\u7fa4<\/a>\u5728 54 \u5929\u5185\u51fa\u73b0\u4e86 419 \u6b21\u610f\u5916\u6545\u969c\uff0c\u5e73\u5747\u6bcf\u4e09\u5c0f\u65f6\u5c31\u6709\u4e00\u6b21\u3002\u5176\u4e2d\uff0c\u4e00\u534a\u4ee5\u4e0a\u7684\u6545\u969c\u662f\u7531\u663e\u5361\u6216\u5176\u642d\u8f7d\u7684\u9ad8\u5e26\u5bbd\u5185\u5b58\uff08HBM3\uff09\u5f15\u8d77\u7684\u3002<\/p>\n

\"cdc29f17-a037-4859-a7f9-0cb603be5e64\"<\/p>\n

\u7531\u4e8e\u7cfb\u7edf\u89c4\u6a21\u5de8\u5927\u4e14\u4efb\u52a1\u9ad8\u5ea6\u540c\u6b65\uff0c\u5355\u4e2a\u663e\u5361\u6545\u969c\u53ef\u80fd\u5bfc\u81f4\u6574\u4e2a\u8bad\u7ec3\u4efb\u52a1\u4e2d\u65ad\uff0c\u9700\u8981\u91cd\u65b0\u5f00\u59cb\u3002\u5c3d\u7ba1\u5982\u6b64\uff0cMeta \u56e2\u961f\u8fd8\u662f\u4fdd\u6301\u4e86 90% \u4ee5\u4e0a\u7684\u6709\u6548\u8bad\u7ec3\u65f6\u95f4\u3002<\/strong><\/p>\n

\uff0c\u5728\u4e3a\u671f 54 \u5929\u7684\u9884\u8bad\u7ec3\u4e2d\uff0c\u5171\u51fa\u73b0\u4e86 466 \u6b21\u5de5\u4f5c\u4e2d\u65ad\uff0c\u5176\u4e2d 47 \u6b21\u662f\u8ba1\u5212\u4e2d\u65ad\uff0c419 \u6b21\u662f\u610f\u5916\u4e2d\u65ad\u3002\u8ba1\u5212\u5185\u7684\u4e2d\u65ad\u662f\u7531\u4e8e\u81ea\u52a8\u5316\u7ef4\u62a4\u9020\u6210\u7684\uff0c\u800c\u610f\u5916\u7684\u4e2d\u65ad\u5219\u4e3b\u8981\u6e90\u4e8e\u786c\u4ef6\u95ee\u9898\u3002 GPU<\/a> \u95ee\u9898\u662f\u5bfc\u81f4\u6545\u969c\u7684\u4e3b\u8981\u539f\u56e0\uff0c\u5360\u610f\u5916\u4e2d\u65ad\u7684 58.7%<\/strong>\u3002\u5176\u4e2d\u53ea\u6709\u4e09\u8d77\u4e8b\u4ef6\u9700\u8981\u5927\u91cf\u4eba\u5de5\u5e72\u9884\uff0c\u5176\u4f59\u7684\u7531\u81ea\u52a8\u5316\u7ba1\u7406\u3002<\/p>\n

\"3f50681d-22d9-4848-b23f-91eec91aa85f\"<\/p>\n

\u5728 419 \u4e2a\u610f\u5916\u4e2d\u65ad\u4e2d\uff0c148 \u4e2a\uff0830.1%\uff09\u662f\u7531\u5404\u79cd GPU \u6545\u969c\uff08\u5305\u62ec NVLink \u6545\u969c\uff09\u5f15\u8d77\u7684\uff0c\u800c 72 \u4e2a\uff0817.2%\uff09\u662f\u7531 GPU \u7684 HBM3 \u5185\u5b58\u6545\u969c\u5f15\u8d77\u7684\u3002\u6709\u8da3\u7684\u662f\uff0c54 \u5929\u5185\u53ea\u6709\u4e24\u4e2a CPU \u53d1\u751f\u6545\u969c\u300241.3% \u7684\u610f\u5916\u4e2d\u65ad\u662f\u7531\u591a\u79cd\u56e0\u7d20\u9020\u6210\u7684\uff0c\u5305\u62ec\u8f6f\u4ef6\u9519\u8bef\u3001\u7f51\u7edc\u7535\u7f06\u548c\u7f51\u7edc\u9002\u914d\u5668\u3002<\/p>\n

\u4e3a\u63d0\u9ad8\u6548\u7387\uff0cMeta \u56e2\u961f\u5f00\u53d1\u4e86\u4e00\u7cfb\u5217\u5de5\u5177\u548c\u4f18\u5316\u7b56\u7565\uff0c\u5305\u62ec\u7f29\u77ed\u4efb\u52a1\u542f\u52a8\u548c\u68c0\u67e5\u70b9\u65f6\u95f4\u3001\u5229\u7528 PyTorch \u7684 NCCL \u98de\u884c\u8bb0\u5f55\u5668\u8bca\u65ad\u6027\u80fd\u95ee\u9898\u3001\u8bc6\u522b\u62d6\u540e\u663e\u5361\u7b49\u3002\u6b64\u5916\uff0cMeta \u8fd8\u5173\u6ce8\u5230\u4e86\u73af\u5883\u56e0\u7d20\u7684\u5f71\u54cd\uff0c\u5982\u5348\u95f4\u6e29\u5ea6\u6ce2\u52a8\u5bf9 GPU \u6027\u80fd\u7684\u8f7b\u5fae\u5f71\u54cd\uff0c\u4ee5\u53ca\u5de8\u91cf GPU \u540c\u65f6\u8fd0\u884c\u5bf9\u6570\u636e\u4e2d\u5fc3\u7535\u7f51\u7684\u5de8\u5927\u538b\u529b\u3002<\/p>\n

\u7136\u800c\uff0c\u968f\u7740\u4eba\u5de5\u667a\u80fd\u6a21\u578b\u53c2\u6570\u91cf\u7684\u4e0d\u65ad\u589e\u52a0\uff0c\u6240\u9700\u7684\u8ba1\u7b97\u8d44\u6e90\u4e5f\u968f\u4e4b\u6269\u5927\u3002\u4ee5 xAI \u8ba1\u5212\u4e2d\u7684 10 \u4e07\u5757 H100 \u663e\u5361\u96c6\u7fa4\u4e3a\u4f8b\uff0c\u6545\u969c\u7387\u53ef\u80fd\u4f1a\u6210\u500d\u589e\u957f\uff0c\u7ed9\u672a\u6765\u7684 AI \u8bad\u7ec3\u5e26\u6765\u66f4\u5927\u7684\u6311\u6218\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"

Meta released a study showing that its cluster of 16,384 NVIDIA H100 graphics cards used to train the 405 billion parameter model Llama 3 experienced 419 unplanned failures in 54 days, an average of one every three hours. More than half of these failures were caused by either the graphics cards or the High Bandwidth Memory (HBM3) they carry. Due to the sheer size of the system and the highly synchronized nature of the tasks, a single graphics card failure could cause the entire training task to be interrupted and need to be restarted. Despite this, the Meta team was able to maintain an effective training time of over 90%. During the 54 days of pre-training, there were 466 work interruptions, of which 47 were planned and 419 were unplanned.<\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[146],"tags":[415,297,3755,239],"collection":[],"class_list":["post-16647","post","type-post","status-publish","format-standard","hentry","category-news","tag-gpu","tag-meta","tag-3755","tag-239"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts\/16647","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/comments?post=16647"}],"version-history":[{"count":0,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts\/16647\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/media?parent=16647"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/categories?post=16647"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/tags?post=16647"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/collection?post=16647"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}