{"id":6464,"date":"2024-03-28T09:28:53","date_gmt":"2024-03-28T01:28:53","guid":{"rendered":"https:\/\/www.1ai.net\/?p=6464"},"modified":"2024-03-28T09:28:53","modified_gmt":"2024-03-28T01:28:53","slug":"%e5%8f%b7%e7%a7%b0%e5%85%a8%e7%90%83%e6%9c%80%e5%bc%ba%e5%bc%80%e6%ba%90-ai-%e6%a8%a1%e5%9e%8b%ef%bc%8cdbrx-%e7%99%bb%e5%9c%ba%ef%bc%9a1320-%e4%ba%bf%e5%8f%82%e6%95%b0%ef%bc%8c%e8%af%ad%e8%a8%80","status":"publish","type":"post","link":"https:\/\/www.1ai.net\/en\/6464.html","title":{"rendered":"DBRX is the world's most powerful open source AI model: 132 billion parameters, language understanding, programming capabilities, etc. are better than GPT-3.5"},"content":{"rendered":"

Startups Databricks<\/a> Recently announced the launch ofOpen Source<\/a> AI Models<\/a> DBRX claims to be the world's most powerful open source large-scale language model to date, more powerful than Meta's Llama 2.<\/p>\n

\"00913eec-49cd-4d7b-a5a6-ad51449ab066\"<\/p>\n

DBRX uses a transformer architecture, contains 132 billion parameters, and consists of 16 expert networks. Each inference uses 4 of the expert networks, activating 36 billion parameters.<\/p>\n

Databricks introduced in a company blog post that in terms of language understanding, programming, mathematics and logic, it compares with mainstream open source models such as Meta's Llama 2-70B, France's MixtralAI's Mixtral, and Musk's xAI's Grok-1.DBRX wins in both categories.<\/strong><\/p>\n

\"a522ab11-f579-49a2-8a42-b31912e1c725\"<\/p>\n

Figure 1: DBRX outperforms existing open source models in language understanding (MMLU), programming (HumanEval), and mathematics (GSM8K).<\/p>\n

existLanguage Comprehension<\/strong>In terms of performance, DBRX's score is 73.7%, which is higher than GPT-3.5's 70.0%, Llama 2-70B's 69.8%, Mixtral's 71.4%, and Grok-1's 73.0%.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
\n

Model<\/p>\n<\/td>\n

\n

DBRX Instruct<\/p>\n<\/td>\n

\n

Mixtral Instruct<\/p>\n<\/td>\n

\n

Mixtral Base<\/p>\n<\/td>\n

\n

LLaMA2-70B Chat<\/p>\n<\/td>\n

\n

LLaMA2-70B Base<\/p>\n<\/td>\n

\n

Grok-11<\/sup><\/p>\n<\/td>\n<\/tr>\n

\n

Open LLM Leaderboard2<\/sup><\/p>\n

(Avg of next 6 rows)<\/p>\n<\/td>\n

\n

74.5%<\/p>\n<\/td>\n

\n

72.7%<\/p>\n<\/td>\n

\n

68.4%<\/p>\n<\/td>\n

\n

62.4%<\/p>\n<\/td>\n

\n

67.9%<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n<\/tr>\n

\n

ARC-challenge 25-shot<\/p>\n<\/td>\n

\n

68.9%<\/p>\n<\/td>\n

\n

70.1%<\/p>\n<\/td>\n

\n

66.4%<\/p>\n<\/td>\n

\n

64.6%<\/p>\n<\/td>\n

\n

67.3%<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n<\/tr>\n

\n

HellaSwag 10-shot<\/p>\n<\/td>\n

\n

89.0%<\/p>\n<\/td>\n

\n

87.6%<\/p>\n<\/td>\n

\n

86.5%<\/p>\n<\/td>\n

\n

85.9%<\/p>\n<\/td>\n

\n

87.3%<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n<\/tr>\n

\n

MMLU 5-shot<\/p>\n<\/td>\n

\n

73.7%<\/p>\n<\/td>\n

\n

71.4%<\/p>\n<\/td>\n

\n

71.9%<\/p>\n<\/td>\n

\n

63.9%<\/p>\n<\/td>\n

\n

69.8%<\/p>\n<\/td>\n

\n

73.0%<\/p>\n<\/td>\n<\/tr>\n

\n

Truthful QA 0-shot<\/p>\n<\/td>\n

\n

66.9%<\/p>\n<\/td>\n

\n

65.0%<\/p>\n<\/td>\n

\n

46.8%<\/p>\n<\/td>\n

\n

52.8%<\/p>\n<\/td>\n

\n

44.9%<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n<\/tr>\n

\n

WinoGrande 5-shot<\/p>\n<\/td>\n

\n

81.8%<\/p>\n<\/td>\n

\n

81.1%<\/p>\n<\/td>\n

\n

81.7%<\/p>\n<\/td>\n

\n

80.5%<\/p>\n<\/td>\n

\n

83.7%<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n<\/tr>\n

\n

GSM8k CoT 5-shot maj@13<\/sup><\/p>\n<\/td>\n

\n

66.9%<\/p>\n<\/td>\n

\n

61.1%<\/p>\n<\/td>\n

\n

57.6%<\/p>\n<\/td>\n

\n

26.7%<\/p>\n<\/td>\n

\n

54.1%<\/p>\n<\/td>\n

\n

62.9% (8-shot)<\/p>\n<\/td>\n<\/tr>\n

\n

Gauntlet v0.34<\/sup><\/p>\n

(Avg of 30+ diverse tasks)<\/p>\n<\/td>\n

\n

66.8%<\/p>\n<\/td>\n

\n

60.7%<\/p>\n<\/td>\n

\n

56.8%<\/p>\n<\/td>\n

\n

52.8%<\/p>\n<\/td>\n

\n

56.4%<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n<\/tr>\n

\n

HumanEval5<\/sup><\/p>\n

0-Shot, pass@1<\/p>\n

(Programming)<\/p>\n<\/td>\n

\n

70.1%<\/p>\n<\/td>\n

\n

54.8%<\/p>\n<\/td>\n

\n

40.2%<\/p>\n<\/td>\n

\n

32.2%<\/p>\n<\/td>\n

\n

31.0%<\/p>\n<\/td>\n

\n

63.2%<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

existProgramming skills<\/strong>In terms of performance, DBRX scored 70.1%, far exceeding GPT-3.5's 48.1%, and higher than Llama 2-70B's 32.3%, Mixtral's 54.8%, and Grok-1's 63.2%.<\/p>\n\n\n\n\n\n\n\n\n\n
\n

Model<\/p>\n<\/td>\n

\n

DBRX
\nInstruct<\/p>\n<\/td>\n

\n

GPT-3.57<\/sup><\/p>\n<\/td>\n

\n

GPT-48<\/sup><\/p>\n<\/td>\n

\n

Claude 3 Haiku<\/p>\n<\/td>\n

\n

Claude 3 Sonnet<\/p>\n<\/td>\n

\n

Claude 3 Opus<\/p>\n<\/td>\n

\n

Gemini 1.0 Pro<\/p>\n<\/td>\n

\n

Gemini 1.5 Pro<\/p>\n<\/td>\n

\n

Mistral Medium<\/p>\n<\/td>\n

\n

Mistral Large<\/p>\n<\/td>\n<\/tr>\n

\n

MT Bench (Inflection corrected, n=5)<\/p>\n<\/td>\n

\n

8.39 \u00b1 0.08<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

8.41 \u00b1 0.04<\/p>\n<\/td>\n

\n

8.54 \u00b1 0.09<\/p>\n<\/td>\n

\n

9.03 \u00b1 0.06<\/p>\n<\/td>\n

\n

8.23 \u00b1 0.08<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

8.05 \u00b1 0.12<\/p>\n<\/td>\n

\n

8.90 \u00b1 0.06<\/p>\n<\/td>\n<\/tr>\n

\n

MMLU 5-shot<\/p>\n<\/td>\n

\n

73.7%<\/p>\n<\/td>\n

\n

70.0%<\/p>\n<\/td>\n

\n

86.4%<\/p>\n<\/td>\n

\n

75.2%<\/p>\n<\/td>\n

\n

79.0%<\/p>\n<\/td>\n

\n

86.8%<\/p>\n<\/td>\n

\n

71.8%<\/p>\n<\/td>\n

\n

81.9%<\/p>\n<\/td>\n

\n

75.3%<\/p>\n<\/td>\n

\n

81.2%<\/p>\n<\/td>\n<\/tr>\n

\n

HellaSwag 10-shot<\/p>\n<\/td>\n

\n

89.0%<\/p>\n<\/td>\n

\n

85.5%<\/p>\n<\/td>\n

\n

95.3%<\/p>\n<\/td>\n

\n

85.9%<\/p>\n<\/td>\n

\n

89.0%<\/p>\n<\/td>\n

\n

95.4%<\/p>\n<\/td>\n

\n

84.7%<\/p>\n<\/td>\n

\n

92.5%<\/p>\n<\/td>\n

\n

88.0%<\/p>\n<\/td>\n

\n

89.2%<\/p>\n<\/td>\n<\/tr>\n

\n

HumanEval 0-Shot
\npass@1
\n(Programming)<\/p>\n<\/td>\n

\n

70.1%<\/p>\n

temp=0, N=1<\/p>\n<\/td>\n

\n

48.1%<\/p>\n<\/td>\n

\n

67.0%<\/p>\n<\/td>\n

\n

75.9%<\/p>\n<\/td>\n

\n

73.0%<\/p>\n<\/td>\n

\n

84.9%<\/p>\n<\/td>\n

\n

67.7%<\/p>\n<\/td>\n

\n

71.9%<\/p>\n<\/td>\n

\n

38.4%<\/p>\n<\/td>\n

\n

45.1%<\/p>\n<\/td>\n<\/tr>\n

\n

GSM8k CoT maj@1<\/p>\n<\/td>\n

\n

72.8% (5-shot)<\/p>\n<\/td>\n

\n

57.1% (5-shot)<\/p>\n<\/td>\n

\n

92.0% (5-shot)<\/p>\n<\/td>\n

\n

88.9%<\/p>\n<\/td>\n

\n

92.3%<\/p>\n<\/td>\n

\n

95.0%<\/p>\n<\/td>\n

\n

86.5%<\/p>\n

(maj1@32)<\/p>\n<\/td>\n

\n

91.7% (11-shot)<\/p>\n<\/td>\n

\n

66.7% (5-shot)<\/p>\n<\/td>\n

\n

81.0% (5-shot)<\/p>\n<\/td>\n<\/tr>\n

\n

WinoGrande 5-shot<\/p>\n<\/td>\n

\n

81.8%<\/p>\n<\/td>\n

\n

81.6%<\/p>\n<\/td>\n

\n

87.5%<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

88.0%<\/p>\n<\/td>\n

\n

86.7%<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

In mathematics, DBRX scored 66.9%, higher than GPT-3.5's 57.1%, and higher than Llama 2-70B's 54.1%, Mixtral's 61.1%, and Grok-1's 62.9%.<\/p>\n

\"39f6e358-5234-4579-a7df-4814c03846da\"<\/p>\n

Databricks introduced that DBRX is a hybrid expert model (MoE) built on MegaBlocks research and open source projects, so it can output tokens extremely quickly per second. Databricks believes that this will pave the way for the most advanced open source model of MoE in the future.<\/p>","protected":false},"excerpt":{"rendered":"

The start-up company Databricks recently announced the launch of the open source AI model, DBRX, which claims to be the world ' s most powerful open source large-language model to date, stronger than Meta ' s Llama 2. DBRX uses the transformer structure, which contains 132.2 billion parameters and consists of 16 expert networks, using 4 of which each reason to activate 36 billion parameters. Databricks, in a company blog post, describes the comparison of Llama 2-70B of Meta, Mixtrali of France, and xA of Mask<\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[146],"tags":[167,1906,219],"collection":[],"class_list":["post-6464","post","type-post","status-publish","format-standard","hentry","category-news","tag-ai","tag-databricks","tag-219"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts\/6464","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/comments?post=6464"}],"version-history":[{"count":0,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts\/6464\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/media?parent=6464"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/categories?post=6464"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/tags?post=6464"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/collection?post=6464"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}