December 6, Bloomberg News.Open source ChatGPT platformMistral AI gets 450 million euros (nearly 3.5 billion dollars)Financing, valued at nearly $2 billion ($14.2 billion). The current investment by NVIDIA, Salesforce and others.
Mistral AI's open-source big language model Mistral7B majors infeatures such as small parameters, low energy consumption, and high performance, and allows for commercialization of the. Support for generating text/code, data fine-tuning, summarizing content, etc. currently has 4500 stars on github.
It is worth mentioning thatMistral AI had secured a $113 million seed round without releasing any productsThis is the first time in the history of European technology thatmaximumOne of the seed rounds of funding for the
Open source address:https://github.com/mistralai/mistral-src
Help file:https://docs.mistral.ai/
API Interface:https://docs.mistral.ai/api
Compared to the meta-universe, ChatGPT, which just celebrated its 1st birthday, has withstood multiple tests such as commercial landing and user audience, and has driven a large number of tech companies to participate in generative AI change.
Currently, there are two main camps, closed source and open source. Llama hits the ground running at MetaFirstAfter the gun.A large number of outstanding companies have emerged in the field of open source large language modeling, such as Writer, Baichuan Intelligence, Together.ai, Mistral AI, etc., while gaining recognition in the capital markets. These vendors are also convinced that open source is one of the shortcuts to AGI for large models.
Mistral AI was introduced to the "AIGC Open Community" back in June of this year, and I was very impressed with it at that time.Since no products have been released, the official website has only 3 sentences:.We are assembling a world-class technical team to develop themostof generative AI models.
Mistral AI's official website content from June this year
We operate in Europe and are based in Paris, France. If you have extensive research and development experience in the AI field, please contact us.
At that time, with these three words, it raised $113 million in seed round financing, valued at $260 million. Usually this kind of enterprise either rub a wave of heat to get money, and then casually change the model to sit and wait for death.
Either that or it's a sweeper-level tech bull that's famous as soon as it hits the ground running. Judging from the results of this financing, Mistral AI belongs to the latter does have two tricks up its sleeve.
Mistral AI's three co-founders, Timothée Lacroix, Guillaume Lample, and Arthur Mensch, are well known for their big factory resumes and successful projects, as well as being university alumni.
Timothée and Guillaume worked in Meta's AI research department and led the development of LLaMA, the originator of the ChatGPT-like open-source model.Arthur worked at DeepMind, Google's AI research lab.
Products.Mistral AI's Mistral 7B, launched on September 27 of this year, is the currentStrongestOpen source large language model that outperforms Llama213B in all benchmarks; outperforms or is comparable to Llama134B on many benchmarks; performance on code tests is comparable to CodeLlama7B.
In order for the model to reason faster and with less energy consumption, theMistral AI uses two main mechanisms, grouped query attention and sliding window attention, respectively.
Grouped query attention is an improvement on the standard attention mechanism that reduces computational complexity by grouping queries. In the Transformer model, the attention mechanism typically involves three sets of vectors for queries, keys, and values.
In the standard self-attentive mechanism, each query is matched against all keys, which leads to a huge computational burden when the sequence is long.
And grouped query attention works by combining multiple queries into a single group. The query vectors of each group then interact with only a portion of the key vectors instead of all of them, making the overall efficiency very efficient.
Sliding window attention is a technique used in sequence processing tasks to limit the scope of the attention mechanism and reduce the amount of computation. In this approach, instead of computing attention for the entire sequence, the attention of each element is limited to elements within a window in its neighborhood.
In this way, each part of the model only needs to process the information within the window, thus reducing the number of elements involved in each attentional computation.
This not only reduces the computational requirements, but also limits the scope of the model's context and helps the model to focus on localized information.