Author | vanilla
Edit | Li Shuiqing
Zhidong reported on April 11 that yesterday, the "European version of OpenAI" Mistral AI once again quietly showed its muscles, throwing out a new MoE (expert hybrid) large model Mixtral 8x22B magnetic link, with a model parameter scale of up to 176 billion, second only to Musk's Grok-1, becoming the second largest open-source model on the market.
▲Mistral AI发布Mixtral 8x22B
Mixtral 8x22B consists of 8 expert models, each with a parameter size of 22 billion and a model file size of about 262GB. In terms of evaluation results, Mixtral 8x22B topped the list of open-source models in MMLU (Massive Multitasking Language Understanding), and Hellaswag, TruthfulQA, GSM8K and other tests surpassed Llama 2 70B, GPT-3.5 and Claude 3 Sonnet.
This is the third important model released by large model manufacturers after the comprehensive update of OpenAI's visual version of GPT-4 Turbo and Google's Gemini 1.5 Pro. In addition, Meta also hinted that Llama 3 will be released next month.
1. Won the top of the MMLU open source list, and 3 A100 graphics cards can run
The Mixtral 8x22B contains 8 expert models, each with a parameter size that soars from 7 billion in the previous generation to 22 billion, with a sequence length of 65,536.
Soon after the release of the magnet link, Mixtral 8x22B was released on the open-source community Hugging Face, with a model file size of about 262GB, which can be further trained and deployed.
In terms of performance, it topped the list of open source models in the MMLU evaluation list, surpassing Llama 2 70B, GPT-3.5 and Claude 3 Sonnet in many tests.
▲Mixtral 8x22B测评成绩
Although it was released in a low-key manner, the Mixtral 8x22B once again detonated the open source community. Both Perplexity Labs, an AI search platform, and Together AI, an open-source platform, were quick to provide support for the model.
AI scientist Jia Yangqing said that through reasonable quantification, Mixtral 8x22B can run on 4 A100/H100 graphics cards, strictly speaking, in fact, 3 A100 graphics cards are enough.
▲Jia Yangqing said that 3 A100 graphics cards can run Mixtral 8x22B
Awni Hannun, a machine learning researcher at Apple, said that the Mixtral 8x22B model works well on Apple's machine learning framework MLX using the M2 Ultra chip, and released a 4-bit quantization model in the MLX community.
▲Apple's MLX machine learning framework runs Mixtral 8x22B
2. Yang Likun forwarded, the French AI ecology is so "against the sky"
Today, the AI ecosystem in Paris, France is at the "Next Level". Yang Likun, Meta's chief AI scientist from Paris, retweeted a post about how Paris became a major AI hub, and the story dates back more than 10 years.
▲杨立昆转发Damien Henry帖文
In 2013, Xavier Niel founded Ecole 42, a computer training school, an unusual school that accepts students from different backgrounds to train programming in a peer-to-peer learning way.
In 2015, Yang Likun founded FAIR Paris, also known as the Facebook AI Research Institute. He is recognized as one of the three major inventors of deep learning, the other two being Canadian.
FAIR sends a signal to French talent that they don't have to leave France to pursue deep learning research, and to other tech giants in the U.S. that it is possible to conduct scientific research in France. Then, in 2018, Google DeepMind also opened a lab in Paris.
In 2016, the first edition of VivaTech was held, which became the EU's premier event for startups and investors, attracting more than 2,400 startups and more than 2,000 investors.
In the same year, the open-source community Hugging Face was founded in France. Perhaps they wouldn't have known at the time that their Transformer library would soon become the industry standard.
In 2017, Xavier Niel founded STATION F, the world's largest startup campus. Under the leadership of Roxanne Varza, it became the technology hub of France. It is worth mentioning that Hugging Face is one of the first startups to join the STATION F program and the first unicorn born from the program.
▲STATION F Pioneer Park
In 2018, Yang Likun won the Turing Award and became one of the most influential figures in the field of AI.
▲Yang Likun
During this time, the author of this post, Damien Henry, formed the Google Arts & Culture team in Paris, focusing on AI and images. He also co-founded Clipdrop, an AI visual generation tool with two co-creators, which was later acquired by Stability AI.
In 2019, Paris had already taken its place on the world's AI map, but it was not as mainstream as it is today.
But in 2020, the pandemic swept the world, and remote work became the norm. This has shifted the tech world: With so many places to choose from, why do I have to rent a room in Silicon Valley? Moreover, Americans are starting to see technology as a threat, as opposed to the opposite in most less technologically advanced countries.
In 2021, Hugging Face grew rapidly to become the most powerful open-source AI platform we know today. DeepTech is accelerating globally, especially in Paris.
Mistral AI was founded in 2023 and has grown rapidly to become one of OpenAI's biggest rivals within a few months. The way they publish their models is low-key and "aggressive", garnering over 4 million views with just one magnet link without any context, making expensive posting videos obsolete.
In the same year, kyutai, Europe's first independent AI research laboratory, was established, and Scaleway, Rodolphe Saade, and others announced their entry into the computing field to bring more GPUs to Europe. ICCV, the world's top computer vision conference, has also chosen to be held in Paris, and top investment institutions such as Redshirt Capital are increasingly focusing on French AI startups.
Conclusion: Another giant has been added to the open source community
With the Mixtral 8x22B model causing a sensation in the open source community, we have witnessed the rapid development of the open source model and the rise of the European AI ecosystem. This achievement not only demonstrates the strength of Mistral AI in the field of large models, but also reflects the deep potential of AI research and innovation in France and Europe as a whole.
From the establishment of Ecole 42 and FAIR Paris, to the rise of Hugging Face and STATION F, Paris has become an important center for global AI innovation. In the future, we look forward to seeing more cities outside of Silicon Valley play an important role in the global AI arena and drive cutting-edge innovation.