Microsoft secretly developed the first 100 billion model, which was actually operated by OpenAI's opponents!

Author | Huawei Wing

Less than two weeks after the release of the Phi-3 Mini model, Microsoft announced the news of its self-developed 100 billion parameter level model.

For the first time since investing more than $10 billion in OpenAI in exchange for the right to reuse its AI models, Microsoft is starting to develop new and large enough AI models in-house that could compete with the most advanced models from Google, Anthropic, and OpenAI.

The new model, known internally as MAI-1, is overseen by Mustafa Suleyman, a former Google AI leader and CEO of AI startup Inflection. According to people familiar with the matter, MAI-1's parameter size will be much larger than any smaller open-source model that Microsoft has previously trained such as Phi-3. But this means that it will require more computing power and training data, so the cost will be higher.

At the same time, Microsoft's move shows that it is now pursuing a "dual track" in the field of artificial intelligence, with the goal of developing "small language models" that can be built into applications cheaply and run on mobile devices, as well as larger, state-of-the-art AI models. Apple seems to be exploring a similar path at the moment, having also released eight small AI language models for device use.

500 billion parameter levels

It may be unveiled this month at the earliest

According to the presentation, MAI-1 will have about 500 billion parameters or settings that can be adjusted to determine what the model learns during training. In comparison, OpenAI's GPT-4 has more than 1 trillion parameters, while small, open-source models released by companies like Meta and Mistral have 70 billion parameters.

This suggests that MAI-1 can be positioned as a model at the level between GPT-3 and GPT-4, which will be able to provide response accuracy that is much higher than open-source models such as Llama and Mistral, but lower than OpenAI's flagship LLM.

To train the model, Microsoft has been allocating a large number of servers equipped with Nvidia GPUs and compiling training data from a variety of sources, including text and public internet data generated by OpenAI's GPT-4, and may also use training data from Inflection and certain other assets to support MAI-1.

At the moment, the exact purpose of the MAI-1 has not been determined (even within Microsoft), and its most desirable use will depend on its performance. If the model does have 500 billion parameters, it's too complex to run on a consumer device. This means that Microsoft is likely to deploy MAI-1 in its data centers, where large language models can be integrated into services such as Bing and Azure.

Microsoft could unveil MAI-1 as early as later this month at the Build developer conference, based on the progress made in the coming weeks.

MAI-1 的研发是基于 Inflection？

"Although MAI-1 is a completely new large language model that is separate from Inflection's previously released Pi, it may be built on technology brought to the table by former Inflection employees." According to two Microsoft employees with knowledge of the situation.

According to a statement on OpenAI's official website, Inflection was once a competitor to OpenAI, but it has now shifted its focus from chatbot Pi to selling AI software to businesses. Sean White, who has held various technical roles, has joined the company as the new CEO.

In March, Microsoft bought most of the startup's employees and intellectual property for $650 million and hired Suleiman to lead a new consumer AI division. The division brings consumer-facing products, including Microsoft's Copilot, Bing, Edge and GenAI, under a team called Microsoft AI, and Suleiman reports directly to Microsoft CEO Satya Nadella.

The new division marks a major organizational shift for Microsoft, and its President of Network Services, Mikhail Parakhin, will report to Suleiman along with his entire team. It's also one of Microsoft's latest moves to capitalize on the generative AI boom.

"I've known Mustafa for several years, and I have great admiration for him as the founder of DeepMind and Inflection, as well as a visionary product maker and pioneering team builder pursuing a bold mission," Nadella said in a statement. ”

Founded in the UK in 2010 and acquired by Google in 2014 for $500 million, Suleiman is one of the company's three founders. At DeepMind, Suleiman was forced to leave DeepMind in 2019 after employees complained about his aggressive and overly aggressive management style. Later, when referring to the employee complaint at the time, Suleiman responded: "I really messed up. I'm very demanding and quite unforgiving. I set some rather unreasonable expectations that led to a very hostile working environment for some people. I very much regret this. ”

A few months later, he moved to Google headquarters, where he was responsible for leading AI product management and policy. In 2022, he left Google to join Greylock, a Silicon Valley venture capital firm, and launched Inflection later that year.

It is reported that Microsoft will also hire most of Inflection's employees, and Inflection's co-founder and chief scientist, Karén Simonyan, will also serve as the chief scientist of its AI team. While Microsoft did not specify the number of employees transferred, it said it included AI engineers, researchers and large language model builders who designed and co-authored "many of the most important contributions to advancing AI over the past five years."

Inflection 的第三位联合创始人、LinkedIn 创始人兼执行主席 Reid Hoffman 将继续留在 Inflection 的董事会。

Last June, Inflection also closed a $1.3 billion funding round led by Microsoft, Nvidia and three billionaires (Reid Hoffman, Bill Gates, and Eric Schmidt). At the time, Kevin Scott, Microsoft's chief technology officer, also said, "Ambitious AI companies like Inflection are leading the industry with transformative products that are easy to use and demonstrate the many possibilities of AI." ”

Conclusion

Microsoft's development of the MAI-1 model also highlights its willingness to explore AI development independently of AI vendors such as OpenAI.

Previously, Microsoft has been working on introducing AI assistants in its products such as Windows, Office software, and cybersecurity tools, but mostly in the form of working with external companies.

Last year, Microsoft invested $13 billion in OpenAI, the maker of ChatGPT, and quickly integrated its technology into products and digital experiences. Currently, OpenAI's technology powers many of Microsoft's generative AI capabilities, including Azure, Copilot, and chatbots with built-in Windows.

Microsoft has also invested in other AI startups, including a €2 billion ($2.1 billion) investment in French AI startup Mistral AI to host Mistral AI's large language models (LLMs) on the Azure cloud computing platform.

In the future, all of this may change, and Microsoft may begin to promote the application of self-developed large models in various products. Suleiman's department will take over projects such as integrating the AI version of Copilot into the Windows operating system and enhancing the use of generative AI in its Bing search engine.

"Microsoft is in an AI race," an AI engineer at Microsoft said recently, but when it comes to ethics and security, Microsoft has taken shortcuts for speed and rushed to launch products without fully thinking about what will happen next. All big tech companies have access to much of the same data, and there is no real moat in the AI space.

Original link: Microsoft secretly developed the first 100 billion model, which was actually operated by OpenAI's opponent! Netizen: You don't want Ultraman anymore? _Featured article by Generative AI_ Huawei _InfoQ