Written on the eve of the release of Baidu's "Wen Xin Yiyan"

Compare, of course, what cannot be compared.

Text / Book Airlines 2023.3.15

This article is written very urgently, because after 12 o'clock it is not pushed, these predictions cannot be sent. But I've been inquiring about multiple sources before, and by this time, I feel that sorting them out is still a little useful for everyone.

It took a little more than a month for Baidu to officially announce that it had such a thing as "Wen Xin Yiyan" to release the finished product, which is not unusual in itself. Because the relationship between Wen Xin Yiyan and Wen Xin Big Model is the relationship between ChatGPT and GPT-3.

Baidu's big model has been trained for a long time, and it is not so difficult to make a front-end application. I believe that most of this month's time is spent talking about business cooperation, finding people to publicize, internal testing of various tricky and weird use cases, and finding bugs.

Baidu's toolbox not only contains the Wenxin large model, but also the underlying technology that supports it, including its use of self-developed chips to train. BAT has announced self-developed chips for internal training, and Baidu's "Kunlun" announced has reached the second generation, using a 7nm process, while also emphasizing versatility. The third generation will be deployed next year.

Although Baidu will also expand its external compatibility and developer ecology with its various modules, such as chip layer, framework layer and model layer, it still believes that it is all its own things working together, and the effect must be better than a bunch of general solutions put together (we can use Huawei's 5G technology and OpenRAN camp, or the comparison between x86, ARM camp and RISC-V camp as evidence for this view).

GPT-4 was released today, but it's worth noting that OpenAI has announced very few technical details this time, which is very different from the previous feeling of being a non-profit research organization that generously serves all of humanity. It also shows that if something is really good, the person making it can't be too generous.

Everyone is watching "China's ChatGPT", and it seems that only China has the opportunity to make competing products developed outside of American companies at this point in time. Whether it can be made with this thing and whether it is of the same quality as this thing are two different questions.

Fudan made the earliest MOSS release, but it was scolded very badly. Of course, as an academic experimental project, word of mouth is not related to future application prospects, so it may not be the most important thing. Companies that really want to produce products are actually beating up vaccinations, constantly reducing the psychological expectations of domestic and foreign followers. Specifically, it is to affirm that you must not be able to do such a good effect as ChatGPT now, there will be more problems, and it will be more clumsy.

For example, Xiaoice said a while ago that the "Xiaoice chain" is that in a full-featured but high-cost, low-efficiency large model, a part of the product that can be quickly commercialized, possibly with some manual tuning, will be more qualified for roles such as chatbots than the current Xiaoice model. Even without considering GPT-like technologies, Xiaoice have previously implemented some of these use cases autonomously, such as the generation of articles, images, videos, and as many rounds of dialogue as possible. These things that have already been made, in fact, there is no need to throw them away, anyway, the outside seems to be in the same black box.

So, what might Baidu say? If we browse Baidu's toolbox, we can find that in addition to chips and full technology stacks, there is another thing that is likely to be overlooked: the knowledge graph.

The latest data of Baidu's knowledge graph that we know so far comes from 2020, and the specific statement is that "Baidu has created the world's largest multi-source heterogeneous knowledge graph, with more than 5 billion entities and 550 billion facts, and is constantly evolving and updating, with more than 40 billion calls per day".

While deep learning is advancing by leaps and bounds, the knowledge graph seems to have been forgotten and has not progressed for a long time. However, many AI researchers have believed that directly establishing the correspondence between entities and events is the key to making the "black box" white and making the AI decision-making process transparent. Of course, we know that this hasn't happened so far.

The main problem is that the process of building a knowledge graph is maddening. If some of them are machine-generated, it may produce false correspondences — like the ironic "answers" that Baidu sometimes extracts below the search box. (The same goes for Google.) A famous example is that it labels scientists who study the WannaCry ransomware virus as the author of the virus. ）

If you comb by hand - what a joke, huh?

But now, assuming that GPT and similar technologies can produce ethical, logical, and accurate answers above the threshold required for commercial applications, then this part of human labeling work, including labeling knowledge graphs and cleaning deep learning corpus, is no longer a seemingly impossible task if it can make mature GPT loop itself.

Therefore, even if the high-quality corpus of Chinese is not as much as English, it is possible to replace the manual cleaning of past materials by mature GPTs, check and build a knowledge graph, and then feed them to achieve self-enhancement. Of course, this is only a theoretical assumption.

In fact, whether or not the knowledge graph, or other existing resources are integrated into the Wenxin model, we now know that its problems are not few. In the "Wen Xin Yi Frame", that is, the drawing app, there are often cases where prompt cannot be correctly recognized, and I may be able to give examples together when I touch "Wen Xin Yiyan" tomorrow.

However, one example that may prove that they introduced the knowledge graph into the larger model is that Wen Xin Yige can correctly understand Chinese dishes such as "rice noodles" and "Buddha jumping over the wall", which may be ambiguous when translated. I believe that Baidu will make special efforts to promote examples in this regard, because when they were first established, their slogan was "better understand Chinese".

In order to maximize strengths and avoid head-to-head confrontation in computing power, the above various statements are all to save some of the abilities that need training to emerge. The emergence ability is all or nothing, and it is expected that it will not work compared to ChatGPT, and if there is a place to win, it is a windfall.

Domestic production is currently at the stage from me too to me better, of course it can be compared, MOSS can be compared, what can not be compared. This is the correct expectation that tomorrow when we face "Wen Xin's words".