Guide
On the evening of April 3, Wu Jun, a computer scientist and natural language model expert, made a live broadcast sharing on hot topics such as artificial intelligence and ChatGPT.
Wu Jun, graduated from Tsinghua University and Johns Hopkins University, Ph.D. in computer science, former senior researcher at Google, former vice president of Tencent, and venture capitalist in Silicon Valley.
Why is ChatGPT not present in China? Wu Jun said that its "hardware cost is almost $1 billion", which ordinary scientific research institutions cannot do, let alone the problem of research level.
Wu Jun | speech
Why does the emergence of ChatGPT cause panic?
I know that ChatGPT has been very popular in China recently, and many people are discussing it, but it is interesting that in fact, there are not many people talking about this topic in the United States. In fact, it's not just ChatGPT, but a decade back, when a lot of new technologies came out, I found that the discussion in the Chinese media was much higher than in the United States. Although that technology actually appears mainly in the United States, Chinese people are more concerned about it. I think that's a good thing, but it's also a bad thing.
The "bad" thing is that these technologies are actually overhyped, and in the process, there are a lot of troubled waters who make money from them. For example, blockchain, it was so hyped up at the time, but this matter is rarely discussed now, right? This is the first one. The second is the meta-universe, which is currently the only Facebook in the United States that is still insisting on doing it. When we get to China, a lot of people are talking about whether we will live in a completely virtual world in the future. Finally, probably from the end of last year to the beginning of this year, Facebook invested tens of billions of dollars in this field, did not listen to a single sound, and finally began large-scale layoffs. Until now, one of the hot topics is ChatGPT, some people are excited, some people are afraid, and I now see that there are still many people in China fishing in troubled waters, trying to cut everyone's leeks again.
Before talking about what ChatGPT is, I'm going to tell you a historical story that makes you laugh, but when you look back, the same is true of how many people perform today.
In 1503, Columbus's son wrote down an incident in which Columbus sailed west to the New World, but when he reached Jamaica, there was no food on board. As a result, Columbus and the crew could only hope that the locals would provide food. However, after a few days of provision, the crew became at odds with the locals - some of the crew members stole the locals' things, so the locals cut off the supply of food.
To get out of this predicament, Columbus thought of a brilliant trick. Columbus was carrying a perpetual calendar with him, and the calendar was marked with all these information that there would be solar eclipses, lunar eclipses, and so on in a certain year, month, and day. Columbus then brought the local tribal leader and said that you had sinned against God by not providing me with food, God would be angry, the moon would turn red, and God would take the moon away.
Of course, we basically know now that when a total lunar eclipse occurs, that is, when the earth has not completely blocked the moon, the moon is indeed red, which is what we call the "blood moon". However, Jamaicans at the time did not know it. As a result, at night, Jamaicans noticed that the moon had indeed turned red, and then slowly disappeared little by little. The locals went into a panic, saying that God was going to punish them.
The tribal leader hurried to Columbus, promising to agree to all of Columbus' conditions. Columbus said, OK, I'll go to the tent and pray to God that He won't punish you, but I need a little time, and then Columbus went into the tent. In fact, after entering the tent, Columbus was holding an hourglass and watching the timekeeping.
Today we have astronomical knowledge, we must know the time of the total lunar eclipse, and it will last for about 48 minutes, when the moon will reappear. However, these Jamaicans do not know. All they saw was that Columbus came out of the tent and the moon came out. Then Columbus said, "God has heard my exhortation and promised to forgive you, but you must provide us with good food." So, the locals are grateful for their constant food.
What does this story say? There are reasons behind the occurrence of the total lunar eclipse, but when people do not know this reason, they can often only attribute this natural phenomenon to the role of a god. And this god himself was created by man. That is to say, after man himself created a god, he then lay at God's feet and became his slave.
That's why I'm giving you a course on the History of World Civilization.
In fact, the development process of this civilization is the process of human beings constantly understanding the laws of nature. We have made a little progress so that we are no longer blindly believing that a person praying to God can really stop the moon from disappearing, like the local indigenous people. We now know that behind the solar and lunar eclipses, it is actually the three laws of the planet Kepler at work, and then behind the three laws of the Kepler planet is Newton's law of universal gravitation. After human beings understand this reason, they are no longer just afraid of nature, and we can do many, many things using the laws of nature.
What is the technical basis of ChatGPT?
From history back to the present, the situation with ChatGPT is similar, behind which is a mathematical model called the language model at work. In other words, behind ChatGPT is a mathematical model. Today, this technology appears powerful for three main reasons:
First, it uses a lot of calculations;
Second, it has a large amount of data;
Third, the way to train language models today is much better than before.
So, what is a language model? Or what era is it a product of?
It was developed in 1972 by a team led by my mentor, Fred Jelinek. Specifically, it was a technique he had with people at IBM at the time to measure how likely a sentence or a linguistic phenomenon could be. So what is it for? Its initial use was to do speech recognition, then to do machine translation, and then to do computer question answering, which is the answer question we are familiar with today.
At that time, it can do abstracts, for example, for example, there is a 10,000-word article, so how do you summarize ten sentences to summarize the content of this article, which is a mathematical problem for people who do this natural language processing. That is, what are your conditions? The condition is these 10,000 words, and then what is the result you want to get? The result could be ten sentences, a hundred words. Then there are many combinations here, you can pick a few sentences at random, or you can split some sentences into two paragraphs, and remove the less important modifications or descriptions later. Then, you can also combine two sentences into one sentence, then when you synthesize a piece of text, the computer will calculate a probability, which sentences are more likely to be synthesized together, and it will help you synthesize according to the probability.
And the ChatGPT we see today is this big language model, which will pick a text with the highest probability and most likely to happen to show you. So in general, the process of generating results by ChatGPT is a process that uses a lot of computing resources to calculate. It requires a very large amount of data to support, and there are many, many GPUs (computer processors). Without these things, ChatGPT can't do it.
And today's ChatGPT, in fact, is not only technology, but also a lot of artificial behind. They also hired a company to review the results produced by ChatGPT. For example, ChatGPT produced a hundred snippets, all so good that I couldn't tell them apart, so these people were responsible for helping me tell which one was more like an accurate summary.
In fact, you can see that behind ChatGPT is a language model, and the technology for this language model has been around since 1972. Now, after fifty years, everyone in the industry now doesn't actually think it's a big deal. Prior to this, this language model has actually done a lot of things.
The term "language model" was first coined by my mentor Jarnick. He arrived at Johns Hopkins in about 1993, and I came to the university in 1996 and became his student. So the Chinese of the word, the four words "language model" you see, was coined when I published my paper in the 90s of the 20th century. Back then, only those of us in the circle knew it could do a lot of things, but you wouldn't think about saying, hey, this thing will be hyped up later.
You can understand that the "language model" is to ChatGPT what Kepler's three planetary laws are to lunar eclipses.
What was the situation when the "language model" was born?
So what was the situation of the language model at the time of its invention?
In fact, in the 90s of the 20th century, the models obtained by simple statistical methods were very inaccurate. This is equivalent to, let me use an analogy, you observe planets, but using Ptolemy's geocentric prediction is very inaccurate. So, that's when we started to introduce a lot of information about syntax, themes, semantics. Then, this language model becomes very complex. After complexity, it brings a big problem.
What problem?
For example, I made a very complex language model at that time, how many parameters did this language model have at that time? 6 million parameters, that is, the size of the language model is basically determined by this parameter. What I did at that time was already the largest and most complex language model that could be done in that era. Instead of a PC, I was using 20 super servers, and it took me about three months to train such a language model. So you see, it's very computationally intensive. So, the first version of ChatGPT, what were the language model parameters it used? That's about 200 billion parameters, and you can see how it has changed over the years.
So, today many people ask, ChatGPT appeared in the United States, when can Chinese research institutions do ChatGPT? In fact, most research institutions in China cannot do it, not because of the level of research, but because ChatGPT is too resource-intensive. Today's ChatGPT, the cost of hardware alone may be almost $1 billion, which is not counting the electricity money, so the cost and cost are very huge. So, if you joke and ask what ChatGPT's biggest contribution is, I think it contributes a lot to global warming.
So, what I want to say is that ChatGPT, its principle is very simple, but it is actually quite difficult to do it in engineering.
What questions are computers good at answering?
Around 2010, that is, 13 years ago, how far can language models go? Let me show you two examples. Both examples were made before I left Google in 2014. I was in charge of Google's automated question answering system, which lets computers answer questions. However, because this product is in English, it is basically not too faceless in the Chinese world.
Let me show you a question Google answers – why is the sky blue?
Its answer is this: sunlight will be refracted when it reaches the earth through the atmosphere, the gas in the air will scatter different colors of light to various places, blue light wavelength is short, higher refractive index than other colors, so the sky looks blue.
This was an answer that computers produced at the time. To be fair, this answer is better than writing a paragraph myself, because to explain this phenomenon, you need to know a lot about physics, and the sentence seems quite reasonable. One of the purposes people use ChatGPT today is to get him to answer questions.
In this, I will make a split for everyone.
In fact, the questions we ask computers can be divided into two categories, the first is called simple problems, and the second is called complex problems. Simple questions are questions about facts, such as who such and such a star is and what year she was born. These are all easy questions because it's a fact with clear answers.
The second category is complex problems, which is what people find ChatGPT very amazing. It can integrate information and answer why the sky is blue, as if it had its own logic. Another is to ask questions about the process, such as how do I bake a cake, can you not write down the steps step by step? Today we asked ChatGPT how to bake a cake, it can write the process to you in detail, how many glasses of water, how many eggs to add, how much flour to add, etc., it can tell you. Then you can actually bake a cake based on the answers it provides, and it may bake pretty well.
This is something that everyone thinks is amazing. But you know, this thing, in 2014, the computer has actually done it, and it has done it very well. So, there isn't much mystery about the technology itself.
Computers and people, who is better at writing?
Another reason people talk about ChatGPT right now is because they think it can be written. For example, write a job brief, which is where Americans use ChatGPT the most today. I did 1234567 this week, these seven things, hey, you see I don't have to bother writing myself, I'll let ChatGPT generate one, and then edit it a bit.
However, the matter of computer writing, in fact, you say that it is difficult and easy to say, I can give you an example.
After I left Google in 2014, I didn't do much programming at the time, but I still had some computing resources, so I would write some programs and play with it in my free time. At that time, I asked the computer to write two poems, which you can read.
The first poem is a five-word poem, which is called a poem in the style of Li Bai in my words, and you can read it. The poem was written by the computer itself. In fact, if you read it, there are really some of Li Bai's characteristics in this poem.
That second poem, I also put the picture below, you can take a look.
First of all, because ancient poems have a ping, but our current pronunciation is different from the pronunciation at that time, so we don't care whether this pingqi is in line with the ancient or not, but this we only from its content artistic conception, you will feel very smooth reading.
Okay, then again. How is the first poem made?
In fact, it couldn't be simpler, you put Li Bai's poem on the computer. Li Bai has more than 1,000 poems in total, which is only about 10,000 sentences, which is too simple for computers. When it is written, it is to divide the sentence into two words and three character groups, such as "empty worry" is a group, "remember Chang'an" three words in a group. Then it goes to spell the language model I just talked about, calculate the probability, which probability is the largest; After dismantling, I made a request to him, saying that I wanted to write a poem remembering Chang'an, and it was arranged and combined to produce this "Remembering Chang'an", which is actually put together in this way. The second poem is a little more complicated.
But do you know how long I wrote these two programs? Two days. What does this mean? It shows that you let the computer write something quite decent, in fact, it is not a very difficult thing, it is not as mysterious as you think, or the computer writing itself is not as mysterious as you think.
So why do these two poems look particularly good? Because this is Tang poetry, the format of Tang poetry is fixed. In the same way, Why is it good to write weekly reports in ChatGPT? Because the format of the weekly report is basically a list of pulls, it is also a fixed format. Including, if you read the Chinese edition of the Wall Street Journal, I'll tell you here, 90% of the content is written by computers, but you don't know it. After writing, of course, people must give it a theme, and then write an introduction to the first paragraph it writes, and then give a summary and a title, which is what people have to do.
Why is it better to write financial articles? Because it has a lot of facts in it and the format is fixed, it does it well.
When I spend so long talking about the background of ChatGPT, I actually want to say that it is not mysterious, not some very sophisticated machine behind it. On the one hand, ChatGPT relies on a mathematical model, which existed in 1972, but today it is very powerful and relies on brute force.
So, how much power does ChatGPT training consume at a time? It may be about 3,000 Tesla electric cars, each running 200,000 miles, running it to death, such a large power consumption, enough to train once, which is a very expensive thing.
What exactly does ChatGPT do to us?
So let's talk about what effect ChatGPT has on people.
This goes back to history, every technological revolution, in fact, it has some impact on people. However, ChatGPT is not a new technological revolution, because as I just talked about, the process is very long, from the 70s to the 90s of the 20th century, we did a lot of things, and from the 90s to the present, a lot of people have done a lot of things. The biggest improvement here is not actually the language model itself, but actually the deep learning produced around 2000, which makes training the language model more accurate than before, not simply doing statistics.
Today's training language model is no longer simply statistics, which is one of the reasons why ChatGPT can produce better results.
As for what kind of impact ChatGPT can have on people, I won't answer you directly to this question, let me ask you first, just show you these two Tang poems, have you found any characteristics? By the way, these two poems are well written, but you originally know about the Tang Dynasty, and you will not have a newer understanding because of these two poems. Because, ChatGPT is somewhat like parroting, you have to say a paragraph before it can follow along. It may sound good, but it doesn't provide much information.
Today, 90% of the content on the Internet falls into this category - not providing more new information, not original content, not one's own perceptions, nothing more than copying from east to west. At present, short videos such as Douyin and Kuaishou, I think 99% of the content belongs to this category, there is no nutrition, you may find it quite interesting after reading it, but in fact, no matter how much you read on it, it actually does not help you.
If ChatGPT really threatens anyone, I think what threatens is the work of this type of people, that is, the one who makes short videos on this TikTok, or publishes some content, ChatGPT will do much better than them. You want to think about such a thing, suppose that there is a group of people who turn the sentences in the three hundred Tang poems over and over every day, and can also make some poems, then ChatGPT must be much faster than people, so this technology will have an impact on this group of people.
So, who won't be affected? It's that the people who create the content won't be affected.
Why do I say that? Remember what I said about "why is the sky blue?" Why does Google answer this question?
Because when Google answered, it probably analyzed almost all decent sentences in English at that time, about 100 billion English sentences. So in fact, you'll find that on some university websites and NASA websites, it has this answer, but we put it together, cut it out, and picked it out. But the earliest physicists to do this research and clarify this truth, this work is meaningful, and ChatCPT cannot be replaced.
So, what does ChatGPT work like? For example, after Ptolemy created this model, every once in a while, they would compile a calendar of about a few decades, and then mark which day there would be a solar eclipse, how the planets would move, and so on. Then people according to these rules, print a lot of this book, this ChatGPT is equivalent to many books, you take it and look at it, say, oh, a certain year, month and day will occur lunar eclipse, the answer will be clear. But the really meaningful work behind it was not printing the book, but doing the research of Ptolemy.
So I think historically, ChatGPT is not really a technological revolution, it affects the lazy people, the people who are too lazy to use their brains and create new things. Those who truly explore the mysteries of human knowledge will not be replaced.
What exactly does ChatGPT do to us?
Many people ask, what are the new opportunities for ChatGPT? Frankly speaking, you don't have a chance, because it's too resource-intensive, you can't afford it. So who can benefit? That's the people who sell resources.
I can use an analogy, that is, during this California gold rush, many people flocked to pan for gold, and we still don't know which gold diggers actually made money, and none of them left their names. But who made the money in the end? It's the water seller and the jeans seller. The same goes for ChatGPT. Everyone goes to pan for gold together, in fact, you can't earn money, but in the process, you still have to buy water to drink, buy jeans to wear, and finally these two groups of people earn money. Levi's, such a company that came into being at that time, it made jeans.
Then in the end you may be paying money to several large cloud computing companies, which may be a result.
Well, after talking about the history of this ChatGPT, I will give you a brief summary.
First, don't be afraid.
Many people today fear ChatGPT, just as they don't fear lunar eclipses like the Jamaican natives Columbus met back then.
Second, don't force yourself to find the so-called opportunity, how you work is how you work.
I saw that some classmates asked me why Apple doesn't do ChatGPT, and I said that's right! That's why Apple is the richest company in the world, with the most profits and the most market capitalization. At present, many companies that so-called do this artificial intelligence are losing money until now. So, this is why many students sometimes ask a lot of questions that are too out of tune, I jokingly ask him, have you paid off your mortgage? If you don't pay it off, you go back to work and do your job well, which is the most meaningful thing for everyone, and historically it has been true.
Third, you need to see the tricks of these so-called conspirators or people who want to cut your leeks.
That is, if another person pretends that Columbus says that he is the representative of God, and then he can pray that heaven will make this moon come out, you don't believe it. So you need to understand some of the science behind ChatGPT. Some of the simplest principles, like the ones I'm talking about today, you still need to understand.
Note: This article is from Dr. Wu Jun's sharing on "Get".