Zhang Tao, head of Copilot technology application at SenseTime: Large models can't solve everything

Guest | Zhang Tao

Interview & Writing | Yun Zhao

出品 | 51CTO技术栈（微信号：blog51cto）

"Programmers' jobs are worried about by AI" is not new. Robin Li said that there is no need for programmers in the future, and Huang said that in the future, children should not learn to code!

Will the coding industry disappear and the developers of AI programming tools have the best say in answering this question.

Recently, SenseTime's Little Raccoon came out of the circle, and we invited Zhang Tao, the leading developer behind Little Raccoon, the head of SenseTime's Copilot application technology, in the AIGC practical school.

"Those parts that still need people to communicate with the machine carefully are called Coding in the past, now they are called Coding, and they will still be called Coding in the future!"

From human language with vague semantics to precisely running machine instructions, this gap will always need to be bridged by Coding.

In Zhang Tao's view, we might as well think about the meaning of AI programming from another angle. AI allows us to move beyond traditional coding and instead work to enable humans to communicate with machines in a more natural and intuitive way. Zhang Tao believes that eliminating Coding is not the goal, and people's real vision is to improve the efficiency and intuitiveness of programming, so that more people can participate in creating the future.

"AI is a new form of interaction, which can be combined with many vertical products. ”

In the past, product development often started from the scenario requirements, and then collected a variety of software components or technical implementation solutions to realize the products they originally wanted to define.

If you want to add AI to a mature product, or introduce AI in the early stages of the product, even if it is not an AI Native product, you need to take into account the limitations of AI in advance.

It is worth noting that the introduction of AI is not so simple, perhaps the way to introduce AI is not eye-catching, it may be a small button, or integrated into an invisible overall process, but behind this are a variety of scenarios and needs.

"You have to take care of the big model to make the product, because your core feature is provided by the big model. This is a clear change of thinking brought about by the AI Native product. ”

However, it is not a matter of course for software programmers to lead a product with a large model application.

As Zhang Tao said when referring to his role change, it is not enough to only understand large models, there are many more, such as product thinking, but the most important thing is to have corresponding domain knowledge and industry cognition.

"We can't be overconfident that we can do everything if we know the big model. ”

Here are the key takeaways from the interview:

• AI products will not eliminate programmers, but will promote the healthy development of the programmer industry.
• In fact, we are doing these two things now: first, to overcome several shortcomings of the current large model, and second, to stimulate the large model to think more.
• AI efficiency improvement tools are more about enabling it to deal with things that can be solved by human slow thinking.
• The cool point is that if there is no sense of accomplishment in a thing, but it must be done, the machine can directly help me complete it.
• You can't be overconfident that you can do everything if you understand the big model. Otherwise, the world may only need the more than 700 people who open AI, and the rest of the people don't need to make software.
• For scenarios with high requirements for accuracy, the cost of SFT is relatively high, and the operability is also limited, so it is better to directly plug in some knowledge bases.

The interview is as follows:

1. Coding will always exist, and the problem brushing monster may disappear

51CTO "AIGC Practical School": Recently, Robin Li, Zhou Hongyi, and Huang Jenxun have all expressed their views on whether the profession of programmer will disappear, what do you think about this issue?

Tao Zhang: Programmers will still exist, but the way they work is different. The programmer community is curious about technology and must have tried a variety of AI-assisted programming products. From a somatosensory point of view, most of the existing AI programming tools exist in the form of Copilot, which replaces developers to complete simple and boring programmatic work and accelerate development efficiency. The most important thing is that it can make your coding ideas smoother and silkier.

If there is a part of the people who will be replaced, "memorizing answers" or "brushing questions" may be dangerous. From another point of view, AI products will not eliminate programmers, but will promote the healthy development of the programmer industry.

On the one hand, you don't have to put too much effort into examining the ingenuity of existing classic algorithms or outlandish code. It's more about programming ideas, communication, collaboration, and problem-solving skills. On the other hand, it will also allow developers to be more active in the exploration of key technologies, and at the same time participate in the development experience and design of products. For example, I used to be a programmer, and now I actually spend half of my energy on product-related things. Therefore, in the era of AI, a worker who implements code will be transformed into a thinker.

51CTO "AIGC Practical School": Will Coding itself be eliminated?

Zhang Tao: I don't think it can be eliminated. The point of eliminating Coding is that people want to communicate with machines through "human" communication, that is, programming in natural language, but there is a natural gap between the two.

It's called natural language because it still has some vague semantics and parts that need to be clarified. It naturally has a gap with strictly formal languages, such as mathematical formulas, codes, etc., that is, no mathematician will use natural language to explain his mathematical findings.

As for the parts of Coding that can be replaced by AI, they are those parts that are organized and have established tasks, and those parts that still need to be carefully communicated by humans and machines.

51CTO "AIGC Practical School": As the efficiency of Coding increases, will enterprises need fewer programmers?

Tao Zhang: This problem does exist. Some of the established tasks or CRUD, in fact, these are things that AI can do with simple instructions from humans, and may even do it better than humans, and it is tireless. The skill or ability component of the programmer profession may change somewhat.

In the past, programmers' skills were more about programming, such as being particularly familiar with a certain programming language, understanding some programming ideas, and some practical experience. In the future, it may shift to setting aside more time to discover new algorithms and think more about new product features, for example, thinking about what new interaction methods and product forms will be brought to the programming field after the AI era, and I am an example.

Second, let the large model admit that he can't do it

51CTO "AIGC Practical School": Are AI programming products mature and what needs to be improved?

Zhang Tao: If we don't consider the limitations of the current model capabilities, it can be said that it is becoming mature. Now, there are two main limitations to the big model that are more critical. One is context length, and the other is hallucination. Nowadays, there are some engineering methods to solve these two types of problems. Although this is not a complete solution, it can be solved to a state of use, such as RAG, which essentially refines some high-quality data and information to increase the density of information in it to solve this problem.

In addition, there are also prompt engineering and SFT methods, so that the large model itself can suppress the illusion, so that when it encounters some unknown knowledge, it can honestly say "I don't know". These are all very important supporting projects in the process of landing large language models.

However, there is no mature state of reaching an established industry standard, and everyone is still groping. It may be clear on the path, but on the actual technology stack, there is no real consensus, such as the things mentioned above, whether langchain can do it or not, but it may be faster and easier to use AutoGen. For different scenarios, people will adopt some different methods. To sum up, in fact, everyone is doing these two things now: first, to overcome several shortcomings of the current large model, and second, to stimulate the large model to think more.

3. Let's talk about AI programming product ideas

51CTO "AIGC Practical School": In terms of stimulating the ability of large models, what kind of product path does the little raccoon family have?

Zhang Tao: The Little Raccoon family has completed two products, namely programming and office scenarios. From the underlying logic, these two products are actually an inherited relationship. AI efficiency improvement tools are more about enabling it to deal with things that can be solved by human slow thinking.

Slow thinking and logical reasoning may not be something that can be done directly by relying on this large model "next token prediction". So what to do?

First, give the product a specific capability that is most suitable for the large model. Code capability is a very powerful intrinsic capability of the large model, because the large model itself runs in the computer. We can internalize it and strengthen the programming ability of large models.

Second, after verifying that it is feasible, you can do more with your code capabilities to solve things that could not be solved by large models before: such as data analysis, data visualization and other tasks. This corresponds to the scene of an office raccoon. In addition to strengthening the code capability, it is also necessary to combine the execution feedback of the sandbox to strengthen the multi-round reflection ability of the large model. In the future, this series of products also plans to activate more capabilities of large models through websearch, knowledge base, multi-agent, etc.

In terms of the form of AI programming products, at present, based on code capabilities, it is natural to think of integrating into the IDE that developers often use in the form of plug-ins, which is the most natural form. The rest of the programming-related work, such as the front-end, is to fork the code against a high-fidelity image, but it can also be done directly for the large model, which is what the multimodal large model is good at.

51CTO "AIGC Practical School": From the perspective of users, where are the pain points and cool points of AI products?

Zhang Tao: Judging from the data of domestic and foreign Copilot products, the acceptance of developers is very high. The cool point is that if there is no sense of accomplishment in a thing, but it must be done, the machine can directly help me complete it.

For example, in the process of writing code, developers have many tasks, simple ideas, and many established methods, but they do not have APIs, and they need to spend time typing. This kind of work is handed over to the AI, and as soon as the code is written there, the little raccoon can directly give the next ten lines of functions. This is a big cool point for programmers.

Devin is a valuable method for developers to search for technical documents, and the credibility cannot only rely on the introspection ability of the large model itself. A better way is to use strong reference information from RAG or web search, which is attributed to the source and can be reviewed.

In addition to helping you write code, there is also a chat with you to solve some problems encountered in the programming process, and you can directly ask questions about certain technical points of the code.

Fourth, role change: it is not enough to understand large models

51CTO "AIGC Practical School": From developers to AI product builders, how has it changed?

Tao Zhang: In the past, I was mainly a programmer, but now I am mainly working on code products, so I will have more requirements in product thinking. But it's not exactly a shift to a product role.

The reason why code products are chosen is because vertical AI products need to have knowledge and industry cognition in this field.

My role here is not so much of a model expert, but more of a practitioner in the industry in which the product is located – a programmer. We can't be overconfident that we can do everything if we know the big model. Otherwise, the world may only need the 700 people who are Open AI, and the rest of the people don't need to make software.

51CTO "AIGC Practical School": Compared with the development of non-large model products before, what are the changes in the process?

Tao Zhang: The previous products were more based on the scenario, and then collected some software components or some technical implementation solutions to implement their own pre-defined products. If you want to add AI to a mature product, or introduce AI in the early stage of the product, even if it is not an AI Native product, you need to take into account the limitations of AI in advance.

In the past, you may start from the requirements design, but now you may have certain considerations and need to take care of the large model, because if some of the core features of the product are provided by the large model, the thinking needs to be changed.

For example, in code products, we first have code scenarios, and then we must consider how the large model works, how to develop its strengths and avoid weaknesses at the first time when it is put in, and make the large model work better for programming scenarios in the preparation stage and in the process of polishing again and again.

5. The polishing of AI products requires the cooperation of both engineering and model

51CTO "AIGC Practical School": What are the tricky challenges in creating an AI product?

Tao Zhang: At present, there are many thorny problems in AI products. For example, how to make large models more practical, the RAG solution we just mentioned actually needs to do a lot of work, how to ensure that the performance effect is improved. Although it is said to be a method of "adding featured data to prompts", the featured data here is also a vague definition, so whether his real understanding of the model is buff or loss or noise requires a lot of research work here. For example, for scenarios with high requirements for correctness, the cost of SFT is relatively high, and the operability is also limited, so it is better to directly plug in some knowledge bases. In addition, there is a lot of experimental work to be done from the perspective of engineering and model coordination, and the process cannot be defined very precisely. All of the above needs to be solved on both the project side and the model side.

Source: 51CTO Technology Stack