Produced by Tiger Sniff Technology Group
Author: Qi Jian
Editor|Wang Yipeng
Header Image|DALL-E 3
Before the advent of ChatGPT, the idea of "AI replacing humans" was always considered a "sci-fi" gimmick for AI companies. However, after seeing the super capabilities of large language models, people are really starting to worry a little about their "jobs", especially programmers.
In March 2024, startup Cognition showcased Devin, one of the company's AI programming tools. Cognition describes Devin as "the world's first fully autonomous AI software engineer."
In Cognition's demo video, Devin is able to build an application from scratch with only natural language instructions. Devin also completed an actual task order on Upwork, a crowdsourcing platform, not only to write and debug the computer vision model, but also to compile a report at the end of the task.
Cognition even posted on social media platforms that Devin had passed an interview with an engineer at a leading AI company.
So far, the remarks that "those who develop AI are the first to be replaced by AI" are rapidly heating up on the Internet.
However, just a few days ago, Carl, an American engineer who claimed to have 35 years of development experience, posted a video on his YouTube account, Internet of Bugs, saying that Devin's demo might be fake.
Carl, an American engineer who claims to have 35 years of development experience, posted a video on YouTube saying that Devin's demo may be fake
Could Devin just be "acting" a programmer?
At present, Devin is not yet open for use, and applications can only be submitted through email, so the outside world's perception of Devin basically comes from the official demo video and the evaluation of a small number of third-party developers and product personnel.
Among the third-party evaluations, the most popular ones include a report published by Bloomberg on March 12 about Cognition AI and several of its Chinese founders. As mentioned in the report, the journalist used Devin to build a website from scratch in 5-10 minutes. The report also quoted Silas Alberti, a computer scientist who tried Devin, as having a high opinion of Devin.
Although Carl, a blogger who questioned the cost of Devin, did not actually test Devin. However, by analyzing the Devin hands-on video released by Cognition AI, he proposed a number of suspected fake content in the demo, including:
1. In the demo video, many of Devin's actions are not actually understood or completed correctly. The reports generated by Devin do not contain the information that the customer actually needs.
2. When fixing the wrong code, Devin also had the problem of "it is not possible to create difficulties". Carl points out that some of the code that Devin fixes is a mistake that "humans never make". So it's likely that these errors don't come from the customer's codebase, but Devin himself.
3. Technically, Devin isn't doing very well either. Sometimes they take an outdated approach to handling files, perform inappropriate operations in a Python environment, or even write their own low-level file read loops instead of using standard libraries correctly.
4. The video looks like Devin completed the task quickly, the video creator was able to complete the requested task in about 30 minutes, but the timestamp in the chat shows that the task lasted for many hours and even into the next day.
While Carl believes that Cognition is exaggerated in Devin's demonstrations and hands-on operations, he also says that Devin has demonstrated impressive capabilities in some technical operations. For example, Devin is able to automatically update and configure files to match the desired library version. He believes that as technology advances, AI tools like Devin can fully understand and perform more complex programming tasks.
Humans are still the key to AI programming?
At present, Devin has not yet been launched, and whether its ability is really as strong as Demo is actually difficult to distinguish between true and false.
What is certain, however, is that while AI can indeed help humans with programming, the mainstream view is still that AI can replace humans.
"AI is nowhere near as smart as rumored to be, and it takes a lot of repeated communication to understand business logic, let alone work. A developer who focuses on AI programming told Tiger Sniff that many programmers are currently using tools like Copilot, but not many people are worried about being quickly replaced.
In fact, there are still two difficult problems in the work of large language models: on the one hand, the input information may not be able to understand the underlying logic correctly, and it may require repeated communication with AI. On the other hand, it is difficult to avoid errors and illusions in the content of AI output, and there is a great risk in the work that requires more rigor and seriousness without human inspection and supervision.
"When a human programmer writes a program, the result is basically certain, either good or bad. But after the big model came, it was hard to say. Chen Liang, CTO of Alipay, said that there is great uncertainty about the results of AI writing programs, "He can write better than you, or he may be worse than you." "It takes a human programmer to find a way to control it or influence it.
Chen Liang believes that although the output result cannot be determined like writing a program before, it can be influenced by translation, corpus, data, etc., so that AI has a greater probability of providing good results.
When Cognition just released the Devin demo video, Andrej Karpathy, former Tesla's AI technical director and OpenAI co-creator, once posted a comment on X:
"In my opinion, automated software engineering looks similar to autonomous driving. The development of autonomous driving is:
1. First of all, humans perform all driving actions manually
2. Then the AI helps to keep the lane
3. You can slow down when you see the car in front
4. It also changes lanes
5. It will also stop at the sign/traffic light and take turns
6. Eventually you get a fully functional solution and continuously improve the quality until fully autonomous driving is achieved.
In software engineering, progress is similar. It's all about AI doing more and humans doing less, but still requiring supervision:
1. First, humans write code by hand
2. Then GitHub Copilot autocompletes a few lines
3. ChatGPT写入代码块
4. Move to larger code diffs (e.g. Cursor copilot++ style)
5....
Andrej Karpathy believes that Devin is an impressive demo that could drive advancements in coding tools, including terminals, browsers, code editors, and more, as well as human oversight.
What is the AI crisis for programmers?
While Devin may not be a substitute for humans, many human jobs are already being threatened by AI.
"AI doesn't necessarily replace people, but people who can use AI must replace people who can't. ”
Since 2023, the importance of AI tools for work has risen to new heights.
Chen Liang said that at present, 60% of Ant programmers use Codefuse when writing code, and about 12% of the code written by Ant programmers using Codefuse is generated by Codefuse.
Yibo, the founder of Yichuang Technology, told a more vivid example.
"At the beginning of 2023, I required all programmers in the company to be able to write code in ChatGPT, and spent one month learning and assessing, and the average efficiency increased by 3-5 times after the assessment. One of our strongest architects is basically 5 to 10 times more efficient at programming. ”
Yibo said that before using Copilot tools, the architect has to take the customer requirements and sort them out, and then arrange 5 to 10 people to help him do the work, and he has to tell these people: you will do the front-end framework. You write the back-end interface.
"Now there is no need for this, after the architect has figured it out now, it is clearly written in the document, and he can directly use ChatGPT to get it out. "It may take a week for a team of 10 people to explain something clearly, do it well, check it clearly. Now it may take less than a day, or even a few hours.
Devin is innocent, and the sin is hype
Devin has propelled his creator, Cognition, to the forefront. Founded at the end of 2023, the company has already received about $21 million in investment from well-known institutions such as Founders Fund. The company was valued at about $350 million in its seed round and is now valued at $2 billion.
Now, with Carl questioning the title of Devin's "World's First AI Software Engineer" as false advertising, Cognition is facing a public relations crisis.
Regarding "false advertising", Carl criticized the promotion of AI products for often lacking transparency and not truthfully showing the actual capabilities and limitations of the product, thus misleading the public and professionals.
In the past year, there have been a lot of cases of "false propaganda" of explosive AI.
One of the most famous is "Google Releases Gemini Demo Video". In a demo video released by Google in its latest AI model, Gemini, it demonstrates the multimodal capabilities of experience. But the company later admitted that the video was actually edited, using static images and text prompts instead of real-time motion processing. Latency has been reduced in the video, simplifying Gemini's output.
Not long ago, Amazon launched a "No Checkout" technology (Just Walk Out), which was also pointed out as artificial "disguised" artificial intelligence. Rather than relying solely on advanced AI and computer vision, the technology is reported to be manually reviewed by more than 1,000 Indian workers to ensure checkout accuracy. However, Amazon has denied this claim, claiming that these workers are mainly training models to help improve the technology.
In an earlier case, a well-known AI company in China was accused of disguising the content of human simultaneous interpretation as AI translation content at a conference site, and displaying the company's product logo.
In a video accusing Devin of fraud, Carl called on manufacturers, media and users to be skeptical of AI technology and make the necessary verifications. Carl believes that this exaggeration not only misleads the public, but also sets unrealistic expectations for AI technology.
Some netizens also expressed support for Carl's views.
rayhere7925 argues that many of these tech start-ups have only one goal: to make money. How do they get funding? through investors and/or hype. I'm sure this bombastic presentation, and all the articles and influencer videos that come with it, must have convinced investors in some places to inject capital into this company – which is exactly what this company is aiming for. If it means more money can be made, companies will lie (if they can get away with it). It's no surprise that this company lies in order to hype and attract investors.
The problem with modern journalism, according to Apexphp, is that people who used to write technical articles usually had some knowledge of the field, they knew the meaning of the jargon, they were able to read scientific papers in their field of writing, and they understood most of them. But right now, we just have a bunch of pay-per-article people who find interesting headlines on the internet and simply write a few sentences because they work for a company that pays the same as they would pay for a well-written, well-researched article, so they don't have to bother.
People who are changing and want to change the world are all in the Tiger Sniff APP