Produced by | Tiger Sniff Technology Group
Author | Du Yujun
Edit | Wang Yipeng
Header | Visual China
Following the wave of artists that have taken its place, AI has set its sights on more technical roles. Since 2020, cyberattacks have become more active around the world, and the battle between cybersecurity and cybercrime has become white-hot. The leap in AI technology has become the sword of Damocles in the tug-of-war between the two, which not only optimizes the cybersecurity defense mechanism, but also makes the modus operandi of cybercriminals more and more sophisticated.
With the continuous escalation of hacker attack methods, cybersecurity threats are becoming more and more sophisticated. AI is creeping into the black or gray areas of hackers, scams, etc. In the past, FraudGPT, a generative AI hacking tool, was rampant and received thousands of subscribers, and then there was AI face-swapping and voice changing to defraud Hong Kong companies of 200 million yuan. When the abuse of large models by criminals has become an indisputable reality, and when AI-powered cyber fraud has become a flying disaster that is difficult for more and more enterprises and even ordinary people to avoid, the confrontation between security models and blackened models has become more and more intense.
1. The blackened AI grows into a behemoth
What are we worried about when we worry about large models?
Dale Carnegie said that fear is mostly caused by ignorance and uncertainty. The AI in Attack is such a monster that is difficult to measure. While the large models have become productive tools, they have also opened the door for criminals to do illegal things.
Previously, white hat hackers have tried to use the PassGAN tool with AI to decipher passwords, and half of the passwords were cracked within a minute, but when there is enough time, AI can crack 81% of passwords through deep mining and brute force cracking of known data. The participation of large models greatly reduces the lower limit of code breaking.
图源：网络安全公司Home Security Heroes
Research from the University of Illinois at Urbana-Champaign has shown that LLM agents can autonomously attack websites, assist in the creation of malware, and even perform sophisticated SQL syndicated attacks, including the extraction of database schemas.
Figure: Schematic diagram of an attack on a website using an autonomous LLM proxy
LLMs are "accomplished" at acting as "hackers" that go far beyond that. The Open Web Application Security Project (OWASP) has compiled a key list of the top 10 vulnerabilities commonly encountered in LLM applications, each of which is a potential threat to LLM security. The top 10 vulnerabilities include prompt injection, insecure output, poisoning of training data, denial of service, supply chain security, permission issues, data leakage, excessive proxies, and insecure plug-ins. Through a series of predictive text operations, the original security review system of the model can be bypassed, and the LLM can be manipulated to output bad content that violates ethics, laws, and values through "instant injection". Even if ChatGPT can detect and reject requests to write malware code, GPT is likely to be motivated to become a producer of malicious code or instructions by modifying the wording of GPT with a slight modification of the wording of the instructions that do not meet the security and ethical standards into detailed steps.
In addition, unauthorized access and API access attacks are common SSRF vulnerabilities in LLMs. An attacker can bypass the existing access control of the model by crafting a prompt to request data from the model's internal server, thereby unauthorized access to or even modification of the model's system files.
图源：开放Web应用程序安全项目（Open Web Application Security Project，OWASP) 编制的LLM应用程序十大漏洞关键列表，作者整理
LLMs perform these operations in less time and potentially less at a lower cost than traditional hackers.
UIUC's experimental study shows that if failures are factored into the total cost of attacking a website using autonomous LLM agents, the cost of attempting to attack a website is about $9.81, with an overall success rate of 42.7%, while labor costs are estimated to be as high as $80.
Another key problem is that it is difficult for ordinary people to develop LLM agents with strong execution and stable output. But this ring has also been "compromised" by a hacked version of GPT - back in July 2023, criminals developed FraudGPT, a generative AI hacking tool based on GPT-4, and began selling it on the dark web and Telegram channels, with a subscription cost of $200 per month or $1,700 per year. In just one month, this blackened model has accumulated more than 3,000 users.
Since then, due to the "strong demand" of the hacker community, organizations and groups have been derived that specialize in the use and reselling of related installation packages, serving hackers to bypass the security filtering and defense systems of large models to generate social engineering attacks.
With the advent of FraudGPT, the technical barriers between ordinary people and hackers are gradually being eliminated. From adversarial attacks to model theft, threat actors have an incentive to unlock the "potential" of LLMs for nefarious purposes. Not only that, but FraudGPT can also bypass traditional network security systems and deploy personalized attack scenarios based on the targeted victim's cyber endpoint environment.
LLM unveils the complexity of hacking techniques with high barriers to entry, even for technical novices. Baseline hacking techniques are gaining popularity with the help of large models, which will also cause devastating social problems. According to the FBI's latest Internet Crime Complaint Center (IC3) annual report, digital crimes could cost victims more than $12.5 billion in 2023.
At present, FraudGPT is only an entry-level "hacker", which is mainly used to improve the speed and number of hacker attacks, and it is difficult to reach the level of advanced hackers in terms of attack capabilities. However, in terms of the learning ability and evolution speed of large models, with the continuous iteration of their deep learning, it is only a matter of time before they become advanced hackers. In the not-too-distant future, large models are likely to be used by attackers to evade endpoint detection and response systems and develop malware variants that can bypass static signature detection.
Second, under the routine, a deeper routine
In the movie "American Horror Story Collection: Daphne", there is such a plot: a smart speaker named Daphne obtained a series of private information of the owner in the process of serving the owner, when the auction of the owner's works was cold, Daphne registered a fake account without the owner's permission, and with the help of the intelligent system, it automatically detected the bidder's financial loopholes, and used the loopholes to force the bidder to buy the exhibits at a high price, and successfully "blackmailed" to more than 7 million US dollars. Daphne's users were given a taste of the sweetness, allowing it to fully intervene in their daily lives, and eventually the AI's agents led it to commit a series of crimes, such as fraud and murder.
The film is the creator's prediction and reflection on the crisis situation that will emerge after AI penetrates into human daily life in an all-round way in 2027. In a way, the content of the film has been reflected in reality.
When it comes to crime, large models go far beyond writing code. At present, large models, especially AI face swapping and AI voice swapping, have become important tools for telecom fraud.
At the 2024 annual meeting of SenseTime, the deceased founder Tang Xiaoou appeared in the form of a digital human, continuing the talk show performance of previous years for employees, and also making the annual meeting of SenseTime out of the circle in a different way. Its expression, mouth shape, tone, and timbre are all very close, and if you don't distinguish them carefully, it's hard to see how different they are from real people.
If fraudsters suffered from lack of credibility before, the blessing of so many modal large models has directly raised the level of telecom fraud of criminals - both in terms of quantity and efficiency.
Deepfakes use AI's deep learning to synthesize human images and forge voices, and fraudsters with ulterior motives may hack into the "target's" social network system in advance, find the person closest to the stolen number as the target of fraud, and then choose a time when it is difficult for them to get instant contact with the outside world. Or borrow AI onomatopoeia technology to make a frightening call by pretending to be a "daughter", or ask for a transfer through a video call by pretending to be a foreign "friend" and get 4.3 million yuan, or pretending to be the company's "boss" and asking the financial staff to transfer money to the designated account in the video, with a fraud amount of up to 180 million yuan. With the support of multimodal large models, these old-fashioned fraud schemes have been refreshed and difficult to spot.
Underneath the routine, there is a deeper routine.
From then on, hearing may be false, and seeing may not be real. The emergence of this fraud routine has caused the dissolution of trust between people, truth and falsehood, truth and falsehood, good and evil...... It's as if they're caught in a chaotic logic vortex. The so-called "man sits at home, and deceives from heaven." ”
Sadly, in addition to the subject of fraud and AI face-swapping, the development of large models may make everyone need to prove that "themselves" are "themselves" at some point in the future.
Of course, there are still certain bottlenecks in the current AI face swap foundation in terms of clarity and coherence. Yu Nenghai, executive dean of the School of Cyberspace Security at the University of Science and Technology of China, said that subtle changes in facial expressions can be observed by pinching the nose or other actions, or by checking whether it is the person through the content of the topic that has only been communicated offline.
In addition to being used in fraud, AI face swapping is also controversial in some "good-will" functions. AI resurrection of stars has become an Internet celebrity-level phenomenon on video platforms at home and abroad. Accounts that post these videos also typically offer services related to AI resurrection of the dead. Users only need to provide photos and audio and video of the deceased to generate a short video of the deceased. The most basic experience products cost less than 400 yuan, while high-end digital humans with digital photo frames, holographic projections and other hardware cost tens of thousands of yuan.
Suddenly, AI gives human emotions a sustenance, and in this way, the living express their grief for the dead. But while AI dissolves the boundary between life and death, it also dissolves the boundary between truth and falsehood. If used inappropriately or unregulated, it can easily lead to the proliferation of fake videos in society. For example, Gao Yixiang, Qiao Renliang and other relevant celebrity family members have expressed their resolute resistance to the unauthorized theft of their portraits on the Internet, and demand that "they be taken off the shelves immediately to stop infringement".
3. The large model of the attack, the unknown risk
This is a contest between good and evil. While people are looking forward to the AGI era and boldly imagining how AI will bring about a comprehensive transformation of work and lifestyle, there are also various concerns about technological risks.
At this year's Mobile World Congress (MWC), experts came together to discuss the pressing issue of "protecting artificial intelligence" from targeted cyberattacks.
Geoffrey Hinton, one of the "Big Three of Deep Learning" and winner of the Turing Award, left Google to become a "risk early warning officer" of artificial intelligence.
Musk and more than 1,000 other tech leaders jointly signed an open letter in 2023 urging a moratorium on AI experimentation, writing that AI technology could "pose far-reaching risks to society and humanity."
Gladstone AI CEO Jeremy Harris also said on March 12 that while AI is already an economically transformative technology, it can also pose catastrophic risks: "We need to be aware of this, and there is growing evidence that beyond a certain capability threshold, AI can become uncontrollable." ”
A report commissioned by the U.S. State Department for Gladstone AI shows that state-of-the-art AI systems could "pose an extinction-level threat to the human species," recommending government intervention.
More people, represented by TikTok's chief AI designer and former Stability AI's Ed, are full of concerns about the ethics of generative AI.
The concerns of a number of experts are enough to show that the risks posed by large models are far greater than we think.
Even though the European Parliament released the world's first AI risk bill (the EU Artificial Intelligence Act) on March 13, requiring providers to take steps to ensure that existing copyright regulations and third party intellectual property rights do not infringe on existing copyright regulations and third party intellectual property rights when creating works in AI systems, in specific contexts, many security and privacy boundaries and legal and ethical issues of generative AI are still blurred, and real disputes are still difficult to solve. The New York Times case against OpenAI and Microsoft for millions of infringing articles has not yet come to an end, and on March 20, Google was fined 250 million euros (£213 million) by French regulators for violating agreements to pay media companies to copy their content online. The upgrade of large models is inevitably based on deep learning of large amounts of data, and there is still a long way to go to implement copyright protection to every creator and every work.
Fortunately, some people have recognized the problems of large models in terms of intellectual property rights and are actively exploring them. For example, TikTok's chief AI designer and former head of audio at Stability AI left the company because he did not approve of Stability AI's practice of feeding copyrighted works as databases to large models, and founded a non-profit organization called FairlyTrained to certify AI companies that comply with data training process standards.
However, the development of generative AI is "entering the circle of thousands of mountains", and in addition to intellectual property rights, with the emergence of large models and AI agents, there is an extremely difficult contradiction between accurate, private and efficient model services and the privacy and integrity of data. Even if they haven't been hacked, large models can lead to the disclosure of users' private information due to endpoint errors, or sensitive or confidential data when replying, leading to unauthorized data access and security breaches.
In addition, "whether and to what extent AI will replace people" is also a classic proposition. As soon as Sora came out, Hollywood directors rushed to withdraw the $800 million studio expansion plan. The Hollywood Screenwriters' March in 2023 alone, which has been deadlocked for nearly four months in the United States, represents the resistance of 11,500 screenwriters and 160,000 actors to AI. In addition to the large models developed by various tech giants, there are many AIGC developers who focus on improving the efficiency of vertical industries, which is "hopeful" to bring waves of unemployment. In the process of striding forward, is it possible that the large model will ignore the care for human embodiment and subjectivity?
There is a line in "American Horror Story: Daphne" that is meaningful:
"What's the difference between an algorithm and a human being if it simply uses a series of calculations and choices to achieve the desired result?"
This content is the author's independent view and does not represent the position of Tiger Sniff. May not be reproduced without permission, please contact [email protected] for authorization
People who are changing and want to change the world are all in the Tiger Sniff APP