The face in the video is facing the camera, barely moving, and the expression on the face is no different from usual.
After 7 seconds of video call, Fang Yang You, a native of Jinan, transferred 300,000 yuan to the other party's designated account. When he called the police, he learned that the familiar face in the original video was "fake", and the other end of the screen was not his relatives at all, but an image video synthesized by fraudsters using AI technology.
This AI scam incident in May this year is not alone. Zhang Zhenhua, captain of the Criminal Police Brigade of the Shanghe County Public Security Bureau in Shandong Province, told the Beijing News reporter that with the open and open source of deep synthesis technology, deep synthesis products and services have gradually increased, and the illegal acts of using fake audio and video such as "AI face change" and "AI voice change" to defraud and slander are increasing. Previously, police in Harbin, Fuzhou and other places have reported fraud incidents related to AI face changing.
Wang Jie, deputy director and associate researcher at the Institute of Rule of Law at the Beijing Academy of Social Sciences, who has long been engaged in anti-fraud research, believes that in the era of artificial intelligence, faces and voices have become a kind of information that urgently needs to be protected. However, with the development of AI technology, the use of AI face changing technology for fraud, rumor-mongering, infringement of portrait rights or intellectual property rights has occurred frequently, which means that the technology is being abused, and the important information of the face is easily "lost".
While AI real-time face changing technology is developing, it is also rapidly "civilian". An investigation by a reporter from the Beijing News found that AI real-time face changing technology is sold on multiple platforms at a price of about 100 yuan, and one of the merchants said that 368 yuan can buy its own real-time face changing program, which can be used unlimited times within 50 years. Considerable profits drive the development of AI gray market, some merchants claim to have a monthly income of up to 200,000 yuan, he claims that he can do any video, arbitrarily change a face into a pornographic video, it only takes 150 yuan a minute.
"Our future will definitely coexist with AI," Wang said, which means that humans will also coexist with AI-related infringements. How to make technology develop for good has become a very important issue at present.
After Fang Yangyou reported the case, the Shanghe criminal police rushed to Guangdong and arrested Xie and three other suspects suspected of helping trust. The case is currently under further investigation. Source: "Jinan Public Security" WeChat public account
AI scam
The video connection lasted only 7 seconds, looking at the familiar face on the screen and listening to the familiar voice coming from the mobile phone, Fang Yangyou believed what the new account that had just added himself as a friend said - he was indeed his relative, and 300,000 yuan was transferred to the bank account provided by the other party.
"Everything turned out to be fake." At the end of June 2023, Zhang Zhenhua, captain of the criminal police brigade of the Shanghe County Public Security Bureau, introduced the case to a reporter from the Beijing News, "This is the recently popular 'AI fraud', a fraud method that uses intelligent AI technology to carry out fraud." ”
It happened on May 29 this year, when Fang Yangyou, a native of Jinan, was scrolling through a short video at home when he suddenly received a message from a stranger. The other party claimed to be a relative of Fang Yangyou, and he sent a new QQ number, hoping that Fang Yangyou would add it. As soon as he added his friend, the other party called Fang Yangyou, and the 7-second video connection was a little stuttered, but the familiar face and the familiar voice made Fang Yangyou believe that this was indeed his "big brother".
Without saying a few words, the other party hung up the video on the grounds that the network quality was not good. In the subsequent text chat, the "big brother" said that he urgently needed to transfer a sum of money, but he couldn't make a direct payment, and wanted to transfer it to Fang Yangyou first, and then ask him to help transfer it to the designated bank card.
"It's urgent over there, you can arrange it right now." Two screenshots of the successful transfer were sent, but Fang Yangyou never received the money. The "big brother" "dumped the pot" to the bank, "that should be 24 hours away, the bank's information will not be wrong." On the grounds of being in a hurry to use the money, the "eldest brother" repeatedly urged Fang Yangyou to pay in advance.
Unable to withstand repeated urging, Fang Yangyou transferred 300,000 yuan to his designated account. Not long after, the other party asked Fang Yangyou to transfer another 350,000 yuan. At this time, Fang Yangyou suddenly remembered the anti-fraud propaganda he had seen, and quickly called his relatives, only then did he know that everything was fake.
Jinan native Fang Yang You encountered an AI fraud, and after 7 seconds of video, he transferred 300,000 yuan to the other party's designated account. Source: "Jinan Public Security" WeChat public account
That night, Fang Yangyou reported the case. According to the flow of the transfer card number, Zhang Zhenhua and his colleagues found that the bank card had been spent at a gold store in Guangdong. The next day, the Shanghe criminal police went to Dongguan, Guangdong, and arrested the criminal suspect Xie and three others. Later, three more suspects were captured by the police.
In a follow-up investigation, the police found that the real source of this AI fraud was in northern Myanmar. This group of people in China is a criminal team that specializes in helping overseas fraudsters to launder money, and they have a clear division of labor, and the purchase of money, payment, docking and online are all operated by different people. At present, they are under follow-up investigation on suspicion of the crime of aiding and trusting.
In fact, cases of using AI technology to commit fraud have occurred in many parts of the country. Wang Jie first learned about AI scams in 2019, when an international student saw scammers using technology to synthesize videos of her parents and believed it. Previously, Harbin, Fuzhou and other Internet police have reported fraud incidents related to AI face changing, and their methods are not exactly the same.
On May 20, the Baotou Municipal Public Security Bureau's official WeChat account, Ping An Baotou, posted that in April this year, criminal gangs used AI face-changing technology to gain the trust of victims through video chat, allowing them to call 4.3 million yuan to the scammers' designated accounts within 10 minutes. After the victim reported the case, the Baotou City police and bank jointly intervened and successfully stopped the payment of more than 3.3 million yuan of suspected telecommunications fraud funds.
On June 15, the official WeChat account of the "Dalian Banking Association" also posted that after fraudsters obtained the image information of a man's ID card, they synthesized a short video through AI technology, and used the video to successfully deal with the face recognition verification of the bank's large-scale fund transfer, cancel the time deposit in his card, and then transfer some of the funds. Fortunately, this man set a daily cumulative transaction limit for the bank's non-counter channels, and some funds that exceeded the limit could not be transferred out in real time, effectively avoiding the loss of some funds.
There are also scammers who target the "voice".
In 2022, lawyer Ma Junzhe revealed his experience of being defrauded by telecommunications by 30,000 yuan in a short video. The scammer used AI technology to synthesize Ma Junzhe's mother's voice and called him pretending to be his mother, saying that his father had broken bones due to a car accident and urgently needed 30,000 yuan in medical expenses, hoping that Ma Junzhe would transfer the money to her friend's account.
At the moment of transferring the money, Ma Junzhe found that the place of ownership of the bank card was not the location of his parents, Ma Junzhe immediately called back the series of calls from his mother, and after his mother confirmed it, Ma Junzhe called his father again, but his father's phone could not be reached. He thought his father was doing a check-up, so he sent the money directly.
But when Ma Junzhe called his mother again the next day, her mother said that there was no such thing. Ma Junzhe was puzzled, "The voice is my mother's, I can't hear it wrong." The number is my, I can't be wrong. After calling the police, he learned from the police that this is a new type of fraud, the fraud gang uses technology to synthesize the mother's voice, and then uses technology to copy a series of virtual numbers exactly the same as the mother's number to make calls, and uses technical means to block the number of relatives during the scam and let them cut off contact, thereby increasing the credibility of the scam.
Who is "losing face"
"Behind AI scams, there is still the problem of people's information leakage." Wang Jie said that with the advent of the era of artificial intelligence, people's faces and voices have also become a kind of information that can obtain benefits, and many people have "lost face" without realizing it. As individual biometric information, facial information is sensitive personal information, with direct identification, uniqueness and uniqueness, and once leaked, it will cause the personal dignity of the information subject to be infringed or the personal and property safety will be harmed, and its harm is more serious than the general personal information leakage.
Anhui native Pan Ziping has personally experienced that after only a few seconds of video call, his face information is lost.
On the evening of March 24, Pan Ziping was fascinated by reading fairy tales on his mobile phone. When turning the page, he accidentally touched the ad on the side, and a game began to download automatically in the background of the phone. After the download was completed, Pan Ziping opened it to see that it was also a game related to Xianxia, and he played it for a while and felt bored, so he uninstalled it directly.
A few seconds later, a phone call from abroad came over, and the other party said that he had obtained all the information in Pan Ziping's mobile phone, including photo albums and address books, and wanted to add QQ to discuss in detail. Pan Ziping didn't think too much about it, so he added friends. As soon as it was added, a video chat came directly, the other party's face never appeared in front of the camera, and a male voice simply reported the names of several people in Pan Ziping's address book, and then hung up the video.
Although neither phone call mentioned the money, Pan Ziping became wary in his heart, and he immediately withdrew the money to the bank card and untied the bank card with Alipay and WeChat.
A few minutes later, a QQ message came. This time it is a video, about ten seconds, in which a naked man with Pan Ziping's face on his head is doing some indecent movements. Pan Ziping was stunned, "Is the current technology already so powerful?" Before he could react, the call came again, "You give me thirty-eight, or I'll send your 'little video' to everyone in your address book." ”
The other party also sent a screenshot, this video is ready in the dialog box, as long as you click send, Pan Ziping's relatives and friends in the address book can receive it. When the video came again, the other party turned on the screen sharing function, Pan Ziping showed the balance of Alipay and WeChat to the other party, saying that he couldn't take so much money, and the two sides started bargaining from 30,000, cutting to 28,000, 18,000, until 8,000 yuan, Pan Ziping also said that he couldn't take it. After that, he began to instruct Pan Ziping to open online loan platforms such as borrowing, in an attempt to let Pan Ziping give them money by borrowing online loans.
Just a second before the transfer, Pan Ziping suddenly woke up, he immediately hung up the video and dialed 110 to call the police. Since no actual damage was caused, the police did not file a case. Pan Ziping asked the police, what to do with the obscene video that uses AI technology to change faces in real time? The police were also unable to destroy the video, but could only tell him that the editor sent a mass message to the address book to inform him of the incident.
Pan Ziping believes that it was the game that leaked his information, and he hurriedly checked the mobile phone permissions of each app to avoid encountering "danger" again.
In Wang Jie's research, there are many people who have lost personal information due to viewing irregular websites, and even been scammed. He pointed out that collecting face information is a very simple thing for existing technology. Cameras everywhere, face detection for authentication in mobile phones, and apps with album reading permissions are quietly obtaining face information.
Jinan native Fang Yangyou encountered an AI scam, and the other party only had a video call with him for 7 seconds. Source: "Jinan Public Security" WeChat public account
In 2021, "Focus Interview" reported that in the Ministry of Public Security's "Clean Net 2021" special operation, the Hefei Municipal Public Security Bureau's Cyber Security Detachment joined hands with the Baohe Branch to destroy a criminal gang that illegally used AI artificial intelligence technology to forge dynamic videos of other people's faces and provide technical support such as registering mobile phone cards for the black and ash industry chain. There are about 10 gigabytes of face data in the suspect's computer, which change hands many times online, and their owners are unaware of it.
Celebrity influencers who are active on the screen are more likely to lose their face information than ordinary people. In August 2021, Liu Haoran's studio issued a statement saying that due to the large number of videos and video screenshots of insults against actor Liu Haoran using AI face-changing technology and chat records containing insulting and defamatory remarks on the Internet, the above videos and screenshots were also continuously sold, and the studio reported the case to the police. Recently, some merchants have used AI face changing technology to apply the facial features of actors Di Li Gerba and Yang Mi to their e-commerce platform live broadcast room.
Internet celebrity "Caro Lai Lai", who has 1.3 million Weibo followers, fell into the dilemma of pornographic rumors due to AI face changing technology. In April, she posted on Weibo that her face had been stolen from pornographic videos. Someone registered an account on overseas social platforms pretending to be her identity, first carrying her daily content, and when the number of fans was almost the same, they began to sell these pornographic videos with her face in the form of membership subscriptions.
"There are cases where AI technology is being abused by humans." Liu Xianquan, dean of the Institute of Criminal Law of East China University of Political Science and Law and president of the Criminal Law Research Association of the Shanghai Law Society, believes that people have entered the "era of weak artificial intelligence", that is, technology is still a tool used by humans. With the development of artificial intelligence technology, infringement incidents have occurred repeatedly, and in addition to fraud and pornographic rumors, there have been many cases of infringement of people's portrait rights and intellectual property rights.
The "face protection" puzzle
Technology continues to be updated, and the threshold for the public to access AI real-time face changing technology is gradually decreasing.
As early as 2019, an AI face-changing APP called "ZAO" became popular, and users only need to provide a photo to become a character in a film and television drama. But not long after, due to copyright infringement, privacy and other issues, the APP was removed.
"In fact, there is no problem with the AI face changing technology itself, the key is the way it is used." Liu said that once technology is misused, it implies huge risks. The Provisions on the Administration of Deep Synthesis of Internet Information Services, which came into effect on January 10, 2023, also mentions that the provision of deep synthesis services shall provide a conspicuous identification function, and remind users of deep synthesis services to make conspicuous identification. Where deep synthesis service providers and technical supporters provide biometric information editing functions such as faces and human voices, they shall prompt deep synthesis service users to inform the edited individual in accordance with law and obtain their separate consent.
However, AI real-time face changing technology is still quietly sold on the Internet, which makes it more difficult to "clearly identify" and "inform the edited individual".
When the keyword "AI for face" was searched on the Taobao platform, no products appeared. However, if you replace it with similar words such as "AI head change" and "AI face change", you can retrieve the product. The price of face swapping for photos ranges from 35, 50, and 100 yuan, while video face swapping is billed on a minute basis, ranging from 70 yuan, 150 yuan, and 400 yuan per minute.
"It can be replaced with anyone's face," said the store's "engineer Lao Zhang", who only needs to provide the original video and any face photo to achieve 100% face-free face replacement. Not only can you change faces with popular stars, but you can also "resurrect" deceased people in videos.
"Engineer Lao Zhang" not only provides AI face changing services, but also teaches face changing technology. "You can buy programs from us and teach yourself to do them," he said, claiming that he had developed algorithms to change faces, and sold related program plug-ins on Taobao platforms, worth 368 yuan. The program is activated by card and is valid for 50 years, during which you can use the program an unlimited number of times.
To replace the face more naturally, many merchants have said that buyers need to provide a face photo that is similar to or exactly the same angle as the original. However, one merchant said that only the buyer needs to provide a photo of the front face, they can achieve the effect of "fake real", "We can automatically model the face photo and calculate the appearance of several other angles." ”
Many merchants will guide users to add WeChat for details. The Beijing News reporter added the WeChat of the owner of the store "Jaylun Video Design Station", as of July this year, the monthly sales of the products sold by the merchant on Taobao were displayed as 23, but in the process of chatting, the other party repeatedly emphasized that its operating income was 200,000 yuan a month in order to persuade the Beijing News reporter to place an order. He also showed a screenshot of his Alipay, showing total assets of 160,000 yuan. At 14:36 p.m. on the day of the consultation, the merchant said that the day's revenue had reached nearly 1,000 yuan.
This businessman claims to be a master in the field of AI real-time face change, and the circle of friends released are all related cases. On May 9, he posted a 19-second video in the circle of friends, a female anchor wearing an off-the-shoulder top and miniskirt, with black stockings, twisting and dancing to the camera, her face looks no different from star Yang Mi, except that when she lowers her head and turns sideways, the picture looks incongruous.
He told the Beijing News reporter that ordinary videos charge 70 yuan a minute, and pornographic videos charge 150 yuan a minute. He showed a screenshot of obscene images and videos, which were quickly withdrawn. When the Beijing News reporter asked if it was all face-changing videos, he replied, "Don't ask so much, just tell you that you can change, don't say more." ”
An investigation by a reporter from the Beijing News found that the above-mentioned merchants were indeed able to successfully produce a one-minute obscene video of changing faces according to customer needs within two hours and complete the transaction. During the conversation, he repeatedly withdrew information he considered "sensitive", never mentioned telling "the edited individual", and the subsequent video did not have any signs about the AI face. However, the merchant sent a "disclaimer" with it, saying that "any picture and video material is prohibited from being disseminated, any consequences have nothing to do with the producer, it is only for entertainment, and I am not responsible for the use of pictures and videos, and I do not assume any responsibility." ”
A Taobao merchant showed a Beijing News reporter a screenshot of pornographic images or videos to prove that he "can do any video." Screenshot of the reporter's investigation
In addition to Taobao, Douyin also has a variety of AI face changing software products, priced from 1 yuan to 196 yuan. In addition, the Xiaohongshu platform also has accounts that sell AI face changing courses, which guide users to pay attention and purchase by publishing AI face changing videos, accompanied by copywriting such as "it has been very popular recently" and "quick learning".
There are also AI face changing software that are free. When searching on Douyin with "AI face changer" as the keyword, a large number of related short video ads will pop up, with a link to the AI face changing software attached. After the link to a software called "AI Follow-up", a variety of short videos of celebrities and amateurs appeared on the homepage. Just watch a 25-second ad, and you can use this app to create a video of more than ten seconds for free.
"Just like people buy knives, do you blame this knife or the person who sold it after buying a knife and killing someone?" Gan Shirong, a partner at Beijing Huacheng Law Firm, pointed out that there is no problem with the technology itself, the key lies in whether the user has committed illegal acts, such as malicious scandalization, suspected infringement of others' reputation rights, sale of face-changing pornographic videos, suspected of producing and disseminating pornographic materials, which may constitute criminal offenses. However, the arbitrary sale of this technology increases the risk of violation of the law and increases the difficulty of law enforcement supervision.
How to protect our "face"
"The advent of AI has made it easier for people who are already vulnerable to deception." Wang Jie said that AI is just a tool that fraudsters use to enhance the credibility of scams and thus increase their success rate. As the Internet becomes a part of people's lives, AI face-swapping scams are not just as simple as encroaching on other people's property, Liu Xianquan believes that if this technology continues to develop, the fraud methods will be more realistic and diverse, making people unpreventable.
According to Zhang's experience, this technology is not "perfect" in current AI fraud cases, and if the video call is longer, or if the face is placed in a complex and unstable scene, it may show clues. However, in an emergency, it is difficult for people to test and observe sound painting many times in the face of familiar faces and voices.
Both experts said that it is not easy to defend rights against AI infringements. Liu Xianquan pointed out that there are currently no relevant regulations on the use and development of AI technology at the legal level.
Wang Jie believes that when infringement incidents such as telecom network fraud using AI technology have already occurred, it is too difficult to pursue them. He has researched a number of anti-fraud police officers, and after spending a lot of time and money, they found that the source of the crime was abroad. For victims, not everyone can devote enough time, money and energy to accountability for violations.
On the Taobao platform, there are still merchants selling AI face changing technology. Network screenshots
After Pan Ziping's face was stolen in the pornographic video, because he was not cheated of money, he did not file a case after calling the police. He couldn't find who had stolen his face, nor could he get them to stop spreading, so he could only edit a text message and send it to each of his friends, reminding them not to spread it or trust the video content.
"Ah Lei, who only wants to make up" is a beauty blogger with more than two million fans on the Xiaohongshu platform, and she has posted many pictures of Chinese style makeup. In April last year, she posted that her homepage video was stolen and used as a face-swapping template in the face-swapping app. She doesn't know who uploaded the templates, "An app may have dozens or even hundreds of uploaders, which makes it difficult to defend rights." ”
"I don't have the money and energy to do it, it's not my original sin." Someone reminded her in the comment area to protect her rights, she said helplessly, rights protection must first start from suing the APP, locate each uploader, and then sue the individual, such time and money costs are very high, the timeline before and after is very long, and the compensation obtained after winning the lawsuit is not much.
"The core issue is the protection of personal information." Wang Jie believes that there is a leakage of personal information behind fraud incidents, and with the continuous upgrading of deepfake technology, protecting personal privacy such as faces has become a problem that cannot be underestimated. Therefore, when investigating and handling cases, public security organs should pay more attention to the channels of information leakage and deal with them from the source. The government should strengthen the supervision of personal information protection, and APP developers and operators should also work together to manage the storage, reading, use, and erasure of facial information in accordance with the law, and promote compliance and self-discipline in the protection of personal information of enterprises.
With the rapid development of artificial intelligence technology, Liu Xianquan proposed that how to prevent illegal infringement in advance. He believes that at present, artificial intelligence technology has not developed to the point of becoming an independent responsible subject, and technology is still a tool that is used by people, but because it has a certain intelligent part, it may increase the harm to society when used by criminals. Therefore, in addition to improving the relevant laws and regulations on new technologies, there should also be relevant restrictions on the process of human use and development of the technology, such as whether a certain field is accessed, and who can use the relevant technology.
Laws and regulations related to AI technology are also constantly improving. On April 11, the Cyberspace Administration of China (CAC) issued the Measures for the Administration of Generative AI Services (Draft for Comments), which provides more detailed provisions on the provision of generative AI services, including access qualifications, responsibilities and obligations of content producers and personal information processors, and relevant penalties.
"Our future will definitely coexist with AI," Wang said, which means that humans will also coexist with AI infringements. He believes that future legislation should pay timely attention to new phenomena and problems in the process of technological development, ordinary netizens should enhance their awareness of high-risk networks, and law enforcement supervision departments should accelerate technological upgrading, develop detection technologies such as "face reversal", and use artificial intelligence technology to improve case-handling capabilities.
He said that "only when all parties in society work together" can the application of technology develop for the better.
(Fang Yangyou and Pan Ziping are pseudonyms)
Beijing News reporter Wang Chang Intern Zou Bingqian
Edited by Chen Xiaoshu
Proofread by Janin