"All or Nothing" was recently in hot release. The film tells the story of programmer Pan Sheng who was abducted to an overseas "company" under the lure of high salary of an overseas Internet fraud team, and was forced to engage in fraud activities under the coercion of Lu Bingkun and An Juncai, and finally helped the deceived Chinese Anna escape, and was successfully rescued with the cooperation of the Chinese police and the Foreign Immigration Bureau.
In the film, Pan Sheng is attracted by overseas work, but accidentally ends up in a fraudulent factory. The first thing Pan Sheng was forced to do after entering the online telecommunications fraud company was to use crawler software to grab the email information of the subtitle group members and send them online gambling links.
At the end of the film, Manager Lu, the leader of the fraud company, was sentenced to death, Anna was sentenced to two years in prison for committing fraud, and Pan Sheng provided a list of more than 2,000 victims, which was a major meritorious performance and was sentenced by the court to be exempted from criminal responsibility.
What is a web crawler
Web crawlers, also known as web spiders, web robots, are programs or scripts that automatically capture network information and data according to certain rules. In layman's terms, web crawlers simulate human behavior, replace human operations with programs, jump from one link to the next, and traverse web pages as if crawling on the web. Crawlers jump, open, browse and other actions are faster than people, and the level of the website browsed is deeper, so they are called web crawlers.
Web crawlers can illegally obtain information, pictures, evaluations, and personal information on the Internet. The stolen data is not only used for commercial sales, but also may be used by black ash products to make fake websites, carry out phishing scams, etc., bringing major economic losses to individuals and enterprises.
Illegal theft of crawlers and platform anti-crawling
The malicious crawling of web crawlers and the anti-crawling of platform websites is a dynamic offensive and defensive process, which is roughly three stages.
The first stage is to restrict IP addresses and accounts. At first, the anti-crawling measure of the website was to directly deny access that did not originate from the browser. When malicious web crawlers access, a 403 error response code appears, or a "Sorry, it cannot be accessed" prompt.
The second stage, CAPTCHA interception. In order to bypass the anti-crawling mechanism, the web crawler sets the header information, simulates the browser, and multi-threaded to carry out large-scale malicious crawling of static pages. In response to malicious crawling behavior, websites and platforms restrict and block accounts and devices that frequently change UserAgent (imitating browser) and frequently use proxy IPs: when the same IP and the same device visit the website within a certain period of time, the system automatically restricts their access to browsing; When a visitor visits too many times, the request is automatically redirected to a verification code page, and the visit can only continue after entering the correct verification code.
The third stage is the technical protection of dynamic web pages. In the face of the upgrade of anti-crawling technology, web crawlers have also upgraded. Web crawlers can automatically identify and fill in verification codes to bypass the interception of secondary verification; Use multiple accounts at the same time, configure the IP proxy tool, and bypass the platform's restrictions on accounts and IP addresses. In response to the changes of web crawlers, many websites and platforms use dynamic web page opening technology. Based on dynamic web page technology, the URL address of the web page is not fixed, and the background interacts with the front-end user in real time to complete user query, submission and other actions. And different pages will be produced at different times, different users, and when visiting the same URL address. Compared with traditional static web pages, dynamic web pages effectively protect important data information and effectively curb the malicious crawling behavior of web crawlers.
To bypass the new anti-crawling measures, web crawlers use Selenium and Phantomjs technology, which fully mimic human operations. At this time, the attack of crawlers is also more intelligent and complex, simply by limiting the number of visits, front-end page display encryption has been unable to effectively protect, it is necessary to improve man-machine recognition technology and other interception to identify black products, increase the cost of illegal evil. Topimage's full-process three-dimensional prevention and control measures effectively prevent malicious crawling behavior to ensure the security of the website platform.
The whole process anti-climbing scheme of the top image
Illegal theft by crawlers is becoming more intelligent and complex, simply by limiting the number of visits, front-end page display encryption has been unable to effectively protect, it is necessary to improve man-machine recognition technology and other interception to identify black products, increase the cost of illegal evil. Dingxiang's full-process three-dimensional prevention and control measures effectively prevent malicious crawling behavior to ensure the security of e-commerce websites.
Regularly test the operating environment of the platform and app, strengthen the security of the app and client, encrypt the communication link, and ensure the security of the end-to-end whole link. At the same time, the top image defense cloud, risk control engine and intelligent model platform are deployed to build a multi-dimensional defense system.
The top image risk control engine realizes the effective identification of malicious "crawler" behavior according to the request of the service query scenario, the device fingerprint information collected by the client, and the user behavior data behavior, and effectively identifies and intercepts the malicious crawling behavior based on the security prevention and control strategy. Based on changes in business, crawling risk, and anti-crawl strategy, the Dingxiang Intelligent Model Platform helps enterprises build exclusive risk control models to achieve real-time changes in security policies, thereby effectively intercepting various malicious crawling risks.
Captcha in the AI era
Captcha is an important technology to prevent data theft, which has also become an important target for black and gray industry to crack. Top image verification code is based on verification of environmental information for defense, and provides double security by producing endless verification pictures + providing verification of environmental information.
First of all, the top image verification code based on AIGC technology can continuously obtain new verification pictures, which greatly increases the identification and cracking cost of black and gray production, and greatly improves the difficulty of verification element identification. Based on deep learning and neural networks, it generates some pictures and elements that are difficult to predict and repeat, and adds dynamic factors such as timestamps or random numbers in the verification process to increase the difficulty of cracking and effectively resist machine cracking.
Secondly, the top image verification code integrates real-time stream computing and scenario strategy, combines machine learning training human-machine model and historical data correlation analysis, and performs machine learning modeling on user-generated behavior trajectory data through graphical algorithms and AI models, and combines access frequency, geographic location, historical history and other dimensional information to quickly and accurately return human-machine judgment results. Collect identifiable environmental information in the verification process of the verification code, configure rules and policies, and screen out requests that may be black and gray for secondary verification or interception. For example, determine whether the verification environment information at the time of verification is consistent with the verification environment information when the token is reported, block the IP address of multiple malicious attacks, and limit the number of verification code inputs.