"Collapsed" in the past 12 hours and lost about 10 million orders Didi apologized late at night Technical person: The paralysis of the whole line of products points to the failure of the lower link
Reporter: Zhao Wenqi
"I thought it was my mobile phone that was in arrears and had no network, and I charged 50 yuan to find that I couldn't call a car!" "An originally ordinary working day, but because countless workers couldn't call a car for work, Didi Chuxing's system failure was on the hot search early in the morning.
In fact, on the evening of November 27, many users reported on social platforms that the Didi App could not be used normally, and many drivers also said that they could not receive orders or that navigation could not be used after receiving orders. In this regard, Didi responded urgently at 23:19 on the same day, saying, "Due to system failure, the Didi App service was abnormal this evening, and it was urgently repaired by technical students and is currently being restored." ”
At 7:31 a.m. on November 28, Didi Chuxing's official Weibo apologized again, and said that the technical team had repaired it overnight, and Didi's ride-hailing service had returned to normal. However, after a whole morning or so, some passengers still reported that they could not hail a taxi on the Didi platform.
Image source: Didi sharing car in operation Image source: Photo by reporter Liu Xuemei
This time it was not only all of Didi's product systems that collapsed, but the "Daily Economic News" reporter learned from a number of Didi employees that Didi's intranet also failed this morning.
A senior technical person analyzed to reporters that the App is only an interface and is used by users, but the App has to access the underlying data, and Didi will definitely isolate between various businesses at the company level, and if all products collapse, it must point to a lower level of problems.
As of press time, most cities have resumed Didi's taxi function, but some passengers reported to reporters that some of Didi's functions still have crashes, and the response time for calling a taxi is also slower than usual.
It is reported that some drivers on the Didi platform began to withdraw money and the withdrawal was successful, and the driver was afraid that there would be problems with the system, and his account would be cancelled. In this regard, Didi responded to the reporter of "Daily Economic News", after verification, the current driver account and withdrawal function are normal, and the majority of drivers do not need to worry. The Didi platform is a daily withdrawal for drivers with high service levels. In addition, Didi said that drivers and masters are concerned about the affected related rewards, abnormal expenses, word-of-mouth values (including travel points), etc., and will also resume and reissue them as soon as possible. As for the previous netizen who said that "a driver's income exceeded 69 billion yuan" and posted a screenshot of the account balance, Didi Chuxing responded that after verification, it was confirmed that the picture was a rumor forged by PS, and I implore everyone not to spread rumors or rumors.
The internal and external networks have "collapsed", and the number of lost orders may exceed 10 million
"When I arrived at the company this morning, I found out that Didi collapsed last night, and we are all writing code on paper now. A Didi employee revealed to reporters. In addition to the intranet crash, the system crash of the whole night plus a morning rush hour caused inconvenience to a wide range of users and drivers.
The reporter of "Daily Economic News" saw from the social platform that many users complained that the morning rush hour taxi crashed, and finally they could only transfer to other platforms to call a car; some passengers reported that the distance of 7 kilometers was taxi, and Didi collected 268 yuan, asking for a refund as soon as possible; and some drivers mentioned that they had been unable to click the "delivery" button when they sent passengers to their destinations.
Image source: Complaints from users around the world on Xiaohongshu Image source: Xiaohongshu
In the system failure, the billing system on the driver's side of Didi was chaotic for a while. There are reports that the background income balance of a contracted driver is shown as 69387199691.71 yuan, but this figure was subsequently debunked as a picture of PS.
There are also hot searches mentioning that some Didi drivers are worried that there will be problems with the system and the account will be canceled and withdrawn on the platform, in this regard, a Didi driver told reporters that there are indeed drivers around him who say they want to withdraw, but there are also many drivers who feel that there is no need to withdraw.
In response to the attendance points and billing issues that drivers are concerned about, Didi's previous statement also mentioned that starting from Monday evening, due to system reasons, the issuance of relevant rewards that have been received/purchased will be delayed, and the platform will calculate normally, and make up the payment one after another, and the word-of-mouth travel points will also be calculated normally, and the display will be resumed one after another, please be patient. Individual drivers and masters report that they cannot start the trip and billing after receiving passengers, so please pick up and drop off passengers normally and deliver them to their destinations safely, and the platform will calculate and reissue fares according to the itinerary in the future.
Didi's nearly 12-hour system failure was also the longest outage in the company's history, and in addition to the impact on the daily lives of drivers and passengers, it would also cause losses to Didi.
According to Didi's latest financial report, the total transaction volume (GTV) of Didi's core platform in the third quarter was 91.5 billion yuan, a year-on-year increase of 37%. Among them, the total transaction value of China's travel was 72.5 billion yuan, a year-on-year increase of 33%, and the total transaction value of international business was 19 billion yuan, a year-on-year increase of 52%. The total number of trips in China was 2.878 billion, up 32% year-on-year, and the total number of international trips was 701 million, up 43% year-on-year. Based on this calculation, the average daily order volume of China's travel in the third quarter reached 31.3 million, breaking through the historical peak of a single quarter.
If you ignore the difference in the number of orders in the morning and evening, the reporter of "Daily Economic News" roughly calculated according to the financial report data that Didi's downtime was close to 12 hours, the number of lost orders also reached the level of 10 million, and the transaction volume was close to 400 million yuan.
R&D investment of 2.1 billion yuan in the third quarter was updated five days ago
Since the cause of the failure has not been officially disclosed, the outside world has only speculation and reasoning about Didi's large-scale and long-term system failure. An industry insider analyzed to the "Daily Economic News" reporter that cloud servers, systems, and interfaces may have bugs. However, Didi is generally relatively stable and mature, and this kind of large-scale inaccessibility rarely occurs, so it is difficult to speculate on the specific reason, and only Didi may be able to disclose accurate information.
However, he also mentioned that the latest version of Didi IOS APP was released 5 days ago, and I don't know if there is any bug that broke out after the new version of the APP ran for a few days. The reporter of "Daily Economic News" noticed that the IOS side of Didi Chuxing, Qingju Bicycle and other apps was updated 5 days ago.
This is not the first time Didi has experienced large-scale system failures.
On September 22, 2022, many users reported that Didi could not be used normally, and Didi responded that "due to the network failure of the computer room, some of Didi's services were affected", and the fault was repaired in less than half a day;
In April 2021, Didi also had a system failure, and Didi officially responded that it was caused by a system upgrade;
In October 2019, "Didi Navigation collapsed" also rushed to the hot search, when there was a large range of drivers and passengers who could not find each other, Didi responded that it was a failure during the system update, which caused problems with navigation;
The day before the National Day in 2020, Didi Taxi had an abnormal network loading situation due to the pre-holiday rush hour, and Didi officially called that day "the most difficult day to take a taxi";
In January 2021, "Latepost" revealed that Didi issued an internal letter to establish a technical committee, and according to Didi insiders at that time, the establishment of the technical committee may be related to a large-scale outage accident on September 30, 2020.
According to Didi's latest financial report, Didi's R&D expenses in the third quarter were RMB2.1 billion, accounting for 4.1% of total revenue, compared with RMB2.5 billion in the third quarter of 2022, accounting for 6.2% of total revenue. Didi explained in its earnings report that the decline in R&D expenses as a percentage of revenue was mainly related to the reduction in product development expenses for the company's smart car business.
In addition, Didi launched its own cloud services in 2017, according to the official website of Didi Cloud, Didi Chuxing's cloud computing services are based on Didi Chuxing's business technology and experience accumulation, using leading cloud computing architecture, high-specification server cluster construction, high-performance resource allocation mechanism, and refined operation model, and is committed to providing developers with simple, fast, efficient and stable, cost-effective, safe and reliable IT infrastructure cloud services.
In February this year, Didi Cloud announced that due to the adjustment of the product line, public cloud services will no longer be provided from 00:00 on March 31, 2023.
How can Internet platforms minimize losses due to technical failures?
In the past two years, not only has Didi had multiple system failures, but most mainstream Internet products have also gone down. On November 12, Alibaba Cloud just experienced a large-scale outage that lasted for more than an hour, and many Alibaba products were affected.
Internet analyst Ding Daoshi said in an interview with the "Daily Economic News" reporter that Baidu, Alibaba, Tencent, Vipshop, Weibo, etc., almost mainstream Internet companies, have experienced various forms of technical failures in the past few years. According to the summary of the causes of the previous technical failures, Ding Daoshi made several possible guesses about Didi's downtime this time.
The first possibility is the edge end failure, that is, the terminal equipment and related optical cable damage, a few years ago Tencent WeChat had a large-scale failure, and later found out that the construction party cut off the optical cable during construction. Another example is that every major earthquake can cause damage to related communication facilities.
The second is a man-made management loophole. In 2020, Weimob had a major failure because of a conflict between internal technicians and the company, which damaged the company's entire system data and brought a loss of 1 billion yuan to Weimob, which is not uncommon in the industry.
The third is that the associated system fails, triggering the protection mechanism, causing the normal service to go offline. There have been cases of the collapse of the telecom network leading to the failure of a large number of Internet platforms, including WeChat Pay, QQ, Vipshop, etc., and finally found that the reason was that the cooling system of one of the network computer rooms of Telecom failed, and the temperature of the entire computer room rose rapidly, triggering this protection mechanism, and a large number of network servers were forced to go offline.
There is also the most common type of hacking. There are many kinds of hacker attacks, in addition to common viruses, Trojans, worms, etc., in the past two years, there has been a rise of cyber extortion attacks, criminals found vulnerabilities, gained control of the server or obtained sensitive user data, and asked the operating company to pay millions of dollars in the form of bitcoin ransom, which is also a more common situation.
"On the one hand, we should recognize that cybersecurity is an ongoing process, not a one-time task. As technology continues to evolve and threats continue to evolve, we need to constantly update and improve our security measures and strategies. This includes regular security training, timely updates to software and systems, security audits and risk assessments, and more. On the other hand, it is necessary to set up more redundancy, and the backup system must be ready at any time, even if it is never needed. Ding Daoshi said.
However, he also mentioned that for a large company like Didi, the user losses caused by the system crash will definitely be properly dealt with after verification, and consumers will not be made to pay for the failure, which is also the bottom line.
National Business Daily