According to the forecast of the consulting firm Yole Développement, the market size of 3D imaging and sensing will grow from $1.3 billion in 2016 to $9 billion in 2022, benefiting from the predictable explosive growth of the consumer electronics market, of which the 3D imaging and sensing market for consumer electronics will grow from $20 million in 2016 to $6.058 billion in 2022, with a compound annual growth rate (CAGR) of 158%.
Global 3D Imaging and Sensing Market Revenue Forecast 2011-2022
Even more convincing than the "staggering" forecasts is the entry of products into the consumer market.
In 2017, Apple removed touch ID from the iPhone X in favor of Face ID with 3D sensing technology. This year, Android camp manufacturers have also begun to explore similar 3D sensing technologies.
At this stage, 3D cameras have evolved three mainstream technology routes: Structured light, time of flight (TOF) and stereo vision technology. Although the technology is different, the ultimate goal is the same - to build a 3D ecological empire.
Before the Empire – What guns to choose and where to find ammunition?
Unlike the "megapixel wars" of the past, most smartphone practitioners are likely to fight harder on the new front of 3D sensing technology.
The first is the question of the choice of options. From a technical point of view, the principle of 3D structured light imaging is that after the structured light projects specific light information to the surface of the object, the reflected information is collected by the camera, and then the position and depth of the object are calculated according to the changes in the light signal caused by the object, and then the entire three-dimensional space is restored. The advantage of this technology is that the technology is mature, and the depth image resolution is relatively high, but the disadvantage is that it is easily affected by lighting.
The TOF imaging principle is to emit a beam of infrared laser through phase modulation to the measured object, when the infrared laser is reflected back to the camera, it will cause a slight change in the phase and the phase at the time of emission because of the delay in the time of flight of the light, by calculating the change in phase, the distance between the measured object and the camera can be calculated. The TOF scheme basically does not need to use an optical prism, the anti-interference performance is good, the viewing angle is wider, and the disadvantage is that the depth image resolution is low. And the current TOF solution to achieve ar function experience is not very ideal, industry analysts believe that this is related to the device does not want to occupy too much CPU resources to reduce the running speed.
The binocular line of sight uses a dual camera system to form a 3D image through the angle between the two cameras, which is similar to the human eye. At present, dual cameras have basically become the standard of mobile phones like fast charging, so this solution should theoretically have the lowest cost. However, the complexity of the overall system, high power consumption, unsatisfactory performance in low-light environments, and low accuracy bring great limitations. Lenovo has tried a similar scenario on a Phab2 ProAR phone that supports Project Tango, but the experience is not optimistic.
Binocular vision, structured light, and time-of-flight technology comparison
Secondly, more important is the extreme complexity of 3D sensing systems and the production of critical components. Because 3D cameras contain many components that need to operate consistently, they also need adequate supply chain management. Some have even described 3D cameras as "sub-devices."
Compared with the traditional camera industry chain, the 3D camera industry chain has added parts such as "infrared light source + optical components + infrared sensor", which makes vertical cavity surface emitting laser (Vertical-Cavity Surface-Emitting Laser, VCSEL), collimated lens (WLO), optical diffraction grating (DOE) and so on become the core device. Although 3D cameras generally used LEDs as infrared light sources in the early days, LEDs do not have resonators, resulting in more divergent beams, and VCSEL has more advantages in terms of accuracy, coupling, miniaturization, low power consumption, and reliability, so now common 3D camera systems use VCSEL as an infrared light source.
Some analysts believe that the large-scale popularization of 3D sensing technology in the Android camp may not be until 2019, mainly because the production yield of 3D sensing modules at this stage is low: first, the production of high-efficiency VCSEL components is not easy, and the current average photoelectric conversion efficiency is only about 30%; second, the necessary components of structured light technology DOE and CIS of infrared light lenses need to have a very high technical background; third, the production process of 3D sensing modules needs to consider the problem of thermal expansion and contraction. Increase the difficulty of module assembly.
The first step of Empire is to design a mobile phone that supports 3D structured light
Find an integral part of the X 3D structured light camera
On June 19, oppo Find X was unveiled, and as striking as the dual-track periscope structure was the OPPO FaceKey 3D structured light system integrated in this structure. At the beginning of May, OPPO announced that it would commercialize structured light in OPPO mobile phones within 6 months. What many people did not expect was that OPPO would release the product within a month. This makes OPPO gain a lot of "first" names. But more importantly, OPPO has advanced the industry's mass production schedule by a year.
OPPO FaceKey 3D structured light
In order to complete the selection, pre-research, trial production and verification of the entire hardware solution including the structured light transmitter, receiver and structured light decoding chip within half a year, and at the same time quickly break through the structural light module selection, baseline selection, structured light depth map drawing, depth map calibration, depth camera and RGB camera calibration, face security program verification and other difficulties, OPPO is facing the difficulty of hardware development can be imagined.
Just as difficult as the hardware solution is the design of the software solution. The core of the secure face framework and scheme, including hardware mapping, image correction, network algorithm verification, etc., must all be run in a secure TEE environment to solve the security of face information, payment environment and algorithm. But before OPPO, the 3D face algorithm had no mature experience to learn from in China. When the algorithm self-development was first launched, the OPPO algorithm team also had zero experience in 3D data, server environment, and algorithm scheme.
However, in the short period of 3 months, the OPPO algorithm team and hardware team used structured light hardware to complete the 3D face data collection of 100,000 people, and the face data distribution covered the world, and completed the construction of a 500TFlops large training server during this period. At the same time, in order to achieve a misidentification accuracy of one in a million, OPPO also discussed the 3D face recognition algorithm in depth with AI algorithm companies Kuangshi and SenseTime, and formed a set of its own algorithm schemes on face detection, face recognition and live detection algorithms.
The security technology included in OPPO 3D face payment
In the end, through 4 rounds of hardware solution iteration and 10 rounds of face algorithm iteration training, OPPO successfully completed the structured light solution of today's top experience. Comparative data show that in the single depth map 3D rendering, 3D rendering accuracy, anti-glare and 50cm-70cm distance recognition, OPPO Find X scheme is more mature than the iPhone X; in 20-50cm distance recognition, power consumption, resolution, frame rate, volume and other aspects are not inferior. The advent of the first 3D face recognition algorithm in China also marks that OPPO has become a force that cannot be ignored in the industry in the field of technology research and development.
In addition to OPPO, other manufacturers in the Android ecosystem are also exploring the future of 3D imaging technology. At the same time as Find X, Xiaomi launched the Xiaomi Mi 8 Exploration Edition at the Xiaomi Mi 8 launch conference, integrating a 3D structured light system in the bangs area of the special-shaped screen. Shortly after the release of the Find X, Vivo also demonstrated its ToF solution during the Shanghai MWC.
Empire Development – 3D data needs to be secure enough to be used for payments
The most important factor in biometrics is security, and there should be no "one". This is also one of the biggest advantages of 3D structured light face authentication compared to 2D face recognition. Although the latter has also become a must-mention feature at mobile phone launches in the past year, 2D face recognition with algorithms as the main implementation method still faces very serious security problems although it achieves a very fast unlock speed. Some review agencies have even found that many 2D face recognition phones can be unlocked just by printing a photo of the A4 with a normal printer.
More serious than the unlocking of the device is the security of personal information. On the one hand, this refers to private sensitive data, including personally identifiable information, as well as financial information such as bank card numbers and passwords; on the other hand, it is biological information itself, such as fingerprint data and facial information. If the bank card is lost, the password can be replaced, but the fingerprint or facial information cannot be changed for a lifetime, and the risk after leakage is permanent. Therefore, this is also the reason why payment institutions must go through a very rigorous authentication process if they want to support face or fingerprint authentication.
From another point of view, the recognition of 3D structured light technology by payment institutions is, on the one hand, an endorsement of the security of this technology, and on the other hand, an important factor in affirming whether this technology can be popularized in the future. Therefore, the significance of the technology OPPO supporting Alipay face brush payment on Find X is even more significant.
The partnership between Find X and Alipay is part of OPPO's strategic partnership with Ant Financial. In addition to payment, the two sides will also set up a joint innovation lab to explore the deep integration of hardware and software underlying technologies, such as supporting the performance optimization of Alipay on OPPO mobile phones and the step-by-step direct access of high-frequency payment scenarios.
Li Sheng, product manager of OPPO, introduced the cooperation with Ant Financial at the Find X launch
Driven by the Joint Innovation Lab, Find X can support Alipay's face payment as soon as it goes public. Based on structured light as the hardware, combined with OPPO's deep learning-based face intelligent recognition self-developed algorithm, 3D face payment can accurately compare the information entered and recognized by users, and control the misidentification rate at one in a million payment accuracy, which is much higher than the current fingerprint recognition misidentification rate of about one in 50,000. Users can expect to take advantage of this technology in more OPPO mobile phones in the future, and safely replace fingerprints, passwords or PIN codes with faces as the main verification method for mobile payment applications.
Empire emerges – connecting end-to-end 3D data flows
If the 3D data obtained by 3D structured light technology is only used for unlocking and payment, then the use of rich data is obviously insufficient, and mobile phone manufacturers will not only stay at this level. Therefore, although the iPhone X and Find X 3D structured light currently bring users a safe and novel experience, what is more meaningful is the imagination of the two manufacturers of the potential layout of the entire 3D data.
In May, shortly before the release of find X, OPPO announced the successful realization of the world's first 5G video call demonstration using 3D structured light technology. Through the oppo mobile phone's structured optical system to collect 3D data, Qualcomm's 5G new air interface terminal prototype is used, and finally the three-dimensional portrait screen is restored at the remote receiving display.
3D video calling
At WWDC 2018 in June, apple not only focused on ARkit 2, but also launched the "USDZ" 3D file format. Some developers commented that "this achievement is comparable to the 'car on the same track' implemented after Qin Shi Huang unified the Six Kingdoms, a 3D file format that makes it possible to transfer 3D data across apps." At this launch, Apple demonstrated that the same set of 3D images can be presented across applications in Safari, News, and cameras. From the first time last year on the iPhone X equipped with a 3D imaging system, to this year's efforts to present 3D images, it is not difficult to see Apple's layout in this field of technology.
As a branch of computer vision, 3D imaging provides the underlying technical capabilities, and thus enables devices to have new functions such as augmented reality (AR) and 3D faces, and these new features empower the classic SoLoMo business and the emerging short video, live broadcast, game and other services, giving users a fresh experience. In addition to the secure payments that were highlighted earlier, 3D reconstruction, AR, and gaming will most likely become breakthrough application scenarios enabled by 3D structured light technology in the future.
Taking AR application as an example, in 3D AR games, structured light is used to accurately 3D model the surrounding environment, and the model is imported into the game in real time to achieve the close integration of virtual games and the real three-dimensional world to achieve realistic three-dimensional effects. Or the three-dimensional information captured by a precise structured light dual-camera camera can achieve accurate AR posts and micro-expression effects. Gartner predicts that by 2021, 40 percent of smartphones will be equipped with 3D cameras that can be used for AR, and analyst Jon Erensen even said, "This kind of feature is very important for AR, and I think that's something you don't want to be left behind." ”
OPPO Find X launched a presentation on 3D structured light application scenarios
So we see that apple and OPPO are both laying out AR. The former has introduced ARkit as one of the key points in the WWDC in the past two years; the latter has also joined hands with SenseTime and the State Key Laboratory of CAD &CG of Zhejiang University to create a development plane called ARunit, which can support features such as instant positioning and map construction (SLAM), multi-plane recovery, placement of virtual objects on non-planar surfaces, plane sign recognition, light estimation, map preservation and state setting, occlusion processing, etc. It can help developers create a new generation of new AR applications. Since its release on March 19, it has attracted 13 partner products including JD.com and King Glory, and is expected to provide AR support for 80 million OPPO mobile phones in 2018.
OPPO ARunit
In OPPO's view, the mobile Internet is blurring the boundaries between the real world and the digital world. The combination of virtual image + virtual image, virtual image + reality image, and real image + real image will bring a new generation of user experience of Ubiquitous Reality (UR). This is far from the vr and AR we see today, and what will really further promote immersive user experience in the future will be 3D content and "coming" 5G. Therefore, through the active structured light perception of the target object position and depth information, the entire three-dimensional space to achieve high-resolution, high-precision restoration, and then through the low-latency 5G network for real-time transmission of 3D video, will become the next generation of mobile user experience ubiquitous reality.
Enable 3D video calls under 5G networks
In layman's terms, OPPO intends to use the high-speed characteristics of 5G networks to support the large data transmission of 3D video signals, thereby providing users with a new call experience. This is obviously no longer just a two-dimensional video call, but a video call experience under the display of a three-dimensional space. Although it is not possible to directly present 3D data because of the limitations of display technology, it can be seen that in the future, after the technology development of the 3D data presentation end is in place, we can experience the digital life of "holographic projection" in science fiction works.
summary
3D imaging technology has indeed become the focus of technology development for many mobile phone manufacturers, and its importance may be no less than 5G and AI in the future. However, because of the problem of the scale of technical applications, there is no new killer application in addition to unlocking and payment.
Manufacturers generally believe that high cost, yield and other issues will lead to 3D sensing cameras in 2018 only suitable for flagship models priced higher than 3,000 yuan, but from the beginning of 2019, it will become the standard for smart phones above 2,000 yuan, just like fingerprints, dual cameras, etc., and by 2020, the coverage will be further expanded, and it will not be a surprising thing for thousands of yuan models to be equipped with 3D sensing cameras.
3D sensing camera market penetration estimates
With the wide application of 3D technology in the largest consumer electronics market of smart phones, the application scenarios will inevitably be more abundant. At that time, what kind of vitality will OPPO and Apple's 3D empire show?
This article is the original of "Electronic Engineering Album", all rights reserved, please do not reprint