Image source: Visual China
"Do you know how many chips a smartphone has?"
As the most inseparable electronic device for everyone at present, although a smart phone is only the size of a palm, it has at least 100 chips integrated inside. Among them, the most familiar nature is the SoC, SoC is called system-level chip, which integrates CPU, GPU, ISP, NPU, etc., so we can say that SoC is the "core processor" of mobile phones.
Among so many chips, the name of NPU (neural network processor) is not as loud as CPUs and GPUs, but it plays the smartest role, specifically responsible for implementing AI operations and AI applications, directly affecting the strength of mobile phone AI capabilities. The AI capabilities of NPU can significantly change the existing mobile phone functions, and imaging is a promising field of application.
At OPPO's 2021 Future Technology Conference on December 14, OPPO released the first self-developed NPU chip, Mariana MariSilicon X, which is the world's first dedicated NPU chip for imaging, and its emergence will greatly enhance the ability and participation of computational photography. Behind it shows OPPO's determination to "self-developed chips", and also reveals new ideas for its future development.
So far, the four major head manufacturers of Chinese smartphones have also joined the "core" track of self-developed chips.
Explore the deeper "Mariana" with your best areas
There are many chips contained in smart phones, which also means that mobile phone manufacturers have many directions in the choice of self-developed chips, why did OPPO choose image NPU as its first self-developed direction? The development logic behind it is inseparable from the technical track that OPPO adheres to.
OPPO Find X3
For a long time, the highest demand for smartphones by users was the pursuit of image quality. For mobile phone brands, in order to meet the needs of users, it is inevitable to achieve key breakthroughs in imaging technology.
In order to enhance the image capability, the most direct way is to switch to a larger sensor, but the limited body space of the smart phone is destined not to allow the existence of three or four huge image sensors; coupled with the larger size of the sensor requires a longer physical focal length, maintaining the equivalent of about 27mm mainstream focal length will make the thickness of the mobile phone surge synchronously, which is not destined to be a sustainable development direction.
After years of exploration of image technology in the past R series, Reno series, and Find series, OPPO found that it has encountered a bottleneck in trying to continue to improve the quality of moving image from the optical aspects of sensors and lenses. Only through the image-specific NPU can we solve the problems of the current ISP function limitations and general computing power, sensor size and optical quality stagnation, which is also the mainstream direction of future image development.
Moreover, at present, the general-purpose chips and NPU chips provided by upstream suppliers cannot meet the computing power and energy efficiency requirements of AI for computing images. Therefore, in order to take into account high computing power and low energy consumption, OPPO also had to develop its own NPU for imaging.
The reason why OPPO set the name of the project as "Mariana" is also quite image, there are two reasons behind it. First, if you want to achieve the mobile terminal high-quality AI image experience that OPPO expects, a more mature chip production process (such as 12nm) can no longer be satisfied, so its first self-developed chip is directly launched on the 6nm process process, which is a great challenge for OPPO's entire self-developed chip team.
On the other hand, the name "Mariana" is also a determination of OPPO to challenge the self-developed chip, and also expresses that OPPO will attack the chips at the bottom of mobile phones with its best field.
What's the magic of Mariana MariSilicon X
What kind of chip is the Mariana MariSilicon X, which OPPO has high hopes for?
First of all, we need to understand that Mariana MariSilicon X is an NPU chip, not an ISP module, with AI computing power that the latter does not have, and can use neural networks to achieve noise reduction and detail improvement of images. It adopts the new golden architecture concept of DSA in the AI era, which aims to give up the compatibility of secondary capabilities for OPPO's AI image needs, but to invest all the chip area and computing power into the imaging field, which is the so-called "dedicated core", the task is more clear, and the performance is naturally stronger. At present, cutting-edge technology companies around the world are using DSA for AI, such as Google's TPU. However, the use of the DSA architecture also means that the specificity of this chip is extremely high for the manufacturer's chip direction planning, and it is necessary to predict in advance where the image outlet is within a few years of chip development to commercial use.
Its internal architecture consists of a "MariNeuro" self-developed AI computing unit, a "MariLumi" self-developed image processing unit, a supporting image-specific memory subsystem, and I/O components.
In addition, mariana MariSilicon X process is directly launched on the current industry's very advanced 6nm process, which is foundry by TSMC. Although choosing 6nm can lead to stronger performance performance, the high investment, high risk, high difficulty and high uncertainty that comes with it are self-evident.
According to industry reports, the cost of 6nm chip 1 tape-out is as high as 100 million, and no one knows whether it is successful before the official tape-out experiment, and once it fails, it faces a reversal. Fortunately, OPPO succeeded once in the tape-out stage, according to Jiang Bo, senior director of OPPO chip products, recalled: "Remember that there is no delay after the tape-out is taken back, the lighting process is 24 hours, the entire R & D team is very excited, we completed a huge challenge!" ”
For the improvement of users' actual use, Mariana MariSilicon X will mainly use AI technology to solve the traditional mobile phone image problem - low-light video quality, pushing computational photography to a new limit of 4K+20bit RAW+AI+Ultra HDR. For people who do not understand photography, this string of complex terms may be difficult to understand, to put it bluntly, it is to make the picture have rich detail (4k resolution is 4 times that of mainstream 1080p, the picture detail is richer), high tolerance (the very bright part of the picture and the extremely dark part are clearly visible), pure picture (through AI calculation to achieve nearly noise-free night video) and lossless computing (image data loss is small, retain more information and detail, more friendly to the later stage), as far as possible to ensure the "original taste" of the picture.
To give a simple example, let's say we take a picture in a dim theater, if there is no AI computational photography, the whole picture may be completely dark, or only faint colors can be seen. In the future, with Mariana MariSilicon X, OPPO can use its "MariLumi" image processing unit to significantly improve the performance of HDR (high dynamic range imaging) capabilities, so that the pictures taken in the theater can be restored to the state seen by the human eye.
It sounds simple, but behind it is a very difficult R&D process. To know that now the Android flagship platform Snapdragon 8 only supports 18bit HDR, OPPO's first self-developed chip can increase its bit width to 20Bit. Between these 2 Bits, the picture tolerance is increased by a factor of 4 times (the 20th power of 2 is compared with the 18th power of 2), directly from the original 250,000:1 to 1 million:1.
In addition, the Mariana MariSilicon X can also achieve 4K 30 frames of AI night scenes. Video is the next competitive track, and the real-time processing characteristics of video make it much more difficult than the first shot and then calculate the photo. OPPO can simultaneously achieve 4K+AI+HDR specification night video, which has touched the ceiling of the current mobile phone image performance.
At the beginning of the release, OPPO also announced a "Mariana MariSilicon X prototype", iPhone 13 Pro Max and OPPO Find X3 Pro three mobile phones in the 4K extreme dark environment real shooting comparison. From this, we can see that whether it is brightness, color and detail restoration, the gap between the three is very obvious.
The reason why it can achieve such excellent performance is also inseparable from the computing power of the Mariana MariSilicon X up to 18TOPS@int8, its performance is 20 times faster than the OPPO Find X3 equipped with Snapdragon 888, and in terms of AI computing power alone, it has surpassed Apple's current latest A15 SoC processor, which has an AI computing power of about 15.8TOPS.
Of course, the Mariana MariSilicon X capabilities are more than that, they also have real-time RAW computing, RGBW Pro, dual-layer storage and many other features. To put it simply, the future application of Mariana MariSilicon X in smartphones will exist independently of SoC, rather than replacing the AI capabilities in SoC, and the ultimate goal is to independently improve the image of mobile phones and achieve the results of 1+1>2.
Dual-core or become the mainstream choice for self-developed chip overtaking
In the current mobile phone industry, most of them use the platform architecture of mainstream SoC, which carries the main processing work in the operation of the mobile phone, in fact, it is also integrated with different chip modules such as CPU, GPU and NPU, but it is only packaged to the same chip.
For mobile phone manufacturers, if the research and development of SoC is directly challenged in the field of self-developed chips, it takes longer time cost, labor cost and input cost, which requires extremely rich experience and trial and error process behind it, including heterogeneous design capabilities, solving algorithm problems and process selection with manufacturers.
For this stage, the chip self-development technology of consolidating a certain function has become more realistic, and more experience can be obtained from it. Through the practice of two-core or even multi-core collaboration, it is also possible to achieve leapfrog improvement of functions on a certain vertical optimization route and build its own technical barriers.
OPPO already plans to ship the Mariana MariSilicon X in the next-generation Find X series flagship smartphone in the first quarter of 2022, a move that also proves OPPO's confidence in its chip capabilities.
However, we can directly select NPU from OPPO, and we can also see that OPPO has incorporated the heterogeneous design capabilities of the chip into the scope of self-research. OPPO's ultimate goal is to open a new era of mobile phone technology innovation and image capabilities through chip self-development, the release of Mariana MariSilicon X is OPPO to test its own technical capabilities and strength of the product, but also like Huawei, Apple, become a technology company with self-developed chip capabilities. It is believed that this image-specific NPU is only an attempt at this stage, and there will be more actions to be revealed in the future. (This article was first published on titanium media App)