Fast forward to December 2021, when the minority team was in Dali, almost every night someone would run up to the terrace and try to record the clear starry sky above the Erhai Sea with a variety of equipment.
Of these, several Google Pixel devices are the most impressive. In the past, we always lamented the charm of computational photography, but only when we used the mobile phone that brushed Weibo and douyin during the day to capture the moment of the brilliant galaxy, we really understood the subtlety and charm of the algorithm.
In addition to being amazed, several editors on the team who have been using Google Pixel have also talked to me about some interesting details: in the matter of computational photography, even the Google Pixel, which has always been regarded as the norm by the industry, actually has some shortcomings that are still difficult to overcome. One of the most representative issues includes color casts, a problem that has been mentioned since the Google Pixel 4 but has not been effectively solved until the recent Google Pixel 6.
Under the support of the night vision algorithm ( right ) the color of the picture has a significant color cast compared to the naked eye perception (left) | Pictured: Clyde shoots with the Google Pixel 4
Even the starry sky samples mentioned at the beginning are the same - for a veteran of computational photography like Google, color reproduction is a problem that is difficult to solve entirely by algorithms. In order to make the calculated starry sky more in line with the aesthetics of the look, in the picture retouching function in Google Photos, Google even prepared a set of "astronomical filters" for photos in night vision mode for post-color retouching.
So why is it so difficult to restore color?
▍The root of the problem is "guessing"
If you look at it in the direction of the Greek root, photograph in English is Photograph, which literally means "a painting made with light". In photography, the original carrier we used to "paint" was film, and the light information recorded on the negative needed to be chemically processed before it could be developed; after the emergence of CCDs and CMOS with photodiodes as the core, more efficient photoelectric signal conversion became mainstream.
But cmOS, whether it's a CCD or a more common one in smartphones, they can only record different luminance information based on light intensity, due to the principle of how photodiodes work. In other words, the photoelectric signal received by the sensor can only restore a black-and-white photo.
In order for the sensor to capture colored lighting information, a color filter array (CFA) appeared.
Common color filtering arrays: Bayer arrays | Image: Wikipedia
You can understand CFA as a layer of "filter" before the light reaches the sensor, taking the most widely used Bayer array filter as an example, the Bayer array takes 2 green pixels, 1 red pixel and 1 blue pixel as the unit, and the light through the filter can retain the corresponding color information at the same time in addition to leaving intensity information on the sensor. It is worth noting that because each pixel only filters and records one of the three colors of red, green and blue, the information obtained from a single pixel cannot fully reflect the composition of different colors in the picture.
Light passes through the array leaving different color messages | Image: Wikipedia
In other words, the loss of picture detail begins the moment we press the shutter button of our phone. In order to obtain a complete color image, the color information obtained through the Bayer array will also rely on the decay algorithm for interpolation and reconstruction. If the algorithm that uses the average of neighboring pixels like the following figure speculates on the missing pixel color information around the collected pixels, after completing the entire de-mosaic process, the reconstructed picture will even occupy 2/3 of the entire picture.
The process of restoring the color information obtained by the Bayer array is also the process of losing details | Pictured: Google
Because the "guess" component is inevitable, the resulting image generally cannot accurately restore the original shooting scene. Even on the basis of the proximity pixel average algorithm, many more complex de-mosaic algorithms have been derived, but the premise of "limited color information obtained through CFA" has always been placed here.
On the other hand, although many manufacturers have tried other CFA arrangements in recent years, such as Huawei's RYYB and OPPO's RGBW based on quadBayer arrays in 2014, these arrangements are more than the traditional Bayer arrays that are also improving the amount of sensor light, in line with the goal of "bloody fight" in the night scene shooting in previous years, but from the actual situation, purely relying on algorithmic guessing, it is not much helpful to solve the color restoration problem.
▍Take google pixel as a guide
After clarifying the basic fact that "color reproduction is difficult because photos are guessed", let's go back to the mobile computing photography giant Google Pixel mentioned at the beginning of the article. If you want to say that in the 6 years since the birth of Google Pixel, what is the most valuable experience left for the development of computing photography in the mobile phone industry, I personally believe that there are two points: the flexibility of making good use of all available algorithmic resources, and the ability to vertically integrate hardware.
Regarding the first point, the best example is the Super Res Zoom introduced by Google Pixel 3 - that is, the generation when Google is still using single cameras to "hang" other manufacturers' dual-camera telephoto.
When solving the problem of digital zoom cropping, reconstructing, and then causing detail to lose, the google camera team first thought of the entry point is also the CFA filter mentioned above. But compared to other peers, Google's engineers chose DRIZZLE, a common shooting technique that has been popular in astrophotography for decades. DRIZZLE achieves a 2x or even 3x digital zoom effect by capturing and compositing multiple photos with slightly varying angles, with the basic idea of merging multiple low-resolution bursts directly into a higher resolution pixel grid. The specific merge alignment process is as follows:
Pixel completion by panning | Pictured: Google
As for the ability to vertically integrate hardware, everyone is even more obvious.
Starting from the second generation of Google Pixel, almost every generation of Google Pixel devices (except a series and Pixel 5) will be equipped with a separate chip dedicated to image algorithm processing, which is called Pixel Visual Core on Google Pixel 2, evolved into Pixel Neural Core on Pixel 4, and finally came to this year's "big move" - Google Tensor.
The original Sensor Model is not very "sensitive" Google Pixel, with the increasing demand and functions of computational photography, the dependence on independent image processing chips is also increasing, embarking on the road of self-developed chips has become inevitable (the same is true for OPPO mentioned below) - after all, in the premise of having increased the RAM from 4GB to 6GB, Google Pixel 4 users will still encounter the situation of "not enough computing power to kill the background" from time to time.
Then again, since the algorithm strength and software and hardware integration are strong, such as google pixels, they still have not been able to solve the color reproduction problem mentioned at the beginning of the article. In recent years, what other ideas do domestic Android manufacturers have to solve problems?
▍The software is not enough hardware to make up
Here comes another manufacturer that is regarded by more people as the "industry benchmark" - Apple. Since the color information collected through the sensor is difficult to restore completely, we may wish to prepare a set of actual data from the real world for the image processor as a color calibration reference.
As for the calibration of color, in fact, there are special sensors responsible for collecting color data in reality, as early as 2017, Apple stuffed a multispectral sensor from AMS into the "bangs" matrix on the iPhone X. This custom version of the 6-channel multispectral sensor is a tribute to the iPhone's 10th anniversary, and the spatial color data collected is used to drive the True Tone True Color display function, which allows the phone to dynamically adjust the screen display based on the ambient light color temperature.
This feature is still somewhat new to this day, and it is not the only use of this type of sensor - as mentioned above, the Apple TV 4K released last year has a magical screen color correction function built-in: the iPhone paired with apple TV is pressed to the front of the TV screen, which can automatically calibrate the display color of the content of this screen In this process, it should be transformed into a simple screen color adjuster through a multispectral sensor.
Then, on flagship models such as Huawei P40 and P50 Pro, we saw the use of multi-channel multi-spectral color temperature sensors to assist shooting, on the one hand, it can be used as an ambient color reproduction standard to provide white balance reference, and on the other hand, it also solves the color cast problem caused by the RYYB array.
OPPO launched a 10-bit full-link color engine on Find X3, which optimizes the color experience from the screen hardware to ensure the integrated look and feel of mobile phone shooting and output.
▍ Find X5 How to deal with color puzzles
Taking the pioneer google pixel in the field of computational photography, as well as huawei and Apple's experience in software and hardware collaborative investment as a reference, OPPO has also made a vertical integration of software and hardware on find X5 based on the advantages of years of deep ploughing in the field of imaging.
The first is the addition of a natural color sensor. The natural color sensor used by OPPO this time mainly relies on a 13-channel spectral sensor, which is equivalent to increasing the electronic retina similar to the structure of the human eye, capturing accurate colors from the hardware level, restoring the real ambient light source information, and providing high-precision data for color reproduction from the source of image processing as a reference.
Of course, multispectral sensors are not new to consumers. The Huawei P50 Pocket, released late last year, enables UV detection, also through a 10-channel spectral sensor. The Find X5 series' 13-channel multispectral sensor should be able to calibrate solid color more accurately, which was not possible with an iPhone in the past.
The self-developed image NPU chip "Mariana MariSilicon X" will also be officially installed in the Find X5 series, bringing a fundamental qualitative change to the computing photography capability. Unlike most manufacturers in the past, which purchased the 3A algorithm directly from the chip manufacturer and then made small repairs, OPPO can do more, deeper and more flexible underlying development based on the Mariana MariSilicon X, with its up to 18Tops computing power and self-developed 3A algorithm, whether it is the integration of data input from the 13-channel spectral sensor, or the color restoration in the tricky scenes such as solid color and large face confusion color, the effect will be greatly improved.
Finally, of course, there is a calibration of the color output style, which requires a long-term experience of nature and aesthetics. OPPO Find X5 series has previously officially announced cooperation with Hasselblad, which has a century-old history, and has also passed Hasselblad's natural color certification, and the color debugging scheme is benchmarked against Hasselblad medium format cameras, and the actual color effect is very exciting.
We once thought that mobile phone photography has been very powerful, but the algorithm-based blind guessing effect has not been able to meet the needs of users, combined with the comprehensive strength of software and hardware natural color era, has just begun.