XR词汇的提出，可以追溯到CO Allen于2014年的申请的一份专利《Method for creation ofaugmented reality medium i.e. extended reality medium, involves detectingaugmented reality contents based on determination that reality data correspondsto given acquisition event data》，文中将XR作为AR的别称进行使用，约2年后，一些国际媒体服务器厂家开始在市场中尝试推广XR的概念。
XR (Extended Reality) is a technology that uses camera tracking and real-time image rendering technology to make the display medium (LED/projection screen) and the virtual scene outside the display medium track the camera angle in real time, and composite it with the real shooting image in front of the camera lens, so as to create a sense of infinite space.
XR is a fusion of virtual reality (VR), augmented reality (AR) and mixed reality (MR) to bring an "immersive" digital experience that seamlessly transitions between the virtual world and the real world.
In 2020, the rise of XR extension technology has brought a new revolution to the film and television industry, and LED virtual production based on LED background walls has become a hot topic that has been discussed by industry insiders. The combination of XR (Extend Reality) technology and LED display has built a bridge between virtual and reality, and has made a big splash in the field of film and television virtual production. This article will focus on an in-depth analysis of the overall production process, integrated equipment applications, and core advantages, so as to provide reference for the successful implementation of more film and television virtualization production projects.
What is LED Studio Virtual Production?
LED Studio Virtual Production is a comprehensive solution, tool and approach. We define LED virtual production as "real-time digital production", and in actual use, LED virtual production can be divided into two types of applications: "VP studio" and "XR extended studio".
VP Studio
VP studio is a new type of film and television shooting method, more used for the shooting of movies and TV series, which can enable film and television producers to replace the green screen with LED screens and display real-time backgrounds and visual effects directly on the set.
The advantages of VP Studio shooting can be reflected in several ways:
1. The photography space is free, and various scenes can be shot in the indoor studio, whether it is forests, grasslands, snow mountains, etc., the rendering engine can be used to create them in real time, which greatly reduces the cost of framing and shooting.
2. The whole production process is simplified, "what you see is what you get", the producer can view the desired shooting picture on the screen in time during the shooting process, and the scene content and narrative space can be modified and adjusted in real time, which greatly improves the efficiency of changing and changing scenes.
3. The performance space is immersive, and the actors on the set can perform in the immersive space while standing on the XR stage with real-time generated scenes, and experience it directly, making the actors' performances more realistic and natural.
At the same time, the light source of the LED display provides real light and shadow effects and delicate color performance for the scene, and the shooting effect is more realistic and perfect, which greatly improves the overall quality of the film.
4. Shorten the return on investment cycle, compared with the traditional time-consuming and labor-intensive film shooting process, virtual shooting and production can realize the release of the film faster because of its high efficiency and greatly reduced cycle, saving the actors' remuneration and the staff's expenses, and greatly reducing the shooting cost.
This virtual production of films based on LED backdrops is considered a huge development in filmmaking and provides a new impetus for the future of the film and television industry.
XR Extended Studio
XR extended shooting refers to the use of visual interaction technology, combining the real and the virtual through the production server, and using the LED display to create a virtual environment that can be interacted with by human and computer, bringing the audience a "sense of immersion" that seamlessly transitions between the virtual world and the real world.
XR extended studios can be used for live webcasts, live TV broadcasts, virtual concerts, virtual evenings, and advertising shoots. XR extended studio shooting can expand virtual content beyond the LED stage, so that virtual and reality can be superimposed in real time, enhance the audience's sense of visual impact and artistic creativity, and allow content creators to create infinite possibilities in a limited space and pursue endless visual experiences.
Whether XR can replace the green screen has been a hot topic. XR has some distinct advantages over green screens:
1. The background presented by XR in real time provides a sense of presence for the actors, which can allow people who lack experience in non-physical performances to intuitively understand the current situation and grasp their own positioning.
2. XR makes the effect of combining the actor's performance and the performance environment present in a timely manner, and it is more convenient for the creative team to modify the creative idea in time.
3. XR can save a lot of time and money spent on light restoration in the film and television production process.
However, the xR technology itself also has limitations, if it is based on the xR technology realized by the LED screen scheme, then there will be no shadow under the feet of the actor standing on the LED screen, which is a big regret of the xR technology.
Workflow for virtual production in LED studios
In the virtual production of LED studios, the entire shooting process of "VP Studio" and "XR Extended Studio" is roughly the same, which is divided into four parts: pre-preparation, pre-production, on-site production, and post-production.
VP Studio Shooting Process
The biggest difference between VP film and television production and traditional film production methods is the change in the process, and the most important feature is "post-production". VP Film & Television Production moves the 3D asset production and other links in the traditional visual effects film to the actual shooting of the film, and the virtual content of the pre-production can be directly used for the visual effects shooting in the live camera, while the post-production links such as rendering and synthesis are moved to the shooting site, and the composite picture is completed in real time, which greatly reduces the workload of post-production and improves the production efficiency.
In the early stages of video shooting, VFX artists use real-time rendering engines and virtual production systems to create 3D digital assets. Next, a seamless spliced LED display with high display performance is used as a background wall (usually presented as a ring screen or dome screen) to build an LED stage in the studio, and the pre-produced 3D rendering scene is loaded onto the LED background wall through the XR virtual server to create an immersive virtual scene with high picture quality. Then, a precise camera tracking system is used to track and shoot objects using object position tracking and positioning technology, and after the final shooting is completed, the captured footage is transmitted back to the XR virtual server through a specific protocol (Free-D) for viewing and editing.
XR extended studio shooting process
The steps for XR extended shooting are much the same as for VP studio shooting, but usually the entire shot is captured in the camera in a VP studio shoot without the need for extension.
In the XR extended studio, because of the particularity of its picture extension, there is an additional link to expand the "background" screen in post-production. After the captured footage is sent back to the XR virtual server, the scene needs to be extended to the outer cone without screen area through the method of image overlay, and the real scene and virtual location are integrated to achieve a more realistic and immersive background effect. Then, through color calibration, positioning correction, color matching and edge transition and other picture processing technologies, the harmony and unity of the inside and outside of the screen are realized, and the overall picture after expansion is finally output. In the background of the broadcast system, you can see and output the completed virtual scene.
On top of extended reality, XR extended shooting can also incorporate motion capture sensors to achieve the interactive effect of AR tracking, and performers can interact with virtual elements in 3D space on stage in real time and without restrictions.
What are the equipment used in LED virtual production?
LED studio virtual production is the integration of technology, and the software and hardware equipment required include LED display, virtual engine, camera tracking system and virtual production system. Only through the perfect integration of these software and hardware systems can we shoot fantastic and cool visual effects and obtain the final effect.
Although the LED display of XR extended studio has a small construction area, it needs to be supported by low latency characteristics in live broadcasting, requires greater data transmission and real-time interaction, and needs a stronger system to support real-time image processing.
The LED construction area of VP studio is large, but because it does not need to expand the picture, the requirements for the system are relatively low, but it requires high-quality picture shooting, and the configuration of other equipment such as virtual engine and camera is high, and it must reach the professional level.
Virtual Production Server
The virtual production server is the infrastructure that connects the physical stage with the virtual set, and is highly integrated with the LED display hardware, control system, content rendering engine, and camera tracking.
The XR virtual production server is the core of the virtual shooting workflow, which is responsible for accessing the camera tracking system + virtual production content + real-time images shot by the camera, outputting the virtual content to the LED wall, and outputting the synthesized XR video images to the broadcast station for live broadcast and storage.
The most common virtual production systems are: Disguise, Hecoos.
Rendering engine
The rendering engine of video production is the performance of various latest graphics technologies, and the application of the rendering engine to video production is like the engine of a car, which determines the realism, smoothness and attractiveness of the program picture.
The picture, scene, color effects, and other content that the viewer sees are directly controlled by the engine, and the implementation of these effects includes a number of rendering techniques: ray tracing, where image pixels are calculated as particles of light; Path Tracing - Ray Reflection Back Viewport Calculation; Photon mapping - light source emitting "photon" calculations; Radiosity - The illumination path of the scattered surface reflected into the camera.
最常见的渲染引擎有：Unreal Engine、Unity3D、Notch、Maya、3D MAX 。
Camera tracking system
Camera positioning tracking refers to the process of tracking the position and pose of a camera in real space. It is composed of a complete set of equipment such as positioning machine, rocker arm, camera stabilizer, PTZ, encoder, binocular stereo vision machine, etc.
The spatial information is transmitted to the rendering engine in real time, which provides the virtual camera in the engine with information about the position of the real camera in 3D space, as well as lens parameters including focal length, focus, and aperture. Relying on real-time rendering and real-time keying and compositing technology, the movement and composition of the virtual camera and the real camera are synchronized, so that the real image and virtual elements can be combined in real time.
There are three types of camera real-time dynamic tracking technology: grid recognition – a real-time tracking method based on the slight movement of the camera in the two-dimensional X and Y axis planes, without modifying the camera's tracking mode; Mechanical Sensing Tracking - mounted on a pedestal or tripod for a very precise and high-resolution solution; Infrared Tracking - Calculates the camera position by IR and emitter.
The most common camera tracking systems are: Mosys, Redspy.
Lighting system
Under the premise that the existing natural light source in the scene is insufficient, it is necessary to artificially enhance the lighting in the scene to enhance the light sensitivity of the film and improve the brightness of the picture, which plays an important role in the establishment of space and the creation of atmosphere.
Basic lighting - In the case that the LED background wall provides basic lighting, it is still necessary to use lighting equipment to supplement the light to make up for the problems of narrow spectrum, low color rendering, no directionality and no hard light in the background wall lighting; Supplementary light - LED light matrix + soft light to make up for the problems of narrow spectrum, low color rendering, no directionality and no hard light of background wall lighting; Special effects lighting - in a specific shooting scene, according to the scene + plot needs and temporary arrangement of lighting; Lighting Control System - CNC lighting can be directly driven and controlled by the virtual engine and XR fusion system through DMX/Artnet and other protocols.
A commonly used lighting system is ARRI.
LED display
LED studio virtual production is a new scene for large-screen display applications, and it is a new market derived from the continuous development of the LED small spacing market and the continuous improvement of the technical level of LED display equipment.
Compared with traditional LED screen applications, virtually produced LED displays have more stringent requirements for accurate color reproduction, dynamic high refresh, dynamic high brightness, dynamic high contrast, wide viewing angles without color shift, high-quality picture display, etc.
Inhibits moiré lines
Moiré is a high-frequency interference formed by the LED screen when imaging on the image sensor of the camera device, which will cause the picture to appear colored streaks. Mini 4-in-1 technology can make the LED display reach more than 80% of the luminous opening rate.
Mini LED with special lenses, depth of field and aperture allows for a wider degree of adjustment freedom and a lower degree of false focus, which cleverly avoids the moiré pattern of the display in the finished film, and finally achieves a realistic and immersive effect from shooting angles.
Low reflections
The smaller the light reflectance has on the surface of the screen, the more realistic the background can be. On the surface of the Mini small pitch screen, there is a unique light-absorbing groove design between the lamp groups to reduce the impact of light; And the surface is matte and matte, which controls the light reflectivity below 5%, and the shooting angle can be close to 180 degrees. It is easy to cope with all kinds of lighting environments on the shooting site, the picture is cleaner, and the shooting is not easy to wear.
Wide color gamut and high CRI
The application of color gamut matching and conversion technology can truly and accurately restore the colors of nature. High CRI ensures good color uniformity, optimizes the continuous spectrum of each color and band, and improves the color rendering index. At the same time, the luminous efficiency is improved, and part of the blue light energy is suppressed under the premise of ensuring the luminous efficiency, and the red (R9) spectrum is compensated.
Wide viewing angle without color deviation
Mini LED products can provide a wider shooting angle of nearly 180° to ensure no color cast; 24bit processing depth, better presentation of the level and details of the image, from any angle of observation or shooting, can present a good visual effect.
Low scan high refresh rate
With a 1/6 scanning rate and a refresh rate of 7680Hz, the Mini LED small pitch screen has a stable shooting picture under high refresh, no ripple black screen, clear image edges, and accurately restores the real information of the image.
Benefits of XR Making:
Immersive and real-time production and editing processes
Actors, hosts, and producers can see the virtual environment on set, allowing them to make adjustments and directions to perform or shoot on set, reducing the time and budget spent on post-production. To put it simply, it is "post-production pre-production". In addition, virtual production solves the cost of crew transportation and environment waiting for on-location shooting, so in short, you can reproduce and modify the desired environment at any time without having to wait for the rain or sunset to set.
Realistic on-site interactive lighting
Lighting is provided by an LED display on the XR site, which facilitates the seamless integration of real people with the virtual environment and reduces the time for further lighting adjustments. This is especially useful for shooting real objects with high reflections, where realistic ambient reflections and lighting need to be provided without the need for post-compositing additions.
There is no color spillage problem and no problem with keying compositing
In a traditional green screen shooting environment, it often takes a lot of time to finely key and fix color spills, but XR doesn't need to be keyed at all, and the shot is straight out of the scene.
Industry applications of XR
Corporate Events
Today's companies are increasingly looking to XR as a new way to solve the problem of remote meetings. Virtual conferences, virtual presentations, and even product launches can all use XR technology to enhance the immersive experience of attendees and give them a real sense of the brand's culture and philosophy.
educate
XR allows students in remote teaching to immerse themselves in realistic scenarios, allowing students to better understand, learn, and impress in simulated virtual situations.
Movie
xR provides opportunities for filmmaking, reducing staff travel time and exposure hazards during the pandemic. In post-production, LED screens offer a faster, easier, and higher-quality alternative to green screens, resulting in shorter production cycles and more creative visual decisions can be made in real-time.
Television studios
In a studio environment, LED screens provide high-resolution, clear content and lighting in real-time, and while cameras are required to capture it at the end, real-time content allows presenters to better interact with the virtual environment, rather than guessing and stepping on points as before.