CC BY
23https://doi.org/10.48417/technolang.2023.04.09 Research article
Mixed Reality as a Technology for Immersive Stage Space
Construction
Jiajun Shen1 (E) and Jing Zheng2 1Zhejiang Conservatory of Music, No. 1 Zheyin Road, Xihu District, Hangzhou, 310024, China
[email protected] 2China Academy of Art, 16 Bld., 352 Xiangshan Road, Zhejiang, Hangzhou, 310000, China
Abstract
In immersive theater, the innovation of stage design is one of the reasons for the great success of the performance. Stage design is to bring ordinary materials to the stage and redefine them through artistic creation, so that the audience can have a new aesthetic experience. Traditional theater stage design uses material materials to create a realistic space. With the development of technology in the digital age, the concept of "Mixed Reality" has been extended to stage design, allowing it to get rid of relying only on material materials. Mixed reality technology plays a significant role in the digital technology field by seamlessly integrating computer-generated virtual objects and scenes into the real world to create unique visual experiences. This paper proposes a heterotopia stage space construction technology based on Mixed Reality. It aims to study how to put virtual information into the real world by way of Mixed Reality in order to make dancers interact with a real virtual space. Then, taking as an example the dance drama Deep Weather - designed and produced by the author - this paper introduces the application of Mixed Reality technology in immersive theater, and focuses on the function and operation principle of the technology module. Finally, it summarizes a set of design principles of immersive theater construction technology based on Mixed Reality, including surprise, connecting physical space, aesthetics, facilitating actions, iteration and extension, and technicality.
Keywords: Stage Design; Immersive Theater; Mixed Reality; Mixed Reality Technology; Mixed Reality Art; Design Principles; Digital Aesthetics
Acknowledgment: Supported by Design-AI Lab of China Academy of Art, CAADAI2022B003. Supported by Key Laboratory of Intelligent Processing Technology for Digital Music (Zhejiang Conservatory of Music), Ministry of Culture and Tourism, 2022DMKLC003.
Citation: Shen, J., & Zheng, J. (2023). Mixed Reality as a Technology for Immersive Stage Space Construction. Technology and Language, 4(4), 104-125. https://doi.org/10.48417/technolang.2023.04.09
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
УДК: 7:004
https://doi.org/10.48417/technolang.2023.04.09 Научная статья
Смешанная реальность как технология иммерсивного построения сценического пространства
Цзяцзюнь Шен 1 (И) и Цзин Чжэн 2 1 Музыкальная консерватория Чжэцзян, 1 Чжэин Роуд, район Сиху, Ханчжоу, Китай
[email protected] 2Китайская академия искусств, Сяншань, 352, район Сиху, Ханчжоу, Чжэцзян, 310000, Китай
Аннотация
В иммерсивном театре новаторство оформления сцены - одна из причин успеха спектакля. Сценический дизайн призван принести на сцену обычные материалы и переопределить их посредством художественного творчества, чтобы зрители могли получить новый эстетический опыт. Традиционный театральный дизайн сцены использует материальные материалы для создания реалистичного пространства. С развитием технологий в эпоху цифровых технологий концепция "смешанной реальности" распространилась и на сценический дизайн, что позволило ему избавиться от опоры только на материальные материалы. Технология смешанной реальности играет важную роль в области цифровых технологий, плавно интегрируя сгенерированные компьютером виртуальные объекты и сцены в реальный мир для создания уникальных визуальных впечатлений. В данной статье предлагается технология построения гетеротопического сценического пространства на основе смешанной реальности. Целью проекта является изучение того, как перенести виртуальную информацию в реальный мир посредством смешанной реальности, чтобы танцоры могли взаимодействовать с реальным виртуальным пространством. Затем, на примере танцевальной драмы "Deep Weather, " разработанной и созданной автором, статья знакомит с применением технологии смешанной реальности в иммерсивном театре и фокусируется на функциях и принципе работы технологического модуля. Наконец, в нем обобщается набор принципов проектирования технологии строительства иммерсивного театра, основанной на смешанной реальности, включая неожиданность, соединение физического пространства, эстетику, облегчение действий, итерацию и расширение, а также техничность.
Ключевые слова: Дизайн сцены; Иммерсивный театр; Смешанная реальность; Технология смешанной реальности; Искусство смешанной реальности; Принципы дизайна; Цифровая эстетика
Благодарность: При поддержке Design-AI Lab Китайской академии искусств, CAADAI2022B003. При поддержке Ключевой лаборатории технологий интеллектуальной обработки цифровой музыки (Музыкальная консерватория Чжэцзян), Министерство культуры и туризма, 2022DMKLC003.
Для цитирования:. Shen, J., Zheng, J. Mixed Reality as a Technology for Immersive Stage Space Construction. // Technology and Language. 2023. № 4(4). P. 104-125. https://doi.org/10.48417/technolang.2023.04.09
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
INTRODUCTION The Technological Transformation of Stage Design
The stage space is not only the space carrier of the narration, but also the spacetime container for the audience to perceive the narrator's psychology. Through the stage space, the actors and the audience interact with each other, thus transcending the limitations of time and space of dance and drama performance, so that the performance is not restricted by the specific stage space but extends to the infinite field.
The innovation of stage space is also one of the reasons for the success of a stage performance. Presenting common materials from reality on stage and reinterpreting them through artistic creation, transcending their everyday symbolism and practical significance, placing them in an otherworldly space, and providing audiences with a simultaneous sense of familiarity and novelty, allowing them to gain a completely new artistic experience. The stage space establishes a specific "situation," which is based on the "situation." In addition to the "concrete space" that can be directly perceived by the audience, there is also an "abstract space" that requires the audience to mobilize their subjective initiative and give full play to their imagination. Stage space is not confined to the exploration of physical space but pays more attention to the imagination space hidden in the stage. Stage space constructs a "heterotopia" in real space.
Since the Renaissance, the design of stage space has been dominated by realistic graphic scenes. Swiss designer Adolphe Appia took the lead in challenging this tradition. He believed that when the three-dimensional actors on the stage touch the (two-dimensional) plane set, visual harmony is immediately broken, and that the set, which appears in the same stage space as the actors, should also be three-dimensional. Appia tried to put this idea into practice in a series of designs for Wagner's operas, using a large number of platforms and staircases on the stage, selecting semi-abstract three-dimensional devices and props to replace the traditional painted sets, and using a large number of symbolic stage devices to fully mobilize the audience's imagination that should fill in the details of the stage set. Appia also used light and shadow to integrate actors and three-dimensional scenery elements into a unified space by changing the intensity, color and projection angle of light (Yu, 2018). At the same time in China, Classical opera uses specific ways to arrange the stage space, for example, setting a door at each end of the backdrop, which dictates the path for actors to enter and exit (Qin, 2003).
In modern times, stage design has tended to move large-scale real scenes onto the stage. For example, the Hippodrome Theatre on Sixth Avenue in New York City is a masterpiece of American stage art with its spectacular set designs. Spanning an entire city block, every feature of the theater leaves the audience in awe, including two water tanks supported by hydraulic lifts for various aquatic performances (Cui, 2006).
Expressionist stage design differs from traditional forms by removing elements unrelated to the spirit of the drama. This includes visual factors that merely reflect the surface of daily life without engaging with soul role of stage art is greatly reduced and becomes increasingly subjective and abstract, evolving into a condensed symbol.
However, whether in Renaissance or modern expressionism, traditional theater stage design takes materials and their processing as the core, and is a realistic space created by material materials. With the development of technology in the digital age, the concept of "Mixed Reality" has been expanded, triggering the reconstruction of the relationship between Virtual Reality and reality, and making art creation get rid of relying solely on material materials. In addition to Mixed Reality (MR), there are also Virtual Reality (VR) and Augmented Reality (AR) in the digital age. If the entity space of the stage, based on materials and material processing technology, is to create a real "heterotopia" for the audience, then Mixed Reality, Virtual Reality and Augmented Reality technology build an immersive virtual space "heterotopia" for the audience.
Immersive Theater and Mixed Reality Technology
The term "immersion" in "immersive theater" was first explained from the perspective of social psychology and was described in terms of "flow" theory by William Lidwell in Universal Principles of Design (Lidwell et al., 2003). In 1990, Mihaly Csikszentmihalyi, a psychologist at the University of Chicago, elaborated on flow in detail in his book Flow: The Psychology of Optimal Experience (Csikszentmihalyi, 1991), systematic description of the immersive psychological experience. His research indicates that when people are fully immersed in a particular situation, they experience a high level of concentration and a continuous sense of excitement and fulfillment, akin to the flow of water. In this state, individuals filter out and exclude all irrelevant perceptions, forget the real world, and enter a realm of self-forgetfulness, namely the state of flow (Gao, 2022). This experience will give people a feeling of enrichment, excitement and happiness, which is also called the best feeling.
The origin of immersive theater can be traced back to the UK, deeply influenced by the art of Broadway in New York. Its concept is rooted in the "environmental theater" theory proposed by the American theater theorist Richard Schechner (Schechner, 1968). With the development of the postmodern context, immersive theater gradually extended and formed. Around 1960, some avant-garde theater artists in New York began to break away from the traditional theatrical model, introducing different forms of expression such as environmental theater, improvisational art, and interactive theater. Among them, improvisational art emphasizes the integration of life and art, presenting the behavior and process of people through fortuitous events (Gao, 2022). In recent years, immersive theater become a global hotspot. Immersive theater combines pre-planned storylines with spontaneous events, breaking away from the traditional separation of audience and performance spaces in theaters, and redefining the relationship between the audience and performers. The traditional fixed stage and audience seating are abandoned in favor of active spaces where actors move and perform, and audiences can freely choose which characters they want to follow. In 2015, Meng Jinghui's Mermaid by the Dead Sea became China's first original immersive theater production, integrating various art elements to depict an emotional tragedy. Audiences can experience the performance up close and personal, a characteristic of immersive theater created through multi-linear narratives.
Nowadays, digital media art mixed with traditional stage technology is widely used in Immersive Theater. By utilizing visual installations, motion capture, mixed reality, as well as comprehensive sound systems, stage lighting, props, and set design, a diverse range of scenescapes is created. For example, Klaus Obermaier's 2012 work Apparition, by employing sophisticated computer vision algorithms in conjunction with 3D motion-capture technology, the system can real-time recognize the movements, outlines, and shapes of performers against virtual backgrounds. It then translates this data into images, precisely projecting them onto the performers, thereby achieving a unique and vivid visual effect.
The adoption of other digital technologies allows the audience to interact with spatial media in a more intuitive way. For example, the piece M.U.R.S created by the Spanish avant-garde theater group Lavra. In the performance, smart mobile devices become a tool for the audience to participate in the performance. Before the performance, the audience needs to download an App that was jointly developed by MIT and Barcelona Lab. During the performance, the audience will receive various instructions from App, guiding them to choose their respective camp (Chen, 2017). The director sets the performance venue as a futuristic city, a closed environment resembling a fortress, where everything is under digital control. Here, the audience will feel the smart life of the future city in advance and become a "puppet" of mobile phones. By way of strongly interactive immersive theater, the creator expressed his concern about the future urban survival of humankind. In Eyes, another young domestic artist - Zhang Daming with his VR dance theater work - used a 360 degree camera to shoot the performance in the traditional stage, then converting it to VR glasses. Eyes is a dance video work that transforms dramatic techniques into Virtual Reality content creation. It explores the Immersive Theater space experience by designing action tracks for dancers and panoramic cameras at the same time. In this work, VR creates a theatrical atmosphere, allowing the audience to be surrounded by this atmosphere. This feeling of being there is different from that of a bounded screen. VR theater compresses time and space, compresses emotion, makes all things more energy-intensive, makes the audience's sensory organs absorb more information within a certain period of time, and heighten the sensitivity of their sensory organs (CISD, 2021). This immersive drama of Virtual Reality brings a more rich and unique immersive experience to the audience.
From a technical point of view, Mixed Reality and Augmented Reality often use the same technical system, but there is a difference at the application level. Augmented Reality (AR) technology, a significant branch in the field of digital technology, enhances real-world information by embedding computer-generated virtual elements into the physical environment. Users can enter this environment and engage in human-computer interaction by wearing special devices such as AR glasses and data gloves. Compared with VR technology, AR can effectively integrate real-world scenes and digital information, thereby enhancing users' real-life experiences (Lei & Xin, 2019).
Augmented Reality technology originated from early computer interface research, and frequent depictions of augmented reality scenes can be found in early movies and science fiction novels, such as Iron Man and Galaxy Guard, semi-mechanical characters
enhance their observed world by overlaying annotations and images using the visual systems in their helmets or glasses (Lei, 2015).
In 1968, Ivan Sutherland, a pioneering figure in computer graphics and Virtual Reality at Harvard University, developed the world's first Augmented Reality system for head-mounted displays called STHMD (see-through Head-Mounted Displays), and published a paper titled "A Head-Mounted Three-Dimensional Display," thereby inaugurating the era of Augmented Reality (Lei, 2015).
In the late 1980s and early 1990s, research into Augmented Reality (AR) gradually gained momentum. In 1990, Bajura and his team developed a system that utilized ultrasound technology to overlay images of a patient's abdominal cavity onto a video perspective display, allowing surgeons to seemingly see internal organ pathologies as if looking through the patient's skin (Lei, 2015).
In 1993, computer science professor Steven Feiner at Columbia University in the United States developed an augmented reality system for guiding hardware maintenance. This system could overlay 3D visual explanations of mechanical principles onto a laser printer, providing convenience for mechanical maintenance personnel. Then, in 1996, Professor Steven Feiner's laboratory launched the first outdoor augmented reality system, marking the beginning of applied research into augmented reality technology in the laboratory (Lei, 2015).
In recent years, Augmented Reality technology has been widely used in the restoration of cultural heritage sites. According to international consensus, the transformation and restoration of cultural heritage sites should be carried out while preserving the original appearance of the sites and without destroying the material form of cultural heritage. The proposal of this consensus highlights the respect and cherish for cultural heritage, and Augmented Reality technology provides an ideal means for this. Taking the #Taull1123 project implemented in St. Clement's Church in Catalonia in 2015 as an example, the project created virtual images of the different historical stages of the wall paintings, including the restoration of wall painting images from over 800 years ago based on historical documents and fragments of wall paintings. A 9-minute augmented reality performance is conducted every half an hour, allowing the audience to seemingly travel through time and witness the brilliant ancient wall paintings firsthand. In the Archaeoguide project, visitors can use digital headsets to see scenes of ancient Greeks racing in the Olympia stadium. This interactive experience with ancient civilizations allows visitors to immerse themselves and feel the allure and remarkable achievements of history. In addition, in January 2017, Baidu's AR laboratory implemented a public welfare technology program, enabling users to open the Baidu App on their phones and point it at the Zhengyang Gate in Beijing to trigger a real-life scene, presenting the ancient daily life of people entering and leaving the city gate. This interaction with historical scenes through a mobile app provides people with a more convenient way to understand and experience the splendor of ancient civilizations (Kang, 2018).
In the above-mentioned applications, Augmented Reality technology demonstrates the capability to overlay virtual information onto the real world, providing users with a completely new perceptual experience. Users are able to perceive information that would
otherwise be inaccessible to their senses, adding new possibilities to human activities in both work and entertainment scenarios in the real world.1
Mixed Reality, which uses the same technology system as Augmented Reality, enhances the real world by placing virtual objects in the real world and locking their positions based on the spatial orientation of the real world. For example, placing virtual pet cats on a table in the real world and adjusting accordingly when users move back and forth. Microsoft's HoloLens, Magic Leap, and Meta 2 are all committed to this style of Mixed Reality, and virtual things do seem like a part of one's real world. Mixed reality is a special type of Augmented Reality, and its implementation requires an environment that can interact with various things in the real world. If everything in the environment is virtual, this is the field of VR. If the virtual information presented is only superimposed on real things, then it is called AR. The key to MR is to interact with the real world, obtaining real-time information.
In Immersive Theater, can the application of Mixed Reality technology stimulate the audience's more complex space-time imagination? Jean Francois Lyotard's "Strange Moments" (Moments Étranges) reveals an unusual presence in everyday reality, so that the shock of viewing can come from the simultaneous appearance of the real and the virtual world in live space (Lyotard, 2017). Through an exploration of the MR dance theater piece Deep Weather, this study shows how to bring virtual objects into the real world through Mixed Reality technology to make dancers interact with a real virtual space, and how to activate a space-time beyond the current physical entity through the spatial reconstruction of a physical digital system.
THE CONSTRUCTION TECHNOLOGY OF IMMERSIVE STAGE SPACE BASED ON MIXED REALITY:
This paper aims to study how to reconstruct the space of the stage through Mixed Reality technology, and to build an immersive "space heterotopia" for the audience.
In recent years, global climate anomalies have become increasingly apparent, with a series of extreme weather events occurring frequently, causing serious impacts on human society. The world is currently undergoing a large-scale climate change, characterized by a significant trend of rising temperatures. Against the backdrop of continuously increasing global temperatures, a number of extreme weather events, such as the El Niño phenomenon, are occurring with greater frequency (Wang et al., 2007). Forest fires are sensitive to climate fluctuations.
1 The real world is the world that subjects can perceive without relying on any tools and only with their senses. It is commonly referred to as the physical world. Virtual Reality is a world created by computers. Through wearable devices, human senses can only obtain perceptual signals provided by computers, thus immersing themselves in the virtual world created by computers (Wang, 2021).
THE EXAMPLE OF DEEP WEATHER
Project Introduction
In the discussion of global warming, we can easily obtain a large amount of scientific data about climate and the environment. However, relying solely on rational thinking does not fully help us understand the profound extent of environmental changes on Earth. Therefore, the dance-drama Deep Weather attempts to present the fragile, complex, poetic and extremely substantial relationship between humans and the earth in the form of a virtual "presence."
Due to the particularity of Mixed Reality dance drama, the first step of the production is to investigate the performance space. We set the performance scene of the dance drama in a White Cube.2 Mixed Reality technology can show 300 trees rising from the ground, changing with the weather and seasons, and finally catching fire due to climate change.
The combination of new technology and dance serves two purposes. First, it is a metaphor for the future. What may happen in the illusion is a real scene of the future, namely a timely warning of an unpredictable future caused by the climate crisis and forest degradation. Secondly, to break the traditional stage spatial pattern, reconstructing the spatial mechanism through Mixed Reality technology, creating a surreal reality, and providing a new way to experiencing dance.
The final venue of the work is located in the Hangzhou Baolong Art Center which is located in the open space of the art museum on the top floor of Baolong city commercial center. This is a conventional white box exhibition space that measures 22 meters in length, 16 meters in width, with a total area of 352 m2 and a usable height of 5 meters. Two columns stand in the two-thirds of the exhibition hall, dividing it into two areas. The exhibition walls are white solid cement walls, opaque. On each side of the display wall, there is a 2-meter wide entrance leading to a corridor, which connects to other exhibition halls. The northeast entrance has been determined as the primary visual placement, as it directly leads to the central hall of the exhibition center, and is the first door visitors see upon entering the venue. The white wall on the other side has been designed as a projection surface, serving as the primary canvas for augmented reality projection in this project.
Deep Weather consists of four main scenarios, where the combination of virtual landscapes and real physical environments creates a unique aesthetic experience. On-site viewing will be available through both projection screens and mobile devices.
Scene 1: a huge abstract forest. Each dancer is dancing in the forest with a piece of black cloth in his hand. On the cloth are animals, such as horses and deer. Modern dance.
Scene 2: urban industrialization, waste, eagles, and physical gestures by actors, expressing a sense of anxiety.
Scene 3: The increase in temperature has resulted in forest fires, with actors situated amidst the flames, using their bodies to express pain. They carry animals painted on black cloth, creating a somewhat occult atmosphere, resembling a form of performance art.
Scene 4: the rise in temperature causes glaciers to melt, large pieces of ice float around the dancers' heads and bodies, the sea level rises, the land is submerged, and people are drowning. A lonely house appears in the scene.
2 White Cube, also known as white space, refers to the mode of a modern art exhibition public space.
The sound scheme of Deep Weather is as follows:
In the first part, a voice at the beginning expresses the good memories of the past in the forest, winter. The sound of this scene expresses a faint sad memory and imagination of the forest, and the sound of human breathing.
In the second scene, someone constantly recites a string of numbers to express the continuously rising data in climate change.The voice of urban industrialization is anxious and gradually tends to be uncontrollable. There are eagles chirping. The sound is mixed with the sound of biting the apple and the sound of sucking, showing desire.
The sound in the third part takes as its main line3 a kind of breathing, mixed with sound from inside the body, the sound of animals, and the sound of celestial bodies.
In the final scene, the sound can exhibit the melting of glaciers, the impact of glaciers, the sound of water, and the deep sound of marine organisms. Xintianyou4 is the end.
After determing the scheme of scenes and space, 3D modeling and 3D scene production will begin. In the MR scene the audience can operate mobile phones, iPads, and other devices to walk in a virtual space. Whether it is a burning forest or a floating planet, it will be as omni-directional as our landscape in the real world. The audience can turn around, look up and look down. Therefore, 3D scene production is indispensable to enhance the authenticity of the MR scene.
The three-dimensional scene is composed of several three-dimensional models. There are certain rules for the production of three-dimensional models. Since the number of "faces" of objects which the MR engine Unity3d and the player hardware can support is limited, the shape of objects should be realistic, and the number of faces of models should be minimized, so that fluency and modeling style can be perfectly unified during MR viewing. In order to allocate resources reasonably in the Unity engine, it is necessary to create low-precision models while preparing high-precision models. According to the construction method of the patch, it can be divided into the following two types:
For low- precision objects, only one patch map can be used to represent the distant landscape. The high-definition perspective seen in some 3D virtual scenes is a kind of material effect, created by drawing maps, which is very appropriate in some large virtual scenes.
Every detail of a high-precision object needs to be faithfully reproduced. For example, every part of the floating ship (fig. 1) in the project needs to be restored. With the addition of mapping materials, the number of faces is far beyond the range of the Unity3d engine, which is difficult to achieve smooth operation. Therefore, in the Unity3d engine, we will control the faces within 200000. Unity3d engine generally does not use models with too many faces. Such models are usually used in the field of film.
3 In Chinese Qi studies, heavenly bodies are connected to the human body, and changes in the natural environment can also bring about a lack of harmony in the body's aura.
4 Xintianyou, a kind of Shanxi local melody, folksong popular in China's Northwest.
3D Scene
Figure 1. High-precision model of Deep Weather (created by the author)
Of course, low-precision 3D models that are close to the real effect can also be achieved by rendering baking maps. The principle of baking maps is to store the visual information of the surface on the map. These visual cues encompass surface color, texture, transparency, reflection and refraction. It is these attributes that give the three-dimensional virtual world similar color and lighting information to the real world. This is our baking result (fig. 2). We use low face models for some scene models, but we use baking maps for rendering. After importing Unity, the visual effect is much better than that of ordinary game images. Of course, it is different from 3D movies. The image quality of MR can be used as a reference for mobile game images, but the current technology cannot achieve a complete real image.
Figure 2. Low-precision model of Deep Weather (created by the author)
Unity3d Effects
In the scenes of Deep Weather there are special effects involved, such as orest fire, thick smoke, and intense shaking Unity3d, as one of the commonly used engines in game development, places particular importance on the implementation of special effects. In Unity3D, the implementation of almost all special effects is directly or indirectly related to the particle system. The particle system is a powerful tool that can be used to simulate and render a wide range of visual effects, such as flames, smoke, splashes, explosions, and more. At the core of the particle system are the particles, which are made up of a large number of simple small images or networks. Each particle represents a small part of a fluid or amorphous entity, and the numerous particles together present a complete physical sensation.
Before commencing the project, we gained an in-depth understanding of the various attributes of the Particle System in Unity3D. The properties of the particle system encompass the appearance, movement, lifespan, emitter shape, speed, color, size, and more. By adjusting these properties, developers can achieve various complex visual effects, adding more audiovisual feasts to the work. Taking the flame effect as an example, the flame effect is typically composed of multiple flame particles that move, rotate, and fade in three-dimensional space, ultimately presenting a realistic flame effect. Additionally, the particle system can also achieve the emission shape of the flame, the change in speed over time, the smoke effect, and more, making the flame effect more lifelike and vivid.
The creative process of crafting a burning flame effect in Unity3D encompasses several key aspects. Firstly, we need to consider the generation position of the flame and how to seamlessly integrate it with the volcanic terrain in the scene. To achieve this, we chose to use a box-shaped emitter to ensure that the flame naturally blends into the terrain environment. Secondly, the color of the flame is influenced by the burning material, causing it to undergo color changes during its lifespan. To simulate this effect, we set multiple particles to mimic the flame at different positions, accurately depicting subtle changes in flame color. Finally, when the flame ignites, the local air becomes less dense due to heating, creating micro-convection with the surrounding cool air. In air of different temperatures, the refractive index also varies, causing light to produce varying degrees of refraction. This phenomenon creates the swaying visual effect of the flame from side to side, enhancing its authenticity.
In the process of combustion, if we want the flight direction of flame particles to be roughly aligned with the direction of the airflow, we control the wind field values of the smoke system and flame system through parameters. And these parameters are based on the physical laws of real world (fig. 3).
Figure 3. Unity 3D combustion effect of Deep Weather (created by the author)
In the course of the project it is also determined that the docking between 3D Max and Unity3d virtual engine should be run in in advance. If you want to express special effects, it is impossible to import Unity3d after finishing in 3D Max, because the special effects of various 3D software are not common. These effects can only be done in Unity3d. Or the special effects can be created in other software, rendered as two-dimensional Gif animation and imported into Unity3d. But this means that the scene is a two-dimensional animation. If the audience's perspective changes, the sense of space achieved by the 3D scene cannot be achieved anymore. Therefore, if the audience's perspective is limited to a fixed dimension during the performance, one can try a two-dimensional if animation. If the perspective changes a lot, you still need Unity3d to produce special effects. Of course, there are not many special effects in the scene. The main reason is that the performance of the mobile terminal is breaks down (there will be jams and flashbacks).
Function of Vuforia
In the development process of the MR dance of Deep Weather, one cannot support the MR effect only by using a Unity3d engine, so a third-party SDK needs to be introduced. In order to give the audience a better immersive experience, we chose Vuforia, which has high stability and the fastest recognition speed. Its core functions include:
1.Target recognition: Vuforia supports multiple types of target recognition, including images, products, text, targets, etc. User devices can recognize these targets in real scenes and interact with virtual content.
2. Target Tracking: Vuforia can track the location and direction of targets in real time, ensuring the perfect combination of virtual content and real-world scenes. The tracking function can also recognize gesture control and voice commands, allowing users to interact with the AR experience more naturally.
Implementation of Mixed Reality
The software used for creating mixed reality scene models is 3D Max. Elements such as forests, volcanoes, glaciers, and small boats are all crafted in 3D Max and then imported into Unity3D for scene assembly.
Occlusion is an issue in mixed reality. Typically, it is not possible to achieve the effect where virtual objects are occluded by real-world objects. For instance, in the case of a forest, within a mixed reality setup, one cannot make the dancer appear in front of the forest. At best, we can only render the forest as semi-transparent, with the dancers moving through it. To address this issue, we have integrated an AR Occlusion Manager script into our AR Camera. This script facilitates environmental depth estimations, human body detection, and environmental scanning to calculate when occlusions should happen, allowing virtual objects to be properly occluded. This enhances the harmony between the virtual scenery and the dancers.
Site Commissioning
Unlike traditional stagings, a digital virtual scene is superimposed on the real stage by using MR technology. In order to ensure the accuracy of the real-time interactive relationship between actors and virtual scenes, virtual scene debugging and dance rehearsal are carried out synchronously during the performance, and they are optimized and adjusted together.
Throughout the entire debugging process in the virtual scene, we meticulously debugged and optimized every aspect, including the position, size, animation effects, special effects presentation, and changes in lighting and shadow in the virtual scene. In order to ensure the precise presentation of details throughout the entire performance process, we conducted multiple repeated rehearsals to ensure that every detail meets the high standard requirements of the artistic creation, thereby achieving the best artistic effect for the entire performance. This series of meticulous work not only played a key role in the perfect presentation of the virtual scene, but also laid a solid foundation for the successful presentation of the entire stage performance.
For example, with three-dimensional production software, we make the shape and texture of the forest close to a forest in real nature, and adjust the size to the towering trees that one needs to look up to. In Unity3d, the position of elements is adjusted in combination with the stage structure and lighting area of the scene. A white line animation is made (fig. 4), so that the multi-level white lines wrap around the forest of trees from the four sides of the stage, and the actors perform in the lit area according to the images that are returned to the big screen.
Figure 4. White line animation in Unity3d (created by the author)
The MR stage is a new technology used to replace traditional stage sets, creating a virtual three-dimensional space in which actors perform on stage while the audience perceives them as being within a virtual environment. This innovation expands the spatial possibilities of stage performance and enriches the expressive power of stage art. Taking Deep Weather as an example, through MR technology, this production successfully created a three-dimensional virtual space on the theater stage (fig. 5-8). The virtual space was also projected onto the walls of the theater, allowing audience members who cannot use mobile devices to immerse themselves in the atmosphere of global ecological change, and providing a strong sense of presence at the scene for all viewers.
/ \ .- < v v -
! - . - i
! ! - jkf—
\шш
> V^j - -
ЭДШЗй Щ
: k - m
'Ml rfl /.. Щ
1 v
Figure 5. A scene of Deep Weather - Dance in the forest (created by the author)
Figure 6. A scene of Deep Weather - The radiation of the sun (created by the author)
Figure 7. A scene of Deep Weather - Floating planet (created by the author)
Figure 8. A scene of Deep Weather - The Last Home of Humankind (created by the
author)
The virtual scene stage is a room of 22mx16m. Since we use an external camera to shoot the scene, and then MR synthesis is carried out by the host computer, the lighting of the site will directly affect the clarity of the virtual scene presentation. A site with sufficient brightness can enable the audience to obtain a clearer virtual and real landscape. To this end, we used a stage lamp to illuminate the dancers from one direction. The real scene illuminated by the stage lamp was clearly captured by the camera and integrated with the virtual landscape, and finally presented on the projection wall.
The work focuses on the construction of auditory soundscape, attempting to employ Cubase to create a 5.1 surround sound setup for a three-dimensional audio experience. It utilizes four speakers to achieve an enveloping sound effect, to enhance the immersive experience on site. In the end of performance incorporates the low rumbling of deep-sea creatures and the sound of human breathing.
In the sound production process, the Ambient White sound source and some sound samples are mainly used, and the Dolby Atmos renderer is used to produce the 5.1-channel panoramic sound, to arrange the sound according to the distance, and to draw the moving route of the sound on the automatic track.
Viewing Mode
We have established two modes of live viewing (fig. 9). One is on mobile devices, and the other is through projection screens. Mobile devices such as smartphones and iPads allow the audience to freely look around in 360 degrees. Projection screens enable viewers to see the content without interacting with any devices, keeping their attention focused on the performers
Figure 9. Analysis graph of Deep Weather, drawn by the author
The Perfomance as Installation Art
After the dance performance, the audience can watch the MR art installation through the application. The MR installation art works exhibited are all from the stage virtual scene, which restores the real world 1:1. Thanks to the immersive sensory atmosphere of MR works, the audience can instantly experience different virtual scenes from the forest to the depths of the ocean. In the process of device display, the natural substances and physical and climatic processes in the scene are no longer just the dramatic background of the narrative story - They step forward from behind the scenes, becoming the leading roles, bearing the fragile, complex, and poetic relationship between humanity and the earth.
The device scenario encompasses seven parts:
1 ^ There are many trees, and there are some interlaced white lines between the trees, and the names of some frequently flashing forests.
2^ The sun's radiation is heating up, the temperature is getting higher and higher, and the sun is moving and vibrating.
3 ^ The forest catching fire slowly, with thick smoke and violent vibrations. Finally, it bursts into flames and dissolves into the air, leaving only smoke behind.
4^ Floating planets.
5^ Suspended ice.
6^ The umbrellas opened one by one and gathered together.
7^ A house like a ship symbolizes the last home of humankind.
SUMMARY: DESIGN PRINCIPLES OF IMMERSIVE THEATER CONSTRUCTION TECHNOLOGY BASED ON MIXED REALITY
The immersive dance theater Deep Weather provides a field of action for dancers by building a digital virtual space. Generally speaking, the semantics of virtual reality can encompass the technology combining the real world with digital virtual techniques, that is to say, Mixed Reality is also a kind of Virtual Reality in a broad sense. However, in this article, it refers specifically to Mixed Reality technology, which is a technology that puts virtual objects into the real world so that it can be presented simultaneously with the real world, that is to say, what the audience sees is the coexistence of the real world and the virtual world. For example, in Deep Weather the actors are real and the stage environment is virtual. The audience sees the integration of the two. The immersive effect of Mixed Reality technology is to place virtual objects in empty space, which is the way and significance of MR innovating the stage space. The current MR technology enables real people and virtual objects to occlude each other, thus allowing for the spatially free distribution of objects and individuals on stage. This ensures that a cluster of virtual objects won't obscure the performers by densely covering the front space.
Through the artistic design practice of this project, combined with previous experiences in immersive theater, we have derived comprehensive design principles for immersive theater based on mixed reality: Surprise, Connecting Physical Space, Aesthetics, Facilitating Actions, Iteration and Extension, and Technicality.
Surprise
The charm of Mixed Reality is that in the real world it will surprise the experimenter by encountering all kinds of scenes beyond real experience. The feeling of surprise can be derived from characters different from real life, or presenting a rare landscape in reality, or having a deeper observation and understanding of the things narrated than ordinary people, or experiencing another life, or realizing a superpower that does not exist in reality. This kind of surprise is also a sense of strangeness (Lyotard, 2017), which breaks the fixed formula of certain orders, presents an unusual reality of the presence.
In reality, reasonable and unexpected designs often produce wonderful results. In the virtual world, we can use reality as a reference to conceive and create amazing scenes and events through exaggeration, association, metaphor, personification, and other methods. From the experience and evaluation of other works, the setting of surprise is the mystery behind creating the fun and allure in virtual reality works.
Connecting Physical Space
MR technology is an integrated technology that superimposes real space and virtual object. The virtual object is implanted in the real space where the audience is located and exists simultaneously with the real environment. Creating a space in reality is often limited by the rules of the physical world, and objects can only be placed in a fixed way. Using MR technology has achieved an effect that cannot be achieved in the real world. Therefore, when constructing an Immersive Theater space based on Mixed Reality, some basic visual principles can be used to achieve amazing effects related to real space.
"Connecting physical space" is an important concept in the creation of art with MR. The real world is governed by inherent physical properties and laws, with time and space constrained by certain fixed mechanisms. The application of mixed reality technology may transcend real-world logic, offering audiences a multi-layered sensory experience. For example, in Deep Weather, the audience can see the beginning of rain in the theater, see many towering trees rising on the floor of the theater, and see a huge burning volcano suspended under the roof of the theater, which is impossible in the real world. This surreal creation is not a random fabrication, but a reasonable surprise based on the structure of real space. For example, the MR game Night Terrors: The Beginning, which is enhanced based on the physical environment, is an MR horror game that can interact with real scenes. The game turns the player's home into a haunted house by way of MR technology. The game makes full use of mobile camera, GPS and other components. The game scene is generated by scanning the unique environmental layout in the home, and various terrorizing elements such as ghosts, demons and zombies are randomly generated (Su & Zhao, 2017). For another example, the young artist Huang Yixian recently created a Mixed Reality interactive immersive space work "VISION." Its final presentation took place in the Media Art Departments large laboratory. In agreement with the existing spatial structure of the laboratory, an underwater virtual world was constructed to realize the emotional healing of the audience.
In the above works, the use of MR technology is based on physical space, which fully combines virtual information with physical space. Through the unique creative
medium of MR technology, the fiction in art transforms into a poetic reality, guiding the audience into a surreal poetic space (Zheng, 2020).
With the rapid development of technology and equipment, virtual reality art has entered a stage that emphasizes interactive, visual, and auditory content demands. At this stage, the focus of the works has shifted from technological innovation to the exploration of content and media language features. Artists believe that creating interesting works is more meaningful than pursuing technically strong ones, so their works pay more attention to the current social reality. Meanwhile, the pursuit of fun and artistry inevitably becomes an important design principle for virtual reality art. However, it is important to note that there are differences and connections between the discussion of artistry in a digital environment and the discussion of artistry in traditional art (Wu, 2019).
Compared with traditional art, digital aesthetics has the possibility of multiple presentation, as witnessed by the development of digital movies and digital devices. For example, the well-known American science fiction writer William Gibson conceived and shaped the concept of cyberspace in his masterpiece Neuromancer (Gibson, 1986), Cyberspace is an abstract philosophical concept involving virtual reality in computers and computer networks. It is a new human living space based on knowledge and information, which not only facilitates the dissemination of knowledge more conveniently and quickly, but also achieves knowledge decentralization. Gibson believes that cyberspace is continuously expanding and will eventually engulf humanity. In cyberspace, individuals can be completely immersed in media and no longer need to concern themselves with actual events happening around them (Su & Zhao, 2017). By contrast, Olafur Eliasson's work Sun Companion (2020), located at the corner of the entrance of UCCA(Ullens Center for Contemporary Art), the shining sun is on the ladder of the entrance, forming a reasonable imagination in real space.
In Neuromancer the story is set in a cyberspace where computers and artificial intelligence control the world, with a strong anti-utopian flavor. In Sun Companion, the texture of the three-dimensional sun is slightly rough and lacks realism, but it can be considered as an aesthetic choice.
From these two examples, we can see that the change of media has brought about a change of aesthetics. In digital comprehensive art creation and design, the works that can move people are not real, orderly and noble, but can be attributed to the "punctum" and "air" of Roland Barthes. The original meaning of "punctum" refers to the small hole and small wound pierced by a needle in a photograph (Barthes, 1980). Roland Barthes proposed the aesthetic concept "punctum" to explain the conflict between unexpected elements and the image content in photographic works. In photographs, unrelated elements collide with each other, creating a sense of strangeness, giving the viewer a feeling of being pricked. This sensation not only exceeds conventional visual experiences but also guides the viewer to transcend the immediate visual impression and enter into a spiritual realm of experience (Zhang, 2020). This "inexpressible" spirit is "air" (Barthes, 1980).
Aesthetics
"Punctum" and "air" provide the key to the source of aesthetic feeling from a higher level. It is internally related to the narrative field set by the work and produces a deeper and multi-dimensional interpretation space. By adopting and paying attention to multiple artistic aesthetics, Virtual Reality art gradually returns to the corresponding aesthetic system from the shock and surprise brought by technology, and is deeply moved and impressed by art.
Facilitate Actions
We have long been used to static appreciation of Mixed Reality art installations, and the artworks related to MR art installations are mostly limited to the design language of static entity modeling. People often do not associate MR art installations with physical behavior. In Deep Weather, we conceive a new MR art vocabulary. For example, we drive actors to cross through the gap between virtual trees, and drive actors to hide under umbrellas through virtual rain and virtual umbrellas. Here, MR art devices are not only static devices for viewing, but also factors that promote dance behavior on the stage through the "permission and invitation" of behavior (Pallasmaa, 2011).
Iteration and Extension
In the optimization process of virtual reality art creations, it is essential to continuously test and iterate on the artwork based on the results of these tests. In the art piece Deep Weather, the virtual environment is linked with the sphere of movement of the dance performers. Therefore, various tests were conducted, including sectional tests and comprehensive tests, depending on the distribution of the performers within the theater space. The outcomes of these tests are utilized for the iterative updating of the artwork, driving its ongoing refinement.
In addition, we have also extended the "affordances" of the works, that is, in addition to the application of dance theater, through the live interaction between the audience and the MR art scene, the audience can enjoy a series of MR art installations. It expands the modes of artistic appreciation of the works.
Technicality
The achievement of a multi-level audio-visual interactive experience in Mixed Reality theater is the technical index or criterion to test Virtual Reality art works. When the scene of Deep Weather is used as an MR art device, the audience walks among it, surrounded by tall trees. When we encounter trees blocking the road, we will instinctively avoid them, which is the reflection of our psychology and cognition in the logic of the real world. If we don't sidestep and keep moving forward, we'll experience a sense of jumping out of the scene, this feeling of discontinuity will remind us that the landscapes before our eyes are not real. It is the realistic scene effect in the MR art installation that greatly enhances the sense of immersion.
Although Virtual Reality art has experienced a period of development, it is still encountering some bottlenecks. Taking the hardware equipment as an example, the display effect of current MR glasses is still far below the expectations of users. The resolution and refresh rate are far from sufficient. The low refresh rate causes the display
to lag and drag. Because the head-mounted display is close to the user's eyes, the resolution requirement is higher than that of the ordinary display placed at a certain distance from the user. However, it is difficult to provide sufficient resolution on such a small display screen, and thus we have no choice but to wait for hardware improvements.
New media technologies have vastly expanded the boundaries of art. They not only enable artists to transcend the limitations of physical materials but also open up a new channel for interaction with audiences. The evolution of artistic mediums poses a direct challenge and stimulation to traditional aesthetic theories. Under the influence of new media, we have witnessed the birth of new art forms and practices. This transformation is far from over, and its impact on future culture and society is as yet immeasurable.
REFERENCES
Barthes, R. (1980). La Chambre Claire: Note sur la Photographie. Dition du Seuil. Chen, Y. (2017). On the Application of Digital Technology in the Practice of
Contemporary Drama Art [Master's Thesis]. Shanghai Theater Academy. CISD. (2021). ZhangDaming: From Dance Beauty to VR. China Academy of Stage Arts.
https://mp.weixin.qq.com/s/xh519gny1exY3rjbGoK6CO Csikszentmihalyi, M. (1991). Flow: The Psychology of Optimal Experience. Harper Perennial.
Cui, P. (2006). Mirror of Technology and Civilization - Analysis of the Development of
Stage Art. Drama Literature, 11, 73-82 Gao, Y. (2022). Form of Expression and Aesthetics of Digital Media Art in Immersive
Drama [Master's Thesis]. Shanghai Conservatory of Music. Gibson, W. (1986). Neuromancer. Ace
Kang, L. J. (2018). Research on the Application of Augmented Reality in the Display of
Cultural Sites. Decoration, 3, 97-99. Lei, H. P, & Xin, L. (2019). Research on the Design of Immersive Teaching Methods
Based on Augmented Reality Technology. Computer Education, 1, 130-133. Lei, Z. L. (2015). Research on the Application of Android based Augmented Reality Technology in Tourism Experience [Master's Thesis]. Beijing University of Technology.
Lidwell, W., Holden, K. & Butler, J. (2003). Universal Principles of Design. Rockport Publishers.
Lyotard, J. F. (2017). The Idea of a Sovereign Film. In Acinemas: Lyotard's Philosophy of Film (pp. 62-70). Edinburgh University Press. https://doi.org/10.1515/9781474418959-010 Pallasmaa, J. (2011). The Embodied Image: Imagination and Imagery in Architecture. John Wiley and Sons.
Qin, Y. (2003). The Form of Stage Performance Determines the Form of Stage Art.
Journal of Beijing Dance Academy, 3, 96-99. Su, K. & Zhao, S. Y. (2017). Technical Principles and Commercial Applications of VR Virtual Reality and AR Augmented Reality. China Industry and Information Publishing Group, People's Post and Telecommunications Press.
Wang, A. R. (2021). Realistic Analysis of the Relationship between Reality and Virtual Reality in the Environment of Augmented Reality (AR) Technology. Dialectics of Nature Communication, 4, 40-46.
Wang, M. Y. Shu, L. F. Wang, J. S. Tian, X. R. & Li, H. (2007). Characteristics of Forest Combustibles in Southeastern Tibet and the Impact of Climate Change on Forest Fires. Fire Science, 1, 15-20.
Wu, N. N. (2019). Research on Interactive Art Design of Immersive Virtual Reality. [Doctoral Thesis]. Central Academy of Fine Arts.
Yu, H. B. (2018). Essentials of Scenography in the West. China Theatre Press.
Zhang, Q. (2020). The "Punctum" of Photos and Bart's "Ontological" Desire -A Reconstruction of the Text Logic of "La Chambre Claire". Zhejiang Academic Journal, 6, 189-190.
Zheng, J. (2020). "Virtual Presence" in Space Art in the Immersed Heterotopia Multimedia Era. New Arts, 5, 129-131.
СВЕДЕНИЯ ОБ АВТОРАХ/ THE AUTHORS
Цзяцзюнь Шен, [email protected], Jiajun Shen, [email protected],
Цзин Чжэн, [email protected] Jing Zheng, [email protected]
Статья поступила 5 августа 2023 Received: 5 August 2023
одобрена после рецензирования 26 октября 2023 Revised: 26 October 2023
принята к публикации 3 декабря 2023 Accepted: 3 December 2023
