CC BY
15Journal of Siberian Federal University. Humanities & Social Sciences 7 (2016 9) 1661-1667
УДК 004.921
Information and Communications Technologies (ICT) and Multimedia Technologies in Art and Culture
Nikolay V. Borisov*, Artem А. Smolin, Denis A. Stolyarov*, Pavel P. Shcherbakov and Vasiliy A. Trushin
ITMO University 49 Kronverksky pr, St. Petersburg, 197101, Russia Saint Petersburg State University 7/9 University Embankment, St. Petersburg, 199034, Russia
Received 17.03.2016, received in revised form 24.05.2016, accepted 19.06.2016
In this article, through the example of the projects of the Centre of Design and Multimedia of ITMO University the possibilities of using a variety of equipment and software in variousfields of culture and art are demonstrated. Information and multimedia technologies have given opportunity comprehensively enrich different creative and cultural industries and arts, introduced more possibilities into creative projects to create a special creative atmosphere and a deeper and more emotional perception of creative content.
Keywords: Keywords: multimedia, culture, Internet Theater, multimedia 360°, virtual reality. DOI: 10.17516/1997-1370-2016-9-7-1661-1667. Research area: culture studies.
Introduction
Museums, theaters, libraries and other cultural institutions for a long time feel at ease in the paradigm of the information society. Information technologies in all their diversity actively contribute to the processes of education, cognition and various game practices.
Modern theater directors and playwrights, actively using multimedia technology in their performances are looking for new ways of communication with the audience.
One of the vivid examples of modern theater production that uses cutting-edge technologies is a theatrical production "Neurointegrum"
(http://art1.ru/teatr/nauka-illyuzii-kak-illyuziya-nauki/), set at the New Stage of Alexandrinsky Theatre (Fig.1).
"Neurointegrum" is a media performance where the emotional state and a set of mental commands used by the actor to operate audiovisual algorithms of production act as a performing instrument. The system of tracking movement and gesture recognition is used as well. The whole production is automatically generated in real time.
Multi-projection - stage is represented by two screens sized 10x4.5 meters for graphical projections (without the use of the previously
© Siberian Federal University. All rights reserved
* Corresponding author E-mail address: nikborisov@gmail.com; denis1900@mail.ru
- 1661 -
recorded video - the graphics is generated in real time), and a separateprojection on theperformer's silhouette.
Multi-channel surround sound-fourteen audiochannelsandtwosubwoofers areuseO indie performance,all ore leperatedaol uoel faathe o^i^chc^iicoi^seicond sdiroand efOerts. Additionollf, for gesture recognition, IR camera and Mieroiofe Krmcl are oieO; theoeii alsd F^oWo^^^^i^^oV^a^^ KOOHohtinaUatoin, whfcOis intergctlvelycontoolledbythe petCormer'sheod io one of thnrceres.
The performance is operated by four compuCers:two person^ comcoterstc gecerate projections (onto the floor, in the rear projection screen and onto the performer's silhouette), one iMac computer with the installed Windows system to work with neural interface Emotiv Epoc and Microsoft Kinect, as well as one Mac to generate sound. The sound of the performance is also gefrraeed m renltimf,meludinn surronnd efiecle.
Multimedia Technologies jnTheat te Ajioviiies
In 2013, in Saint Petersburg, the staff of the Cenlie of Design and Multimedia od iTMO Univereieytookpart if itf «^eaei^i^i^g uf 0ha rheatfronOcultural NewOtnaeef
AleoandridsUe The^t^se". laise offd^ ma^ni onc^ls of the complex is active use of modern multimedia iechnologtoim tneatrioalfrndootions.
Wiahinonrbihhrfamous film director Andrey MogenCy usodCheworPsof the preit Rpssiom wrltedFyodooMIObaClpvioCDosPeyevsky
"New technologies. They return us to the modern man. We are taking the essence of the technology - a new structure of thinking, a new attitude to space and time.
We will start testing of the New Stage with the key figure of civilization, the bearer of the code of nation - Dostoevsky (7/test.thenewstage. ru/ru/project/21/tc-dostoevskiy)".
Wrihinthefeama sofotesho e^enmglhe ose of naak)us teohuoiogihar defies, us web an
3. Sources of media streams
Server room
7 Portal "Theatrical workshop"
1. Synchronization server
Fig. 2. Graph of themultipurposehardwareandsoftwa
the creation of various software and hardware c omplexes wasof a fualoature.
Ow the onehand, tOeiusk to Implemift tetiinoiogical s^t^^utioi^s that enfhla te provide information teaices No tie NewStmge oS AiexindrlnskpTieatre waa sit tothe ang^^a^re of tdr Retrstah tTMO UniapraiSy, c^n ihe o tOerhand ttwas decessafyto tdtrodont anirisiy of hrtdwaie undsnftwaraoamptefraas tleoiage design for the performance.
A multifunctional hardware and software c o mplex (Fig.2), which allows carrying out online viewing of the theater and other thematic events (lectures, workshops, etc.) [3] has become one of the implemented projects.
The following interrelated objectives were set and implemented:
1. Capture and delivery of a few high-quality video streams to the viewer at the same time;
2. Flexible connection of various sources of video streams and their prompt redirection;
tyitemontheNhwStaweofAieiaidrinaky Theatre
3. Remote diagnostics of the state of the videt sSreoma tources;
4. Cnnfiftiratien awd rationatplanning of the aietworf iuftastrueturr ase to onsure rtaular ttwbledNiratian uf the wdoinsystam.
The algarithm oi the hrrdwaae ani roiiwaae comatox operatsvn ^noianv, 20t3) meisa stseamstecetasd Wrom vorious devioat ianthameha,wenalmatat, video camesasanO 4 ideo streams that are used for retranslation from other media servers, etc.), through the local computer network of the New Stage of Alexandrinsky Theatre are transmitted for processing to the media server, located near the core of the computer network of the New Stage. Due to the open architecture of the selected software for the media server (Media Server Wowza v.3.4), plug-in written in Java was developed and connected to the media server, which allowed to develop the system of media data sources registration through the Web. Using it, the media server administrator
еанВоггалгааав^к,скуЛВками ДИМА. кваа „
ВАНЯ Г ДЭ6Ылатиз™й'
I "НЯ- А, склизкий уже . " ментовку меня едаш^1 А "" "чхиий. да? мокет, еще , ДИМА, и вдам' я ил? Е [ ВАНЯ Т^Г ' 603,3 " ■
! малыш? ™вЛМ- 4111 » <™ «валил? А
ДИМА • кто? ужа от«™ п^од^. ИЛИЯ. Ну, ты. Димочка, ты! Включи голову
НАТАЦщ ТАМЯ "»А
ËÏÏÏÏL™"1"™™
"»««то,»,,,
КОТТП
А-ШГДАСОИА^ЛЬЧО- ЭП тлсос Этоспчилл^СТВО
лея >£П
ЧАТЛ^ МЫ KEtUjtoiCtrtWM
ooísinwt
«Я: мм и ауало Фкж* ЧЛТЛ_> Г "JOrTT^-l -и*
Fig. 3. The fragment of NET DRAMA performance
has the ability to register any of the media data sources, to verify its condition and make it available at the media server, as well as to create sets of the desired quality video streams from any registered media data sources and promptly manage their translation, recording andconfiguration.
Informationof the outflrw strecms registratkcnsystem istrt^i^smhgrCrth^ trancrtions th^ct^^aidwe asl th
video and aaciio sSreoms. foemlna a nm(h of iOe projectsCsoadcasting and decides which audio and video streams will be available to directors who work with specific projects.
The direction of the software and hardware compCehenusehs poOof thcmeWhomanceetself ran ha aeherret to tOesD-oalleW non-httoiol objecgvfOjn-he ujeofwodemtechnojogiesih -heatrW^ t^r°Ouc^^^wns.
One of sucf prwlectr nt the opening oO Chc New Othf eol Alexandrinsliy Tlinateo watthe theater nf texl-tOe "NET DRAMA" prnjecj (^m.3tB0eiS^^^, h013).
The conception of this project is that the process of writing a script for a new play becomes the subject ofa theatrical performance. Fouryoung
playwrights, right in front of the spectators' eyes, simultaneously wrote the script of the modern interpretation of the work by Fyodor Mikhailovich Dostoyevsky "The Brothers Karamazov".
Through the use of two projectors the audience watched three components of the performance:
f Immadiotdy, hhe psogramme -tself, forwriting the script simultaneously by four participants in rrnUimew^WghHghl coktars;
C. What, ii^a^chf^lt^m^eh to cnmmenton the written and give comments in response to the cries of the spectators from the audience, whichcontributedto the creation of a good communication environment between the aEaienrenтd She pCamwrights;
S.A finual oni^gnn^^ n to broadcast the nlarwrights'faegs in the processof their week.
TOws, an applioettonwith an Intuitive iaierface, visually repretemting tUs wltaie procf si oh r^it^c^Iiwc work was developed
An a result oh iwhlementinr o number of projects on implanting information and multimedia technologies, the team of ITMO University, in collaboration with the colleagues from theatres have formed a number of principles
- 1664 -
of the modern Internet-theater (Borisov, Internet and Multimedia..., 2015):
1. Various creative projects (shows, performances, video art) that use information-communication- multimedia and other technologies;
2. Multi-stream online broadcasting of shows with various interactive elements (voting, chat, screen management, etc.);
3. Interactive performances with the direct participation of Internet users;
4. An archive of previous stage productions with the option of watching them online.
Multimedia 360° Technologies
One of the promising directions, which started being actively used in culture and art, isChciechnoiocyof nideoteoordinh in the 36C° format.
Em)Ooyees oftheCeiOer ofDesign aod MulümediaoflTMOUclversity have dcheleped a spociallzed coiera tcrccosd 360° video(htmei/ video360prdduction.comC).Conten3hsefSsdwiihCie help oh this eqmcmoiUcanOe viewedwithavsriety of virtual reality headsets (Oculus Rift, Samsung
Gear VR, Homido, etc.), as well as on various mobile devices and from the monitor screen.
With the help of the created equipment recordings of various concerts, musical, cultural and special events were carried out (Mariinsky and Alexandrinsky theaters, Peter the Great's Museum of Anthropology and Ethnography "Kunstkamera", band "Picnic", Jazz Philarmonic Orchestra and band "Leningrad"), recorded a number of excursions in cooperation with Solovetsky State Historical-Architectural and Natural Museum-Reserve". Figure 4 shows the recording of excursions on the rivers and canals of St. Petersburg in 360° Video format (Borisov, Ispol'zovanie tekhnologii...., 2015).
In addition to the passive viewing of the created content, there is a possibility to create interactive 360° video, with an option to continue viewing using various devices (a mouse, joysticks and moteo ncepturedevices)or ac tive virtuaieagr alaced diteceiy inage anocaofvideomaterial. Becidrm, inirodaciion of aoffertni conieut of "aiciuro in pictnae functioa"into36B°vidro,ao well as the transition to the simulated 3D space is possible as well.
Fig. 4. A fragment of the tour on the rivers and canals of Saint Petersburg in 360° video format
- 1665 -
Within continuing work on the grant of the Russian Humanitarian Foundation "Multimedia Information System "Ancient Fortress of the North-West of Russia", the Center of Design and Multimedia employees have created a prototype of interactive tour of Korela Fortress Museum (Borisov, Multimedia 360° - New Technology ..., 2015).
Resume
The information society is inconceivable without the use of computer technologies that find reflection in all kinds of activities. And
creative activity is not an exception. Information and multimedia technologies introduced into any creative product are designed not to compete and not to resist the classical works, but to create a decent alternative, as they bring new experimental approaches and new ways of perception.
This work was partially financially supported by Government of the Russian Federation, Grant 074-U01.
This work was partially financially supported by Saint-Petersburg State University development program.
References
Nikita, S. Nauka kak illiusiia nauki [Science as an Illusion of Science]. Available at: http://art1.ru/ teatr/nauka-illyuzii-kak-illyuziya-nauki/
TTs "Dostoevskii" Forum-otkrytie Novoi stsen y Aleksandrinskogo teatra [TC "Dostoevsky" Forum-Opening of the New Stage of Alexandrinsky Theatre]. Available at: http://test.thenewstage.ru/ ru/project/21/tc-dostoevskiy
Borisov, N., Smolin, A., Stolyarov, D., Shcherbakov, P. [2014]. Interactive Internet Theatre (Interactive Multimedia Solutions at the New Aleksandrinsky Theatre Stage), In Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. 145, 100-
Borisov, N., Smolin, A., Stolyarov, D., Shcherbakov, P. [2013]. Interactive multimedia solutions Developed for the Opening of the New Stage of the Alexandrinsky Theatre, In Proceedings of the Third International Conference Digital Presentation and Preservation of Cultural and Scientific Heritage -DIPP2013. Veliko Tarnovo, Bulgaria, September 18-21. (III), 153-159.
Borisov, N., Smolin, A., Stolyarov, D., Tscherbakov, P. [2015]. Internet and Multimedia Technologies for the Theater Performances, In Eva 2015 Saint Petersburg. Electronic Imaging & the Visual Arts International Conference, St. Petersburg, June 24-25, 2015 Conference Proceedings. SPb ITMO University, 163-169.
Project Video 360 production. Available at: http://video360production.com/ Borisov, N.V., Smolin, A.A., Stolyarov, D.A. [2015]. Ispol'zovanie tekhnologii multimedia 360° v kul'ture i iskusstve [Use of Multimedia 360° Technologies in Culture and Art], In Informatsionnye sistemy dlia nauchnykh issledovanii: Sbornik nauchnykh statei. Trudy XVIII ob 'edinennoi konferentsii "Internet i sovremennoe obshchestvo" [Information Systems for Scientific Researches: Collection of Scientific Papers. Proceedings of the 18th Joint Conference "Internet and Modern Society"]. Saint Petersburg, 23 - 25 June 2015. SPb., 184-196.
Borisov, N., Smolin, A., Stolyarov, D. [2015]. Multimedia 360° - New Technology for Arts and Culture, In Eva 2015 Saint Petersburg. Electronic Imaging & the Visual Arts International Conference, St. Petersburg, June 24-25, 2015 Conference Proceedings. SPb ITMO University, 18-31.
- 1666 -
Информационные и коммуникационные технологии (ИКТ) и мультимедийные технологии в искусстве и культуре
Н.В. Борисов, А.А. Смолин, Д.А. Столяров, П.П. Щербаков, В.А. Трушин
Университет ИТМО Россия, 197101, Санкт-Петербург, Кронверкский пр., 49 Санкт-Петербургский государственный университет Россия, 199034, Санкт-Петербург, Университетская набережная, 7/9
В данной статье на примере проектов Центра дизайна и мультимедиа Университета ИТМО продемонстрированы возможности использования различного оборудования, а также программного обеспечения в различных областях культуры и искусства. Информационные и мультимедиа технологии позволили всесторонне обогатить различные креативные и культурные индустрии и искусства, привнесли в проекты больше возможностей для создания особой творческой атмосферы и более глубокого и эмоционального восприятия творческого контента.
Ключевые слова: мультимедиа, культура, интернет-театр, мультимедиа 360 °, виртуальная реальность.
Научная специальность: 24.00.00 - культурология
