CC BY
23Cellular Therapy and Transplantation (CTT). Vol. 9, No. 2, 2020 doi: 10.18620/ctt-1866-8836-2020-9-2-78-82 Submitted: 19 May 2020, accepted: 26 June 2020
Augmented reality technology for auricular reconstruction in the treatment of microtia
Аndrey I. Yaremenko ', Anna V. Lysenko ', Elizaveta A. Ivanova ', Oleg V. Galibin 2
1 Department of Maxillofacial Surgery, Pavlov University, St. Petersburg, Russia
2 RM Gorbacheva Research Institute of Pediatric Oncology, Hematology and Transplantation, Pavlov University, St. Petersburg, Russia
Dr. Elizaveta A. Ivanova, Department of Maxillofacial Phone: +7 (953) 144 1508
Surgery, Pavlov University, L. Tolstoy St. 6-8, 197022, E-mail: [email protected]
St. Petersburg, Russia
Citation: Yaremenko АI, Lysenko AV, Ivanova EA et al. Augmented reality technology for auricular reconstruction in the treatment of microtia. Cell Ther Transplant 2020; 9(2): 78-82.
Summary
Facial defects and deformations occupy a significant place in the practice of maxillofacial and dental surgeons. Nevertheless, maxillofacial surgery is developing rapidly and requires improvement of existing treatment methods, and introduction of new approaches to reconstructive surgery. Augmented reality is a promising direction of computer technology development which is actively used in medicine and education. Modern computer technology allows to create a 3D model of a lost organ and use it for preoperative planning, as well as apply a virtual model for intraoperative navigation. Recently, the method of augmented reality has been actively developed, when a virtual image of the zone of the surgical area or a dedicated organ is used, which is compared with its real prototype in static mode, or in real-time using computer devices. The benefits of using augmented reality technologies in reconstructive surgery is associated with preoperative virtual planning, simplification of the surgical intervention itself, as well as with a reduction in the risks of intra- and postoperative complications. The aim of our work was to study the opportunity of using the augmented reality technology in reconstructive surgery for microtia correction based on pre-operative computer simulation.
At the preoperative stage, a photometric analysis of the patient was carried out, then a computer simulation of the missing auricle was performed. Using a 3D printer, a virtual model of the reconstructed auricle was obtained. The image in three-dimensional format was loaded into augmented reality glasses, which made it possible to project the shape and position of the simulated auricle to the area of the defect of the auricle when preparing
for surgery. During the surgery, a marker was installed near the surgical field, in order to display the three-dimensional model in a destined position. During surgical intervention, an autogenous costal cartilage was taken, from which the auricle was formed using augmented reality approach and three-dimensional modeling. Subsequently, the graft was introduced to the formed bed in the area of the right ear auricle.
The obtained 3D model of the auricle before the operation enabled planning of the forthcoming operation and determine the amount of autograft needed for reconstruction. Using the augmented reality glasses, the exact shape of the auricle is reproduced during the operation, and its proper position is assessed in relation to the healthy side. No complications were observed over the postoperative period.
Virtual modelling of a lost or absent organ based on a preoperative examination provides important information about its spatial structure. Preoperative virtual planning allows you to predict the individual features of the operation, its difficult stages, to anticipate possible complications. The use of augmented reality technology during reconstructive surgery is a promising method requiring further development and improvement.
Keywords
Ear, microtia, cartilage, transplantation, augmented reality, 3D printing.
Introduction
Microtia is a congenital malformation of the outer ear, which is most often inherited [1]. Microtia is characterized by unilateral (79-93% of cases, 60% of which are associated with the right ear) or bilateral anomaly of the development of auricles or their reduction in size and which is often associated with atresia or stenosis of the external auditory canal [2]. The size of auricles varies from degree I, when the auricle is only slightly smaller than usual, to degree IV, also known as ano-tia [3]. Disproportional shape or location of the auricle in relation to other parts of the face alters aesthetic appearance of a person and perception by other people, thus negatively affecting psycho emotional condition of the patient [4]. High-quality surgical correction of the auricle is a difficult task for the surgeon [5]. One should not only reconstruct a new organ, but also to do this in accordance with the shape, size and position of the contralateral auricle [6, 7]. The complex individual structure of the auricle sufficiently complicates the work of surgeon [8]. At the moment, there is no single approach to preoperative planning for the auricle reconstruction. The final aesthetic result is largely determined by imagination and practical experience of the surgeon [5]. In addition, the surgical intervention itself is quite traumatic.
In most cases, the costal cartilage tissue of the patient is used to create the auricular framework [9]. The graft extraction technique is often more traumatic than the reconstruction of the auricle and may be complicated, e.g. by pneumothorax [10]. Competent preoperative planning is of high importance for successful reconstructive surgery [8]. Modern computer technology is able to provide ideal shape proportions and location of the auricle, as well as to reduce the injury at the first stage of the operation, i.e., extraction of costal autograft [7, 8]. Usage of augmented reality technology is of great interest in this aspect [7].
Augmented reality (AR) is a kind of virtual reality [11, 12, 13]. This technology embeds a virtual object in the environment, integrates and complements the real world, but does not replace it [12, 14]. Ronald Azuma in 1997 defined augmented reality as a system that, being in three-dimensional space, combines real and virtual objects and allows a person to interact with them [15]. Therefore, this technology could present an accessory tool for reconstructive surgery [14, 16].
The aim of our study was to determine the opportunity of using AR-technology in surgical interventions for reconstruction of the auricle with a cartilage autograft.
Materials and methods
A 20-year-old patient was admitted to the department of maxillofacial surgery of the First Pavlov Medical University with a diagnosis: congenital atresia of the right ear auricle. At the preoperative planning stage design, a series of photographs of the patient's face and auricles were made (Fig. 1).
Photogrammetry of the face in profile was performed using the "PhotoScan" program (Agisoft). 3D model of the right ear auricle was reproduced of the ideal shape, size and location, in accordance with the auricle at the opposite side (Fig. 2).
Figure 1. Photo of the patient at the surgery planning stage. Marking is required to determine the face proportions and architectonics as a 3D model of the future auricle
The grid of the 3D model was subjected to retopology using software "3D Max" to optimize the 3D model and further 3D printing.
Figure 2. Computer model of the patient's auricle
Based on the obtained data, a three-dimensional model of the right ear auricle was produced of polylactide (PLA) by means of 3D printer "Picaso Designer" (Fig. 3).
Figure 3. 3-D model of the patient's "ideal" right auricle
The cartilages of right ear auricle were simulated, being based on the PLA model (Fig. 4). The necessary sizes of costal cartilage autograft were determined. Before starting the surgery, an augmented reality marker was placed in the close proximity to the surgical area (Fig. 5).
Figure 4. Volumetric model shows the components of the cartilage of the auricle
Figure 5. Virtual image of the right ear auricle obtained by analyzing the marker and displayed in AR glasses (Epson Moverio BT-300 model)
This marker represents an object made of laminated paper and located in the surrounding space, which is captured and analyzed by special software for subsequent visualization of a virtual object relative to the marker. Based on information about the spatial position of the marker, the program projects a virtual object into a real environment. The virtual auricle suggests the ideal position of the intended organ and allows you to perform a precise surgical access, thus reducing potential injury to surrounding tissues.
After excising of the autograft from the costal cartilage, reconstruction of the auricular cartilage was performed according to the previously created virtual model (Fig. 6).
Figure 6. Cartilaginous frame of the auricle (left) and its comparison with the virtual model (right)
The simulation included the margin of autograft volume, since there are always risks of transplant resorption. A three-dimensional cartilaginous frame was placed subcuta-neously in a preformed bed and fixed in accordance to the virtual image. Maintenance of the costal graft volume was noted during early and later postoperative terms (up to 6 months) (Fig. 7A, Fig. 7B). There were no other complications over the postoperative period.
Figure 7. A - 7 days after surgery, B - 6 months after surgery
Discussion
Recently, 3D modeling and 3D printing technologies are widely used in aesthetic and reconstructive maxillofacial surgery [16, 17]. AR is actively used in plastic surgery during facial contouring for congenital and post-traumatic defects and deformations of the facial skeleton, for example, underdevelopment of the body of the lower jaw, deformation of the zygomatic bone [18].
At the preoperative planning stage, computer modeling made it possible to create a three-dimensional model of the reconstructed organ. This stage enabled to achieve ideal anatomic location of the reconstructed organ, in accordance with anatomical pattern of the patient's face. Based on these
data, it was possible to create a virtual model of the reconstructed organ and calculate the required volume of auto-graft, thus allowing to extract a transplant of optimal size and minimize surgical trauma.
At the stage of surgical intervention, the virtual model provided exact reference points for the of the cartilage graft positioning, thus making it possible to limit the area of soft tissue detachment when forming the graft bed and reducing the time of the operation.
Conclusion
The formation of a virtual organ model based on preoper-ative planning provides important information about its placement, anatomy and spatial structure.
Preoperative spatial planning allows to predict the course of surgery and minimize the amount of surgical intervention.
A combination of computer modelling, real-organ imaging, and an operation area using the AR method during an auricle reconstruction operation is a promising method that requires further development and improvement.
Acknowledgments
We are grateful to Rustam Mirzakhmedov for programming support and hardware.
Conflicts of interest
None declared.
References
1. Romo TIII, Fozo MS, Sclafani AP. Microtia reconstruction using a porous polyethylene framework. Facial Plastic Surgery; 2000; 16(1): 15-22.
2. Harris J, Kallen B, Robert E. The epidemiology of anotia and microtia. J Med Genet. 1996; 33(10): 809-813.
3. Brent B. The correction of microtia with autogenous cartilage grafts: II. Atypical and complex deformities. Plastic Reconstruct Surgery. 1980; 66(1): 13-21.
4. Casas CQ. Image-guided surgery with surface reconstruction and augmented reality visualization: U.S. Patent No. 10,154,239. 2018.
5. Zim SA. Microtia reconstruction: an update. Curr Opin Otolaryngol & Head Neck Surg. 2003; 11(4): 275-281.
6. Nagata SA. New method of total reconstruction of the auricle for microtia. Plastic Reconstruct Surg. 1993; 92(2): 187-201.
7. Nuri T, Mitsuno D, Otsuki Y, Ueda K. Augmented reality technology for the positioning of the auricle in the treatment of microtia. Plastic Reconstruct Surg Global Open. 2020: 8(2).
8. Sabbagh W. Early experience in microtia reconstruction: the first 100 cases. J Plastic, Reconstruct & Aesth Surgery. 2011; 64(4): 452-458.
9. Thomson HG, Kim TY, Ein SH. Residual problems in chest donor sites after microtia reconstruction: a long-term study. Plastic Reconstruct Surg. 1995; 95(6): 961-968.
10. McCarn KE, Weber SM. Ultrasonography for rapid detection of pneumothorax after costal cartilage harvest. Arch Facial Plast Surg. 2011; 13(1): 57-59.
11. Cao C, Cerfolio RJ. Virtual or augmented reality to enhance surgical education and surgical planning. Thorac Surg Clinics. 2019; 29(3): 329-337.
12. Huang TK, Yang CH, Hsieh YH, Wang JC, Hung CC. Augmented reality (AR) and virtual reality (VR) applied in dentistry. Kaohsiung J Med Sci. 2018; 34(4): 243-248.
13. Wang J, Suenaga H, Hoshi K, Yang L, Kobayashi E, Saku-ma I, Liao H. Augmented reality navigation with automatic marker-free image registration using 3-D image overlay for dental surgery. IEEE Transact Biomed Engeneering. 2014;61(4): 1295-1304.
14. Wang J, Suenaga H, Liao H, Hoshi K, Yang L, Kobayashi E, Sakuma I. Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation. Comp Med Imaging and Graphics. 2015;40: 147159.
15. Azuma RT. A survey of augmented reality. Presence. 1997; 6(4):355-385.
16. Glover JC. 3D bioprinting applications for in vitro modeling of cellular interactions and tissues. Cell Ther Transplant. 2016; 5(2): 8-11.
17. Kim Y, Kim H, Kim YO. Virtual reality and augmented reality in plastic surgery: a review. Arch Plastic Surg. 2017; 44(3):179.
18. Muraev AA, Dymnikov AB, Korotkova NL, Kobets KK, Ivanov SY. Planning Technique in Maxillofacial Plasty. Klin-icheskaya Medizina. 2013; 5(3): 57-62 (In Russian).
Технология дополненной реальности для реконструкции ушной раковины при лечении микротии
Андрей И. Яременко ', Анна В. Лысенко ', Елизавета А. Иванова ', Олег В. Галибин 2
1 Кафедра челюстно-лицевой хирургии, Первый Санкт-Петербургский государственный медицинский университет им. И. П. Павлова, Санкт-Петербург, Россия
2 НИИ детской онкологии, гематологии и трансплантологии им. Р. М. Горбачевой, Первый Санкт-Петербургский государственный медицинский университет им. акад. И. П. Павлова, Санкт-Петербург, Россия
Резюме
Дефекты и деформации занимают значительное место в практике челюстно-лицевых хирургов и стоматологов. Тем не менее, челюстно-лицевая хирургия развивается быстрыми темпами и требует совершенствования существующих методов лечения и введения новых подходов в ходе реконструктивных операций. Одним из перспективных направлений развития компьютерных технологий является дополненная реальность, которая активно используется в медицине. Современные компьютерные технологии позволяют создать трехмерную модель утраченных органов и использовать ее для предоперационного планирования, а также применить виртуальную модель для интраоперационной стратегии. В последнее время активно используется метод дополненной реальности, когда используется виртуальное изображение зоны хирургической области или выделенного органа, которое сравнивается с его реальным прототипом в статическом режиме или в режиме реального времени с использованием компьютерных устройств. Актуальность использования технологий дополненной реальности в реконструктивной хирургии связана с предоперационным виртуальным планированием, упрощением самого хирургического вмешательства, а также с уменьшением риска интра- и послеоперационных осложнений. Целью нашей статьи является изучение возможности использования технологии дополненной реальности в процессе реконструктивной хирургии микротии на основе предоперационного компьютерного моделирования.
На предоперационном этапе был проведен фотометрический анализ пациента, затем выполнено компьютерное моделирование отсутствующей ушной раковины. Используя ЭБ-принтер, была получена модель реконструированной ушной раковины. Изображение в трехмерном формате было загружено в память дополненной реальности, что позволило спроецировать форму и положение имитируемой ушной раковины на область дефекта ушной раковины пациента во время подготовки к операции.
Во время операции возле операционного поля был установлен маркер для отображения трехмерной модели в заданном положении. Во время операции взят аутогенный реберный хрящ, из которого ушная раковина была реконструирована с использованием дополненной реальности и трехмерного моделирования. После этого, полученный трансплантат был введен в сформированное ложе в области правой ушной раковины.
Полученная ЭБ-модель ушной раковины перед операцией позволила спланировать предстоящую операцию и определить объем аутотрансплантата, необходимое для реконструкции. С помощью результатов дополненной реальности точная форма ушной раковины воссоздается во время операции, и устанавливается необходимая симметрия по отношению к здоровой стороне. В послеоперационном периоде осложнений не наблюдалось.
Виртуальное моделирование отсутствующего органа на основе предоперационного обследования дает важную информацию о его пространственной структуре. Предоперационное виртуальное планирование позволяет прогнозировать индивидуальные особенности операции, ее сложные этапы, предвидеть возможные осложнения. Использование технологии дополненной реальности во время реконструктивной хирургии является перспективным методом, который требует дальнейшего развития и совершенствования.
Ключевые слова
Ухо, микротия, хрящ, трансплантация, дополненная реальность, ЭБ-печать.
