CC BY
38
E.V. KOVALEV et al., 2021
smartphone-assisted augmented reality technology for preoperative planning
in spine surgery
E.V. Kovalev1, S.I. Kirilenko1, A.N. Mazurenko2, A.E. Filiustin3, V.V. Dubrovsky4
1Gomel Regional Clinical Hospital, Gomel, Republic of Belarus 2Republican Scientific and Practical Center of Traumatology and Orthopedics, Minsk, Republic of Belarus 3Republican Scientific and Practical Centre for Radiation Medicine and Human Ecology, Gomel, Republic of Belarus 4Institute of Mechanics of Metal-Polymer Systems n.a. V. A. Beliy, Gomel, Republic of Belarus
Objective. To present a virtual three-dimensional model of pathologically altered segments of the patient's spine and to analyze the results of its application when planning a surgical intervention in the smartphone-assisted augmented reality.
Material and Methods. A three-dimensional modeling of the target area of the intended surgical site was performed based on computed tomography data of five patients with various spinal deformities. A smartphone application has been developed that allows displaying a three-dimensional object of the intended surgical site in the form of augmented reality.
Results. The created virtual three-dimensional models were successfully used in five cases for preoperative planning and simulation training before surgery, which allowed to see in detail the anatomical features of the spine, the location of vascular structures when contrasting them, and to plan the direction of the screws. The potential of using augmented reality in clinical practice was demonstrated. Conclusion. The advantages of the smartphone-assisted augmented reality technology for preoperative planning in spine surgery are the simplicity of creating a computer model, the possibility for a surgeon to use a three-dimensional model for orientation in complex anatomical zone at any time of surgery, and the reduction in the risk of technical errors. Key Words: three-dimensional modeling, spine surgery, augmented reality.
Please cite this paper as: Kovalev EV, Kirilenko SI, Mazurenko AN, Filiustin AE, Dubrovsky VV. Smartphone-assisted augmented reality technology for preoperative planning in spine surgery. Hir. Pozvonoc. 2021;18(3):94—99. In Russian. DOI: http://dx.doi.org/10.14531/ss202L3.94-99.
Today, the use of innovative technologies is the main guarantee of the development and competitiveness of any industry. The digital industry is one of the rapid-growing segments of modern computer engineering, which is increasingly used in medicine.
Augmented Reality (AR) is a technique in which computer graphics are superimposed on a video or drawing of the real world. Both objects coexist integrally in the final image [1]. It was Caudell who first proposed the term "augmented reality" in 1992. This technology was applied in the aircraft assembly. The assembly workers could see the data via helmets with semi-transparent display panels [2]. The end of the 90s and the beginning of the 2000s - the age of prototypes of augmented reality mobile devices. One of the pioneers of the numerical tagging of objects using augmented reality is Rekimoto, a professor in the field of information research. The device was a
display with a camera mounted on the back side, from which the image was transmitted to a computer. If a tag was found, data regarding the object was displayed on the screen. The augmented reality became widespread in the consumer market in the 2010s. In 2014, Google Corporation began selling the Google Glass headset. In 2016, Microsoft introduced HoloLens - a new device for augmented reality.
An annual study of the digital acumen index is performed with the determination of the amount of investment in digital technologies. According to statistics, 10 % of investments in the world are put into augmented reality; in Russia this value is 15 %. Meanwhile, it is anticipated that in three years this indicator will increase to 20 % [3]. The development and application of the augmented reality in medicine regards modern achievements in the field of 3D modeling and 3D prototyping. The technique is based
on the use of modern digital information and communication technologies. It may improve the quality of preoperative planning and intraoperative navigation.
Two main software components are needed to implement the augmented reality technique: tracking and visualization. Tracking is understood as monitoring the position of the device's camera relative to the coordinate system. The virtual camera in three-dimensional space is synchronized with the physical camera. Thus, the correct display of the three-dimensional model in the surrounding environment is achieved [4, 5]. A marker technology is more often used to recognize objects. The following things may act as markers: logos, QR codes, generated dots, and physical objects.
To date, there are no papers describing the extensive experience of augmented reality application in spine surgery using a smartphone. These circumstances prove the relevance of this article.
94
The objective is to present a virtual three-dimensional model of pathologically altered segments of the patient's spine and to analyze the results of its application when planning a surgical intervention in the smartphone-assisted augmented reality.
Material and Methods
According to CT scans of five patients with various spinal deformities (degenerative multiplanar deformity, n = 2; cra-niovertebral deformities, n = 3), virtual three-dimensional models were created. A method of forming an augmented reality system that enables surgeons to see a virtual three-dimensional model against the background of reality is suggested. The software applied is freely available on the developers' websites. The augmented reality application can run on smartphones with iOS and Android operating systems, as well as in HoloLens augmented reality headset [6, 7]. The study developed an application for Android 4.1 version "Jelly Bean", compatible with the Android system up to version 9.0 "Pie".
The process of creating a three-dimensional model of spinal structures as a new technique for preoperative planning includes a number of stages (Fig. 1). Prior to the surgery, a CT scan of the intended surgical site is performed on a LightSpeed 16 Pro (General Electric) device, the slice thickness is 1.25 mm (Fig. 1.1). A tomograph workspot generates images of the patient's individual anatomy in Dicom format as a result of the study. The images are transmitted to a computer, where a virtual three-dimensional model is created in special software, which is then exported to a file with the STL extension [8, 9]. The graphic editor processes the three-dimensional model and saves it in OBJ format (Fig. 1.2) [10]. After that, the software of the augmented reality platform Vuforia Software Developer Kit (PTC, Inc., Boston, Massachusetts, USA) for mobile devices registers the name of the project and downloads a graphic tool (QR code), the so-called marker. Using the latter, a virtual three-dimensional object is cre-
ated (Fig. 1.3) [11]. After combining the objects, this data in the application format is transmitted to the source software running on the smartphone (Fig. 1.4, 1.5) [12, 13].
The Augmented Reality app is compatible with a wide range of devices running Android and iOS operating systems. When launching the installed application and pointing the smartphone camera at the previously selected graphic image (marker), after reading and processing data by the software, elements of augmented reality are formed as a three-dimensional model and displayed on the smartphone screen. The application enables to view the virtual model from all sides, including internal structures.
Results
This technology has been tested clinically. Virtual three-dimensional models of the surgical site have been created in patients with craniovertebral deformity (n = 3). This made it possible to see the anatomical features, for example, the course of the vessels when they are contrasted. In patients with degenerative multiplanar spinal deformity (n = 2), in addition to visualization of anatomical structures, guide cylinders simulating the course of the proposed screws for transpedicular fixation (TPF) were additionally modeled. It was done at the stage of graphic processing of the three-dimensional model. The construction of virtual guide cylinders gives additional data on the trajectory of the screw to perform a stabilizing operation. This technology made it possible to view a virtual 3D object at 360°, as well as internal structures when zooming in on a smartphone camera. An additional visual information and simulation training help to improve the skill level of the surgeon at the preoperative stage.
We introduce one of the examples of the practical application of smartphone-assisted augmented reality technology.
Patient R., 62 y.o.: a closed ununited fracture of the odontoid process of the C2 vertebra with displacement and spinal stenosis, cervical myelopathy, and a moderate tetraparesis (Fig. 2).
Considering the difficulty of surgery performed in the craniovertebral area, a virtual three-dimensional model in the form of augmented reality was applied as a technique of preoperative planning (Fig. 3). It took 30 minutes from the moment the CT scan was performed to the creation of the smartphone application.
A posterior decompression of the spinal canal was performed at the level of the occipital bone and the posterior arch of the C1 vertebra with the fixation of the craniovertebral area with a hook-and-screw instrumentation system for occipitospondylodesis. A CT scan of the patient was done after the surgery (Fig. 4). The surgery duration was 3.5 hours.
After the surgery, the patient underwent early rehabilitation in the neuro-surgical unit for 7 days, then he was sent to the rehabilitation unit for specialized treatment.
Discussion
The development of the surgery stages on a three-dimensional model was successfully applied in 5 cases during preoperative planning and simulation training before surgery. It allowed to specify the anatomical features of the spine, the location of vascular structures when contrasting them and to plan the direction of the screws. These activities provided an opportunity to predict and reduce the risks of iatrogenic damage to vascular and nerve formations.
Digital technologies form a new type of education and increase the level of already existing; they enhance the level of accessibility and quality of medicine. Their active development has been observed in recent years. The features of modern processors and a full-fledged smartphone operating system have enabled augmented reality to go beyond the entertainment fields and cover a new level of human activity, especially medicine [14, 15].
The augmented reality application in surgery using a smartphone has become progressively relevant in recent years. In 2020, a group of authors in the field of
95
Fig. 1
The scheme of the AR system (explanation in the text): 1 - CT data, MRI data; 2 -personal computer; 3 - graphic tool (QR code); 4 - smartphone camera; 5 - augmented reality operation scheme
vascular surgery developed and applied a new hybrid augmented reality optical and gyroscopic tracking system, which enhanced image stability. The work was performed using a smartphone [16]
Projection of virtual objects onto the physical body is broadly used in preop-erative planning and in neurosurgery. Meanwhile, the authors use projectors
and tablets to create augmented reality [15, 17]. The studies indicate accurate and ergonomic intraoperative navigation.
Using the example of 20 patients, the work from the field of maxillofacial surgery with the projection of a pathological virtual object through a fixed marker on the lower jaw is given. The positional error ranged from 0.52-2.00 mm
(mean value 0.96 mm; standard deviation 0.51 mm). The authors regard these error values as insignificant and conclude that the augmented reality system is deemed accurate [18].
The challenge of modern monitoring and tracking systems is the limitation of tracking markers on the movable or deformable anatomy of the patient. The static anatomy (for example, bone) gives a high degree of accurate positioning of a virtual object in orthopedic and neuro-surgical augmented reality systems. The dynamic structures (for example, intraabdominal, thoracic organs) that move and deform during respiration and peristalsis are inappropriate for the application of considered techniques.
The benefit of the presented augmented reality system, in comparison with conventional stationary navigation systems, is the direct and improved visualization of the static areas of interest. It should be added that the system is inexpensive and easy to reproduce. Nevertheless, the computer competence may limit its wide distribution. Currently, there are no publications describing the extensive experience of smartphone-assisted augmented reality in spinal surgery. These circumstances prove the relevance of this paper. The article represents a pilot application of the smartphone-assisted augmented reality in the Republic of Belarus in preoperative planning.
Conclusions
The developing computer and software technologies, including augmented reality platforms, are being actively introduced into practical medicine. Their features are the advantages of virtual planning or simulation of the operation plan, in increasing the skills of the surgeon from the position of three-dimensional planning at the preoperative stage, as well as reduction of the surgeon's training time.
The advantages of the smartphone-assisted augmented reality technology in preoperative planning in spinal surgery are the simplicity of creating a computer model, the ability to use a three-dimensional model to navigate in a complex
Fig. 2
CT scan of patient R., 62 y.o.: a - three-dimensional modeling before surgery; b - sagittal projection before surgery; a displaced type II C2 odontoid fracture and spinal stenosis are defined
96
general issue
Fig. 3
The acquisition stages of a virtual three-dimensional model of a spinal fragment: a -a model based on CT scan; b - a model prepared to work in an augmented reality program; c - connecting a three-dimensional model with a QR code and writing an application for a smartphone; d - application operation and displaying the spine model on the phone screen
anatomical area in any moment of surgery, as well as reducing the risk of technical errors.
The study had no sponsors. The authors declare that they have no conflict of interest.
97
Fig. 4
Postoperative CT scan of patient R., 62 y.o: a - CT examination in sagittal projection; b, c - CT examination in axial projection; the correct location of screws in the roots of C2 and C3 vertebrae is defined on both sides; d - intraoperative image with installed craniovertebral structure and fixed autograft from the iliac crest
References
1. Lovo EE, Quintana JC, Puebla MC, Torrealba G, Santos JL, Lira IH, Tagle P.
A novel. inexpensive method of image coregistration for applications in image-guided surgery using augmented reality. Neurosurgery. 2007;60(4 Suppl 2):366-372. DOI: 10.1227/01.NEU.0000255360.32689.FA.
2. Caudell TP, Mizell DW. Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of the Twenty Fifth Hawaii International Conference on System Sciences, Kauai, HI, USA, Feb 7-10, 1992: Vol. 2. P. 659-669. DOI: 10.1109/HICSS.1992.183317.
3. 2017 Global Digital IQ® Survey. A decade of digital. Keeping pace with transformation [Electronic resource]. https://www.pwc.ru/ru/publications/globaldigital-iq-survey-rus. pdf. Date of access: 10.11.2020.
4. Leger E, Reyes J, Drouin S, Popa T, Hall JA, Collins DL, Kersten-Oer-tel M. MARIN: an open-source mobile augmented reality interactive neuronavigation system. Int J Comput Assist Radiol Surg. 2020;15:1013-1021. DOI: 10.1007/ s11548-020-02155-6.
5. Kravtsov AA, Loiko VI. Improving three-dimensional object visualization user interface with augmented reality technology. Scientific Journal of KubSAU. 2014;(100):1408-1420. In Russian.
6. Deng W, Li F, Wang M, Song Z. Easy-to-use augmented reality neuronavigation using a wireless tablet PC. Stereotact Funct Neurosurg. 2014;92:17-24. DOI: 10.1159/000354816.
7. Hou Y, Ma L, Zhu R, Chen X, Zhang J. A low-cost iPhone-assisted augmented reality solution for the localization of intracranial lesions. PLoS ONE. 2016;11:e0159185. DOI: 10.1371/journal.pone.0159185.
8. 3D Slicer [Electronic resource]. URL: https://www.slicer.org. Date of access: 10.11.2020.
9. Ungi T, Lasso A, Fichtinger G. Open-source platforms for navigated image-guided interventions. Med Image Anal. 2016;33:181-186. DOI: 10.1016/j.media.2016.06.011.
10. Blender [Electronic resource]. URL: https://www.blender.org/download. Date of access: 10/11/2020.
11. Vuforia [Electronic resource]. URL: https://developer.vuforia.com. Date of access: 10.11.2020.
98
12. Perez-Pachon L, Poyade M, Lowe T, Groning F. Image overlay surgery based on augmented reality: a systematic review. In: Rea P.M. (ed). Biomedical Visualisation. Advances in Experimental Medicine and Biology, Vol 1320. Springer, Cham, 2020:175-195. DOI: 10.1007/978-3-030-47483-6_10.
13. Unity [Electronic resource]. URL: https://unity.com/ru. Date of access 10.11.2020.
14. Guha D, Alotaibi NM, Nguyen N, Gupta S, McFaul C, Yang VXD. Augmented reality in neurosurgery: a review of current concepts and emerging applications. Can J Neurol Sci. 2017;44:235-245. DOI: 10.1017/cjn.2016.443.
15. Watanabe E, Satoh M, Konno T, Hirai M, Yamaguchi T. The trans-visible navigator: a see-through neuronavigation system using augmented reality. World Neurosurg. 2016;87:399-405. DOI: 10.1016/j.wneu.2015.11.084.
16. Aly O. Assisting vascular surgery with smartphone augmented reality. Cureus. 2020;12:e8020. DOI: 10.7759/cureus.8020.
17. Besharati Tabrizi L, Mahvash M. Augmented reality-guided neurosurgery: accuracy and intraoperative application of an image projection technique, J Neurosurg. 2015;123:206-211. DOI: 10.3171/2014.9.JNS141001.
18. Zhu M, Liu F, Chai G, Pan JJ, Jiang T, Lin L, Xin Y, Zhang Y, Li Q. A novel augmented reality system for displaying inferior alveolar nerve bundles in maxillofacial surgery. Sci Rep. 2017;7:42365. DOI: 10.1038/srep42365.
Address correspondence to:
Kovalev Evgeny Vladimirovich Gomel Regional Clinical Hospital,
5 Bratjev Lizyukovykh str., Gomel, 246029, Republik of Belarus, kovalevsurgery@mail.ru
Received 17.03.2021 Review completed27.05.2021 Passed for printing 02.06.2021
Evgeny Vladimirovich Kovalev, neurosurgeon, Gomel Regional Clinical Hospital, 5 Bratjev Lizyukovykh str., Gomel, 246029, Republik of Belarus, ORCID: 0000-0002-2358-2897, kovalevsurgery@mail ru;
Sergey Ivanovich Kirilenko, neurosurgeon, Gomel Regional Clinical Hospital, 5 Bratjev Lizyukovykh str., Gomel, 246029, Republik of Belarus, ORCID: 0000-00032681-1533, ortoped74@bk.ru;
Andrey Nikolayevich Mazurenko, MD, PhD, neurosurgeon, assistant professor, doctoral student, Head of Laboratory for vertebrae and spinal cord injuries, neu-rotraumatologist, Republican Scientific and Practical Centre for Traumatology and Orthopedics, 60 (site 4) Leytenanta Kizhevatova str., Minsk, 220024, Republik of Belarus, ORCID: 0000-0002-4883-5598, mazurenko@mail.ru;
Andrey Evgenyevich Filiustin, radiologist, Head of the Department of Radiation Diagnostics, Republican Scientific and Practical Center for Radiation Medicine and Human Ecology, 290 Ilyicha str., Gomel, 246040, Republik of Belarus, ORCID: 0000-0002-5358-245X, filiustsin@mail.ru;
Vladislav Viktorovich Dubrovsky, researcher, Institute of Mechanics of Metal-Polymer Systems n.a. V.A Beliy, 32a Kirova str., Gomel, 246050, Republik of Belarus, ORCID: 0000-0002-6172-3371, vlad.mpri@gmail.com.
99
