CC BY
8
VGG Convolutional Neural Network Classification of Hyperspectral Images of Skin Neoplasms
Boris V. Grechkin, Vseslav O. Vinokurov, Yulia A. Khristoforova, and Irina A. Matveeva*
Samara National Research University, 34 Moskovskoe shosse, Samara 443086, Russia
*e-mail: [email protected]
Abstract. The article is devoted to the problem of early diagnosis of cancer. In last five years, various optical methods have been increasingly used to study biological tissues. This study aims to investigate the capability of a convolutional neural network classifier to diagnose skin cancers. The article analyzes hyperspectral images of malignant melanoma and pigmented nevus. A hyperspectral image classifier based on a deep learning neural network was developed. The results show a classification accuracy of diagnose prediction (on test data) at the level of 95%, which demonstrates the possibility of using machine learning for the classification of hyperspectral images of skin diseases. The results of the study can be applied in medical decision-making systems. © 2023 Journal of Biomedical Photonics & Engineering.
Keywords: hyperspectral imaging; malignant melanoma; classification; medical diagnostics; neural network; pigmented nevus; skin cancer.
Paper #8964 received 28 Apr 2023; revised manuscript received 23 Sep 2022; accepted for publication 24 Sep 2023; published online 11 Nov 2023. doi: 10.18287/JBPE23.09.040304.
1 Introduction
Skin cancer is one of the most common cancers in the world. According to the World Health Organization, 1,200,000 new cases of skin cancer have been identified in 2020 [1]. The vast majority of skin cancer deaths in recent decades have been caused by malignant melanoma (MM), despite the fact that it accounts for only one percent of all skin cancer cases [2]. MM is a dangerous, rare and deadly type of skin cancer. It is possible to increase the patient's chances of survival with skin cancer only through early diagnosis. However, the main difficulty in the qualitative detection of MM tumors is its external similarity with such a benign neoplasm as a pigmented nevus (PN).
As a gold standard, doctors usually use biopsy to diagnose skin cancer. This procedure takes a sample from a suspected skin lesion for medical examination whether it is cancerous or not. However, this procedure takes a lot of time and is also painful for the patient. Besides, the diagnosis of the skin can be held by dermatologists with using dermatoscope cameras. Difficulties in diagnosing skin cancers by general practitioners are associated with the difficulty of interpreting clinical signs and the inability to distinguish MM from benign neoplasms such as PN. In order to avoid unnecessary surgical procedures due to the uncertainty of current diagnoses and overcome
the limitations of conventional imaging, new methods to improve skin cancer diagnosis need to be explored.
In last decade, optical diagnostic methods are gaining popularity. One of these methods is hyperspectral imaging (HSI) [3, 4], which allows not only obtain optical images of the studied areas, but also spectral data on the scattering or absorption of radiation by this area. This imaging technique combines digital imaging with spectroscopy techniques to provide improved spectral properties of the captured area within and beyond the visible range of the electromagnetic spectrum.
A hyperspectral image is a three-dimensional data array (hypercube) consisting of successive images at different wavelengths [5]. This image is obtained by photographing the object at different wavelengths. In such image, each pixel contains the so-called spectral signature of a material or substance, located in the corresponding spatial coordinates. Quantitative information about tissue physiology can be extracted using spectral analysis [6]. HSI provides information on the intensity of reflected light at different wavelengths at each point on the sample surface. To solve the classification problems, it is necessary to use different ranges of registered two-dimensional images that form a hypercube.
HSI for skin pathology studies solves two problems at once: (1) examination of the skin surface to detect the
This paper was presented at the IX International Conference on Information Technology and Nanotechnology (ITNT-2023), Samara, Russia, April 17-21, 2023.
boundaries of lesions and (2) examination of the morphological and chemical properties of the skin to detect and identify the type of tumor. This technology improves the accuracy of diagnosing skin pathologies, taking into account the physiological characteristics of the patient, as well as the composite and morphological features of the skin surface. The effectiveness of the applied method was demonstrated in the study by Sherendak et al. Authors implemented manual processing of hyperspectral images [7]. Based on the results of this study, it can be concluded that even in the absence of a large number of samples and additional preprocessing of the training set, it is possible to obtain a high accuracy of classification of malignant skin lesions.
Machine learning has been revolutionized in recent decades by deep learning. It is considered to be one of the most difficult areas of machine learning related to artificial neural network algorithms. These algorithms were largely inspired by the functions and structure of the human brain. Deep learning methods can be applied in various fields such as speech recognition [8] and pattern recognition [9]. In these applications, deep learning systems have achieved impressive results by comparison to other classical machine learning methods.
The advantage of using a neural network classifier for hyperspectral images is the ability to use the same architecture and the same code to diagnose various types of diseases and pathologies after performing the necessary training. This means that with successful training, the artificial neural network will make it possible to obtain the correct diagnosis based on data that was not in the training set.
Automated deep learning algorithms based on convolutional neural network (CNN) have achieved remarkable performance in the detection, segmentation, and classification operations of medical imaging. Lequan et al. [10] proposed a very deep CNN for melanoma detection. A fully convolutional residual network (FCRN) having 16 residual blocks was used in the segmentation process to improve performance. The proposed technique used an average of both support vector machine and softmax classifier for classification. It showed 85.5% accuracy in melanoma classification with segmentation and 82.8% without segmentation. DeVries and Ramachandram [11] proposed a multi-scale CNN using an Inception-V3 deep neural network that was trained on an ImageNet dataset. For skin cancer classification, the pre-trained Inception-V3 was further fined-tuned on two resolution scales of input lesion images: coarse-scale and finer scale. The coarse-scale was used to capture shape characteristics as well as overall contextual information of lesions. In contrast, the finer scale gathered textual detail of lesion for differentiation between various types of skin lesions [12].
This paper considers hyperspectral images of MM and PN based on the architecture of a five-block CNN neural network. The aim of the study is to investigate the capabilities of a neural network for the classification of hyperspectral images of skin pathologies.
2 Materials and Methods
To record hyperspectral images in vivo, an acousto-optical hyperspectral camera was used, which makes it possible to obtain an image of the area under study at an arbitrarily set wavelength in the range of 440-750 nm (Fig. 1). In this case, the spectral resolution is 51 = 2.5 nm (at 1 = 633 nm), and the spatial resolution is 0.14 mm. A feature of the installation is double sequential filtration. It provides almost complete compensation of spectral and spatial image distortions in a single cell caused by Bragg diffraction, which usually leads to a change in the spectra at individual points [13].
Registration of hyperspectral images was carried out in the Samara Regional Clinical Oncology Center. Informational consents of each subject were obtained. The study was approved by the ethical committee of the Samara State Medical University.
The result of registration with using the described setup of one sample is a hypercube, that consists of 151 optical images in .tiff format at wavelengths from 430 nm to 730 nm with a step of 2 nm. Each sample contained an area of neoplasm and healthy skin. Formation samples were taken from patients with European skin type. The final diagnosis was made for each examined sample on the basis of histological examination. Before further analysis, the hypercubes were reduced to 51 images in the range from 530 nm to 630 nm in 2 nm steps. The choice of this interval is conditioned by the best absorption of light by melanin and hemoglobin, as well as the high contrast of images, which facilitates image analysis. Using the software, individual grayscale images were obtained from the original hyperspectral images of the studied skin samples. The submission of images in this color scale allows to simplify the mechanism of interaction by reducing the input channels from three to one.
In total, 50 samples were used: 25 MM and 25 PN. This corresponds to 2550 optical images: 1275 MM and 1275 PN (see Table 1).
A CNN of the VGG architecture was used for image classification [14]. Unlike AlexNet, which focuses on smaller window sizes and steps of the first convolutional level, VGG addresses image depth [15]. The architecture of the neural network is shown in Fig. 2. The classifier consists of 13 layers. The number of trainable parameters increases as the layers deepen. The neural network classifier is divided into 5 blocks. The first 4 blocks are convolutional and each of them end with a maximum pooling layer.
Table 1 Composition of the DataSet.
Number of Number of samples_images
№ Diagnosis
1
Malignant melanoma (MM) Pigmented nevus (PN)
25 25
1275 1275
2
Я
V
9
0
200
400
600
800
1000
В
Fig. 1 Malignant melanoma at different wavelengths (the axes indicate pixels).
The input of the neural network is a two-dimensional RGB image, represented by a spectral slice and reduced to a size of 224 x 224 pixels due to additional scaling. The training and test samples were formed from images extracted from registered hypercubes in the specified range from 530 nm to 630 nm (see Table 1), in a ratio of 90% (2295 images) and 10% (255 images), respectively. The test sample set included only those images that did not participate in the training of the neural network classifier. In the training and test samples, a uniform distribution of samples across classes was made. This is caused by the need to obtain a more accurate classification at the stage of testing the operation of the neural network. Our classifier is trained using the backpropagation method. All layers of the network are fine-tuned using the same global learning network.
The neural network architecture also provides a feedback trigger that aborts training if the accuracy does not increase within three training epochs. This was provided to avoid retraining the model at the training stage.
3 Results and Discussion
The neural network was trained and tested on a computer equipped with an Intel(R) Core(TM) i7-10700KF 3.80 GHz processor, an NVIDIA GeForce RTX 3080 graphics card, and 32 GB of random access memory (RAM). The training time of the classifier was about 75 s (~5 s/epoch).
In the study, various variations of the last layer neuron deletion function were used, which consists in assigning zero to randomly selected features at the training stage. Thinning was added to the network through the Dropout layer, which you can see in Fig. 2. We used different thinning ratios of 10%, 15%, 20%, 25%, 30%, 40%, 50% of the features in the training phase. This was done to reduce the conspiracy of the model with the original data, thereby weakening the effect of overfitting. We found that increasing the removal percentage by more than 15% led to a decrease in the accuracy of the model. When thinning out half of the weights, the accuracy of the neural network classifier decreased to 76%. The highest accuracy, equal to 95%,
c onv2 dSiuput: BiputLayer input: [(Noue. 224, 224, 1)]
output: [(Noue. 224. 224.1)]
conv2d_8 Conv2D input: (None. 224, 224, 1)
output: (None. 224. 224. 32)
max_pooling2d_8 MaxPooling2D input: (None, 224, 224, 32)
output: (None, 112. 112. 32)
conv2d 9 Conv2D input: (None. 112,112. 32)
output: (None, 112, 112,64)
max_pooling2d_9 MaxPoolina2D input: (None. 112,112, 64)
output: (None, 56, 56, 64)
conv2d_10 Couv2D input: (None, 56, 56, 64)
output: (None. 56. 56.128)
max_pooling2d_10 MaxPooling2D input: (None, 56, 56, 128)
output: (None, 28, 28,128)
conv2d 11 Couv2D input: (Noue. 2S. 28.12S)
output: (None, 28, 28, 256)
max_pooliug2d 11 MaxPooling2D input: (None. 28. 28. 256)
output: (None, 14, 14. 256)
dropout 2 Dropout input: (None, 14, 14, 256)
output: (None, 14, 14. 256)
r
flatten 2 Flatten input: (None, 14, 14, 256)
output: (None, 50176)
input: (None, 50176)
output: (None, 256)
deuse_5 Dense input: (None. 256)
output: (None. 2)
Fig. 2 The architecture of the used convolutional neural network.
was presented by the neural network when 15% of the features were removed. The model also presented the smallest loss (0.09) with the removal of 15% of features. To obtain accuracy and loss metrics, the values obtained during the validation test of the neural network were used. Accuracy metric is the ratio of the number of
correctly classified samples to their total number on the validation test and loss metric represents how well our model performs by comparing what the model predicts with the actual value.
The metrics obtained at the training stage of 15 epochs are presented in the Figs. 3 and 4. Then, class activation maps (CAM) were obtained on the example of a hyperspectral image of a pigmented nevus (see Figs. 5 and 6). This map of intermediate activations helps to debug the decision-making process in the neural network, especially in the classification error. It also helps to determine the location and specificity of objects in the image and draw a conclusion about the correctness of the neural network classifier. As it can be seen in Fig. 5, the neural network classifier focuses directly on the nevus itself, which indicates the correctness of the problem classification algorithms.
The obtained image classification accuracy of 95% on our test dataset allows us to say that the selected regions of the spectrum are informative enough to identify skin neoplasms.
Fig. 3 Accuracy of the classification model with thinning out 15% of features: the solid line is the training set, the dotted line is the test set.
Fig. 4 Losses of the classification model with thinning out 15% of the features: the solid line is the training set, the dotted line is the test set.
Fig. 6 Hyperspectral image of a pigmented nevus at a wavelength of 564 nm.
The obtained high accuracy of the binary classification of skin neoplasms is not only comparable with the results of the use of Raman scattering, but also exceeds the efficiency of fluorescence and dermoscopic examination [3]. An analysis of spectral indices calculated from hyperspectral images and characterizing differences in the relative content of skin chromophores for MM, basal cell carcinoma, and PN presented in Ref. [16] made it possible to obtain an accuracy of no higher than 78%.
Taking into account the small size of the training and test samples, the achieved accuracy values can both
increase and decrease as they increase. However, the obtained results demonstrate the possibility of using neural networks for the analysis of hyperspectral images in order to detect MM.
4 Conclusion
In this work, a binary classification of hyperspectral images of 50 samples of skin pathologies in the visible region was carried out using a neural network of the VGG architecture. The spectral ranges of 530-630 nm selected for differential image analysis showed high information content, which confirms the possibility of reducing the recorded spectral data and, accordingly, increases the speed of computational processes. The classification accuracy for MM and PN on test dataset was around 95%. The use of a hyperspectral image classification algorithm based on neural networks has improved the accuracy of skin pathology separation compared to other classification methods [7, 10, 11, 16].
Further development of technology is possible in several directions. The first is to conduct mass studies both with skin samples and with samples of other areas of human tissue to establish and improve the accuracy of diagnosing oncological diseases, as well as to compare the proposed method for classifying hyperspectral images with data on the accuracy of diagnosing other optical methods.
The second is the development of technology and the creation of an autonomous algorithm for the control of oncological diseases, which would allow monitoring the presence and condition of various unfriendly neoplasms.
The third is the optimization of the neural network architecture, the increase in the training sample and the modernization of hardware for registering hyperspectral images.
Acknowledgments
This paper was prepared within the framework of the scientific research for scientific and educational organizations of higher education in Russia "Development of a software and hardware complex for non-invasive diagnosis of neoplasms using spectroscopy" ("YERG-2023-0001" 2- 23-2025).
Disclosures
The authors declare no conflict of interest.
References
1. "Cancer," World Health Organization, 3 February 2022 (accessed 1 November 2022). [https://www.who.int/news-room/fact-sheets/detail/cancer].
2. J. Ferlay, M. Colombet, I. Soerjomataram, D. M. Parkin, M. Pineros, A. Znaor, and F. Bray, "Cancer statistics for the year 2020: An overview," International Journal of Cancer 149(4), 778-789 (2021).
3. V. O. Vinokurov, I. A. Matveeva, Y. A. Khristoforova, O. O. Myakinin, I. A. Bratchenko, L. A. Bratchenko, A. A. Moryatov, S. G. Kozlov, A. S. Machikhin, I. Abdulhalim, and V. P. Zakharov, "Neural network classifier of hyperspectral images of skin pathologies," Computer Optics 45(6), 879-886 (2021).
4. S. Karim, A. Qadir, U. Farooq, M. Shakir, and A. A. Laghari, "Hyperspectral imaging: a review and trends towards medical imaging," Current Medical Imaging 19(5), 417-427 (2022).
5. C. I. Chang, "Hyperspectral imaging: techniques for spectral detection and classification," Springer New York, NY (2003).
6. G. Lu, B. Fei, "Medical hyperspectral imaging: A review," Journal of Biomedical Optics 19(1), 10901 (2014).
7. V. P. Sherendak, I. A. Bratchenko, O. O. Myakinin, P. N. Volkhin, Y. A. Khristoforova, A. A. Moryatov, A. S. Machikhin, P. E. Pozhar, S. V. Kozlov, and V. P. Zakharov, "Hyperspectral in vivo analysis of normal skin chromophores and visualization of oncological pathologies," Computer Optics 43(4), 661-670 (2019).
8. H. Rashid, M. A. Tanveer, and H. A. Khan, "Skin lesion classification using GAN based data augmentation," Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 916-919 (2019).
9. D. Bisla, A. Choromanska, J. A. Stein, D. Polsky, and R. Berman, "Towards automated melanoma detection with deep learning: data purification and augmentation," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2720-2728 (2019).
10. L. Yu, H. Chen, Q. Dou, J. Qin, and P. A. Heng, "Automated melanoma recognition in dermoscopy images via very deep residual networks," IEEE Transactions on Medical Imaging 36(4), 994-1004 (2017).
11. T. DeVries, D. Ramachandram, "Skin lesion classification using deep multi-scale convolutional neural networks," arXiv Preprint arXiv:1703.01402 (2017).
12. M. Dildar, S. Akram, M. Irfan, H. U. Khan, M. Ramzan, A. R. Mahmood, S. A. Alsaiari, A. H. M. Saeed, M. O. Alraddadi, and M. H. Mahnashi, "Skin cancer detection: a review using deep learning techniques," International Journal of Environmental Research and Public Health 18(10), 5479 (2021)
13. A. Machikhin, V. Batshev, and V. Pozhar, "Aberration analysis of AOTF-based spectral imaging systems," Journal of the Optical Society of America A 34(7), 1109-1113 (2017).
14. K. Simonyan, A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv Preprint arXiv:1409.1556 (2014).
15. L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamaría, M. A. Fadhel, M. Al-Amidie, and L. Farhan, "Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions," Journal of Big Data 8, 53 (2021).
16. I. A. Bratchenko, L. A. Bratchenko, A. A. Moryatov, Y. A. Khristoforova, D. N. Artemyev, O. O. Myakinin, A. E. Orlov, S. V. Kozlov, and V. P. Zakharov, "In vivo diagnosis of skin cancer with a portable Raman spectroscopic device," Experimental Dermatology 30(5), 652-663 (2021).
