CC BY
22
High-Resolution Ultra-Spectral Imager for Advanced Imaging in Agriculture and Biomedical Applications
Maria M. Antony#, C. S. Suchand Sandeep*#, Hoong-Ta Lim, and Murukeshan Vadakke Matham+
Centre for Optical and Laser Engineering, School of Mechanical and Aerospace Engineering, Nanyang Technological University, 50 Nanyang Avenue, 639798, Singapore # These authors contributed equally to this work
* 1" e-mail: [email protected], [email protected]
Abstract. Conventional imaging practices used for the inspection and monitoring of biological specimens employ RGB cameras with limited capabilities for early identification of diseases or abnormalities. In this work, we demonstrate and validate a quick, non-destructive, and precise inspection method utilizing an in-house developed push broom ultra-spectral imager. Precise image classification based on ultra-spectral signatures can provide fully automated machine vision capabilities, reducing inspection time, human errors, and man-hours. The proposed method has high spectral resolution (AÀ < 1 nm) with 756 spectral bands, improved detection sensitivity, and high spatial resolution, which could potentially enable early-stage detection and accurate classification of abnormalities or diseases. Two potential applications of the developed system in agriculture and biomedical fields are demonstrated. © 2023 Journal of Biomedical Photonics & Engineering.
Keywords: hyperspectral imaging; spectral mapping; machine vision; non-destructive testing; smart farming; datacube; spectral library.
Paper #8928 received 8 Mar 2023; revised manuscript received 31 May 2023; accepted for publication 31 May 2023; published online 27 Jul 2023. doi: 10.18287/JBPE23.09.030304.
1 Introduction
Automated detection of abnormalities and diseases is of utmost importance in modern agro-food and medical applications [1]. It has been reported that early detection of diseases reduces complications caused by human errors, delays, and inaccuracies [2]. Traditional techniques for inspecting and monitoring biological specimens are based on transmission, absorption, fluorescence, or electron microscopy, and have limited capabilities for automated disease identification at an early stage [3-8]. In general, these techniques only provide spatial information about the sample under study and are thus insufficient for automation. There are a few techniques that combine spectroscopy and imaging that can enable automated disease detection using imaging spectroscopes [4, 9, 10]. Imaging spectroscopes can provide information from multiple spectral bands and hence are excellent candidates for automated and early detection of diseases or abnormalities. Such imagers, also known as spectral imagers, are divided into three categories based on the number of spectral bands they can resolve: multispectral (a few spectral
This paper was presented at the International Conference on Nanoscience and Photonics for Medical Applications - ICNPMA, December 28-30, 2022, Manipal, India.
bands), hyperspectral (tens of spectral bands), and ultra-spectral (hundreds of spectral bands) [11, 12].
»
Sample stage
Fig. 1 Schematic diagram of the pushbroom ultra-spectral imaging system developed.
These spectral imaging systems store images as datacubes, with the x and y dimensions containing spatial information about the sample and the z dimension containing spectral information.
Numerous works utilizing hyperspectral imaging technology have been reported in the last two decades, revolutionizing several application scenarios [13-18]. However, there have not been many reports on ultraspectral imagers for agriculture and biomedical applications. Ultra-spectral imagers can record data from hundreds of spectral bands, increasing the spectral detection selectivity, particularly for molecular absorption and emissions, which can potentially improve automated abnormality and disease detection and classification capabilities. This paper discusses the bio and agro-food applications of an ultra-spectral imaging (USI) system that offers high spectral resolution, improved detection sensitivity, and high spatial resolution. This technique has several advantages over existing imaging methods used for bio and agro-food applications. For example, fluorescence microscopy, one of the gold standards used for biomedical and agricultural applications, needs the samples to be tagged with fluorescent markers, or are applicable only to samples that possess autofluorescence. Hence, fluorescence imaging would be invasive or has limited applicability. The proposed method on the other hand is non-invasive, does not require any kind of sample preparation, suitable for large area inspection, and in general is applicable to all types of samples. It is anticipated that biomedical and agricultural industries will benefit greatly from automated inspection using such systems.
2 Materials and Methods
A schematic diagram of the in-house developed USI system used for the investigations is shown in Fig. 1. The system works in the line scanning (pushbroom) spectral imaging configuration [14]. The sample is illuminated using a broadband light source (Edmund Optics, MI-150). The light reflected from the sample is collected by a fore lens on to a narrow silt, which is then dispersed by a spectrograph. The dispersed light from the spectrograph
is imaged by an electron multiplying charge coupled device (emCCD) camera (Andor, LucaEM). The use of the emCCD improves the sensitivity of the system compared to traditional charge coupled device (CCD) or complementary metal oxide semiconductor device (CMOS) cameras. The emCCD camera's electron-multiplying gain and exposure time can be fine-tuned depending on the lighting condition to optimize the image quality. The spectrum from each sample point in the line of view of the USI system was spread along the detector camera's y-axis. The USI system was initially calibrated for wavelength mapping of the spatial positions on the detector using standard spectral lines. Calibration was performed by imaging a flat sample illuminated with a tunable laser source (NKT Photonics, SuperK Extreme EXR-15, SuperK Select 4xVIS/IR) at twelve calibration wavelengths. A second-order polynomial regression model given in Eq. (1) was used to calibrate the USI system.
Fig. 2 Photographs of the samples investigated. (a) Chinese cabbage leaf, (b) enlarged view of the ROI showing a bruise on the leaf, (c) phantom tissue sample, and (d) enlarged view of the ROI on the phantom tissue under inspection showing the abnormal tissue region as black.
Fig. 3 (a) Datacube recorded from the Chinese cabbage sample. (b) Reflectance mapping at 550 nm showing clear distinction between normal and abnormal leaf areas. x &y axes represent the sample dimensions in ^m.
0.91-1-1-1-1-r
08
nl-1-'-1-1--=J
400 500 600 700 800 900 1000
Wavelength (nm)
Fig. 4 Average reflectance spectra of the two sample classes in Chinese cabbage sample.
^cai = a x C + b x Cy + c, (1)
Cy represents the detector row, Kai denotes the calibrated wavelength, and a, b, c are constants. The values of these constants were obtained by numerically fitting the calibration data and were found to be, a = 7.345 x 10-5 nm, b = 0.726 nm, and c = 331.871 nm. The calibration is cross-verified by imaging the standard spectral line from a helium neon (He-Ne) laser.
During the measurements, the sample was mounted on an automated high-resolution 3-axis translation stage (Physik Instruments) and was scanned in micrometer steps to record the ultra-spectral image of the region of interest (ROI). The system's spectral range, 400-1000 nm, spans the visible to near-infrared wavelength band with a spectral resolution of ~ 1 nm over 756 spectral bands. The spatial resolution of the system was evaluated by imaging a USAF 1951 resolution test target and the system was able to resolve vertical and horizontal lines of Group 3 Element 5 of the test target, corresponding to a lateral resolution of
39.37 ^m. The whole system was automated using an in-house developed LabVIEW® code. The datacube generation as well as all the data processing were done using custom written MATLAB® scripts.
Two representative bio samples, namely Chinese cabbage (Pak Choi) and a phantom tissue sample are chosen from agriculture and biomedical fields respectively, for the investigation using the USI system. A fresh leaf from a Chinese cabbage plant with a tiny bruise damage was used for the first set of investigations. The biophantom tissue sample was procured from Simulab Corporation and the abnormal area on the biophantom tissue was simulated by pasting a black tape on to the sample surface. Representative images of the samples used in the measurements are shown in Fig. 2.
3 Results and Discussion
The Chinese cabbage leaf sample with visible bruises was mounted on the translation stage and the images were recorded using the USI system in line scanning mode. The ultra-spectral datacube recorded and the reflectance mapping at 550 nm for the Chinese cabbage sample are shown in Fig. 3.
The reflectance mapping of the image at 550 nm shows a clear distinction between normal and abnormal regions as shown in Fig. 3(b). The average reflectance spectra extracted from the datacube for the two areas are shown in Fig. 4. The reflectance spectrum labelled class 1 represents the average reflectance spectrum captured from normal leaf area while the spectrum labelled class 2 represents the average reflectance spectrum captured from the abnormal area. The clear distinction between the two can be utilized for the automated quality inspection of Chinese cabbage crops.
The ultra-spectral datacube recorded for the biophantom tissue with simulated abnormal area is shown in Fig. 5(a). The wavelength mapping at 700 nm shows distinct changes for class 1 (normal tissue) and class 2 (abnormal tissue) as shown in Fig. 5(b).
Class 1:
0 1000 2000 3000 3850
Fig. 5 (a) Datacube recorded from the biophantom tissue sample. (b) Wavelength mapping at 700 nm. x & y axes represent the sample dimensions in ^m.
Fig. 6 Average reflectance spectra from the normal tissue (class 1) and abnormal tissue (class 2) regions in the biophantom tissue sample.
The average spectra extracted for the different sample classes (shown in Fig. 4 and Fig. 6) can be used for developing the spectral libraries required for automated classification. Detection using spectral imaging largely depends on robust reference libraries. The detection of pathological sites and diseases will be through the comparison of the spectral variations to the reference libraries using classification algorithms, deep learning, and specific indices. Several algorithms have been developed by various research groups for this purpose [19, 20]. Details of the algorithm utilized for the automated abnormality classification based on the spectral libraries created is shown in Fig. 7. Briefly, the USI datacubes recorded undergoes appropriate spectral pre-processing depending on the reference spectral libraries loaded to the algorithm. The dimensionality of the extracted spectral data is then reduced using specialized techniques such as independent component analysis (ICA), principal component analysis (PCA), or non-negative matrix factorization (NMF). Subsequently, selected mapping and further feature extraction from the data using techniques such as normalized difference index (NDI) can enable quick automated classification and detection using the algorithm shown in Fig. 7.
Distinct features are seen in the reflection spectra extracted from class 1 (normal tissue) and class 2 (abnormal tissue) areas of the sample as shown in Fig. 6, which can be utilized for the automated classification of the abnormalities in the phantom tissue sample.
To illustrate the viability and selectivity of the proposed algorithm, the classification of a Chinese cabbage plant was carried out. For this purpose, the well-known classification method, spectral angle mapper (SAM) was used. SAM is an automated method for evaluating the image spectra similarity with a known spectra or endmember from the reference spectral library. In this method, the spectra are treated as vectors and the spectral angle between them is calculated. Initially, the angle between each endmember vector and the vector defined by the pixel values for each pixel is calculated. This results in a raster layer for each endmember containing the spectral angle. The more closely a pixel
resembles a particular endmember class, the smaller is the spectral angle. Based on the pre-defined tolerance level for the spectral angle, the classification is established. As the vector's direction is only considered in this algorithm for classification, this method is insensitive to illumination intensity [21]. Fig. 8(a) shows the RGB image of the Chinese cabbage plant inspected and Fig. 8(b) shows the classified image obtained by applying the SAM classification algorithm. The green shaded areas correspond to healthy/normal region while the red shaded areas represent abnormal region. This figure clearly illustrates the automated detection capability of the proposed algorithm. It is envisioned that such techniques and algorithms have immense potential for automation in biomedical and agro-food sectors.
Fig. 7 Block diagram showing the algorithm for automated classification/detection based on reference spectral libraries created.
4 Conclusion and Future Work
A USI system with high spectral and spatial resolutions was developed and the applications of the developed system for agro and biomedical applications are demonstrated. An algorithm for the automated detection of abnormalities using the spectral signatures is detailed. Spectral libraries generated using the USI system is utilized for the automated classification of agro-food samples.
technique for automated and early detection of diseases in biomedical domain could enable real-time result oriented solutions with detection, diagnosis and treatment approaches that can find potential applications in related diagnostic medical fields.
Acknowledgements
This research is supported by the National Research Foundation, Singapore and Singapore Food Agency, under its Singapore Food Story R&D Programme (Theme 1: Sustainable Urban Food Production) Grant Call (SFS_RND_SUFP_001_03). The authors also acknowledge financial support received through COLE-EDB funding at COLE, NTU.
Disclosures
The authors declare no conflict of interest.
References
1. J. Goecks, V. Jalili, L. M. Heiser, and J. W. Gray, "How Machine Learning Will Transform Biomedicine," Cell 181(1), 92-101 (2020).
2. E. S. Shenoy, E. S. Rosenthal, Y. P. Shao, S. Biswal, M. Ghanta, E. E. Ryan, D. Suslak, N. Swanson, V. M. Junior, D. C. Hooper, and M. B. Westover, "Real-Time, Automated Detection of Ventilator-Associated Events: Avoiding Missed Detections, Misclassifications, and False Detections Due to Human Error," Infection Control & Hospital Epidemiology 39(7), 826-833 (2018).
3. C. S. Suchand Sandeep, S. Sarangapani, X. J. Hong, T. Aung, M. Baskaran, and V. M. Murukeshan, "Optical sectioning and high resolution visualization of trabecular meshwork using Bessel beam assisted light sheet fluorescence microscopy," Journal of Biophotonics 12(12), e201900048 (2019).
4. T. Vo-Dinh, "A hyperspectral imaging system for in vivo optical diagnostics," IEEE Engineering in Medicine and Biology Magazine 23(5), 40-49 (2004).
5. J. James, V. M. Murukeshan, and L. S. Woh, "Integrated photoacoustic, ultrasound and fluorescence platform for diagnostic medical imaging-proof of concept study with a tissue mimicking phantom," Biomedical Optics Express 5(7), 2135-44 (2014).
6. S. B. Colak, M. B. Van der Mark, G. W. t Hooft, J. H. Hoogenraad, E. S. Van der Linden, and F. A. Kuijpers,"Clinical optical tomography and NIR spectroscopy for breast cancer detection," IEEE Journal of Selected Topics in Quantum Electronics 5(4), 1143-1158 (1999).
7. N. Sujatha, V. M. Murukeshan, L. S. Ong, and L. K. Seah, "An all fiber optic system modeling for the gastrointestinal endoscopy: design concepts and fluorescent analysis," Optics Communications 219(1-6), 71-79 (2003).
8. S. M. Perinchery, A. Shinde, C. Y. Fu, X. J. Jeesmond Hong, M. Baskaran, T. Aung, and V. M. Murukeshan, " High resolution iridocorneal angle imaging system by axicon lens assisted gonioscopy," Scientific Reports 6, 30844 (2016).
9. F. Strasser, M. Offterdinger, R. Piestun, and A. Jesacher, "Spectral image scanning microscopy," Biomedical Optics Express 10(5), 2513-2527 (2019).
10. A. F. Goetz, G. Vane, J. E. Solomon, and B. N. Rock, "Imaging spectrometry for Earth remote sensing," Science 228(4704), 1147-1153 (1985).
11. V. Kravets, P. Kondrashov, and A. Stern, "Compressive ultraspectral imaging using multiscale structured illumination," Applied Optics 58(22), F32-F39 (2019).
12. Y. Garini, I. T. Young, and G. McNamara, "Spectral imaging: principles and applications," Cytometry A 69(8), 73547 (2006).
13. S. Mahesh, D. S. Jayas, J. Paliwal, and N. D. G. White, "Hyperspectral imaging to classify and monitor quality of agricultural materials," Journal of Stored Products Research 61, 17-26 (2015).
14. G. Lu, B. Fei, "Medical hyperspectral imaging: a review," Journal of Biomedical Optics 19(1), 010901 (2014).
15. H. T. Lim, V. M. Murukeshan, "A four-dimensional snapshot hyperspectral video-endoscope for bio-imaging applications," Scientific Reports 6, 24044 (2016).
16. H.-T. Lim, V. M. Murukeshan, "Hyperspectral photoacoustic spectroscopy of highly-absorbing samples for diagnostic ocular imaging applications," International Journal of Optomechatronics 11(1), 36-46 (2017).
Fig. 8 SAM classified image of a Chinese cabbage plant showing automatically segmented normal and abnormal regions.
The proposed method offers possibilities for early-stage detection and classification of abnormalities or diseases. In addition, possible applications of the proposed
17. M. M. Antony, C. S. Suchand Sandeep, and V. M. Murukeshan, "Hyperspectral z-scan: Measurement of spectrally resolved nonlinear optical properties," Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy 261, 120005 (2021).
18. M. M. Antony, C. S. Sandeep, H. T. Lim, and M. V. Matham, "Hyperspectral Vision Based Probe for In Situ Corrosion Monitoring in Saline Environments," IEEE Transactions on Instrumentation and Measurement 71, 1-7 (2022).
19. S. Ortega, M. Halicek, H. Fabelo, G. M. Callico, and B. Fei, "Hyperspectral and multispectral imaging in digital and computational pathology: a systematic review [Invited]," Biomedical Optics Express 11(6), 3195-3233 (2020).
20. A. Lowe, N. Harrison, and A. P. French, "Hyperspectral image analysis techniques for the detection and classification of the early onset of plant disease and stress," Plant Methods 13, 80 (2017).
21. F. Van Der Meer, M. Vazquez-Torres, and P. M. Van Dijk, "Spectral characterization of ophiolite lithologies in the Troodos Ophiolite complex of Cyprus and its potential in prospecting for massive sulphide deposits," International Journal of Remote Sensin 18(6), 1245-1257 (1997).
