CC BY
23
Hyperspectral Camera - Attachment for Microscopy
Mohammed M. Hamza1, Ali Hamandi1, Andrey R. Makarov1,2*, Vladimir V. Podlipnov1,2, and Roman V. Skidanov1,2
1 Samara National Research University, 34 Moskovskoe Shosse, Samara 443086, Russian Federation
2 IPSI RAS - branch of the FSRC "Crystallography and Photonics" RAS, 151 Molodogvardejskaya str., Samara 443001, Russian Federation
* e-mail: andre makar@bk.ru
Abstract. The paper presents a microscope equipped with a hyperspectral ocular nozzle with a phase diffraction grating, which is used as a dispersing element. The spectral range of the device is 400-1000 nm, with the spectral resolution of 5 nm. The paper describes a microscope software that is used to solve problems of spectral analysis. © 2021 Journal of Biomedical Photonics & Engineering.
Keywords: microscopy; hyperspectral imaging; hyperspectral analysis.
Paper #3429 received 24 May 2021; revised manuscript received 25 Sep 2021; accepted for publication 25 Sep 2021; published online 30 Sep 2021. doi: 10.18287/JBPE21.07.030405.
1 Introduction
Hyperspectral equipment that can distinguish a large number of spectral channels, in contrast to conventional three-color imaging, when using hyperspectral image processing, allows you to see subtle color differences in the visible range of optical radiation, as well as wavelengths invisible to the human eye in the near-infrared range [1]. This ability makes it possible to distinguish a number of chemical compounds of the underlying surface in remote sensing of the Earth, and is also increasingly used in biology and medicine [2]. In addition to the classical search signatures, in biology and medicine, a number of compounds with the use of illumination of certain wavelengths can cause the effect of photoluminescence, fluorescence, Raman scattering, etc., which allows you to use the hyperspectral equipment in criminology [3], materials science [4], mineralogy [5]. Such studies can be carried out only in the laboratory, and the use of a combination of microscopy and hyperspectral equipment allows you to see the smallest particles that cannot be seen by the eye [6].
A microscope with hyperspectral imaging can use narrow-band radiation of an object, or radiation transmitted through an optical monochromator [7]. Hyperspectral imaging systems based on acousto-optic modulators are known [8]. Most often, hyperspectrometers use an optical scheme of the scanning type, similar to that described in Ref. [9]. Using a microscope-spectrometer, it is possible to obtain the spectrum of individual pixels in the wavelength range from 400 nm to 1000 nm with a spectral resolution of 5 nm [10-11], depending on the dispersing element used and the resolution of the recording matrix.
We analyze well-known microscopes equipped with spectrum recording systems, and some of their shortcomings become obvious. One of them is the complexity and high cost of the optical or electronic part of the devices. For example, the LOMO MSFU-K microscope-spectrophotometer uses a photoelectron multiplier PMF R928 from Hamamatsu (Japan). Many devices do not allow you to get spectral data in each individual pixel of the image. Most instruments do not have an option of selecting a microscope. The complex can only be installed on certain microscopes.
In this regard, in this paper, the task is to develop and study a hyperspectrometer adapted to work as part of a microscope in the form of an ocular nozzle. Scanning can be carried out by moving the movable table of the microscope. This concept makes it possible to equip almost any microscope for biological research with a hyperspectral imaging system.
2 Materials and Methods
2.1 Optical scheme of a hyperspectral camera to a microscope
The optical unit of a hyperspectral camera for a microscope is made according to a fairly well-known scheme of a slit hyperspectrometer based on a grating dispersion element. Most schemes have a prism or lattice dispersion element. Since the grid has a more compact size, it was decided to use it. A standard microscope lens with a magnification of 10*, 20*, 40*, 100* is used as an input lens. This set of magnifications is most often used in biomedical researches [12-13]. The image from these lenses is built on a slit aperture having a slit width of about
10 ^m. Next, a telescopic system based on two lenses builds an image of the gap on the matrix. A binary lattice with a period of 6 ^m is installed inside the telescope. Such grating provides about 250 channels in the focal plane in the spectral range of 400-1000 nm.
The design elements of the hyperspectral nozzle were developed according to the optical scheme (see Fig. 1) using three-dimensional modeling of 3D printing parts made of plastic on a 3D lapper.
Fig. 1 Optical scheme of a hyperspectral measuring device for a microscope.
An optical hyperspectrometer consists of the following elements: 1) light source; 2) motorized table; 3) focusing lens; 4) entrance slit; 5) collimator; 6) dispersion grid; 7) focusing lens; 8) optical sensor, L - wavelength; a, b - 2D detector arrays.
For illumination, both the built-in halogen lamp of the microscope and the external one with an upper position can be used. In this case, the generated spectrum on the sensor is recorded as the transmission spectrum and the reflection spectrum, respectively.
2.2 Hyperspectral survey control system as part of a microscope
The fact that the camera is based on slit scanning will be used as the basis of the shooting control system for
obtaining hyperspectral images (hypercubes). In our case, the hyperspectral camera remains stationary, while relative to it, the movement is carried out by scanning a sample located on a movable microscope table. The microscope has precise mechanical movement elements. Usually, the operator manually rotates these handles and visually observes the area on the slide table. For automated control of these movements, it is possible to install a stepper motor. Thus, with the help of a special control system, it is possible to install the step and speed of movement from the control unit.
The control unit of the complex is based on a singleboard microcomputer with software for shooting and an optical sensor of the camera synchronized with the shooting and a motion control system for the scanning table. The control is carried out by sending control signals to the stepper motor driver. The camera registers the signal during the time allotted for the accumulation of light by the optical sensor, at the end of which the next signal for movement is formed. At the end of the movement, the next frame begins to be shot synchronously. The time of accumulation of the signal from the matrix, the number of shooting frames, etc. shooting parameters are set using the developed user interface. The structure of the complex is shown in Fig. 2. In order to control the shooting, a compact, sufficiently powerful computer is needed.
The complex uses a Raspberry Pi 4 microcomputer due to the large number of GPIO pins, a large set of USB and HDMI interfaces, the presence of Wi-Fi and Bluetooth. The manufacturer is "Raspberry Pi Foundation", England.
Also, the A3967SLB stepper motor driver (Allegro MicroSystems, USA) is used in the complex. The complex uses a monochrome camera "Basler acA 192040 um" (BASLER Fashion GmbH, Germany).
r J
Bajlci as<r l920-40um
St KJ^Hzi j_1 1 ^^ Microscope
Dispersion part ^^^^^^ "Motic"
Optical part >>
Stepper motor "Atom"
Fig. 2 Block diagram of the software and hardware complex.
The monochrome camera was chosen because of its greater light sensitivity and it can capture a larger range, unlike a color camera.
The ATOM stepper motor (ATOM, Russia) was chosen:
1) Angular pitch, deg -1.8.
2) The error of the angular step, deg ±5% (full step, without load).
The information collected in this way is processed by special software for recognizing objects with similar spectral characteristics.
A stepper motor is attached to the movable microscope substrate. The stepper motor is connected to the movable table of the microscope through a gearbox consisting of two gears. Gear ratio equals to 3. The image of the engine design is shown in Fig. 3.
(b)
Fig. 3 (a) The image shows a microscope, (b) motor design for moving the microscope table.
The stepper motor driver is controlled by a Raspberry Pi 4 single-board computer via GPIO ports [14].
The Python program controls the signals "1" and "0" on the GPIO ports using the python library RPi.GPIO.
The presented complex uses the stepper motor driver "A3967LBT".
The signals are fed to the inputs of the A3967LBT driver and, thus, the stepper motor is controlled [15]. The
presented stepper motor driver allows you to work with step crushing modes in the range from full-step to 1/8 step, which on the microscope under study provides an accuracy of movement from 32.4 |im per step in full-step mode, to 4.05 |im in 1/8 step mode.
The Basler camera is connected to the computer via a USB 3.0 port. The camera parameters were changed using the Python SDK using the pypylon library. Shooting settings: shutter speed, if using a color camera, the white balance value [16].
To set the settings for shooting, you need an interface (Fig. 4). To create it, we used the PyQt5 program.
E MiinWindow -OX
Number of steps i
Direction
Operating mode
Step speed
White balance Name photo
ExporsureTroe Number of photos taten
Preview (OFF-esc)
ON
Start over
START
Fig. 4 Interface for managing shooting parameters.
The "ON Preview" button allows you to see what is in real time on the microscope substrate.
This may be necessary to adjust the focus. The "Start over" button clears the fields. The "START" button starts shooting.
Shooting can be controlled remotely via Wi-Fi, via a separate computer or smartphone using the VNC viewer remote access program.
3 Results and Discussion
We tested the resulting scanning and hyperspectral shooting control system, carried out the necessary tests, calibrations, scanning speed settings, exposure, etc. We also determined the optical characteristics of the hyperspectral attachment to the microscope. A hyperspectral image (hypercube) is obtained by assembling frames from a sequence taken at each step of the stepper motor using specially developed software.
When testing the complex (see Fig. 5), a multicolored paper printed on an inkjet printer was photographed (see Fig. 5a).
The image of a single frame fixed on the CCD matrix of the hyperspectral camera, illuminated by a halogen lamp (Fig. 5b).
ХРШЛймимцнрнрм
Fig. 5 (a) An object, (b) the image of a single frame fixed on the CCD matrix of the hyperspectral camera attachment of the microscope, (c) spectrum image on a hyperspectrometer camera with additional illumination by a laser with a wavelength of 532 nm, (d) an RGB image of colored paper synthesized from hypercube channels in the ENVI program The dimensions of the shooting area are 4.5 mm wide and 14 mm long.
In order to carry out the correct color synthesis for this hyperspectral camera, laser calibration was carried out with a wavelength of 532 nm. The view of the image of the spectral decomposition of the image in the slot from the object that is additionally illuminated by the laser is shown in Fig. 5c. From this image, it is also possible to estimate the spectral resolution of the hyperspectrometer, which, as can be seen, allows distinguishing about 120 spectral channels in the range of 400-1000 nm. In this case, it is easy to get that the spectral resolution in this case becomes about 5 nm.
The spectrum data was obtained and converted into a hypercube using special software. The three layers of the
hypercube, which are responsible for the blue, green, and red components, were used to create the RGB image (see Fig. 5d).
Some "stripes" were skipped due to the 1/2 shooting mode, and therefore the proportions of the length and width of the image in Fig. 5d are equal to each other. You can set a smaller step (shooting mode 1/8), then all the proportions will be observed.
We can observe the spectrum at each point of this image. An example of the spectrum of pixels from the green area of colored paper (Fig. 5b) "rainbow" is shown in Fig. 6.
9 »1Z-. - □ X
Fig. 6 Spectral analysis in the "ENVI" program.
af <1 spectr 1142 000 Light intensity (ISO) 128.000 114.000 99.400 85 200 71.000 56 800
i'* 11 _ 14.200 1
420 000 4X 000 532 000 58 7 000 1 643.000 699.000 755.000 811.000 866.000 922.000 Wavelength (nm)
Fig. 7 The result of the classification of the object in the image. The dimensions of the shooting area are 4.5 mm wide and 14 mm long. Some "stripes" were skipped due to the 1/2 shooting mode, and therefore the proportions of the length and width of the image in Fig. 5d are equal to each other. You can set a smaller step (shooting mode 1/8), then all the proportions will be observed.
Fig. 6 shows the spectral analysis of the "rainbow" image - the green pixel is selected.
In this case, the ENVI program was chosen for spectral analysis. ENVI is a software for image processing and analysis. It is used by image analysts, GIS specialists and scientists to extract timely, reliable and accurate information from geospatial images.
The developer of the ENVI program "L3 Harris Technologies" is an American technology company, a defense contractor and an information technology service provider.
For example, we also carried out the classification of color bands with threshold processing by the well-known Euclidean distance method [17]. The result of selecting one strip from the image of colored paper (Fig. 5) is shown in Fig. 7.
For this purpose, a special software has been developed, with the help of which it is possible to quickly find objects of interest to research with a spectrum similar to the spectrum specified as the search standard. The spectrum can be loaded as a file or specified as an object in the image.
(a)
(b)
(c)
Fig. 8 An example of a hyperspectral image of a biological histology sample of a rat's spleen stained with hematoxylin-eosin: (a) an image of a sample of a rat spleen taken with a conventional camera through a microscope, (b) an image of a sample of a rat's spleen made by a hardware and software complex, (c) the spectrum of the selected point (wavelenght in nm).
We also obtained medical hyperspectral images of histological specimens for studying of rat's spleen from a set of micropreparations according to the catalog [18], produced by the company "Retinoids", produced in compliance with ethical requirements and international recommendations for conducting biomedical research using animals. An example of an image obtained in a microscope with an 40* magnification and its color-synthesized image is shown in Fig. 8.
Fig. 8a shows an image of a histological sample using an 10* lens and a ToupCam U3CMOS05100KPA microscope camera. The field of view is 0.9 * 0.9 mm.
Fig. 8b shows the image obtained by scanning in the 1/8 step mode, the size of the scanning area is 0.4 mm high and 1.2 mm wide. The proportions of the image are not observed due to the limited scanning step. Fig. 8c also shows an example of the spectrum in the cross-hair marker (Fig. 8b) area in the image. The hyperspectral
additive for the microscope gives an additional magnification of about 2*.
In the future, we plan to continue our research to obtain methods for classifying objects in the image of medical samples. In this work, the task of determining the accuracy of the classification of anomalies associated with diseases was not set. To conduct this kind of research, we plan to conduct a survey of a large set of various biological preparations. In this work, the task of determining the accuracy of the classification of anomalies associated with diseases was not set. To conduct this kind of research, we plan to conduct a survey of a large set of various biological preparations. Previously, studies of hyperspectral imaging were conducted in Refs. [19, 20] to determine the types of human skin neoplasms. These studies have shown a fairly high accuracy in determining pathologies. Also, a hyperspectral camera based on an acousto-optic monochromator was described in Ref. [21]. This design
eliminates the need for additional scanning systems. At the same time, the production of a control unit and shooting for a hyperspectral camera in this case turns out to be more expensive than in the device presented in this paper. Also, the used wavelength range was limited to the range of 400-750 nm, while in our study, the spectral wavelengths range is limited only by the sensitivity range of the sensor. The presented control system and an optical camera can be equipped with almost any microscope, which is almost 10 times cheaper in comparison with expensive analogs of microscopes.
4 Conclusion
The article presents a hardware and software complex for controlling hyperspectral photography, as well as software for spectral analysis. During the research, the optical characteristics of the complex were determined, software was developed for controlling the shooting and obtaining primary spatial-spectral data that are then converted into hyperspectral images by processing, which allows, as we hope, to obtain detailed and highly reliable data for establishing an accurate diagnosis from histological samples. During the study, the parameters of the shooting
speed were obtained depending on the magnification of the lens.
Advantages of this complex:
1) simplicity of the scheme;
2) flexibility in the selection of specific components;
3) remote and automatic shooting control;
4) obtaining individual spectra in individual pixels of the image.
A software for spectral analysis was also developed.
Disclosures
All authors declare that there is no conflict of interests in this paper.
Acknowledgments
The development of the hardware of the hyperspectral settop box was carried out within the framework of the state task of the Ministry of Education and Science of the Russian Federation (Project No. 0777-2020-0017), the development of the hyperspectral image analysis program was carried out with the financial support from an RSF grant No. 20-69-47110.
References
1. K. S. Tkachenko, "Application of aerospace imagery in hydrobiological research," Proceedings of the Samara Scientific Center of the Russian Academy of Sciences 14(1), 15-31 (2012) [in Russian].
2. V. S. Putilina, I. V. Galitskaya, and T. I. Yuganova, "Influence of organic substance on heavy metal migration in municipal solid waste disposal sites," Ecology. Analytical Review 76, State Public Scientific and Technical Library of the Siberian Branch of the Russian Academy of Sciences, Novosibirsk (2005) [in Russian]. ISBN: 5-94560-0911.
3. G. Lu, B. Fei, "Medical hyperspectral imaging: a review," Journal of Biomedical Optics 19(1), 10901 (2014).
4. M. Akiko, "Hyperspectral prism-grating-prism imaging spectrograph," PhD thesis, University of Oulu (2001).
5. P. Lypaczewski, B. Rivard, G. Lesage, K. Byrne, M. D'Angelo, and R. G. Lee, "Characterization of Mineralogy in the Highland Valley Porphyry Cu District Using Hyperspectral Imaging, and Potential Applications," Minerals 10(5), 473 (2020).
6. S. Ortega, M. Halicek, H. Fabelo, G. M. Callico, and B. Fei, "Hyperspectral and multispectral imaging in digital and computational pathology: a systematic review," Biomedical Optics Express 11(6), 3195-3233 (2020).
7. I. P. Gurov, A. I. Lopatin, and A. V. Melnikov, "Formation of hyperspectral data on microscopic objects with high degree of spatial non-uniformity of optical density distribution," Scientific and Technical Journal of Information Technologies, Mechanics and Optics 2(84), 103-107 (2013) [in Russian].
8. M. M. Mazur, Yu. A. Suddenok, and V. N. Shorin, "Dual acousto-optical image monochromator with adjustable width of the hardware function," Pisma v ZhTF 40(4), 56-62 (2014) [in Russian].
9. A. G. Orlov, V. V. Egorov, A. P. Kalinin, and I. D. Rodionov, "Aviation hyperspectrometer: architecture and method of calculating elements," in 5th All-Russian Open Annual Conference on the Modern Problems of Remote Sensing of the Earth from Space, 280-287 (2007) [in Russian].
10. V. V. Podlipnov, N. A. Ivliev, and R. V. Skidanov, "A compact imaging hyperspectrometer," Journal of Physics: Conference Series 13689(2), 022053 (2019).
11. S. B. B. Ahmadi, Y. A. Nanehkaran, and S. Layazali, "Review on hyper-spectral imaging system," International Journal of Scientific and Engineering Research 4(5), 253-258 (2013).
12. J. Beach, "A richer view of bio structures," BioOptics World 2, 68-71 (2009).
13. V. Leroi, J.-P. Bibring, and M. Berthe, "MicrOmega a VIS/NIR hyperspectral microscope for in situanalysis in space," Planetary and Space Science 57(8-9), 1068-1075 (2009).
14. Raspberry Pi Documentation, Description of the operation of GPIO ports on a Raspberry Pi computer (accessed 15.07.2020) [https://www.raspberrypi.com/documentation/computers/os.html].
15. 3967 Microstepping driver with translator, Datasheets Allegro MicroSystems, Worcester, Massachusetts (20022003) [https://www.sparkfun.com/datasheets/Robotics/A3967.pdf].
16. Basler ace, User's manual for USB 3.0 cameras, AW001234 (2015) [https://www.micropticsl.com/wp-content/uploads/2013/09/basler ace usb manual.pdf].
17. E. Myasnikov, "Evaluation of Nonlinear Dimensionality Reduction Techniques for Classification of Hyperspectral Images," In AIST (Supplement), 147-154 (2018).
18. V. I. Nozdrin, T. A. Belousova, and G. V. Trunova, "Microscopy slides for studying histology," Catalogue, Retinoids, Moscow (2016). ISBN: 978-5-93118-046-5.
19. L. A. Zherdeva, I. A. Bratchenko, O. O. Myakinin, A. A. Moryatov, S. V. Kozlov, and V. P. Zakharov, "In vivo hyperspectral imaging and differentiation of skin cancer," Proceedings of SPIE 10024, 100244G (2016).
20. I. A. Bratchenko, O. O. Myakinin, V. P. Sherendak, P. N. Volkhin, Y. A. Khristoforova, L. A. Bratchenko, D. N. Artemyev, A. A. Moryatov, O. V. Polschikova, A. S. Machikhin, V. E. Pozhar, S. V. Kozlov, and V. P. Zakharov, "In vivo hyperspectral analysis of skin hemoglobin and melanin content for neoplasia detection," Journal of Biomedical Photonics & Engineering 4(4), 040301 (2018).
21. V. Sherendak, I. A. Bratchenko, O. O. Myakinin, P. Volkhin, Yu. Khristoforova, A. A. Moryatov, A. S. Machikhin, V. E. Pozhar, S. V. Kozlov, and V. P. Zakharov, "Hyperspectral in vivo analysis of normal skin chromophores and visualization of oncological pathologies," Computer Optics 43(4), 661-670 (2019).
