DK202170004A1

DK202170004A1 - A spectral imaging system arranged for identifying benign tissue samples, and a method of using said system.

Info

Publication number: DK202170004A1
Application number: DKPA202170004A
Authority: DK
Inventors: Poulsen Knud; Laust Elbæk Frederik; Rensch-Jacobsen Filip; Nikolajsen Thomas; Tuelo Pedersen Løbner Sheller Nicolai
Original assignee: Hyspec Medtech Ivs
Priority date: 2021-01-04
Filing date: 2021-01-04
Publication date: 2022-07-06
Also published as: WO2022144457A1

Abstract

The present invention relates to a spectral imaging system (1) arranged for detecting if a tissue sample (4) is obviously benign or should proceed to histopathological investigation, said imaging system (1) comprises at least one sample container (3) for accommodating a tissue sample (4), at least one light source (5) arranged for sending light through said tissue sample (4), a light detecting device (7) arranged for capturing spectroscopic data based on light transmitted though the tissue sample, and a processing unit (10) arranged for evaluating the captured spectroscopic data and automatically classify whether the tissue sample is obviously benign or should proceed to histopathological investigation. Use of the spectral imaging system (1) ensures that obviously benign tissue samples are sieved out, i.e. said samples (4) need not be send to the pathology department for further investigation. In this way it is ensured that the pathologist can focus on the more difficult-to-diagnose and/or potentially suspicious tissue samples (4), thereby saving time in the manual examination of histological samples.

Description

. DK 2021 70004 A1 A spectral imaging system arranged for dåidentifying benign tissue samples, and a method of using said system. In clinical medicine, tissue biopsy is a widely used technique for diagnosing and monitoring diseases, including different kinds of cancer. At many pathology departments around the world, a large number of small biopsy samples are examined and evaluated every day.

The samples typically go through a histopathological investigation, where the samples are sliced, placed on microscope slides and dyed for microscopic investigation.

In practice, the majority of samples turn out to be tumor free or benign. As an example can be mentioned that approximately 80% of the one million prostate biopsies performed in the US every year are benign; suggesting that pathologists are spending 80% of their time analysing benign tissue.

A further problem is that the conventionally histopathological methods are characterized by being highly labor intensive, time consuming and expensive. Accordingly, the investigation of the biopsies constitutes a bottleneck at the pathology departments, and practitioners and surgeons are therefore limited in the number of samples they can collect from a patient.

Furthermore, due to the large number of samples to be investigated test results are often delayed, resulting in undesirable set backs in relevant treatment and/or patients may experience prolonged anxiety awaiting the diagnosis based on the test results.

Increased focus has therefore been on computational pathology (digital pathology) which aims at providing quantitative diagnosis of pathological samples, reduction of inter-observer

, DK 2021 70004 A1 variability among pathologists, and saving time in the manual examination of histological samples.

It has in this respect long been known that the study of light propagation through biological tissues is useful to identify several diseases. As an example can be mentioned that for many years physicians applied a light bulb directly to the surface of the breast and observed the pattern of light transmission on the far side of examined breast. Usually, this approach could only detect the presence of a large mass, and it was not possible to detect if said mass was a fluid filled cysts, a benign tumor or a malignant tumor, see e.g. M. Cutler "Transillumination of the breast” Ann. Surg., 93(1), 223-234 (1931).

However, the properties of the interaction between light and biological tissue has motivated the use of technologies that exploit the obtained information to develop tools for diagnosis support. Special focus has in this respect been on applying fast spectroscopic methods to effectively identify malign tissue. For instance, development Multispectral imaging (MSI) and Hyperspectral imaging (HSI) to effectively analyze samples for malign tissue has in recent years received increased focus. Many studies have demonstrated the feasibility of using such techniques to detect cancer infected tissue and several comprehensive reviews have summarized the work, see e.g. "Medical hyperspectral imaging: a review”, Journal of Biomedical Optics 19(1), 010901 (January 2014). Reaches has primarily focused on methods for implementing real- time in vivo screening, especially in order to help surgeons distinguish malignant tissue from benign tissue during surgery. When cancer infected tissue is surgical removed success of the operation relies on ensuring that all malign tissue is removed.

2 DK 2021 70004 A1 The ability of MSI and HSI, to non-invasively, and in real time distinguish benign and malign tissue have increased success rates of cancer operations, see "In-Vivo and Ex-Vivo Tissue Analysis through Hyperspectral Imaging Techniques: Revealing the Invisible Features of Cancer”, Cancers 2019, 11, 756. However, even though it is an advantage to enable an in-vivo indication of whether or not the tissue is malign, the results is based solely on information obtained from light reflected from the surface of the examined tissue. Thus, if e.g. small island(s) of malign tissue is hidden below the tissue surface, the screening using MSI and HSI will not detect said islands, thereby providing a false negative. Recently the suggestion to use HSI for analysis of bulk pathological samples has been discussed, see "Tumor margin assessment of surgical tissue specimen of cancer patients using label-free hyperspectral imaging”, Proc. SPIE Int. Soc. Opt. Eng. 2017; 10054. Here it is demonstrated how a HSI system analyzing light, diffusely reflected from the surface of a sample, can be used to effectively distinguish malign and benign tissue regions. Again only data obtained from light reflected from the surface is evaluated, and accordingly false negatives may be obtained, especially if the samples are not sufficiently thin, i.e. in the micrometer range.

Finally, hyperspectral imaging using transillumination has been used for histopathological examination of excised tissue, see "Transillumination hyperspectral imaging for histopathological examination of excised tissue” F. Vasefi et al, Journal of Biomedical Optics 16(8), 086014 (August 2011). However, in this study it is essential that light passing though the sample pass through an angular filter array with a very small acceptance angle. Accordingly only light with an exit angle corresponding to the acceptable angle can be accepted by the filter and thereafter be captured on an image for further evaluation, while scattered light with exit angles outside the specific

2 DK 2021 70004 A1 acceptance angle are rejected. Thus, the data obtained using said method is based solely on a limited amount of light passing through the sample. Furthermore, the samples still needs to be slices into multiple thin slices and sandwiched between two glass slides prior to analysis, making this evaluation method time-consuming and expensive. Thus, is a first aspect of the present invention to provide spectral imaging system and a method of using said system, which in a fast and effective manner is capable of detecting if a tissue sample under investigation is obviously benign. It is a second aspect of the present invention to provide a spectral imaging system and a method of using said system arranged for evaluating the entire tissue sample directly, i.e. without any sample pre-treatment step. It is a third aspect according to the present invention to provide a spectral imaging system and a method of using said system that provides quantitative evaluation of tissue samples. It is a forth aspect according to the present invention to provide a spectral imaging system and a method of using said system that provides a reduction of inter-observer variability for at least a large number of tissue samples. It is a fifth aspect of the present invention to provide a spectral imaging system which is cost-effective and simple and easy to operate.

These and further aspect are achieved according to the present invention by providing a spectral imaging system arranged for detecting if a tissue sample is obviously benign or should proceed to histopathological investigation, said imaging system comprises

É DK 2021 70004 A1 — at least one sample container for accommodating a tissue sample, — at least one light source arranged for sending light through said tissue sample, — a light detecting device arranged for capturing spectroscopic data based on light transmitted though the tissue sample, and — a processing unit arranged for evaluating the captured spectroscopic data and automatically classify whether the tissue sample is obviously benign or should proceed to histopathological investigation. Use of the spectral imaging system according to the invention ensures that obviously benign tissue samples are sieved out, i.e. said samples need not be send to the pathology department for further investigation. In this way it is ensured that the pathologist can focus on the more difficult-to-diagnose and/or potentially suspicious tissue samples, thereby saving time in the manual examination of histological samples.

There are generally two major detection modes relating to spectral imaging depending on the incidence of light within the tissue: light reflection or light transmission. The spectral imaging system according to the present invention is arranged for detecting the light which is transmitted though the tissue sample, 1.e. it obtains one or more spectroscopic data based primarily on the photophysical processes scattering and absorption of the light which is send though and/or allowed to pass the tissue sample (trans-illumination).

Scattering of light occurs when there is a spatial variation of the reflective index within the tissue sample. Since a tissue sample comprises a large number of different biological components variations within the tissue may be useful for diagnostic purposes.

; DK 2021 70004 A1 The absorption of light involves the extraction of energy from light by molecules and can be used as an indication of the molecules’ response to light. This also provides information that also can be used for diagnostic purposes.

Thus, when a tissue sample is investigated using the spectral imaging system according to the invention, light will be delivered to the biological tissue and transmitted though said sample. As the light propagates through the tissue, the light will both be scattered, due to the inhomogenecity of the biological structure of said sample, and absorbed due to the presence of some biological components, e.g. hemoglobin, melanin and water. The absorption and scattering characteristics of the tissue will change not only in response to the wavelength used, but also in the presence of abnormalities and/or with the progression of diseases, and the absorbed and transmitted light measured at a number of different wavelengths from the tissue, therefore carry quantitative diagnostic information about tissue physiology, morphology, and composition. It is said information (spectroscopic data) that among others are used to determine if the tissue sample is obviously benign or potentially suspicious.

Within the scope of the present invention the term “obviously benign” relates to tissue samples in which spectroscopic data provides a very high degree of likelyhood/propability that the tissue sample is benign. In the present invention the threshold for determining if a tissue sample is obviously benign is at or above 95%, preferably at least 97%, and even more preferred at least 99%, when compared to a predefined set of tissue samples obtained from a traditional histopathological investigation, and therefore accordingly known to be benign or malignant. All tissue samples falling outside this range, is considered to be potential malignant and should proceed to a conventional histopathological investigation.

; DK 2021 70004 A1 The spectral imaging system comprises a processing unit arranged for evaluating the captured spectroscopic data and automatically classifies whether the tissue sample is obviously benign or should proceed to histopathological investigation.

Said processing unit may comprise an algorithm arranged for determining the likelihood of a sample being benign, and for classifying the tissue sample as being obviously benign if the likelihood of said tissue sample being benign is at or above a threshold defined by the user e.g. at least 95%, when compared to a predefined set of tissue samples obtained from a traditional Lhistopathological investigation, and therefore accordingly known to be benign or malignant.

The algorithm will typically be a machine learning modules (artificial deep learning neural network, e.g. a conventional neural network (CNN)) trained to recognize benign and malign patterns in tissue samples obtained from a traditional histopathological investigation. The algorithm preferably takes all the relevant wavelengths into account. E.g. either by using an ensemble network of several machine learning algorithms all being trained on images obtained on different wavelengths, or a CNN build with capacity to have enough input channels needed to train on all wavelengths at the same time. Once sufficiently trained, the machine learning modules can be used to interpret previously unseen spectroscopic data and determine if such spectroscopic data is obviously benign or should proceed to further histopathological investigation.

The machine learning module is considered sufficiently trained when the spectral imaging system has classified a predefined set of tissue samples of at least 1000 as being either benign or malignant, and even more preferred at least 10.000 tissue samples known as being either benign or malignant.

2 DK 2021 70004 A1 In a preferred embodiment the algorithm will be continuously trained over time, and thereby improve, as results from tissue samples that have undergone histopathological investigation are fed back into the system. In order to improve the algorithm even further tissue samples classified as being obviously benign may be randomly selected and send for further histopathological investigation, whereby said results also can be used to train the algorithm even further.

In a preferred embodiment the processing unit comprises data transmitting means arranged for informing the user of which tissue sample is considered potentially malignant, and should be investigated further. Such data transmitting means is well known in the art and will not be discussed further in the present application.

The use of imaging spectroscopy in the spectral imaging system according to the present invention provides the advantage that the system integrates conventional imaging and spectroscopy thereby providing both spatial and spectral data from the tissue sample under investigation.

The preferred imaging spectroscopy to be used in the present invention are multispectral imaging (MSI) and/or hyperspectral imaging (HSI), i.e. the spectral imaging system is a multispectral imaging (MSI) system or a hyperspectral imaging (HSI) system. MSI and HSI are optical spectroscopy imaging modalities, which directly measure the incoming radiance spectra of light, and the technologies are divided into MSI and HSI according to their spectral resolution, number of bands, width and contiguousness of bands, and include the acquisition of image data in both visible and non-visible bands. The invention is however not limited to the MSI and HSI technologies, and the imaging spectroscopy may in principle be any relevant spectroscopic method e.g. near infrared reflection and/or transmission spectroscopy, Infrared (IR) spectroscopic

. DK 2021 70004 A1 investigation as well as Raman spectroscopy. Furthermore, the spectral imaging technology could be a combination of one or more of the methods mentioned above, the only requirement being that the imaging spectroscopy allows transmission of light though the sample, and that spectroscopic data can be captured based on the light passing though said sample. MSI and HSI have the advantages that they are label-free imaging technologies, do not require a contrast agent for obtaining the desired spectroscopic data, and the inventors of the present invention has found that said technologies provides an objective assessment of whether a tissue sample is obviously benign or potentially malign.

In order to send light through the tissue sample, the spectral imaging system according to the invention comprises a light source arranged for sending light through the tissue sample. Said light source may in principal be any suitable light source, it is however preferred that the light source covers the full spectral window of interest, e.g. wavelength between 400 nm and 2500 nm, with a high brightness, and a flat, stable spectrum, and without any wavelength "gaps”.

A person skilled in the art will understand that the light source and light detecting device in a preferred embodiment are selected in dependence of each other, and optionally also in relation to the dimensions of the tissue sample under investigation. In this way it 1s ensured that the light detecting device is capable of capturing all relevant data from the light transmitted through the tissue sample. However, in a preferred embodiment the spectral density of the light source is at least 1 mV/nm, preferable at least 10 mw/nm, and even more preferred at least 100 mw/nm.

The light source may e.g. be a broad spectral device, or one or more lasers having a broadband spectrum covering visible, nIR,

1 DK 2021 70004 A1 and SWIR wavelengths from less than 400 nm to beyond 2500 nm.

It is however preferred to have a light source that does not generate any heat which potentially could influence the tissue sample under investigation, and it is therefore preferred that the light source is one or more light emitting diodes (LEDs) as such lamps does not generate heat during operation.

One preferred LED is USIHO LIR45A, obtainable from Ushio Opto Semiconductor, Inc. having a spectrum from 450 - 1000 nm with a spectra density of >lmW/nm.

If several LEDs are used in the spectral imaging system according to the invention, they may be selected from the Ushio EDC high power LED family with wavelength form 365 nm to more than 1000 nm The spectral imaging system according to the invention may in one embodiment comprise at least two light sources, preferably at least five light sources, and even more preferred at least ten light sources, and wherein each light source covers a separate sub-band, of the spectral coverage of the spectral imaging system.

In such an embodiment the spectroscopic data may be acquired by turning the respective light sources on sequentially.

The spectral imaging system according to the invention comprises a light detecting device arranged for capturing and/or acquiring spectroscopic information (data) from visible and/or non-visible bands from the light that via the light source propagates though the tissue sample.

Said light detecting device is preferably selected depending on the desired spectroscopic data that should be captured by said light detecting device.

In a preferred embodiment the light detecting device is a multispectral camera, a hyperspectral camera and/or a polarization camera arranged for detecting the degree of polarization and state of polarization for each pixel.

i.

DK 2021 70004 A1 The multispectral camera is preferably arranged for collecting data in a few and relatively non-contiguous wide spectral bands, typically measured in micrometers, or tens of micrometers.

These spectral bands are selected to collect intensity in specifically defined parts of the spectrum and are optimized for certain categories of information most evident in those bands.

The inventors of the present invention has found that a combination of spectral bands in both the UV (380 nm - 450 nm), Visible (450 nm - 700 nm) and Near Infrared (700 nm - 1100 nm) åis of particular interest e.g. when the light detecting device of the present invention is a multispectral camera.

When the light detecting device is a hyperspectral camera hundreds of wavelength bands for each pixel is captured, i.e. within a complete spectrum from 350 nm to 1100 nm.

As examples of light sources and light detecting devices, can be mentioned that in one preferred embodiment the light source has a continuous spectrum covering the range from 350 nm - 1000 nm, and the light detecting device is a multi- or hyperspectral camera matching said spectrum.

In an alternative embodiment the light source comprises a number light sources with different wavelengths in the area from 350 nm to 1100 nm, and the light detecting device is a conventional RGB (red, green, blue) camera or a B&W (black and white) camera.

A further embodiment the light source is a tunable light source where the wavelength can be varied continuously or discretely across the entire wavelength range of interest, i.e. between 350 nm and 1100 nm.

In this case the light detecting device may also be a conventional RGB or B&W camera.

One of the important advantages of both the MSI and HSI technique is that said technologies can acquire a large number of information for each pixels in the image, thereby detecting

> DK 2021 70004 A1 changes in the tissue sample that cannot be identified with traditionally gray or color imaging methods. Fach pixel of an HS/MS image represents the light measured by the camera at each specific wavelength, creating a set of light measurements which comprise the spectral signature of the tissue sample. This spectral signature can be understood as a fingerprint of each material in said tissue sample, and allow differentiation of elements/components/materials based on the spectral imaging data by using the processing unit of the spectral imaging system according to the invention.

The spectral imaging data can be visualized as a three- dimensional cube or a stack of multiple two-dimensional images.

The cube face is a function of the spatial coordinates and the depth is a function of wavelength, i.e. each picture represents a spectral sub-band of the spectroscopic systems spectral coverage.

A person skilled in the art will understand that the light detecting device need not be a multispectral camera or a hyperspectral camera, but could be a different kind of camera, e.g. a Si-based charge-coupled device (CCD) camera, and that the hyperspectral cube e.g. may be provided by combining the camera with a light dispersive element such as, but not limited to, a grating a series of optical filters or a Fabry Perot interferometer.

In order to obtain the relevant spectroscopic information the tissue sample is placed in a sample container, which is arranged such that the sample preferably is placed between the at least one light source and the light detecting device.

In order to ensure an optimal light intensity the sample container is preferably placed in close proximity to the at least one light source, preferably less than 1 cm. It is

1 DK 2021 70004 A1 furthermore preferred that the light source is aligned (the longitudinal axis of the light source is placed on the same axis as the longitudinal axis of the sample container) with the sample container such that the light source sends light directly into the tissue sample under investigation. Alternatively an optical lens or optical lens system can be used to collect the light from the light source and project it onto the sample.

It is further preferred that the spectral imaging system according to the invention comprises a calibration unit, arranged for providing a reference for the light transmitted though the tissue sample. Said reference comprises, but is not limited to the intensity, brightness, wavelengths of the light emitted from the at least one light source and captured by the light receiving unit. Said calibration unit may be an integrated part of the sample container, or arranged close to, e.g. less than 2 mm, from the sample container, such that the light passing thought the sample and the calibration unit are directly comparable. The calibration unit may be one or more aperture (s) allowing light to be transmitted directly (i.e. only through the air) from the light source to the light detecting device.

In order to provide a fast and effective way of evaluating if a tissue sample is obviously benign or should proceed to histopathological investigation, it is preferred that the spectral imaging system is arranged such that the tissue sample can be evaluated without any sample pretreatment step, i.e. the tissue sample can be placed in the sample container directly after removal/excision of the tissue sample.

Within the context of the present invention the term “pretreatment step” refers among other to slicing the tissue sample into thin slices, placing the tissue sample on or between glass-slides, dying said tissue sample, and/or any

> DK 2021 70004 A1 other ways a tissue sample conventionally needs to be processed before it can go through e.g. a histopathological investigation.

It is however important that the tissue sample has a size that can be accommodated in the sample container. The sample container 1s preferably dimensioned in order to match the characteristics of the light source. In this way it is ensured that sufficient light is transmitted through the tissue sample whereby the relevant spectroscopic data can be obtained/acquired from the tissue sample. It is however preferred that spectral imaging system according to the invention is arranged for evaluating tissue samples having a thickness of at least l0um, preferably at least 100um, even more preferably at least 1 mm, and even more preferred at least 5 mm and even more preferred at least 10 mm. Within the context of the present invention thickness is defined as the dimension/thickness of the sample taken along the direct axis from the at least one light source, to the light detecting device, when the tissue sample is placed in the sample container. The inventors of the present invention have found that when using the spectral imaging system according to the dimensions up to 1 cm' can be investigated with success, and without any pretreatment. The spectral imaging system according to the invention thereby provides the advantage that a tissue sample can be taken from a patient, and transferred directly to the sample container for investigation without any human interaction other than the interaction by the person, e.g. surgeon or doctor, removing the tissue sample from the site of interest. A person skilled in the art will understand that if the tissue sample is larger than what can be accommodated in the sample container, the i DK 2021 70004 A1 tissue sample may be cut into smaller pieces; however tissue samples are conventionally smaller than 1 cm’. Such a simple division is not considered to be a conventional pretreatment step. If a larger tissue sample is divided into smaller pieces, each piece may advantageously be processed for investigation in the spectral imaging system. In an alternative embodiment the tissue sample may be embedded in a transparent rigid medium such as e.g. paraffin and/or wax which provides a known scattering/absorption of light in said rigid medium. The inventors of the present invention have found that by embedding the tissue sample in such a transparent rigid medium the absorption and scattering properties of light passing through the tissue sample are “controlled”, i.e. the spectroscopic data obtained from such embedded tissue samples may be more reliable. Embedded tissues samples further have the advantage that a reference may be set for lights transmitted though the transparent rigid medium, i.e. without passing through any tissue.

Furthermore, embedded tissue samples may be relevant if the tissue sample needs to be preserved for later evaluation, if a reevaluation 1s desired or for comparative or learning purposes. It is important to stress that the tissue samples are not pre-treated prior to embedding in the transparent rigid medium, i.e. each sample is directly placed/cast in said medium. Embedding tissue samples in e.g. paraffin is a step in a conventional histological investigation, as this enables the pathology to slice the tissue ample in micrometer thin slices.

However, the spectroscopic system according to the present invention ensures that the embedded tissue samples can be investigated directed, i.e. without the additional slicing, conventional required.

In order to prevent any interference in the spectroscopic data from surrounding light, i.e. light not passing though the

16 DK 2021 70004 A1 tissue sample under investigations the spectral imaging system according to the invention may comprise an optical aperture arranged for discarding surrounding light, i.e. light not passing though the tissue sample.

This may for instance be relevant if/when light passes though a part of the sample container not occupied by the tissue sample, e.g. due to irregularities in the tissue samples form/shape, but wherein such surrounding light still is captured by the light detecting device, accordingly influencing the spectroscopic data.

Thus,

in a preferred embodiment the optical aperture has a dimension which is smaller than the tissue sample under investigation, i.e. light is only allowed to pass though the tissue sample, such that the light detection unit only can capture spectroscopic data from the light that is passed though the tissue sample, thereby effectively preventing any interferences with the surrounding light.

Said optical aperture may either be a part of the sample container, be a screen and/or diaphragm placed between the light source and the sample container, i.e. in the light path from the light source to the tissue sample.

The optical aperture may in a preferred embodiment have a fixed dimension, but in an alternative embodiment the dimensions of the optical aperture can be varied depending on the size of the tissue sample.

In a preferred embodiment the optimal aperture is provided by an iris diaphragm which is placed between the light source and the sample container.

The opening of the iris diaphragm can easily be adjusted e.g. depending on the dimensions of the tissue sample, to screen away all light from the at least one light source not passing through the sample.

In order to also screen the light detecting device from external light, e.g. day-light or lamps in the examining room, it is preferred that at least the central parts of the spectral imaging system according to the invention, i.e. the light

- DK 2021 70004 A1 source, sample container and light detecting device is placed in a housing arranged for blocking out external light, thereby in a fast and effective way preventing external light to interfere with the spectroscopic data.

In order to ensure that the tissue sample in the sample container is aligned with the light source, the optical aperture and/or the light detection unit the sample container may be placed on a motorized XY or XYZ stage, which ensures that the sample can be correctly positioned. It is however preferred that the processing unit is arranged for detecting the location of the tissue sample in the sample container, and adjust the position of the sample container automatically if the placement is not optimal.

The spectral imaging system according to the invention may in a further embodiment be arranged for capturing more than one set of spectroscopic data for each tissue sample, e.g. when the tissue sample is placed in different positions relative to the light source, the optical aperture and/or the light detection unit using the motorized XY or XYZ stage. In one further embodiment the spectral imaging system according to the invention further comprises at least one polarization filter. Said polarization filter may either be a first polarization filter arranged for polarizing the light from the light source before reaching the tissue sample or a second polarization filter for controlling the polarization state of light reaching the light detecting device. It is however preferred that the spectral imaging system according to the invention comprises both the first and a second polarization filter, in order to determine the deterioration of polarization of the light when going through the sample. The polarizing filters are typically arranged for cross polarization microscopy.

Lo DK 2021 70004 A1 The filters are arranged with the first polarization filter placed between the at least one light source and the sample container, and the second polarization filter placed between the sample container and the light sensitive device.

In one preferred embodiment the first and second polarization filter are aligned with mutually orthogonal polarization axes. A person skilled in the art will based on the present application understand that the spectral imaging system may utilize a polarization camera instead of, or in combination with the first and/or second polarization filter. The polarization camera may e.g. be a XCG-CP510 polarized camera obtainable from Sony. Said camera has a polarization filter deposited directly on the pixels in the camera.

The spectral imaging system may further comprise further units and/or devices arranged for optimizing the system. As an example can be mentioned that the system may comprise one or more diffuser plate(s) arranged for ensuring that the light transmitted from the light source is homogenized and free of spatial variation and/or one or more collimating lens (es) arranged for ensuring that the light transmitted though the tissue sample is accurately aligning/parallel with the camera. Use of a collimating lens further ensures that the light has minimal spread as it propagates into the tissue sample. The present invention also relates to an automatic tissue sample system comprising the spectral imaging system described above. The tissue sample system may comprise a number of sample containers arranged to move along a process line and wherein each sample container will be investigated individually. The number of sample containers may e.g. be arranged in a row, in an array or in a circle, the only requirement being that each sample container is arranged for accommodating a tissue sample which can be individually investigated. In this way the surgeon/doctor can easily place a larger number of tissue

19 DK 2021 70004 A1 samples in the respective samples containers and either continuously or simultaneously receive information relating to a large number of sample.

The invention also relates to a method of using the spectral imaging system according to the invention, and wherein said method comprises the following sequential steps: - placing a tissue sample in the sample container, - sending light through the tissue sample, - collecting spectroscopic data, and - determining if the tissue sample is obviously benign by comparing the photophysical data with a predefined set of tissue samples obtained from a traditional histopathological investigation.

Thus, using the spectral imaging system according to the invention a tissue sample can automatically and in less than a few minutes determine if the tissue sample under investigation is obviously benign or is potentially suspicions, and accordingly if said sample needs to be investigated further. This fast and efficient detection provides an obvious advantage for a large number of applications. First of all, the number of tissue samples to be manually investigated is significantly reduced since the pathology department only has to focus on the suspicious samples. This will not only save time and cost for the investigation of the samples, but the patient will also receive a diagnose earlier than using the conventional methods, thereby ensuring a fast initiation of the relevant treatment.

Furthermore, the system according to the invention has an obviously advantage for cancer margin assessment during surgery. A complete resection of the tumor is the single most important predictor of patient survival for almost all solid cancers. However a larger number of patients leave the operating room without a complete resection due to positive or close margins.

Using the spectral imaging system according to the invention, tissues samples from the margin can easily be investigated and may accordingly be used to navigate cancer resection, thereby aiding in improve the number of complete resections.

The invention will be explained in greater detail below, describing only exemplary embodiments of the spectral imaging system with reference to the drawing, in which

Fig. 1 shows schematically a preferred embodiment of the spectral imaging system according to the invention, Fig. 2 shows schematically a second embodiment of the measuring unit of fig. 1, Fig. 3 shows schematically a third embodiment of the measuring unit of fig. 1, Fig. 4 shows schematically an iris diaphragm of the measuring unit of fig. 3, Fig. 5 shows schematically a fourth embodiment of the measuring unit of fig. 1,

Fig. 6 shows schematically a fifth embodiment of the measuring unit of fig. 1, Fig. 7 shows schematically a sixth embodiment of the measuring unit of fig. 1, Fig. 8 is a flow diagram showing how the algorithm in the form of an artificial deep learning neural network is trained, Fig. 9 is a flow diagram showing how the algorithm can be used to interpret previously unseen spectroscopic data,

> DK 2021 70004 A1 Fig. 10 shows a spectral imaging system according to the invention in more details. Fig. 11 shows a number of images captured using a spectral imaging system according to the invention, and Fig. 12 shows schematically the machine learning architecture used in the examples.

The invention will be described below with the assumption that the spectral imaging system a multispectral imaging (MSI) system or a hyperspectral imaging (HSI) system. However, this assumption is not to be construed as limiting, as the system also could be based on e.g. Infrared (IR) spectroscopy or Raman spectroscopy.

Fig. 1 shows schematically a first embodiment of a spectral imaging system 1 according to the invention. The system 1 comprises a measuring unit 2, comprising a sample container 3 for accommodating a tissue sample 4, a light source 5 arranged for sending light 6 through said tissue sample, a light detecting device 7 for capturing spectroscopic data 8 based on light transmitted 9 though the tissue sample 4 and a processing unit 10 arranged for evaluating the captured spectroscopic data 8 and automatically classify whether the tissue sample 4 is obviously benign or should proceed to histopathological investigation.

Depending on the technology used, the light detecting device 7 is a multispectral camera or a hyperspectral camera. The multispectral camera is preferably arranged for collecting data in the UV (380nm - 450nm), visible (450nm- 700nm) and Near Infrared (700nm- 1100nm) spectra, and the hyperspectral camera is arranged for collecting a large number, i.e. at least 20,

> DK 2021 70004 A1 wavelength bands for each pixel, and preferably within a complete spectrum from 350 nm to 1100 nm.

The captured spectral imaging data 8 may via a Charged Coupled Device (CCD) of the camera visualized as a three-dimensional cube or a stack of multiple two-dimensional images, and said data is analyzed in the processing unit 10. A CCD is a sensor arranged for capturing light and converts it to digital data that is recorded by the camera.

Said processing unit comprises computer 11 with an algorithm 12 arranged for determining the likelihood of a tissue sample 4 being benign, and for classifying the tissue sample as being obviously benign if the likelihood of said tissue sample being benign is larger than a threshold defined by the user e.g. 95%, when compared to a predefined set of tissue samples obtained from a traditional histopathological investigation, and therefore accordingly known to be benign or malignant.

In the embodiment of fig. 1 the algorithm 12 is an artificial deep learning neural network trained to recognize benign and malign patterns in tissue samples obtained from a traditional histopathological investigation.

In order to prevent any interference in the spectroscopic data 8 from surrounding light the sample container 3 comprises an optical aperture 13 which is slightly smaller than the tissue sample 4 under investigation and which thereby ensures that any surrounding light, i.e. light not passing through the tissue sample is discarded.

In this way light can only pass though the tissue sample, effectively preventing any interference with the surrounding light in the captured data.

Fig. 2 shows a second embodiment 2a of the measuring unit shown in fig. 1. Said embodiment corresponds basically to the embodiment shown in fig. 1 but the sample container 3 incorporates a calibration unit 14, arranged for providing a

2 DK 2021 70004 A1 reference for the light transmitted from the light source and is captured at the light detecting unit, e.g. the amount of transmitted light and/or the wavelengths of the collected data. The calibration unit 14 in the embodiment shown is a single aperture 15 allowing light to be transmitted directly (i.e. only through the air) from the light source to the camera 7. Fig. 3 shows schematically a third embodiment 2b of the measuring unit of the invention. Said embodiment corresponds to the embodiment of fig. 1, but in fig. 3 the optical aperture 13 is not part of the sample container 3, but is an adjustable iris diaphragm 16 placed between the light source 5 and the sample container 3, i.e. in the light path 6 from the light source to the tissue sample. The aperture 13’ in the iris diaphragm 16 can be adjusted stepless to correspond to the dimensions of the tissue sample 4 such that any light not transmitted though the tissue sample is prevented from reaching the camera 7. In fig. 4 the iris diaphragm is shown with three different sizes of the adjustable aperture, and the aperture 13" of fig. 3 corresponds to the dimensions shown in fig. 4b. The embodiment of fig. 3 further has the advantage that the sample container is placed on a motorized XYZ stage 17, which ensures that the sample can be correctly positioned relative to the light source 5 and/or the adjustable aperture 13’ of the iris diaphragm 16. The embodiment of fig. 3 may in a fourth embodiment 2c shown in fig. 5 also comprise a calibration unit 14. Said calibration unit is described for fig. 2 and the same principals apply for this embodiment. In fig. 5 the aperture 13" of the iris diaphragm has been adjusted to correspond to the dimensions of the aperture shown in fig. 4a.

Fig. 6 shows a fifth embodiment 2d of the measuring unit of fig. 1. This embodiment corresponds to the embodiment of fig. 5

> DK 2021 70004 A1 but further comprises a first polarization filter 18 placed between the at least one light source and the sample container, and the second polarization filter 19 placed between the sample container and the light sensitive device. The first polarization filter 18 is arranged for polarizing the light © from the light source 5 before reaching the tissue sample 4 and the second polarization filter 19 is arranged for controlling the polarization state of light 9 reaching the light detecting device 7, whereby the system 1 according to the invention can determine the deterioration of polarization of the light when going through the sample. In a further embodiment 2e corresponding to the embodiment of fig. 6, the tissue sample 4 is embedded in a transparent rigid medium 20 such as paraffin. This embodiment is shown in fig. 7, The paraffin ensures that the absorption and scattering properties of light passing through the tissue sample are “controlled”, i.e. the spectroscopic data 8 obtained from such embedded tissue samples may be more reliable.

Fig. 8 is a flow diagram showing how the algorithm in the form of an artificial deep learning neural network is trained to recognize benign and malign patterns in tissue samples. Said tissue samples are obtained from a traditional histopathological investigation in which a large number of hyperspectral images and their matching classifications obtained through traditional histopathological methods are matched. The machine learning module is considered sufficiently trained when the spectral imaging system has classified a predefined set of tissue samples of at least 1000 as being either benign or malignant, and even more preferred at least

10.000 tissue samples known as being either benign or malignant.

Once sufficiently trained, the machine learning modules can be used to interpret previously unseen spectroscopic data and

J DK 2021 70004 A1 determine 1f such spectroscopic data is obviously benign or should proceed to further histopathological investigation. This is shown in fig. 9, in which classification of a tissue sample is performed by the trained deep learning neural network, which returns a classification of whether the sample is obviously benign, or whether the sample requires further investigation. An example of a spectral imaging system according to the invention is shown in more details in fig. 10. The system comprises a light source 5, consisting of eight high power light emitting diodes (LEDS) 21 placed on a rotating wheel 22. During use the wheel 22 will rotate and sequentially position each of the LEDs 21 underneath a collimating lens 23. Said lens 23 will collimates the light transmitted from the LED and project it onto a diffuser plate 24 located underneath the sample container 3. The purpose of the diffuser plate is to ensure that light from the LEDs is homogenized and free of spatial variation. When a sample 4 placed in the sample container 3 and the light source 5 is turned on, light will be transmitted thought the tissue sample and be collected in the camera lens 25. The camera lens is chosen such that it is achromatic and with a field of view large enough to project an image of the entire sample onto a CDD chip of the camera 7.

The captured spectral imaging data 8 is analyzed in the processing unit 10 wherein the investigated tissue sample will be classified as being obviously benign if the likelihood of said tissue sample being benign is at or larger than a threshold defined by the user e.g. 95%, when compared to a predefined set of tissue samples obtained from a traditional histopathological investigation, and therefore accordingly known to be benign or malignant.

Accordingly the spectral imaging system according to the invention provides a fast and efficient way of screening a large number of samples, whereby the number of tissue samples

Je DK 2021 70004 A1 to be manually investigated is significantly reduced since the pathology department only has to focus on the suspicious samples.

EXAMPLE In order to evaluate the spectral imaging system according to the invention the following experiment were performed, using a spectral imaging system corresponding to the system shown in fig. 10.

The light source comprised eight high power light emitting diodes (LEDs) with center wavelength spanning from 395 nm to 940 nm. The diodes consisted of two UV diodes with a center wavelength (CW) of 395 nm and 425 nm and a full width half maximum spectral width (FWHM) of 20 nm, two diodes with CW of 525 and 600nm and a FWHM of 20 nm and three IR LED with CWs 730 nm, 850 nm and 940 nm respectively with a FWHM of 30 nm. Finally a diode emitting white light covering the entire range from 400-700 nm is also include in the system.

The LEDs are placed on a rotating wheel, arranged with an angle of 40 deg. between each diode, whereby the wheel will rotate during use e.g. by means of a motor, such that each of the LEDs, in the order shown in fig. 11b to 11i, is placed directly under a tissue sample accommodated in the sample container. The order of the LEDs must not to be construed as limiting, and the order of the LEDs could in principal be any order. On the opposite side of the LEDs a RGB CCD camera is placed, such that when a tissue sample is placed in the sample container, light will be transmitted through the tissue sample and collected in a camera lens placed in front of the camera. The camera has a 1/3” CCD chip and is equipped with a compact 25 mm lens from Tameron having a horizontal field view of 11 deg. and a vertical field of view of 8.2 deg. At a working distance of 100 mm this yield an image size of about 15x20 mm,

>; DK 2021 70004 A1 thereby ensuring that an image of the entire sample can be projected onto the CDD chip of the camera. In order to collimate the light transmitted from the respective LED, light will be transmitted first though a collimating lens and then though a diffuser plate before reaching the tissue sample. The lens is a plano convex lens from Thorlabs with a focal length of 25,4mm and diameter of 1” arranged in such a way as to collimate the output of the LED onto a sheet of Teflon acting as a diffuser plate.

The multispectral imaging system is used by sequentially turning the wheel and recording one image for each of the diodes for each sample. The collection of the recorded images for one sample is referred to as the stack of images. Each diode transmitted light though the tissue sample for about two second, which was enough time for the camera to capture an image.

In the present experiments the multispectral imaging system did not comprise an optical aperture since the experiment used samples embedded in paraffin. In this configuration light can only get from the LED side to the camera side by passing through the paraffin which has similar optical properties as the paraffin eliminating the need for an optical aperture.

In order to provide a normal reflection image of the tissue sample, a white LED with a continuous spectrum covering the range from 400-700nm is placed on the same side of the camera arranged to facilitate the recording of a normal reflection image of the sample, whereby it is possible to compare the obtained/captured data to with the conventional reflection technology normally used for tissue samples.

50 samples with a confirmed diagnosis of being either benign or malign were obtained by embedding the samples in paraffin (in a

J DK 2021 70004 A1 conventional manner) and thereafter remove the top layer by a planer. Of these 25 of the tissue samples were confirmed to be benign and 25 confirmed to be malign. The paraffin blocks were approximately 3 cm long and 2 cm wide and varied in thickness from a few to about 5 mm. The spectral imaging system was then used to analyze the samples, and for each sample a stack of two-dimensional images, one image for each LED on the rotating wheel, and one normal reflection image for the white LED placed beside the camera, were captured. An example of the reflection images obtained is shown in fig. 1lla, and the images obtained for each diode is shown in fig. 11b - 111.

The captured images were then used to train a machine learning algorithm to distinguish between malign and benign samples. The machine learning part of the experiment was implemented by sorting the image stacks according the overall labels "Malign" and "Benign”. Each image stack was tiled into smaller fragments e.g. of 224 x 224 pixels. Each tile from a sample is then labeled malign or benign according to the label of the whole sample.

In this experiment the samples were then subsequently split into a training set consisting of benign and malign samples and a validation set consisting of approximately 20% of the entire sample set.

The machine learning architecture used is known as an ensemble network, as illustrated in fig. 12. For training the network the method of back propagation is used The implementation used is to train a network, in this case a ResNet50 neural net (Convolutional Neural Network), for each wavelength individually. For each sample, each tile is fed into

29 DK 2021 70004 A1 the neural network system one by one.

The system is constructed in such a way that each of the images of the tile stack is fed into their own ResNet50 network.

Output from all of the resulting 9 networks are then feed into a so-called, fully connected neural network, who then predicts the final outcome for each tile, respectively "Malign” or "Benign”. For each neural network, which is predicting on each wavelength images/tiles the loss (difference between the label and the output of the neural network) is used to train the specific network using the well-known Adam optimizer function (Adam: A Method for Stochastic Optimization.

Diederik P.

Kingma and Jimmy Ba; https://arxiv.org/abs/1412.6980). The effect of this is that the network learns to get closer and closer to the label.

When applying this, the inventors have used a dropout functionality of around 50% which means that in order to avoid overfitting, 50% of the weights was not trained during backpropagation.

Number of epochs and which learning rate to apply can change depending of the image samples and the sample size.

In this experiment the inventors used the learning rate of 0.0001, because in this case it gave the best results measure on the ability to predict benign and malign tiles on, for the software, unknown samples.

After training, the spectral imaging system was tested by feeding each of the tiles from the validation set into the network.

By making sure that the same tile (determined by position) for each image and each wavelength were fed correctly, the inventors could afterwards map out which tiles of the sample the machine learning software predicts is either "Malign” or “Benign”. An example of an image obtained this way is shown in fig. 11k.

Here each of the tiles identified as malign are marked with a red square (unbroken line), each of the benign samples with a green square (dotted line) and each

30 DK 2021 70004 A1 of the tiles where the system was not able with sufficient confidence is left without a square. In this example 100% of tiles from the validation set were classified correctly as either benign or malign.

A person skilled in the art will based on the present invention understand that the images may be prepared for subsequent Neural network training in different ways. As an example can be mentioned, that a principal component analysis (PCA) may be used for removing or reducing the duplication or redundancy in the obtained multispectral images and for compressing all of the information that is contained in the original multispectral images into their principal components. An example of such a PCA image is shown in fig. 11j and said image may be used instead of the original data for image analysis and interpretation. Modifications and combinations of the above principles and designs are foreseen within the scope of the present invention.

Claims

31 DK 2021 70004 A1 Claims,

1. A spectral imaging system (1) arranged for determining if a tissue sample is benign, said imaging system (IL) comprises — at least one sample container (3) for accommodating a tissue sample (4), — at least one light source (5) arranged for sending light through said tissue sample (4), — a light detecting device (7) arranged for capturing spectroscopic data based on light transmitted through the tissue sample (4), and — a processing unit (10) arranged for evaluating the captured spectroscopic data and automatically classify whether the tissue sample (4) is obviously benign or should proceed to histopathological investigation.

2. The spectral imaging system (1) according to claim 1, wherein the processing unit (10) comprises an algorithm arranged for determining the likelihood of a tissue sample (4) being benign, and for classifying the tissue sample (4) as being obviously benign if the likelihood of said sample is benign is at or above a threshold defined by the user e.g. 95%, when compared to a predefined set of tissue samples (4) obtained from a traditional histopathological investigation.

3. The spectral imaging system (1) according to claim 2, wherein the algorithm is a machine learning module trained to recognize benign and malign patterns in the predefined set of tissue samples (4).

4. The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system

3 DK 2021 70004 A1 (1) is a multispectral åimaging (MSI) system or a hyperspectral imaging (HSI) system.

5. The spectral imaging system (1) according to any of the preceding claims, wherein the at least one light source (5) covers the full spectral window of interest, such as between 350 nm and 1100 nm.

6. The spectral imaging system (1) according to any of the preceding claims, wherein the at least one light source (5) is a light emitting diode (LED).

7. The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system (1) comprises at least two light sources (5), preferably at least five light sources (5), and even more preferred at least eight light sources (5), and wherein each light source (5) cover a separate sub-band, and wherein the spectroscopic data is acquired by turning the respective light sources (5) on sequentially.

8. The spectral imaging system (1) according to any of the preceding claims, wherein the light detecting device (7) is a multispectral camera.

9. The spectral imaging system (1) according to claim 8, wherein the multispectral camera is arranged for collecting spectroscopic data in a few and relatively non-contiguous wide spectral bands, such as in areas between 380 nm and 450 nm, between 450 nm and 700 nm and/or between 700 nm and 1100 nm.

10. The spectral imaging system (1) according to any of the preceding claims 1 - 7, wherein the light detecting (7) device is a hyperspectral camera.

DK 2021 70004 A1

11. The spectral imaging system (1) according to claim 10, wherein the hyperspectral camera is arranged for collecting spectroscopic data in a number of wavelength bands for each pixel within a complete spectrum from 350 nm to 1100 nm.

12. The spectral imaging system (1) according to any of the preceding «claims, wherein the light source (5) is aligned with the sample container (3) and the light detecting device (7).

13. The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system comprises a calibration unit (14), arranged for providing a reference for the light transmitted though the tissue sample (4).

14. The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system is arranged for evaluating tissue samples (4) having a thickness of at least 1 mm, and even more preferred at least 5 mm, preferably up to 10 mm, and wherein the thickness of the sample is taken along the direct axis from the at least one light source (5), to the light detecting device (7), when the tissue sample (4) dis placed in the sample container (3).

15. The spectral imaging system (1) according to any of the preceding claims, wherein the tissue sample (4) is embedded in a transparent rigid medium (20) such as paraffin and/or wax which provides a known scattering/absorption of light when transmitted though said rigid medium (20).

16. The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system

32 DK 2021 70004 A1 comprises an optical aperture (13) arranged for discarding surrounding light.

17. The spectral imaging system (1) according to claim 16, wherein the optical aperture (13) has a dimension which is smaller than a tissue sample (4) in the sample container (3).

18. The spectral imaging system (1) according to any of the claims 16 and 17, wherein the optical aperture (13) is integrated with the sample container (3), or is a screen and/or diaphragm placed between the at least one light source (5) and the sample container (3).

19. The spectral imaging system (1) according to any of the claims 16 - 18, wherein the dimensions of the optical aperture (13) is fixed, or varied depending on the size of the tissue sample (4) placed in the sample container (3).

20. The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system comprises at least one polarization filter (18,19), e.g. a first polarization filter (18) arranged for polarizing the light from the light source (5) before reaching the tissue sample (4) and/or a second polarization filter (19) arranged for controlling the polarization state of light reaching the light detecting device (7).

21. An automatic tissue sample system comprising the spectral imaging system (1) according to any of the claims 1 - 20.

22. The automatic tissue sample system according to claim 21, wherein said automatic tissue sample system comprises a number of sample containers (3) arranged for

35 DK 2021 70004 A1 moving along a process line and wherein each tissue sample in a sample container (3) will be investigated individually.

23. A method of determining if a tissue sample (4) is benign by using the spectral imaging system according any of the claims 1 - 20 or the automatic tissue sample system according to claim 21 or 22, and wherein said method comprises the following sequential steps: - placing a tissue sample in the sample container (3), - sending light through the tissue sample (4), - collecting spectroscopic data, and - determining if the tissue sample (4) is obviously benign by comparing the spectroscopic data with a predefined set of tissue samples (4) obtained from a traditional histopathological investigation.

24. The method according to claim 23, wherein the tissue sample (4) is determined as being obviously benign if the likelihood of said sample is benign is at or above a threshold defined by the user e.g. 95%, when compared to a predefined set of tissue samples (4) obtained from a traditional histopathological investigation.

25. The method according to claim 23 or 24, wherein the spectroscopic data from the tissue sample (4) can be collected without any sample pre-treatment step, such as slicing the tissue sample (4) into thin slices, placing the tissue sample (4) on a glass-slide, between glass- slides, adding contrast agents to the sample and/or dying said tissue sample (4).