HK1171508B - Multi-modality contrast and brightfield context rendering for enhanced pathology determination and multi-analyte detection in tissue - Google Patents

Description

Multi-modal contrast and bright field background rendering for enhanced pathology determination and multi-analyte detection of tissue

CROSS-REFERENCE TO RELATED APPLICATIONS

The benefit of U.S. provisional application No.61,250,809 filed on 12/10/2009 and U.S. provisional application No.61/278,936 filed on 13/10/2009, which are hereby incorporated by reference in their entireties, is claimed.

Technical Field

The present disclosure relates to methods of providing contrast of tissue sections for pathology determination.

Background

Microscopic clinical examination of routinely histologically stained tissue sections can be used to assess morphological patterns and tissue structure of diagnostic significance. A skilled physician can view such histologically stained tissue sections for diagnosis and to design and evaluate treatments. The contrast of the structures provided by such images using typical staining is familiar and allows the physician to focus on interpreting the features and abnormalities of anatomical and morphological tissue sections rather than trying to explain how the staining method appears to be related to her medical training and experience.

Additional tissue imaging techniques are being developed that hopefully enhance the relevant diagnostic information available to physicians regarding valuable biopsy material and archived tissue samples. For example, fluorescence microscopy can be used to detect specialized molecular markers, but fluorescence-based images often lack familiar structural and anatomical background information found in tissues stained with hematoxylin and eosin (H & E) and viewed using bright field microscopy.

While fluorescence-based images provide useful molecular information for the confirmation and characterization of disease states, conventional histologically stained sections are still necessary for pathological determination of tissues. Typically, serial tissue sections through the sample must be prepared and evaluated. Generally, serial sections include conventionally H & E stained sections and specially stained section(s) for diagnostic molecular markers. Comparing successive slices not only increases the cost and time necessary for assessment, it may be difficult or impossible to correlate features found in one slice with features found in another slice. Serial sections may also be lost or destroyed in the staining process pipeline.

Disclosure of Invention

The techniques described herein provide methods and apparatus that use multi-modal contrast to generate complementary contrast components segmented and displayed in a manner relevant to physician training and experience for pathology analysis. Such complementary contrast modalities may be streamed to show changes that allow navigation, focus, and magnification of tissue structures. The tissue section may contain one or more probes that target a particular molecule or chemical of interest. Color contrast of tissue structures is provided that is comparable to that produced using conventional color absorbing histological stains, such as hematoxylin and eosin stains (hereinafter "H & E"). The images produced by one or more of the disclosed methods may also include features that are visualized using additional markers and optical or chemical contrast modalities. In general, the relationship of the characteristics of the differential markers between different tissue sections becomes unnecessary. The images are presented in a digitally reproduced bright field background of color to provide an image appearance that is comparable to that produced in a conventional histology slide that has been stained to reveal the same structural features.

In some disclosed examples of multi-modal contrast, the contrast is derived from the fluorescence labeling and refractive index properties of tissues prepared specifically for the labeling of specific molecules. These examples demonstrate the complementary combination of transmitted light dark field refractive contrast imaging with simultaneous incident light fluorescence imaging of nuclear counterstaining, and interrogation of multiplex molecular probes. The corresponding correlation images are acquired simultaneously (in parallel) or sequentially (consecutively). In some examples, the illumination and detection wavelengths used to produce contrast on unstained or stained tissue can be tightly controlled to facilitate unambiguous segmentation and prevent interference with multiplexed probes. Molecular specific probes for protein antigens, mRNA expression or gene rearrangement of DNA can be positioned overlapping the sample structure. This contrast is associated with changes in refractive index due to tissue structures preserved and resolved using a particular histological process. In a typical example, the disclosed method provides image contrast based on refractive index changes of tissue parts (moieties) in combination with fluorescent counterstaining to provide a color pathological background for molecule specific multiplex probes. Examples include formalin-fixed, paraffin-embedded tissues and frozen tissues. The refractive index contrast may be derived directly from the refractive or scattering properties of the probe portion and tissue, or from the magnitude of the phase shift or the rate of change of the phase shift gradient.

Some disclosed methods include exposing a fluorescently stained sample to a selected excitation beam to produce fluorescence by the fluorescent stain, and producing a corresponding fluorescence image. The same sample is also exposed to high NA annular dark field illumination and a corresponding dark field image is produced representing the change in refractive index and light scattering moieties. In some examples, the fluorescence excitation beam exposure and the dark field refraction illumination field exposure are applied simultaneously, and complementary images are obtained in parallel. In other examples, the fluorescence image and the dark-field refraction contrast image are recorded sequentially. An imaging device according to an example includes a multi-modal optical system configured to produce a transmitted dark field illumination field and an incident illumination fluorescence excitation optical system. These subsystems are configured to produce a plurality of complementary images that can be combined for correlation analysis: a refraction contrast image based on the properties of the prepared tissue, a fluorescence image of nuclear counterstaining, and a plurality of fluorescence images representing various molecular markers that can be segmented by emission wavelength. At least one image capture device is coupled to receive the dark field and fluorescence images, and an image processor is configured to record and process the dark field image and the fluorescence image and produce a combined image.

The computer-readable storage medium includes computer-executable instructions for receiving images associated with a plurality of modalities of contrast (which are associated with preparing a common portion of a sample slice for pathology examination), and overlaying the plurality of modalities of contrast to produce a combined image.

In some examples, the image processor is configured to produce a pseudo-color bright field representation of the combined image based on the recorded refraction contrast dark field image and the fluorescence image. The fluorescence image and dark-field refraction image are separately stained, combined and inverted (inverted) to produce a combined color image in a clear bright-field background with contrast associated with conventional staining. Specific color mapping, which facilitates direct physician interpretation, is applied to refraction contrast images, fluorescent nuclear counterstains, and specific fluorescent probes. These images are then combined to produce a combined color recorded image in bright field reproduction. In some examples, the color mapping is based on a quantitative measurement of human perception of a preferred color for pathology determination associated with at least one color absorbing histological stain, such as an eosin stain. In other examples, a color lookup table is applied to the fluorescence image, wherein the color mapping is associated with at least one contrasting color-absorbing histological stain, such as a hematoxylin stain. In some examples, a color lookup table is applied to the dark-field refraction image and the fluorescence counterstain image to produce an image having inverted contrast associated with complementary hue, inverted values, and inverted saturation compared to those encountered in ideal hematoxylin and eosin staining. In other examples, the optically imaged sample is prepared for further imaging using mass spectrometry to provide molecular mapping (mapping).

These and other features and aspects of the disclosed technology are described below with reference to the drawings.

Drawings

FIG. 1 is a schematic diagram of a representative imaging system that provides both refraction contrast dark field and fluorescence based images.

Fig. 2 is a schematic block diagram of a method of processing and combining recorded dark-field and fluorescence-stain-based images.

Fig. 3 is a schematic block diagram of a representative method for generating a sample image from multiple modalities of contrast, where contrast corresponds to contrast used in pathology assays with hematoxylin and eosin (H & E) staining.

Fig. 4A is an image of representative conventional H & E staining of human prostate sections.

Fig. 4B is a multi-modal contrast image of a human prostate section based on a combination of dark-field refraction images and fluorescence counterstain images reproduced in a bright-field background.

Fig. 5A-5B are dual-illumination multi-modal contrast (refractive contrast and fluorescence) images recorded using a monochromatic CCD with sequential exposures using an interference filter to select the dark field wavelength (fig. 5B) of the blue DAPI fluorescence wavelength (fig. 5A) or longer wavelength transmission.

Fig. 5C is a pseudo color image obtained by applying an inverted color look-up table of a pseudo color to the image of fig. 5A-5B and the image with the inverted color added.

Fig. 5D is a bright field reproduction of a pseudo color of an image corresponding to the image of fig. 5C after inversion of the mapped color space.

Fig. 6 is a brightfield background reproduction image by overlaying the positioning of the quantum dot fluorescent probes with peak emission wavelengths of 565nm and 655nm from DAPI counterstained formalin-fixed paraffin-embedded samples (also imaged for refractive contrast).

Fig. 7 includes other example images for multi-modality imaging for bright field background display. The dark field refraction contrast image and the DAPI fluorescence image of the DAPI counterstained tonsil section were obtained, overlaid, and reproduced as the color bright field image shown in fig. 7(1a-3 a). Protein-specific immunoprobes (localized in fluorescence using quantum dots with peak emissions at 565nm (for CD20 antigen) and 655nm (for Ki67 antigen)) were applied to DAPI counterstained tonsil sections to generate corresponding immunoprobe fluorescence-based images. The probe images were overlaid in the contrasting pseudo-colors (red and green) shown in fig. 7(1b-3 b). Fig. 7(1c-3c) shows the combined images of fig. 7(1a-3a) and 7(1b-3 b).

Figures 8A-8B are additional representative images in which simulated bright field histological images were obtained and fluorescent probe images combined using an alternative method. Fig. 8A is an additive overlay of multi-modal pseudo-bright field images from DAPI counterstained formalin-fixed paraffin-embedded samples using QDot probes with emission wavelengths of 565nm and 655 nm. FIG. 8B is a subtractive (negative) overlay in which the false color probe image is subtracted from the photocopy H & E rendered image.

Fig. 9 illustrates an example of CIEL a b color space for mapping the preferred color characteristics of H & E to refractive contrast and DAPI fluorescence for reproduction in bright field for pathology determination.

FIGS. 10A-10B are gray scale refractive index contrast and DAPI fluorescence contrast images, respectively. Fig. 10C-10D are CIEL a B false color eosin-converted and hematoxylin-converted images, respectively, based on the images of fig. 10A-10B. Fig. 10E is a merged image obtained by combining the converted images of fig. 10C-10D.

FIG. 11 is a schematic diagram of an optical system for simultaneously producing side-by-side refracted dark field images and fluorescence-based images using a single CCD camera.

Fig. 12 includes side-by-side refractive index (dark field) images (a) and DAPI images (B) of the same tissue section acquired and displayed simultaneously.

Fig. 13 includes a superimposed image (with pseudo-color and image inversion) based on two-color bright field reproduction of the side-by-side image of fig. 12. Note that this image is rotated relative to the image of fig. 12.

FIGS. 14A-14B are images of mouse kidney tissue samples prepared for cryosectioning with deposited mass spectrometry imaging tags. Contrast results from refraction at the tissue edges and tissue autofluorescence (blue).

FIGS. 15A-15B are images of mouse kidney tissue samples prepared for mass spectral imaging by deposition of an ionized matrix. The autofluorescence appears blue and the refractive index contrast associated with the ionized matrix crystals is evident.

FIG. 16 is a schematic diagram illustrating a computing environment for the apparatus and methods described herein.

Figure 17 is a multimodality image providing cellular and nuclear context of Calu-3 xenografts detected for mRNA in situ hybridization of two probes, one for ribosomal RNA (cyan, dashed black arrow) and the other for HER2mRNA expression (black, solid black arrow), reproduced in bright field.

Figure 18 is a representative image of prostate cancer for pathology determination using bimodal contrast imaging and presented in a bright field background at 20x magnification. The prominent nucleoli and irregular growth pattern characteristic of prostate cancer is very evident.

FIG. 19 is a portion of the same area imaged at 40 magnification using the same simultaneous bimodal method of combining refractive contrast with fluorescent nuclear counterstaining and rendering in bright field background.

Detailed Description

As used in this application and the claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Furthermore, the term "comprising" means "including".

The systems, devices, and methods described herein should not be construed as limiting in any way. Instead, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments (individually and in various combinations and sub-combinations with each other). The disclosed systems, methods, and apparatus are not limited to any specific aspect or feature or combination thereof, nor do the disclosed systems, methods, and apparatus require that any one or more specific advantageous aspects or problems be solved.

Although the operations of some of the disclosed methods are described in a particular, sequential order for convenient presentation, it should be understood that this manner of description encompasses rearrangement, unless a particular ordering is required by specific language set forth below. For example, operations described as sequential may be rearranged or performed concurrently in some cases. Moreover, for the sake of simplicity, the attached figures may not show the various ways in which the disclosed systems, methods, and apparatus can be used in conjunction with other systems, methods, and apparatus. Furthermore, the description sometimes uses terms such as "producing" and "providing" to describe the disclosed methods. These terms are high-level abstract generalizations of the actual operations performed. The actual operations corresponding to these terms will vary depending on the particular implementation and can be readily discerned by one of ordinary skill in the art.

For a better understanding, a theoretical or other theoretical description of the operation, scientific principles, given herein with reference to the apparatus or methods of the present disclosure, is provided and is not intended to be limiting in scope. The apparatus and methods in the appended claims are not limited to those apparatus and methods that function in the manner described by such theories of operation.

Introduction to the design reside in

The multiple modalities resulting from complementary contrast of tissue may allow visualization of anatomical and morphological tissue backgrounds presented in a brightfield background familiar to trained physicians, as well as probe localization for specific molecules or variations in histochemistry. Multimodal contrast can use (leverage) multiple light-tissue and probes to detect interactions as long as the information provided is complementary. The tissue prepared for pathological examination has constitutive optical activity and the optical properties produced by a particular protocol can be optimized to produce useful contrast properties when combined with an appropriate imager.

Image contrast for non-fluorescent structures can be provided by optically active components designed to be fabricated into or preserved in specific tissue preparation protocols (e.g., preparation protocols used in automated staining protocols) for formalin-fixed, paraffin-embedded and frozen tissues. This enhanced optical activity can be digitally recorded and reproduced in artificial bright field contrast to visualize and highlight structures such as extracellular matrix, nucleoli and cell membranes. Such visualization capabilities can be used to diagnose morphological features and abnormal growth patterns of pathological conditions of tissues. Multiple modalities of chemical or reactive or optical properties of the prepared tissue can be recorded, either serially or in parallel, and digitally converted to bright field image contrast for visualization, and referred to herein as "pseudo-bright fields.

Representative imaged structures are of pathological importance and can be used by physicians for tissue screening and for diagnosis of pre-cancerous and cancerous disease states, as well as for other diagnostic purposes, and for assessing therapeutic efficacy. In unstained or specially stained tissue sections, such morphological structures and anatomical backgrounds can be rendered virtually invisible under monomodal contrast methods such as conventional transmitted light bright field or fluorescence detection. Complementary multi-modality imaging methods can produce medically relevant structural information and present this information in a readily interpretable form without the use of conventional light absorption staining. Quantitative values may be measured and recorded based on the use of one or more sets of computer-executable instructions provided by one or more computer-readable storage media. Morphological measures can be utilized to correlate such morphological properties with molecular information contained in the same tissue; the method can help in the ongoing effort to stratify disease conditions and prognostics and monitor treatment efficacy. Digital multimodal images of tissue slices can be captured simultaneously and rendered using different colors for complementary feature components and streamed, or otherwise stored or delivered for near real-time review by a pathologist or other clinician. Such methods facilitate the development of highly complex tissue-based diagnostics and allow the use of medical training and experience of physicians with routine histological staining. Molecular data for individual tissue sections, including data from immunohistochemistry, DNA hybridization, mRNA hybridization probes, lectins, and mass spectrometry and other analyses, can be integrated and rapidly reported in a format familiar and appropriate to the practicing pathologist.

The examples provided herein utilize a multi-modality imaging strategy that utilizes dual illumination paths (to provide images with complementary contrasts of protein structure and DNA counterstaining) and molecular specific markers for medical diagnosis and assessment. The example method includes a combination of dark field refraction and fluorescence contrast; these complementary contrast modalities are digitally reproduced using specialized color tables derived from physician preference for typical histological staining properties. By using such a combination, it is possible to provide a polychromatic contrast of tissue samples similar to those obtained in samples stained using typical histological methods, such as hematoxylin and eosin (H & E) staining. Such images can be used to visualize regions of interest for further molecular analysis by using luminescent, fluorescent, scattering or absorbing probes for protein, lipid or carbohydrate antigens, mRNA or DNA, probes for charge properties or mass spectrometry (IMS). The multi-modal contrast illumination contrast scheme illustrated herein can provide background information of tissue sections in a manner consistent with common stain/counterstain combinations used in conventional histological methods. For convenience, the beam of optical radiation directed toward the sample to obtain an image is referred to herein as the excitation beam. In some examples, the excitation beam is selected to produce fluorescence at one or more portions of the sample, and may or may not be at a visible wavelength. Other excitation beams include illumination beams at visible wavelengths for direct viewing. The excitation beam may also be based on other types of radiation, including in other wavelength ranges as well as a charged particle beam or an acoustic beam.

In some examples, such methods and apparatus have been used for Fluorescence In Situ Hybridization (FISH), Immunohistochemistry (IHC), and mRNA in situ hybridization (mRNA-ISH) applications in formalin fixed paraffin embedded tissues. Quantum dot (QDot) -labeled FISH probes, QDot-labeled IHC probes, and QDot-labeled mRNA-ISH probes were exclusively detected on tissue by using multi-modal contrast and digital pseudo-bright field reconstruction for visualizing probe localization in the context of tissue anatomy. In a typical example, a dark field refraction contrast image, an image of a counterstain obtained using fluorescent nuclear staining, and one or more probes imaged using fluorescent QDot detection are combined. These and other examples are described below.

Representative imaging systems

A representative example of a suitable imaging system 100 is illustrated in fig. 1. A fluorescent excitation light source 102 is positioned to deliver an excitation light beam 103 along an axis 105 to a wavelength dependent beam splitter (dichroic) 104. Light source 102 is typically a Light Emitting Diode (LED), metal halide or other arc lamp, but other incoherent or coherent light sources such as lasers may be used. As shown in fig. 1, the color separator 104 reflects the excitation beam 103 to the objective lens 106, which objective lens 106 in turn directs the excitation beam 103 to the sample 108. In a typical example, the sample 108 is selectively labeled with one or more fluorophores that fluoresce in response to the excitation light beam 103. A portion of the fluorescence is collected by objective 106 and directed along axis 113 through color separator 104 to optional beam splitter 110. The beam splitter 110 directs a portion of the fluorescence to the camera 112 so that an image of the sample can be recorded, viewed, or analyzed at the computer system 130. Another portion of the fluorescence is directed to an eyepiece 114 to directly view the sample 108 based on the fluorescence. In addition, a shutter (shutter)132 or other beam modulator may be provided to substantially block the excitation beam 103 from reaching the sample 108, or the fluorescence source may be controlled (either by the computer system 130 or manually) so that no excitation beam is generated. The wavelength of light used to excite the light beam may be conveniently selected. Typically, the excitation beam comprises primary optical radiation at a wavelength or within a range of wavelengths suitable for producing fluorescence in a fluorescent dye or fluorophore associated with any selectable marker applied to the sample. Typical wavelengths of the excitation beam range between about 300-550nm, although shorter or longer wavelengths may be used.

In addition to fluorescence imaging systems, refractive contrast imaging systems using annular tilted dark field illumination are provided. In the example of fig. 1, the annular oblique field illumination 117 is chosen such that in the absence of refractive index differences or scattering moieties in the sample, the light flux does not reach the CCD camera 112 or the eyepiece 114 and only refractive index transitions (transitions) occur. The same illumination optimization for refractive imaging can be used at multiple optical magnifications, either by using objectives of different magnifications (acceptance angles) with the same numerical aperture, or by using a secondary magnifying lens. There are a number of strategies for generating contrast based on refraction or scattering of the illumination field. Such refractive index contrast images are referred to herein as "dark field" images. The under-stage condenser system is positioned to deliver oblique-field illumination 117 to the sample 108 at substantially the same location as illuminated by the excitation beam 103. The under-stage condenser system 116 may direct an appropriate light source such as an LED, tungsten halogen lamp, arc lamp or other light source and one or more lenses, mirrors, filters, polarizing elements, phase plates, prisms, rings or apertures that produce an appropriate beam. In the example of fig. 1, oblique field illumination 117 is produced by a carefully sized annulus, but in other examples, different methods for field illumination or point scanning, line scanning, edge illumination, or other strategies designed to produce refractive contrast may be provided. In the example of fig. 1, so-called "dark field" illumination is provided, in which only a portion of the field illumination scattered or redirected by the specimen 108 is collected by the numerical aperture of the objective 106 and reaches the camera 112 or eyepiece 114. The camera 112 and eyepiece 114 are positioned so as to form an image of the sample 108 based on the redirected portion of the transmitted light. In general, the transmission illumination system 117 may be turned off or its light source deactivated if desired so that fluorescence-based images may be viewed or acquired independent of transmission illumination. The example microscope system 130 of fig. 1 thus allows for simultaneous or sequential recording of an image of a sample and observation of the sample based on fluorescence, dark-field refraction contrast, or both.

By using transmissive annular oblique illumination (such as illustrated in fig. 1), contrast can be generated based on interfaces and transitions between portions of the sample having different refractive indices. Generally, the condenser system 116 includes an appropriately sized annulus 118 in the transmitted light path of a compound microscope equipped with a transmitted light source (and may be added to a conventional condenser). In this way, the structures that refract and scatter light have appreciable contrast in the image without the use of light absorbing color stains. The transmitted illumination wavelengths may be spectrally filtered using, for example, a near infrared filter or other filter or combination of filters, so that a spectral image of the fluorescent probe may be obtained with transmission contrast collected at longer wavelengths in the same data acquisition with both illumination sources simultaneously active. The refractive field illumination is typically selected to provide a suitable visual image for recording and is in a wavelength range between about 400nm and 900nm, although different spectral regions in this range may be used if desired. In other examples, refractive dark field illumination is used, where the oblique illumination and the objective lens are placed on the same side of the sample.

The sample dark field image may be obtained by itself using one or more filters to split the fluorescence, turn off or temporarily modulate or otherwise block the excitation beam. In some examples, fluorescence-based images may be obtained using appropriate filters tuned to the fluorescence wavelength and refractive contrast filtered to different wavelength ranges; these different wavelength ranges may be separated into different sensors, directed to different parts of the same sensor or recorded sequentially. Undesirable contributions of dark field illumination to one or more fluorescence images or vice versa can be reduced by spectral filtering, but it is possible to turn off the dark field illumination or the fluorescence light path. In addition, the dark field and fluorescence images can be viewed separately or simultaneously.

The camera 112 is typically a monochrome charge-coupled device (CCD) or Complementary Metal Oxide Semiconductor (CMOS) camera, although other image sensors such as electron multiplying CCD (emccd) and enhanced CCD (iccd) sensors may be used. Wavelength filters, dispersive elements, phase plates, prisms, polarizing elements, tunable optical crystals, and other optical and electro-optical methods may be used to alter the optical radiation reaching the CCD and/or eyepiece so as to produce one or more corresponding monochromatic images within a selected wavelength range. In some cases, fluorescence reaching the camera 112 may be spectrally resolved in multiple wavelength receivers (bins) and a corresponding plurality of fluorescence images obtained for analysis. The spectral analysis may be performed using a plurality of absorbing or refracting filters, prisms or diffraction gratings inserted in the path of the fluorescent light. Typically, spectral resolution can be achieved using interferometric, dispersive, or absorptive optical systems under the control of the computer system 130 or manually inserted. While the image may be recorded as a one, two or three dimensional array of picture elements having values associated with received luminous flux intensity (from fluorescence or other contrast illumination modalities), the image may be recorded in other forms and complex data structures, if desired. For convenience in this specification, an image refers to a 2-D mapping (mapping) of data in a structured array, as viewed by a clinician through a microscope or other viewing device, and a recorded image refers to data values stored, processed and/or displayed based on images received through a CCD or other image sensor.

As noted above, multiple spectral images may be obtained based on fluorescence and transmission illumination, or both. Spectral information at each pixel of the image can be collected and the resulting data analyzed with spectral image processing software. A series of images representing the intensity at different wavelengths (which are electronically and continuously selectable) can be obtained and subsequently evaluated using an analysis program designed to process such data. In certain examples, quantitative information from multiple fluorescence signals and/or optical contrast modalities may be evaluated simultaneously.

The image sensor 112 may be coupled to a computer system 130 that includes a keyboard 152, a processing unit 154, and a display 156. In some examples, one or more additional user input devices (e.g., a joystick, a mouse, or a digitizing tablet) and one or more additional output devices (e.g., a printer or a display) are provided. Processing unit 154 typically includes a microprocessor and one or more computer-readable storage media such as Read Only Memory (ROM), Random Access Memory (RAM), a hard disk, a floppy disk, a compact disk, or a digital video disk for storing image data and computer-executable instructions for recording, transmitting, analyzing, and processing the image or image data. In a typical example, the computer system 100 is coupled to one or more other computer systems via a wired or wireless network connection, and in some examples, the computer system 100 is coupled to the Internet. While image processing operations may be performed on a single computer system, in some examples, image data or images are processed on multiple computing systems, which may be located in a common location or distributed over a network. While a laptop computer may be convenient, other computing devices, such as desktop computers, workstations, handheld computers, notebook computers, or other devices may also be used for image capture and processing. In some instances, the image data may be processed and the sample evaluation may be provided without a display and communicated over a network connection (e.g., by email), sent to a printer or delivered as a text or multimedia message using a mobile phone network.

Imaging system 100 is one example of a suitable imaging system. In other examples, a refractive or catadioptric objective lens may be used instead of the objective lens 106, and a short pass filter (shortpass filter) may be used instead of the long pass filter 104 by rearranging the fluorescence excitation source 102 and the camera 112 and eyepiece 114. In some instances, only a camera or eyepiece is provided for image recording or image viewing. Additional mirrors or prisms may be used to overlap the optical axes (fold), which may be very convenient. Different strategies using phase masks, phase contrast (phasecontrast), Rotterman contrast, oblique illumination contrast, Rheinberg contrast, interferometric difference schemes, adaptive optics, laser scanning, time or frequency domain lifetime imaging, structured illumination, optically switchable probes, polarization and anisotropy, second harmonic imaging, two-photon excited multi-modal contrast, and others may be applied. Sample positioning hardware is not shown for ease of illustration, and in many instances binocular viewing using binoculars may be provided, and appropriate filters and beam splitters may be provided so that the different image outputs receive image light fluxes associated with only one of a plurality of contrast modes, polarization states or wavelength bands. Additional filters (reflective or absorptive) may typically be provided to reduce the luminosity of any excitation light reaching the camera 112 or eyepiece 114 or to control the relative light intensity or spectral content for viewing or recording.

Another representative imaging system is illustrated in fig. 11. As shown in fig. 11, a combined image light flux 1102, comprising a refraction modulated flux 1102A and a DAPI fluorescence modulated flux 1102B corresponding to dark field and DAPI images, is directed along an axis 1103 through an aperture 1104 to a collimating lens 1105. The collimated combined flux is incident on a dichroic mirror 1106, which dichroic mirror 1106 reflects a portion of the modulated flux (DAPI modulated flux 1102A in the example of fig. 11) to mirror 1108A and to filter 1107A, which preferably passes DAPI fluorescence. Lens 1110 receives the DAPI modulated flux and forms a sample image on a first portion 1112A of CCD or other image sensor 112. Dichroic mirror 1106 passes longer wavelength refraction modulated beam 1102B to filter 1107B, which is selected to suppress DAPI fluorescence and the associated DAPI excitation beam. Mirror 1108B directs modulated flux 1102B to lens 1110, which lens 1110 forms a dark field image on a portion 1112B of CCD 1112. The CCD1112 is coupled to a computer or other processing device that can store image data from the CCD1112 in memory and provide the image data to a monitor 1118 or other display. By utilizing such an imaging system, dark field and fluorescence images can be acquired simultaneously and displayed side-by-side as the original image or quickly split into two images, color mapped and overlaid on the monitor 1118 in near real time. The configuration of fig. 11 is merely illustrative, and sample modulated refractive and fluorescent light fluxes may be separated and used in other arrangements for image formation using more, fewer, and different components. The images may be side-by-side on CCD1112 or processed by computer 1114 so that a combined 2-color overlay in a brightfield rendered background may be displayed on monitor 1118. As shown in fig. 11, multiple fluxes (dark field and DAPI) are diverted from the initial optical axis, but in other examples, one flux may be passed along the initial axis and the CCD1112 placed accordingly. Dark field and DAPI images may be produced using different lenses, which may be arranged to produce a common magnification or different magnifications. Additional filters, light sources, and other components may be provided so that the molecular detection markers and other tissue contrast image light fluxes may be provided to and imaged in one or more CCDs or portions 1112A, 1112B of a single CCD.

Color lookup table and image inversion

The system of fig. 1 allows for multi-modal viewing and acquisition of images based on fluorescence or dark field illumination or both. The acquired images may be processed using the representative method illustrated in fig. 2 to present sample features in a common background. In step 202, the dark-field refraction image is typically recorded as a monochromatic image, and in step 204, one or more fluorescence-based images are recorded. The number of such fluorescence-based images may depend on the number and type of fluorescent labels or dyes applied to the sample. These images may use different wavelength bands corresponding to the emission wavelength of the fluorescent marker. In some examples, the different wavelength bands may be overlapping, non-overlapping, or a combination thereof.

In step 204, one or more fluorescence-based images corresponding to fluorescence from the corresponding fluorophore may be obtained. Appropriate spectral segmentation of the fluorescence can be used to obtain multiple fluorescence-based images that can reveal different sample characteristics (which typically depend on the particular probe associated with the fluorescent detection marker).

After the images are acquired (either as each image is acquired or after all or some of the images have been acquired), one or more color mapping look-up tables (LUTs) may be applied to the intensity values of the monochromatic images in step 206 to produce pseudo-color reproduced images, and these reproduced images may be overlaid in step 208. One or more or all of the hue, intensity or saturation of the acquired overlaid image is inverted in step 210 to produce an image with the appearance of a colored structure on the bright field. In a typical application of a pseudo-color LUT, pixels of a monochrome image are assigned RGB color intensity values based on gray-scale pixel intensity values, and vice versa. Such inversion may also invert the color coordinates to produce a complementary color mapping. Image inversion typically maps large pixel intensity values to smaller pixel intensity values. For example, in an image where the pixel intensity is represented by 8 intensity values (3-bit depth), the intensity values may be remapped as shown in Table 1.

Original	Remapping
		0	7
1	6
		2	5
3	4
		4	3
5	2
		6	1
7	0

Table 1. image inversion with 3 bit value,

such mapping schemes may be extended to other bit depths (e.g., 8-bit, 10-bit, 12-bit, 16-bit, and others) and may be applied to different components of a given color space (e.g., hue, saturation, value).

In step 210, image values that appear dark are inverted to appear bright, and image values that appear bright are inverted to appear dark. Step 210 may be referred to as generating a pseudo bright field image.

The order of the image inversion and the pseudo color LUT may be changed as necessary. The particular color LUT may be selected so that, for example, the dark-field image appears with a color contrast similar to histological staining. In this strategy, the image modality is carefully selected to visualize the same structure as an image produced with a conventional staining process, such as conventional H & E staining. The images may be overlaid in step 208 with or without color mapping of the contrast components or inversion to a bright field appearance. Additional color mapping images that are compared to different structures in step 212 may be applied to (typically overlapping) the combined image. The combined and processed image may be stored and/or displayed in step 214. One or more of these steps may be omitted, duplicated, or performed in another order if more convenient.

In many practical instances, it may be advantageous to simulate the staining of specific tissue tissues produced with conventional histological staining in a multi-modal contrast image. Such a simulation provides an analysis and diagnosis device that is familiar to the physician, while still allowing additional information to be visualized in relation to additional specific markers. The simulation also allows for the elimination of light absorbing stains so that the stains do not interfere with the application of other markers or the evaluation of image features visualized by these markers. For example, refractive contrast may be used to visualize extracellular and membrane proteins, while nuclear specific fluorescent dyes such as DAPI may be used to visualize details of nuclear chromatin distribution. Thus, with appropriate image processing, the refraction/DAPI combination can be used to visualize sample features in a manner similar to that obtained with eosin (eosinophilic or protein specific) and hematoxylin (nucleic acid or DNA specific). Because these images are obtained on the same sample, the characteristics of each sample can be spatially registered and included in the displayed image for convenient analysis. An optimized color map may be utilized that allows the images to be displayed at the clinician's preference to best visualize features of interest in the context of medical training and experience. Such color mapping may be conveniently described with reference to CIE1976L a b color space, other color spaces such as Hunter1948L, a, b color space, CIE1931XYZ color space, CIE1976L u v, HSV, HSI, HSV, HSB color, or RGB color or CMYK color values, or a panton or MUNSELL colorimetric scale may also be used.

Fig. 3 illustrates a representative method 300 of sample imaging that allows interrogation of sample features based on refractive index contrast and DAPI fluorescence. In step 302, a gray scale intensity map image of the refractive contrast in the sample is recorded, typically using a monochrome CCD camera. In step 304, a DAPI-fluorescence based gray scale intensity map image of the sample is recorded. Although color filters are used to acquire the two images, the images are recorded as intensity values of an array of pixels (e.g., a grayscale image on a monochrome CCD). In step 306, the index contrast image is processed and mapped to a color having an appearance similar to that produced using eosin color absorption under transmitted white light illumination. Eosin staining generally produces image contrast in the extracellular matrix and protein fraction in the membrane. In some instances, step 306 may be configured such that the processed image has an appearance (e.g., quantified and converted to CIEL a b color space) based on the clinician's subjective preference for eosin staining. These preferred color maps may be based on a group of clinicians or an individual clinician. For convenience, the image produced by step 306 is referred to as a converted image. For eosin stain based processing, such images may be referred to as eosin-reversed images. Such converted images may be displayed images, recorded images, or both.

In step 308, the DAPI recorded image is processed to produce an image associated with a similar appearance to hematoxylin absorption under white light transmitted illumination. As noted above, the image may be generated based on individual or groups of subjective preferences, or matched using quantitative spectral color measurements and mapping to a digital color space. The resulting image of step 308 may also be referred to as a converted image, or a hematoxylin converted image.

The converted image is typically generated using one or more color maps or a specialized look-up table (LUT). The image is typically pseudo-colored and inverted so that the converted image is a complementary color image with inverted saturation, hue and/or value. A combined image is generated by combining the complementary images, e.g., by addition, in step 312, and the combined image is displayed or otherwise analyzed in step 314.

The method of fig. 3 is illustrated using the human prostate sample images shown in fig. 10A-10E. FIGS. 10A-10B are grayscale refraction contrast and DAPI fluorescence contrast images, respectively. Fig. 10C-10D are eosin-converted and hematoxylin-converted images based on the images of fig. 10A-10B, respectively. Fig. 10E is a merged image obtained by combining the converted images of fig. 10C-10D. The eosin converted image of fig. 10C and the hematoxylin converted image of fig. 10D are generated by applying the color LUT and image inversion.

The color space of physician-preferred hematoxylin and eosin stained tissue has been obtained for more closely matching the pseudo-color mapping of the refractive image, and DAPI counterstaining was performed to produce a preferred image appearance. Such a color mapping is illustrated in fig. 9. Referring to fig. 9, CIEL a b color space 900 includes a-axis 902, b-axis 904, and L-axis 906. CIEL a b coordinates are represented as positions on the color sphere 910. Generally, color arcs 912, 914 are assigned to refractive index contrast (eosin-analog) and DAPI fluorescence contrast (hematoxylin-analog), respectively. The color arcs 912, 914 are conveniently selected to produce a similar contrast in tissue as the absorption of white light with hematoxylin and eosin, respectively. A color map may be provided by assigning a, b coordinates based on the measured intensity (L x-value). Color arc 912 corresponds to a longitudinal arc on color sphere 910 that is at an angle of about 30 degrees from the-b-axis. Color arc 914 corresponds to a longitudinal arc on color sphere 910 that is at an angle of about 60 degrees from the-b-axis. Other arcs may be used as well. Representative coordinate ranges that produce H & E staining-like comparisons for selected tissue types are summarized in table 2 below.

Table 2. CIEL a b preferred coordinate ranges for selected tissues.

Computing environment

FIG. 16 and the following discussion provide a brief, general description of a suitable computing environment for software (e.g., a computer program) configured to perform the methods described herein. The methods may be performed in computer-executable instructions organized in program modules. The program modules include routines, programs, objects, components, and data structures that perform the tasks and implement the data types described above to implement the techniques.

Although FIG. 16 shows a typical configuration for a desktop computer, the invention may be practiced in other computer system configurations, including multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed in parallel by processing devices to enhance performance. For example, tasks related to measuring characteristics of candidate anomalies may be performed simultaneously on multiple computers, on multiple processors in a single computer, or both. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

The computer system shown in fig. 16 is suitable for implementing the techniques described herein and includes a computer 1620 having a processing unit 1621, a system memory 1622, and a system bus 1623 that interconnects various system components including the system memory 1622 to the processing unit 1621. The system bus may include any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using a bus architecture. The system memory includes Read Only Memory (ROM)1624 and Random Access Memory (RAM) 1625. A non-volatile system 1626, such as a BIOS, may be stored in ROM1624 and includes the basic routines that transfer information between elements within the personal computer 1620, such as during start-up. The personal computer 1620 may also include one or more other computer-readable storage devices 1630 such as a hard disk drive, a removable memory (thumb drive), a magnetic disk drive (e.g., to read from or write to a removable disk), and an optical disk drive (e.g., for reading a CD-ROM disk or to read from or write to other optical media). The hard disk drive, magnetic disk drive, and optical disk drive can be connected to the system bus 1623 by a hard disk drive interface, a magnetic disk drive interface, and an optical drive interface, respectively, or they can be connected in some other manner. The drives and their associated computer-readable media provide nonvolatile storage of data, data structures, computer-executable instructions (including program code such as dynamically linked libraries and executable files), and so forth, for the personal computer 1620. Although the description of computer-readable media above refers to a hard disk, a removable magnetic disk and a CD, it may also include other types of media that are readable by a computer, such as magnetic cassettes, flash memory cards, digital video disks, and the like.

A number of program modules can be stored in the drives and RAM1625, including an operating system, one or more application programs, other program modules, and program data. A user can enter commands and information into the personal computer 1620 through one or more input devices 1640, such as a keyboard, or a pointing device, such as a mouse. Other input devices may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 1621 through a serial port interface that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, game port, ethernet, IEEE1394, gigabit ethernet, camera link, or a Universal Serial Bus (USB). One or more output devices 1645, such as a monitor or other type of display device, may also be connected to the system bus 1623 via an interface, such as a display controller or video adapter. In addition to the monitor, personal computers typically include other peripheral output devices (not shown), such as speakers and printers.

One or more communication connections 1650 are typically provided, such as wireless connections, wired connections (e.g., ethernet connections), so that the personal computers 1620 can communicate over a communication network. Further, while the personal computer 1620 includes a variety of input devices, output devices, memory, and storage, in some instances some of these components are remotely located for access over a network. For example, processed image data obtained as discussed above may be forwarded over such a network to a remote terminal or processing system for display, evaluation, and further processing by a physician. Also the data storage means may be remote. The personal computer 1620 may be configured to record data in memory, process the data according to the methods disclosed herein, and display the processed data on a local monitor. However, it may be convenient to perform these functions by different processing units at different locations.

The computer systems described above are provided as examples only. The techniques may be implemented in a variety of other configurations. Furthermore, a wide variety of methods for collecting and analyzing data related to processing image data are possible. For example, data may be collected, characterized, colored, and processed to provide bright field background images as appropriate for storage and display on different computer systems. Furthermore, various software aspects may be implemented in hardware and vice versa.

Tissue analysis and tissue processing optimization

Histological protocols are intended to preserve tissue structure and enhance contrast between structures of interest for microscopic examination. To achieve this, many methods are used and have historically been used. Tissue fixation may include various chemicals, examples include, but are not limited to, the use of formalin, Bouin's fixative, ethanol, glutaraldehyde, cryopreservation, microwaves, heat, acetone, acids, alkali solutions, detergents, heavy metals, and many other cross-linking agents or preservatives, for example. These various chemicals have been used to reveal details, preserve cellular and tissue structure, aid in labeling and antigen retrieval and other such efforts to enhance contrast for single modality imaging of pathology. The materials used to infiltrate and embed the tissue and to provide support to the structure for microscopic and ultrathin sectioning also contribute to the optical properties. Subsequent processing, staining and mounting strategies all contribute optical and chemical properties to multi-modality imaging. With this in mind, studies are conducted to optimize multimodality imaging parameters and select appropriate imaging modalities specific to the particular fixation, embedding, labeling and encapsulation conditions typically used for histopathology. This can be done using archived tissue prepared by different conventional methods and adjusting imaging parameters to enhance image quality.

Inversion methods are also continually sought to optimize the tissue preparation protocol to the imaging modality. Image quality is the synergy between tissue preparation, labeling reagents and imaging instrumentation; a multi-modality imaging strategy takes this into account. Thus, the tissue and methods of preservation and preparation are considered to be part of an optical or chemical imaging system. Many vital physical and chemical steps are involved in tissue processing for histopathology. The main stages of automated tissue processing represent many parameters in the processing pipeline that affect image quality. In order to best utilize the particular imaging modality that produces the complementary information, the optical and chemical properties of tissue processing, labeling and sealing must be carefully controlled. The use of automated equipment and optimized protocols for specialized staining and consistency (consistency) of reagents and chemicals is used to allow significant improvements in contrast quality and structural/chemical resolution between complementary imaging modalities. In the context of the examples outlined herein, methods of tissue preparation such as cross-linking proteins by formalin fixation, embedding in paraffin, deparaffinization steps, preservation of nuclear chromatin, counterstaining, specific molecular probes, sealants for tissue preparation, and glass are all used to contribute to multiple imaging modalities. The multiple imaging modalities used in the examples include refractive contrast mass and fluorescence signal and/or molecular mass resolution.

Representative Probe

The pseudo-color bright field rendered image based on multi-modal contrast can be combined with additional detection schemes using various signal generation methods. Some representative probes have been described, but the disclosed technology is not limited to these examples. Some probes configured to specifically bind one or more rakes of interest may be coupled to labels that may be interrogated based on a number of optical and chemical physical properties, such as light absorption, emission, fluorescence lifetime, chemiluminescence, electronic properties, chemical properties, photoswitching capability, intermittent blinking, radioactivity, birefringence, or label quality.

A cross-linker comprising a signal-generating moiety, e.g., a cross-linker of a specific binding moiety and a signal-generating moiety, can be used to detect a specific target molecule in a biological sample. The signal-generating moiety may be used to provide a detectable signal indicative of the presence and/or location of the target. Examples of signal generating portions include, for example and without limitation: enzymes such as horseradish peroxidase, alkaline phosphatase, acid phosphatase, glucose oxidase, beta-galactosidase, beta-glucuronidase or beta-lactamase.

When the signal-generating moiety comprises an enzyme, a chromogenic (chromogenic) compound, a fluorophore compound or a light-generating (fluorogenic) compound may be used to generate the detectable signal. Specific examples of the color-producing compound include bis-aminobenzidine (DAB), 4-nitrophenyl phosphate (pNPP), fast red (fasted), bromo-chloro-indole phosphate (BCIP), Nitrotetrazolium Blue (NBT), BCIP/NBT, fast red, AP orange, AP blue, Tetramethylbenzidine (TMB), 2' -azino-bis- [ 3-ethylbenzothiazoline sulfonate ] (ABTS), o-dianisidine, 4-chloronaphthol (4-CN), nitrophenyl-beta-D-galactopyranoside (ONPG), o-phenylenediamine (OPD), 5-bromo-4-chloro-3-indolyl-beta-galactopyranoside (X-Gal), methylumbelliferyl-beta-D-galactopyranoside (MU-Gal), p-nitrophenyl-alpha-D-galactopyranoside (PNP), 5-bromo-4-chloro-3-indolyl-beta-D-glucuronide (X-Gluc), 3-amino-9-ethylcarbazole (AEC), fuchsin, Iodonitrotetrazole (INT), tetrazole blue, and tetrazole violet.

One type of detectable cross-linker is a covalent cross-linker of an antibody to a fluorophore. The excitation of the cross-linker that directs photons toward a wavelength that is absorbed by the fluorophore can be detected and used to qualitatively, quantitatively, and/or localize the fluorescence of the antibody. Some examples described herein are based on semiconductor nanocrystals (also known as quantum dots or QDots). Quantum dot biocrosslinks are characterized by quantum yields comparable to the brightest conventional dyes available. In addition, these quantum dot based fluorophores absorb 10-1000 times more light than conventional dyes. Quantum dots are generally stable fluorophores, generally resistant to photobleaching, and have a wide range of excitation wavelengths and narrow emission spectra. Quantum dots having particular emission characteristics, e.g., emission at a particular wavelength, can be selected such that a plurality of different quantum dots having a plurality of different emission characteristics can be used to identify a plurality of different targets. The emission from the quantum dots is narrow and symmetric, which means that overlap with other colors is minimized, resulting in minimal bleed through into adjacent detection channels and reduced cross-talk, despite the fact that many more colors can be used simultaneously. The symmetric and tunable emission spectra may vary depending on the size and material composition of the particles, which allows for different quantum dots to be flexibly and densely spaced without significant spectral overlap. Furthermore, their absorption spectra are broad, which makes it possible to excite all quantum dot color variants simultaneously using a single excitation wavelength, thereby minimizing sample autofluorescence. Quantum dots are nanoscale particles that exhibit size-dependent electronic and optical properties due to quantum confinement. Quantum dots are, for example, composed of semiconductor materials (e.g., cadmium selenide and lead sulfide) and are derived from grains (grown by molecular beam epitaxy), and the like.

A variety of quantum dots having various surface chemistries and fluorescent properties are commercially available from invitrogen corporation of ewing, oregon (see, e.g., U.S. patent nos. 6,815,064, 6,682596, and 6,649,138, each of which is incorporated herein by reference). The quantum dots can be coupled to a binding moiety selected for a target of interest. Upon binding to the target, the quantum dots can be detected based on, for example, their fluorescent properties, absorption properties, excitation properties, or fluorescence lifetime.

While many examples of contrast agents that utilize multiplexed probes for multi-modal contrast imaging may be used, including quantum dot-based tags such as those described above, tags configured for imaging mass spectrometry are also highly useful. These so-called "mass tags" can be configured for specific binding of one or more chemical substances or molecules of interest; and subsequently detected using Matrix Assisted Laser Desorption Ionization (MALDI) mass spectrometry or other mass spectrometry techniques. One or more quality tags may be applied to a sample, such as a tissue section, that will be or has been evaluated using the refractive index contrast and/or fluorescence described above. In one example, a ligand or antibody is selected that binds to the target molecule and is ensured to bind to the gold nanoparticles or other nanoparticles. The ligand or antibody present on the nanoparticle binds to the target protein. After binding the target, laser desorption ionization time of flight mass spectrometry (LDI-TOFMS) can be used to subsequently analyze the small molecules on the nanoparticles. U.S. Pat. No. 7,202,472 discloses representative nanoparticles having specifically target-binding antibodies coupled thereto. In this manner, multiple analytes can be detected by providing corresponding specific antibodies or ligands that are bound to respective nanoparticles (where typically each nanoparticle provides a different mass characteristic). In some examples, a photo-cleavable mass tag labeled antibody such as described in U.S. patent application publication 2009/0088332 may be used. In other examples, this is disclosed in US2002/0150927, coupling a probe to a mass modifier, cleaving the mass modifier using an enzyme, and detecting the released mass modifier. In other examples, such as those disclosed in WO00/68434, which is incorporated herein by reference, liposome-encapsulated specific binding oligonucleotides (oligos) are provided, each oligonucleotide having a specific, distinct mass that can be separated by MALDI.

Representative examples

In some additional examples, images of formalin-fixed, paraffin-embedded histological tissue sections prepared according to the Ventana medical system (tussin, arizona) protocol were obtained. In the examples of fig. 4, 5, 6, 10, 12, 13, 20 and 21, the tissue sections were reproduced from prostatectomy and treated with semiconductor nanocrystal quantum dots (QDot) for Fluorescence In Situ Hybridization (FISH) and counterstained with the fluorescent dye 4', 6-diamidino-2-phenylindole (DAPI). Qdot detection and DAPI fluorescence can be generated with an ultraviolet excitation beam in the wavelength range of 370+/-20 nm. Such an excitation beam is well suited for simultaneous multiple excitation of UV absorbing nuclear counterstains such as DAPI and multiple QDot probes. The refractive index contrast in the example contrast scheme is bright versus dark field and does not rely on light absorption staining and thereby allows for simultaneous observation and recording of fluorescence contrast.

Direct observation of such samples using a microscope system, such as that of fig. 1, was found to be useful for direct visualization using a long pass (410nm) filter mounted directly in the microscope eyepiece. The resulting observed image includes nuclei that appear blue in gold/silver histology. Additional colors for direct viewing (e.g., yellow or red) can be induced by using one or more wavelength filters in the transmitted light path. The contrast provided by either illumination (fluorescent or transmitted dark field) can be conveniently turned off to enable imaging with a single contrast method that is independent of the other. The light source intensity (excitation beam, dark field illumination field) can be controlled to balance the contrast of direct two-color visualization or to record normalized integration time on a single sensor. A representative combined dark field (i.e., refractive)/fluorescence contrast image is shown in fig. 4B with 2-color overlap and brightfield rendered along with images of successive slices of the same sample produced using conventional H & E staining (fig. 4A). The image of fig. 4B is based on both color LUT and image inversion. The type of data visible in the unstained tissue sections (fig. 4B) can be used to diagnose Prostate Intraepithelial Neoplasia (PIN) and abnormal growth patterns present in prostate cancer (fig. 20, fig. 21) in a manner similar to those of conventional H & E stained images (fig. 4A), but also interrogation of molecular probe localization can be performed. Furthermore, additional features such as prominent nucleoli are not apparent for DAPI fluorescence alone. Such combined images and processing thereof may therefore be used for diagnosis and therapy, and may not only provide the same information as images based on conventional staining, but also generate additional information.

In these examples, a double contrast (respectively refracted dark field and fluorescence) image was recorded using a monochrome CCD (sequential exposures were made using interference filters to select the refracted contrast light flux at the blue DAPI fluorescence wavelength or longer). FIG. 5A is a monochromatic image using DAPI fluorescence, and FIG. 5B is a monochromatic image of the refractive contrast w/dark field illumination obtained. Additional images based on the combination of the images of fig. 5A-5B are shown in fig. 5C-5D. Fig. 5C is a pseudo-color image obtained by overlaying the monochrome images of fig. 5A-5B and applying a pseudo-color mapping based on a contrast color look-up table (LUT). Fig. 5D is an image corresponding to the image of fig. 5C after inversion and coloring of the image of fig. 5C. The image of fig. 5D may be referred to as a "bright field reproduction" of the image of fig. 5C.

As discussed above with reference to fig. 2, multiple fluorescence images may be obtained and combined. In the context of the ERG gene disruption FISH assay on DAPI counterstained prostate tissue prepared according to the procedure outlined in fig. 18, the ability to locate and reproduce DNA sequence specific probes using the pseudo-bright field method was tested using a 3'5' ERG disruption probe. The ability to apply a pseudo-color look-up table to probe intensity levels and overlaps in an additive color scheme prior to inversion was found to produce a contrast sufficient to identify at least two fluorescent probes simultaneously, with the images of refractive contrast and DAPI counterstain being reproduced in a pseudo-bright field. Successive acquisitions of QDot probe locations at 565nm and 655nm were obtained from samples stained with DAPI and processed to generate the images shown in figure 6 and fluorescence and refraction contrasts of DAPI counterstains. As shown in the inset of fig. 6, such acquisition allows bright field reproduction and display of dual probe FISH localization (see arrows pointing to green and red regions corresponding to probe localization at 565nm and 655nm, respectively). In this case, the probe intensity is superimposed onto a pseudo-color, pseudo-bright field index/DAPI contrast image. Thus, image features similar to those obtained with conventional H & E staining and additional molecular chromosomal rearrangements visualized by the QDot probe can be observed. The overall appearance is familiar to those accustomed to H & E stained images and little or no retraining is required to allow the physician to easily evaluate the sample based on these images.

In another example, protein-specific immunoprobes (QD 565 for CD20 antigen and QD655 for Ki67 antigen) were applied to DAPI counterstained tonsil tissue sections to generate the images shown in fig. 7. The generalized processing steps for tissue processing and contrast optimization are summarized in fig. 18. The dark field refraction image and the DAPI fluorescence based image are acquired, superimposed, and reproduced as a pseudo color pseudo bright field image shown in fig. 7(1a-3 a). Images based on the fluorescence of the immuno probes detected for each probe were obtained and overlaid together as fluorescence images shown in comparative colors in 7(1b-3 b). Probe localization images for QD565 and QD655 probes were pseudo-stained in green and red, respectively, as shown in fig. 7(1b-3 b). The images of fig. 7(1a-3a) and 7(1b-3b) are combined to produce an image, so that the final image reveals the positioning of the probe against the background of the tissue structure, fig. 7(1c-3 c).

Other examples illustrate two methods for overlapping probe localization in bright field backgrounds. Fig. 8A is an additive overlay of a pseudo-bright field image with fluorescent probe images obtained by using QD565 and QD655 probes on DAPI counterstained samples used to obtain the image of fig. 7. FIG. 8B is a subtractive overlay in which the probe image color map is subtracted from the pseudo H & E image. Subtractive superimposition can more closely approximate images obtained using light absorbing stains and is advantageous in contrast generation and multiple image superimposition.

In another example, mRNA-specific ISH probes (QD 605 for 18s ribosomal RNA and QD625 for HER2 mRNA) were applied to DAPI counterstained Calu-3 xenograft tissue sections to generate images as shown in fig. 19. The generalized processing steps for tissue processing and contrast optimization are summarized in fig. 18. Dark field refraction images and DAPI fluorescence based images are obtained, overlaid, and rendered as pseudo-color pseudo-bright field images as shown, and fluorescence based images detected for each probe are obtained and combined with bright field background rendering. As shown in fig. 19, the probe localization images of the QD605 and QD625 probes were pseudo-colored in cyan (black arrow) and black (green arrow), respectively. The final image of fig. 19 thus reveals the positioning of the probe against the background of the tissue structure.

To illustrate video rate imaging, the 2-color imaging method was tested by separating the DAPI emission wavelength from the longer wavelength refraction contrast using an imaging beam splitter similar to that outlined in fig. 11 and projecting two wavelength components of exactly the same field of view side by side onto a single monochromatic CCD sensor. The use of a secondary beamsplitter allows simultaneous acquisition and streaming of images of both color channels to a computer display and recording of a fast time-lag sequence that is only limited by the required integration time and readout time of the camera. Fig. 12 includes side-by-side refractive index (dark field) images (a) and DAPI images (B) of the same tissue section taken simultaneously.

The use of monochromatic intensity capture of different wavelength bands used to generate complementary multimodal images allows the convenient application of specialized look-up tables to the various grayscale intensity images, the transmitted dark field image, and one or more probe locations of DAPI counterstain. The method of mapping the lowest pixel intensity to white in RGB space and the brightest pixel to full saturation of a given hue was tested in the context of the acquisition of the streaming image. This alternative rendering of the transmitted dark field image can be navigated in real time at various magnifications and the snapshot image can be recorded at will. Fig. 13 includes two-color superimposition (by pseudo-color and image inversion) based on the side-by-side images of fig. 12. (note that the image is rotated with respect to the image of fig. 12). Such images can be generated and overlaid quickly, allowing 'live' bright field colour viewing of tissue structures and the concept of counterstaining. The method can be extended to field probe overlap by using multiple sensors or by splitting the light into multiple wavelength bands to project on different regions of a single sensor or a combination of multiple sensors with multiple wavelengths being projected on one or more sensors. Alternatively, sequential detection filters, sequential illumination, dark field refraction and fluorescence images can be recorded by using spectral imaging devices as described in Malik et al 1996, Hoyt et al 2002, both of which are incorporated herein by reference, or by using a single-shot Bayer-mask color camera.

While an optical-based contrast (fig. 22) generated using refractive index, fluorescence, or other methods can be used in the microscope systems of fig. 1 and 11, samples evaluated in this manner can also be prepared for further analysis using mass tags in mass spectrometry. In representative examples, uncoated and matrix-coated mouse kidney tissue was prepared using standard mass spectrometry imaging protocols. Nuclear counterstaining was not used, but it was possible to image gross section morphology and tissue presence based on refractive index differences at tissue edges and by detecting autofluorescence using fluorescence detection optics subsystem. A representative slice is illustrated in fig. 14. Refraction at the exposed tissue edge makes the edge appear bright and blue autofluorescence is associated with the tissue itself. Figure 15 includes images illustrating a combined dark field/autofluorescence image of mouse kidney tissue after deposition of an ionization matrix for mass spectrometry. The host crystal is yellow and autofluorescence is blue. These images show that dark field and fluorescence images can be obtained even after application of the ionization matrix.

Additional discussion

Dark field refractive index contrast and fluorescence have been used in some disclosed examples simultaneously to produce images with multi-modal contrast in tissue samples stained with fluorescent nuclear counterstains. The method is useful in determining pathological conditions using multiple molecular specific probes for IHC, FISH and mRNA-ISH and Qdot detection on the same tissue section, and can also be used to image tissues prepared for mass spectral imaging. This multi-modal contrast scheme has been shown to provide complementary structural context information in a manner similar to conventional histological brightfield/counterstain combinations such as H & E. The structures visible by refractive index contrast comprise protein moieties, and such images allow visualization of structural abnormalities and growth patterns of known pathological significance; including structures such as nucleoli, extracellular matrix, and cell membranes and nuclear membranes. Such structures are not apparent under fluorescent illumination alone. The specific structure visualized using refractive index/fluorescence contrast provides a background for viewing molecular probe signals on the same tissue section and will help physicians screen tissues and diagnose pre-cancerous and cancerous disease states. Dark field refractive index contrast is particularly useful because the method provides bright features as opposed to dark fields and does not use light absorbing stains. Refractive contrast is therefore compatible with direct combination of multiple fluorescence emission probes for cancer marker localization on clear tissue sections prepared using specialized tissue fixation, embedding and staining protocols. When combined with quantitative spectral imaging of the QDot probe, the method does not interfere with probe chemistry or quantitation. By limiting the illumination wavelength for refraction-contrast to the wavelength that is red-shifted from the probe emission, the illumination method can be used simultaneously in the context of spectral image data acquisition for multiple probes. Refraction-contrast combined with fluorescence also allows imaging of tissue background and pathological states on transparent tissues intended for mass spectrometry imaging.

The combined contrast method (refractive contrast and fluorescence) can be visualized directly through the eyepiece in contrasting colors simultaneously. In addition, 2-color image data can be recorded and/or directly displayed in a streaming manner for real-time output of regions of interest and for convenient snapshot recording. The use of simultaneous multi-wavelength acquisition on a monochrome camera provides a convenient means of applying specialized color lookup tables to the gray scale intensity images of streaming dark-field refraction images and fluorescent nucleus counterstain images. Applying a CIEL a b lookup table of known color values corresponding to physician preferences in the context of a particular tissue type also refines the presentation of the tissue structure to the practicing physician. Taken together, careful tissue processing, multi-modal contrast acquisition, and image data processing may provide information similar to that available from conventional hematoxylin and eosin (H & E) stained tissue sections. Such images can also be combined with probe-based image data on otherwise unstained human tissue associated with nuclear, cytoplasmic, and extracellular genes, mRNA expression, and protein antigen markers, as well as other specific probes. By using appropriate color mapping and image inversion, image data can be presented and displayed to a trained pathologist in a familiar manner, and optically active or chemically soluble data, such as mass spectral data, from the same field of view can be overlaid on the familiar background.

Conclusion

As described above, multi-modal contrast can be preserved, enhanced and visualized in cells and tissues. These contrast elements can be combined and reproduced to produce an image similar to that produced with wavelength absorbing stains viewed under transmitted white light illumination. Multi-modal contrast images take advantage of various optical and chemical properties incorporated into tissue by specialized processing. The contrast components can be effectively segmented and presented digitally using a designed color scheme based on typical contrast methods historically used to visualize the same anatomical structures and histochemistry, thereby providing relevance to medical training and experience. The resulting structural background can be used for pathology assays, and it also provides a background for multiple molecular and chemical labels. The method provides important relevance information that may otherwise be difficult or impossible to obtain. In some examples, dark field contrast from the refractive index is combined with a fluorescent DAPI counterstain image to produce an image similar to that obtained using conventional H & E staining. These multi-modality data images have been shown to be useful in the pathological interpretation of tissue sections. Further, such multi-modal image data can be streamed to monitor to allow for the manipulation of histological samples in situ. In other examples, this structural background is subsequently combined with molecular localization of genetic DNA probes (FISH), location of mRNA expression (mRNA-ISH), and Immunohistochemical (IHC) probes localized on the same tissue sections. Multimodal contrast can also be used to evaluate and map tissue sections prepared for mass spectrometry.

Although refractive contrast is a convenient example, other methods are also suitable. Table 3 below lists comparative polymorphisms that can be used to generate complementary information that can be combined to provide a useful context of tissue structure in combination with molecular information for pathological assays. Table 4 lists the major stages of automated tissue preparation for immunocytochemistry, molecular labeling of DNA and mRNA probes on tissues. The details of these normalization stages affect the optical and chemical properties that allow multi-modal imaging for pathology determination.

By using such contrast modalities, the diagnostic method includes two or more modalities that provide contrast to features of medical diagnostic relevance in tissue, wherein the two or more modalities of contrast provide complementary relevant information and the two or more modalities provide contextual information related to tissue level structure, anatomy, or morphology. Typically, images related to two or more modalities of the contextual background are rendered in a manner consistent with medical training and familiar to medical experts (e.g., pseudo-H & E). Such images may be recorded simultaneously or sequentially (before, during, or after rendering) and streamed for rendering on a display, allowing live visualization of the tissue for manipulation. In some examples, two or more independent illumination paths are used. In other examples, incident light fluorescence contrast is utilized while acquiring or processing a transmitted dark field refraction contrast image. In some applications, the dark field refraction contrast is segmented by limiting the wavelength of light used. In other examples, incident light fluorescence contrast is used simultaneously with transmitted dark field contrast.

In some examples, complementary contrast images are provided to directly view two or more colored components through an eyepiece or to directly direct the images to a display. In some cases, it is convenient to acquire two or more complementary contrast components in a single acquisition and simultaneously record the complementary components of multiple illumination paths in a single spectral acquisition. In some examples, the complementary components are recorded by simultaneously wavelength splitting and splitting the optical path.

In other examples, the complementary contrast components are rendered to provide a histologically stained bright field background that is typically based on a color map generated according to physician preference for light absorbing staining slides. In some examples, color mapping of eosin analogs is used for refractive imaging of the eosinophil protein fraction. Typically, an eosin color mapping is applied, followed by image inversion. Furthermore, color mapping of hematoxylin analogs for fluorescent DAPI imaging of nucleic acid moieties may be used, followed by image inversion. These and other complementary contrast components may be colored and streamed. An inverted eosin color map and an inverted hematoxylin color map may be provided and the combined image displayed in a bright field background.

The spatially registered probe locations and chemical mappings may be overlaid on the structure bright field background and colors may be assigned to the probe locations to observe the probe locations and chemical mappings on the structure bright field background. The imaging modality, color look-up table, inversion, and sample preparation may be configured to provide an image appearance selected based on pathologist preferences. Physical, optical and chemical tissue slice preparation protocols can be configured in accordance with multi-modality imaging strategies. Multiple optical magnifications can be used with the same dark-field refractive illumination setup, and multimodal image contrast can be used to provide structural context for subsequent MALDI-TOF mass spectrometry imaging.

The foregoing disclosure and examples included herein are for convenience of explanation only and are not to be taken as limiting the scope of the technology. We claim as our invention all that is encompassed by the appended claims.

Claims (29)

1. A method, comprising:

generating at least one first image and at least one second image of the sample, wherein the generated first and second images comprise complementary contrast image parameters; and

generating a contrast image for viewing;

further comprising obtaining the at least one first image and the at least one second image by subjecting the sample to a first excitation beam and a second excitation beam, respectively, wherein the at least one first image is a fluorescence image of the sample and the at least one second image is a refracted dark field image of the sample.

2. The method of claim 1, wherein the first and second images are combined to produce a combined image.

3. The method of claim 1 or 2, wherein the first and second excitation beams are applied to the sample simultaneously and associated contrast complementary images are obtained simultaneously.

4. The method of claim 1 or 2, further comprising displaying the first and second images side-by-side.

5. The method of claim 2, further comprising displaying the combined image.

6. The method of claim 1, further comprising recording the first and second contrast images as corresponding recorded images.

7. The method of claim 1 or 2, wherein the sample is fluorescently stained and the first excitation beam is selected to produce fluorescence by fluorescent staining such that the first image is a fluorescent image of the sample, and further wherein the second excitation beam is applied to the sample such that the second image is a refracted dark field image.

8. The method of claim 7, further comprising recording the fluorescence image and the refracted dark field image as corresponding recorded images.

9. The method of claim 8, further comprising:

applying a color map to the recorded dark-field image to produce a pseudo-color dark-field image; and

combining the pseudo-color dark field recorded image and the recorded fluorescence image to produce a combined recorded image.

10. The method of claim 9, wherein the color mapping is based on a color look-up table associated with at least one absorption dye.

11. The method of claim 10, wherein the staining is eosin staining.

12. The method of claim 9, further comprising applying a color lookup table to the fluorescence image, wherein the color lookup table is associated with at least one absorption stain.

13. The method of claim 12, wherein the fluorescence is based on DAPI fluorescence and the color lookup table associated with the fluorescence image is based on hematoxylin staining.

14. The method of claim 9, further comprising generating a pseudo bright field recorded image based on the combined recorded image.

15. The method of claim 14, further comprising applying a color lookup table to the refracted dark-field image and the fluorescence image to produce an image having an image contrast associated with hematoxylin and eosin staining.

16. The method of claim 1, further comprising imaging the sample using mass spectrometry.

17. An image forming apparatus comprising:

a first imaging system configured to generate a first image of a sample; and

a second imaging system configured to generate a second image of the sample, wherein the first and second images are complementary images;

wherein the first imaging system is a refractive dark field optical system configured to produce a first image as a refractive dark field image by using a first excitation beam, and the second imaging system is a fluorescence optical system configured to produce a second image as a fluorescence image by using a second excitation beam.

18. The imaging apparatus of claim 17, further comprising: at least one image capture device coupled to receive the first and second images; and an image processor configured to record the first and second images.

19. The imaging device of claim 18, wherein the image processor is configured to generate a combined image based on the first and second images.

20. The imaging device of claim 19, wherein the image processor is configured to apply a color look-up table to at least one of the first and second recorded images to produce a combined image based on a pseudo-color rendition of the at least one of the first and second recorded images.

21. The imaging device of claim 20, further comprising rendering the combined image as a pseudo bright field image.

22. The imaging device of claim 21, wherein the image processor is configured to process the first image based on a color lookup table associated with eosin staining and to process the second image based on a color lookup table associated with hematoxylin staining, wherein the combined image is based on the processed first and second images.

23. The imaging device of claim 18, wherein the image capture means is configured to receive the first image and the second image as side-by-side images.

24. The imaging device of claim 19, wherein the image processor is configured to overlay the first and second images to produce the combined image.

25. The imaging device of claim 24, further comprising a display configured to receive and display the combined image.

26. An image processor comprising:

an image input configured to receive at least a first image and a complementary second image;

a color look-up table input configured to receive at least one color look-up table;

an image combiner configured to process at least one of the first image and the complementary second image based on the at least one color lookup table and generate a pseudo-color image, and to combine the pseudo-color image with one of the first image and the second image;

wherein the first and second images are obtained using first and second excitation beams, respectively, and the first image is a fluorescence image and the second image is a refracted dark field image.

27. The image processor of claim 26, wherein said image combiner is configured for generating a bright field representation based on said combined image.

28. The image processor of claim 27, wherein the color lookup table input is configured to receive a first color lookup table associated with absorption staining and a second color lookup table associated with fluorescence staining, and the image combiner is configured to process the first and second images based on the first and second color lookup tables, respectively, and wherein the combined image is based on the processed first and second images.

29. The image processor of claim 28, wherein the absorption staining is eosin staining and the fluorescence staining is hematoxylin staining.