JP2009112468A - Image alignment apparatus and method, and image alignment program - Google Patents

Abstract

To improve the alignment between three-dimensional reference image data and three-dimensional target image data including ultrasonic image data such as ultrasonic images or ultrasonic images and other modality images.
For example, reference image data R such as US image data 19-1 before treatment and target image data F such as US image data 19-2 after treatment have anatomically characteristic images. The sub-regions E ₁ and E ₂ of the alignment candidates are set by the region setting unit 15, the image similarity between the small regions E ₁ and E ₂ is obtained, and the pre-treatment is set so that the image similarity is maximized. The alignment unit 16 aligns the reference image data R such as the US image data 19-1 and the target image data F such as the US image data 19-2 after the treatment.
[Selection] Figure 1

Description

  The present invention relates to an image alignment apparatus and method for aligning 3D reference image data including ultrasonic image data acquired by imaging a subject such as a patient and 3D target image data, and an image thereof. It relates to an alignment program.

It is generally known that automatic spatial alignment of ultrasound image groups is more difficult than CT images and MR images. This is because the ultrasonic image is inferior to the CT image or MR image in the variety of imaging conditions, the non-uniformity of the gain, and the image resolution and S / N are inferior.
In order to determine a change with time and a therapeutic effect of a lesion using an ultrasound image, ultrasound images with different imaging times and procedures are targeted. For example, an ultrasonic image taken before the operation and an ultrasonic image taken after the operation are compared to determine the temporal change of the lesion and the therapeutic effect. There is a difference in the position of the subject such as the patient at the time of imaging, and the presence or absence of contrast enhancement between the ultrasound image taken before the surgery and the ultrasound image taken after the surgery. There is a change in the shape of the subject itself due to progress. These factors make it difficult to automatically spatially align ultrasound images.

  In image alignment between images such as an ultrasound image and other modality images such as an X-ray CT image, an ultrasound B-mode image, a contrast agent arterial phase image, a contrast agent vein phase image, and an MR image, each image generator Since the order is different, there is no one-to-one relationship between pixel values, and spatial alignment is more difficult.

  As the types of images to be aligned, that is, the types of target images to be aligned with respect to the reference image increase, for example, the types of target images include X-ray CT images, ultrasound B-mode images, contrast agent arterial phases, and contrast media. It is a problem to be solved that if the number of the venous phase images increases, the area and anatomical features that can be commonly observed between these images are reduced.

  Mutual information is reported as a similarity index between images with different modalities. This mutual information amount has been reported to be relatively good particularly in CT images and MR images. The deviation from the correct answer of the relative initial position where the alignment between each image is successful is called a capture range. This capture range has (1) good resolution, and since the entire body including the body surface is imaged, it is easy to determine common anatomical feature regions such as the head, brain, and trunk, 2) It is easy to set the initial position within the capture range since the standardization of the body position of the subject at the time of imaging is progressing. Thereby, comparatively good results are obtained in the alignment between the CT images and the MR images. However, in general, the image similarity calculation area becomes large, and therefore it takes a long calculation time.

  Some attempts have been made to align ultrasonic images, but the number is smaller than other modalities. This is because, in addition to the characteristics of the ultrasound image described above, (1) imaging by manually operating the ultrasound probe, (2) due to the presence of ribs when imaging the liver of the subject, for example. (3) The subject's imaging position is flexible and its standardization has not progressed, and (4) the obtained ultrasound image is in the body of the subject. It is difficult to recognize the relative orientation between the ultrasound images because the part is cut out, for example, it is obtained as a quadrangular pyramid area and the outside is outside the field of view. One of the reasons is considered to be difficult to set within the range.

As positioning problems peculiar to the ultrasound images, the ultrasound images include the compression of the subject such as the affected area by the ultrasound probe and the deformation of the organ due to the posture of the subject such as the patient at the time of imaging. Is included. For example, in order to determine the temporal change of the lesion and the treatment effect before and after surgery, when acquiring an ultrasound image by applying an ultrasound probe to a subject such as an affected part, before and after surgery, It is difficult to make the posture state of the subject, for example, the degree of twisting when the posture is in the horizontal direction, and to make the direction in which the ultrasonic probe is applied to the subject the same. In the first place, there is no correct position in the body position of the subject or the direction in which the ultrasound probe is applied, and even if the orientation is adjusted to the same extent as the CT image or MR image, the same effect as in the case of the CT image or MR image is obtained. I can't expect.
JPWPluim, JBAMaintz, “Mutual-Information-Based Registration of Medical Images: A Survey”, IEEE Trans. Med. Imag., Vol. 22, pp. 986-1004, August 2003 R.Shekhar, and V.Zagrodsky, “Mutual Information-Based Rigid and Nonrigid Regitration of Ultrasound Volumes”, IEEE Trans. Med. Imag., Vol. 21, pp. 9-22, January 2002 R.Shekhar, V.Zagrodsky, J.Garcia, and D.Thomas, “Registration of Real-Time 3-D Ultrasound Images of the Heart for Novel 3-D Stress Echocardiography”, IEEE Med.Imag., Vol.23, pp. 1141-114, September 2004

  An object of the present invention is to make it possible to satisfactorily align three-dimensional target image data with three-dimensional reference image data including ultrasonic image data such as ultrasonic images or ultrasonic images and other modality images. An image alignment apparatus and method, and an image alignment program are provided.

  The image registration apparatus according to claim 1 of the present invention selects feature points on the three-dimensional reference image data and on at least one three-dimensional target image data to be aligned with the reference image data. A region setting unit that sets each region of the alignment candidate formed in a required size, and an image similarity between the regions set by the region setting unit is obtained, and the reference image is maximized. An alignment unit that performs alignment between the data and the target image data, and a display unit that displays the reference image data and the target image data aligned by the alignment unit on a display.

  According to a 28th aspect of the present invention, in the image registration method, feature points are selected on the three-dimensional reference image data and on at least one three-dimensional target image data to be aligned with the reference image data. Each region of the alignment candidate formed in the required size is set, the image similarity between the set regions is obtained, and the position of the reference image data and the target image data is maximized so that the image similarity is maximized. The registered reference image data and target image data are displayed on the display.

  An image registration program according to a twenty-ninth aspect of the present invention selects a feature point on three-dimensional reference image data and on at least one three-dimensional target image data to be aligned with the reference image data. Each area of the alignment candidate formed in the required size is set, and the image similarity between the set areas is obtained, and the reference image data and the target image data are set so that the image similarity is maximized. The registered reference image data and target image data are displayed on the display.

  According to the present invention, it is possible to satisfactorily align the three-dimensional target image data with the three-dimensional reference image data including the ultrasonic image data such as the ultrasonic images or between the ultrasonic image and another modality image. An image alignment apparatus and method, and an image alignment program can be provided.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram of an image alignment apparatus. A plurality of modalities, for example, an ultrasonic diagnostic apparatus (hereinafter referred to as US diagnostic apparatus) 3, an X-ray CT apparatus 4, an MR apparatus 5 and the like are connected to the image alignment apparatus 1 via a network 2.
The image alignment apparatus 1 has a main control unit 6 composed of a CPU or the like. A program memory 7, a data memory 8, an image database 9, a transmission unit 10, a display 11, and an operation unit 12 are connected to the main control unit 6.

  The program memory 7 stores, for example, three-dimensional (3D) US image data (also referred to as US volume data) before treatment and three-dimensional US image data after treatment, or three-dimensional US image data and other three-dimensional US image data. An image alignment program for performing alignment with modality images, for example, X-ray CT image data (also referred to as CT volume data) and MR image data (also referred to as MR volume data) is stored. The image registration program is, for example, a three-dimensional target image such as at least one US image data after registration on the three-dimensional reference image data such as US image data before treatment and the reference image data. In the data, each small region of the alignment candidate formed to the size required for selecting the feature point is set, and the image similarity between the set small regions is obtained. The reference image data and the target image data are aligned so as to increase, and the aligned reference image data and target image data are displayed on the display 11.

The data memory 8 temporarily stores data necessary when the reference image data and the target image data are aligned.
The image database 9 stores image data of each modality received from the US diagnostic apparatus 3, the X-ray CT apparatus 4, the MR apparatus 5 and the like through the network 2. The image data of these modalities is, for example, US image data before treatment, US image data after treatment, US contrast image data, CT image data before treatment, MR image data, and the like.
The transmission unit 10 receives image data of each modality from the US diagnostic apparatus 3, the X-ray CT apparatus 4, the MR apparatus 5, and the like via the network 2.
The display 11 displays, for example, pre-treatment US image data and post-treatment US image data before alignment, or after alignment, for example, pre-treatment US image data and post-treatment US image data. .
The operation unit 12 is, for example, a keyboard or a mouse.

The main control unit 6 executes an image registration program stored in the program memory 7 to thereby execute a data reading unit 13, a data attribute changing unit 14, a region setting unit 15, a registration unit 16, and a display unit. 17 and the image storage unit 18.
The data reading unit 13 uses, for example, US image data before treatment, US image data after treatment, US contrast image data, CT image data before treatment, MR image from image data of each modality stored in the image database 9. Read data etc. Of these image data, image data that serves as a reference for alignment is referred to as reference image data, and image data to be aligned with respect to this reference image data is referred to as target image data. For example, US image data before treatment is set as reference image data, and US image data after treatment is set as target image data.

  The data reading unit 13 directly receives, for example, US image data before treatment, US image data after treatment, US contrast image data, pre-treatment from each modality from the US diagnostic device 3, the X-ray CT device 4, the MR device 5, and the like. CT image data, MR image data, and the like may be read out. In this case, the data reading unit 13 sends an image reading instruction to the transmission unit 10, and directly transmits images from the modalities such as the US diagnostic apparatus 3, the X-ray CT apparatus 4, and the MR apparatus 5 through the transmission unit 10. Read data. The read image data is displayed on the display 11 by the display unit 17. FIG. 2 shows a display example of US image data 19-1 before treatment of the liver and US image data 19-2 after treatment of the liver displayed on the display 11, for example. These US image data 19-1 and 19-2 are, for example, liver cross-sectional images.

  In addition, each US image data 19-1 and 19-2 before treatment and after treatment respectively select characteristic portions anatomically. The pre-treatment and post-treatment US image data 19-1 and 19-2 differ in the spatial imaging direction when the ultrasonic probe is applied to a subject such as a patient.

  The data reading unit 13 reads, for example, reference image data and target image data acquired from the image database 9 with the same or different modalities. For example, the data reading unit 13 reads at least one of the reference image data and the target image data from the image database 9 as ultrasonic image data. For example, the data reading unit 13 reads reference image data and target image data having different pixel sizes from the image database 9.

  Specifically, the data reading unit 13 simultaneously aligns image data having different volumes such as 3D US image data and other 3D modality images such as X-ray CT image data and MR image data. Read as. For example, in order to determine the treatment of liver cancer by ultrasonic puncture, it is desirable to compare and reference X-ray CT image data or MR image data before treatment, US image data before treatment, US image data after treatment, and US contrast image data. There is a request. For this reason, two volume data are not sufficient for volume data to be handled at one time, and usually four or more types of volume data may be required in some cases.

  Image data read by the data reading unit 13, for example, US image data before treatment, US image data after treatment, X-ray CT image data or MR image data before treatment, and US contrast image data are displayed by the display unit 17. It is displayed on the display 11. FIG. 3 shows a display example of US image data 19-1 before treatment, US image data 19-2 after treatment, X-ray CT image data 20 before treatment, and US contrast image data 21. These image data 19-1, 19-2, 20, 21 are displayed on the display 11 in a display window having a pixel size of 256 × 256, for example.

  The data reading unit 13 includes the pre-treatment US image data 19-1, the post-treatment US image data 19-2, the pre-treatment X-ray CT image data 20, and the US contrast image data 21 shown in FIG. As described above, volume data having different pixel sizes are read out simultaneously. Each of these image data 19-1, 19-2, 20, and 21 sets a display window having a pixel size of 512 × 512, for example, and displays it on the display 11 with this display window. For example, the pixel size of each volume data generated from these modalities is generally different depending on the modalities of the US diagnostic apparatus 3, the X-ray CT apparatus 4, the MR apparatus 5, and the like. In order to handle data of these different pixel sizes, for example, display windows having a pixel size of 512 × 512 for the X-ray CT apparatus 4 and the MR apparatus 5 and a pixel size of 256 × 256 for the diagnostic apparatus 3 are used.

The data attribute changing unit 14 changes at least attributes relating to rotation and translation with respect to the reference image data and target image data read by the data reading unit 13 and displays them on the display 11.
The data attribute changing unit 14 uses, for example, an alignment algorithm (image alignment program) for data converted by the brightness value setting function used in each modality such as the US diagnostic apparatus 3, the X-ray CT apparatus 4, and the MR apparatus 5. Make it manageable data. The brightness value setting function used in each modality will be described. For example, the brightness information of CT image data is expressed by 16 bits. In diagnosis, data in a range set by an operator such as a doctor using parameters such as WC (window center) and WW (window width) is set to an 8-bit luminance value. An operator such as a doctor operates the keyboard and mouse of the operation unit 12 to adjust the CT image data having a desired luminance by the luminance value setting function.

  That is, in the registration algorithm, a joint histogram is generated from two types of volume data such as US image data 19-1 before treatment and US image data 19-2 after treatment, and estimated from the joint histogram. An alignment index is calculated using the probability density function. If the original image data of the CT image data is used as it is, a sparse histogram is formed, and the probability density function cannot be estimated appropriately. For this reason, it is necessary to compress the range by some method. It is appropriate that an operator such as a doctor diverts a desired brightness value setting for diagnosis.

  The data attribute changing unit 14 reads out image data, for example, US image data 19-1 before treatment, US image data 19-2 after treatment, X-ray CT image data 20 or MR image data before treatment, US contrast image Image discrimination data 22 such as volume name, photographing date, and pixel size of the data 21 is displayed on the display 11 as shown in FIGS. 2 and 3, for example, and is clearly shown to an operator such as a doctor. The display positions of these image discrimination data 22 are, for example, US image data 19-1, US image data 19-2 after treatment, X-ray CT image data 20 before treatment, and US contrast image data 21, respectively. It is diagonally upward to the left. That is, the volume name is necessary for specifying each image data. The date is necessary for determining the diagnosis history. The pixel size is necessary for determining the reference image size described later.

The data attribute changing unit 14 is a plurality of image data (volume data) read by the data reading unit 13, for example, US image data 19-1 before treatment, US image data 19-2 after treatment, X before treatment. The pixel sizes between the line CT image data 20 or the MR image data and the US contrast image data 21 are matched. The data attribute changing unit 14 determines a reference pixel size when matching pixel sizes between a plurality of image data (volume data).
Specifically, the data attribute changing unit 14 determines a reference pixel size by one of the following methods. In other words, the data attribute changing unit 14 selects, for example, volume data having a pixel size based on the alignment between the US image data 19-1 before treatment and the X-ray CT image data 20 before treatment. The data attribute changing unit 14 is, for example, a pixel size manually operated by an operator such as a doctor on the operation unit 12 during alignment of the US image data 19-1 before treatment and the X-ray CT image data 20 before treatment. Enter. The data attribute changing unit 14 presets image data having a reference pixel size, for example, US image data 19-1 before treatment, using a preset function. In this case, for example, the pixel size of the other image data is matched with the pixel size of the US image data 19-1 before treatment. The data attribute changing unit 14 sets a reference pixel size in advance by a preset function.

  When adjusting the pixel size, temporary data for processing may be created with the set pixel size, or calculation may be performed in consideration of the pixel size every time the alignment calculation is performed. . The data attribute changing unit 14 displays a size adjustment button 23 for adjusting the pixel size on the display 11 as shown in FIG. When the size adjustment button 23 is instructed by, for example, a mouse click operation, the data attribute changing unit 14 determines a reference pixel size by any of the above methods.

  Pixel size is an important factor in alignment. The pixel size differs not only between image data acquired with different modalities but also between image data acquired with the same modality because of differences in shooting conditions at each time. There are a plurality of image data to be aligned for each photographing condition even in different modalities or the same modality. Therefore, it is necessary to specify which image data is to be matched with the pixel size.

For example, as shown in FIG. 2, the data attribute changing unit 14 displays a scale 24 corresponding to the image size on the display 11. The scale 24 is formed to have an actual length, for example, a length corresponding to 2 cm.
The data attribute changing unit 14 changes the spatial orientation / direction of the volume of the target image data such as the US image data 19-2 after the treatment, for example. Specifically, the data attribute changing unit 14 includes, for example, a first mirror operation unit that is reversed in the left-right direction on the display 11, a second mirror operation unit that is reversed in the vertical direction, and a reverse unit that is reversed in the depth direction. Have. If the posture information of the subject such as a patient is correctly recorded in the volume of the target image data, the data attribute changing unit 14 changes the spatial orientation / direction of the volume of the target image data based on the posture information. Do it automatically.
In this case, for example, as shown in FIG. 3, the data attribute changing unit 14 has a first mirror operation unit setting button 25, a second mirror operation unit setting button 26, and a reverse unit setting button on the display 11. 27 is displayed. An operator such as a doctor manually operates the operation unit 12 while observing the target image data displayed on the display 11, and selects and instructs one of the setting buttons 25, 26, and 27. Thus, the data attribute changing unit 14 changes the spatial orientation / direction of the volume of the target image data in response to an instruction from the operation unit 12.

  That is, not only different modalities, but also the same modality causes differences in the posture of the subject such as a patient and the imaging direction. However, three-dimensional US image data and other three-dimensional modality images, such as X-ray CT image data and MR image data, generally have different volume data directions and directions. It is necessary to align these image data after aligning the direction and direction of the target volume data.

  The data attribute changing unit 14 selects a set of volume data to be aligned, that is, reference image data and target image data, from volume data such as three-dimensional US image data serving as N target image data. The data attribute changing unit 14 further determines reference image data and target image data for a set of volume data to be aligned. The data attribute changing unit 14 can distinguish the selected reference image data and target image data from other volume data. For example, the data attribute changing unit 14 may change the colors of the image display frames of the reference image data and the target image data, or may change the displayed volume name.

The region setting unit 15 includes, for example, the pre-treatment US image data 19-1 and the post-treatment US on the three-dimensional reference image data and at least one three-dimensional target image data to be aligned with the reference image data. In the image data 19-2, each region of the alignment candidate formed in a size required for selecting a feature point is set. FIG. 4 shows a diagram in which the small regions E ₁ and E ₂ of the alignment candidates are set in the US image data 19-1 before treatment and the US image data 19-2 after treatment, respectively. The small regions E ₁ and E ₂ of the alignment candidates are anatomically characteristic. For example, the US image data 19-1 before treatment and the US image data 19-2 after treatment are set to the same site. Is done.

The setting of the small areas E ₁ and E ₂ of the alignment candidates includes setting and automatic setting by manual operation. First, manual setting of each of the small regions E ₁ and E ₂ of the alignment candidates will be described.
The region setting unit 15 receives a manual operation on the operation unit 12 by an operator such as a doctor, and displays on the display 11 a set of reference image data and target image data to be aligned according to the manual operation. The pair of reference image data and target image data are rotated on the display 11 with the X, Y, and Z directions as the central axes, respectively, and translated in the X, Y, and Z directions, respectively.
For example, the pre-treatment US image data 19-1 and the post-treatment US image data 19-2 shown in FIG. 4 are each three-dimensional volume data. Therefore, one piece of the US image data 19-1 before the treatment and the US image data 19-2 after the treatment, which are the three-dimensional volume data, that is, two-dimensional US image data 19-1 before the treatment and the post-treatment, respectively. The US image data 19-2 is initially set. Then, the region setting unit 15 sets reference image data and target image data to be aligned, and displays each information of the reference image data and target image data. The set of reference image data and target image data is an initial tomographic plane of a subject such as a patient. The region setting unit 15 is not limited to the initial setting of the two-dimensional pre-treatment US image data 19-1 and the post-treatment US image data 19-2, but again a set of reference image data and target image data. Are rotated about the X, Y, and Z directions on the display 11, respectively, and are translated in the X, Y, and Z directions, respectively. You may set the US image data 19-2 after a treatment.

  The region setting unit 15 sets, for example, the shooting direction of US image data in the shooting direction / offset state using a preset ultrasonic probe. That is, X-ray CT image data and MR image data are image data of a cross section parallel to the body axis of a subject such as a patient. On the other hand, US image data is often imaged from an imaging direction limited by anatomical restrictions of a subject such as a patient. For example, in the case of liver cancer diagnosis, US image data is often taken from a specific imaging direction, such as under the arch or between the intercostals.

  Therefore, the relative position to the coordinate system perpendicular to the body axis in these specific photographing directions is preset and registered. For example, it is possible to preset the shooting direction when shooting from a specific shooting direction such as under the bow or between the eyelids. The preset of the relative position to the coordinate system perpendicular to the body axis in the specific imaging direction is performed by displaying a preset button 28 on the display 11 as shown in FIG. 3, for example, and operating the preset button 28 by an operator such as a doctor. This is done by clicking the mouse in the section 12. As a result, the region setting unit 15 can align the shooting directions of a set of reference image data such as US image data and target image data to a rough position within a predetermined misalignment range in one process. it can. Note that the region setting unit 15 can reset the relative position to the coordinate system perpendicular to the body axis in a specific photographing direction preset and registered in advance.

  The region setting unit 15 selects each image display mode of each volume data such as US image data 19-1, US image data 19-2 after treatment, X-ray CT image data 20 before treatment, US contrast image data 21, and the like. For example, switching to normal MPR (multi-planar reconstruction) display, maximum value display with thickness, minimum value display with thickness, MPR display with thickness, or volume rendering display is performed. This improves the visibility of the image structure over a larger area when selecting appropriate image data to start alignment, and helps doctors and other operators recognize the same anatomical feature points Therefore, the convenience can be improved.

The region setting unit 15 receives a click operation on the mouse of the operation unit 12 by an operator such as a doctor, and initially sets reference image data and target image data, for example, US image data 19-1 before treatment and post-treatment shown in FIG. The small regions E ₁ and E ₂ of the alignment candidates are set on each image data with the US image data 19-2. Each of the small regions E ₁ and E ₂ of the alignment candidates can be set by a manual operation by a so-called operator such as a doctor.

  The area setting unit 15 is manually operated by an operator such as a doctor, with one basic area and a plurality of auxiliary areas used for alignment on the reference image data such as the US image data 19-1 before treatment. One area corresponding to the basic area is specified on the target image data such as the US image data 19-2 after the treatment.

Region setting unit 15, positioning each small region depending always on the pixel size of the target image data to be constant in the actual size of the object in each subregion E _1, the size, for example, a patient, etc. E ₂ candidate E ₁ , to adjust the size of the _{E 2.} Specifically, the area setting unit 15 registers the actual size size to be set by presetting. That is, the pixel sizes of the target image data are different. On the other hand, there is a demand for the size of an anatomically characteristic form used for alignment to be about 2 cm in actual size of a subject such as a patient. For this reason, the size of each of the small areas E ₁ and E ₂ used for alignment is automatically adjusted so that the actual size of the subject such as a patient is always 2 cm.

The area setting unit 15 displays the position and size of each of the small areas E ₁ and E ₂ as alignment candidates as the center and the area boundary. Each of the small regions E ₁ and E _{2 that} are candidates for alignment is formed in a rectangular shape, for example. In this case, in each of the small areas E ₁ and E ₂ , for example, as shown in FIG. 5, the boundary of the rectangle is displayed by a solid line “□”, and the center of the rectangle is displayed by a cross “+”. The shapes of these small regions E ₁ and E ₂ are not limited to a rectangle, and may be formed in a circle or the like.

The area setting unit 15 manually changes the sizes of the small areas E ₁ and E ₂ without moving the positions of the small areas E ₁ and E ₂ of the alignment candidates. That is, the region setting unit 15 changes the sizes of the small regions E ₁ and E ₂ in response to an operation on the operation unit 12 by an operator such as a doctor. When each of the small regions E ₁ and E _{2 of the} alignment candidate is rectangular, the operator operates a mouse, for example, by an operator such as a doctor and moves the pointer on the display 11 to the rectangular vertices of the small regions E ₁ and E _2. Grip at the apex to change the size.

The area setting unit 15 changes the positions of the small areas E ₁ and E ₂ without changing the sizes of the small areas E ₁ and E ₂ of the alignment candidates. When each of the small regions E ₁ and E _{2 of the} alignment candidates is a rectangle, the operator operates a mouse, for example, by a doctor or the like to move the pointer on the display 11 to the rectangular centers of the small regions E ₁ and E _2. The position of the small areas E ₁ and E ₂ is changed by moving while gripping at the center.

Next, automatic setting of each of the small areas E ₁ and E ₂ of the alignment candidate will be described.
The area setting unit 15 sets each sub-area of the registration candidate based on the image similarity between the reference image data and the target image data, for example, the US image data 19-1 and the US image data 19-2 after treatment shown in FIG. E ₁ and E ₂ are set automatically. The following methods are used for automatic setting of the small areas E ₁ and E ₂ of the alignment candidates.

The first method is a method of searching for a square region that maximizes entropy.
When the reference position is (x ₀ , y ₀ ), a square S (l) having a side length l centered on the reference position (x ₀ , y ₀ ) is considered. The length l is changed in the range of l _min ≦ l ≦ l _max , the relative frequency Pi is obtained from the histogram of the pixel value i in the square S (l), and the entropy H (l) is calculated by the following equation (1). To do.

Entropy H (l) is maximized

Is one side of each of the small regions E ₁ and E ₂ .

  It is desirable to use anatomically characteristic image data for the region to be aligned. The pixel values of anatomically characteristic image data are generally distributed over a wide range. In monotonous image data having no anatomically characteristic image data, the histogram is biased. The first method uses entropy H (l) calculated from the frequency distribution of pixel values as an index for extracting anatomically characteristic image data.

The second method is a method using an index based on a difference in average value from the surrounding area.
Consider, for example, eight squares Sj (l) adjacent to a square S (l) having a side length l. j is 1 ≦ j ≦ 8. The average value and variance of the pixel values in the square S (l) are μ, and the average value and variance of the pixel values in the eight squares Sj (l) are μ _j .

Feature index

Is maximized

Is calculated from l _min ≦ l ≦ l _max to determine one side of each of the small regions E ₁ and E ₂ used for alignment.
However, the final length l is
l = a · l _max : 1.5 <a <3
And a is determined empirically. In the second method, an area having a marked difference in the surrounding area as a discrimination criterion is used as a feature area.

The region setting unit 15 is not the maximum value in a certain section l _min ≦ l ≦ l _max in the length l of one side, but the value when the increase in entropy H (l) or feature index first stops. Then, each of the small areas E ₁ and E ₂ is determined.
References include T. Kadir and M. Brandy, “Saliency, scale and image description”, Int. J. Comput. Vision, Vol. 45, no. 2, 83-105, 2001, and G. Wu, F.Qi, and D.Shen, “Learning-Based Deformable Registration of MR Brain Images”, IEEE Trans. Med. Imag., Vol. 25, pp. 1145-1157, September. 2006.

The area setting unit 15 that automatically sets the small areas E ₁ and E ₂ of the alignment candidates includes, for example, 1 basic area used for alignment on the reference image data such as the US image data 19-1 before treatment. The individual and plural auxiliary regions are automatically determined, and one region corresponding to the basic region is automatically determined on the target image data such as the US image data 19-2 after the treatment.

The area setting unit 15 automatically sets the small areas E ₁ and E ₂ as alignment candidates by the following method. The region setting unit 15 sets a set of reference image data and target image data for alignment, for example, US image data 19-1 and US image data 19-2 after treatment shown in FIG. After the initial tomographic plane image data is manually determined by rotating and translating in the Y and Z directions, the small areas E ₁ and E ₂ as alignment candidates are automatically set.
The region setting unit 15 first determines the center of the small region E ₁ having characteristic image data used for alignment from the reference image data, for example, US image data 19-1 before treatment, and then , determining the area size with respect to the center of the determined small region E _1, then, the image similarity is greatest small region between the small area E ₁ set in the US image data 19-1 E ₂ Is searched from, for example, US image data 19-2 after treatment, which is target image data.

Among them, the center of the determination of the small area E ₁ is aligned on the basis of the image similarity between US image data 19-2 after treatment with US image data 19-1 shown in FIG. 2, for example by the area setting unit 15 A method of automatically setting each of the candidate small regions E ₁ and E ₂ is used. In this method, a method of searching for a square region that maximizes entropy, which is the first method, or a method of using an index based on a difference in average value from the surrounding region, which is a second method, is used.
For example search small areas E ₂ from the US image data 19-2 after treatment, the method of alignment algorithms used in the alignment unit 16 to be described later is used.

However, the alignment parameters are 2D, that is, four rotations in the X and Y directions and each parallel movement, and six at most including the shear direction. For the determination of the center of the small region E _1, it is necessary to extract a characteristic portion in the image data, Kitchen-Rosenfeld operator used in this extraction of the image feature points in the computer vision, Hariss operator, SUSAN Its transformation based on operators is promising. As an example, the Moravec operator expanded to the region is shown below. The pixel having the maximum value in the obtained feature image data is used as the center of the candidate area.

An index based on the difference in average value from the surrounding area is used.
Consider a square S (x ₀ , y ₀ ) having a side length l with respect to a point (x ₀ , y ₀ ) on the image data. Consider, for example, eight squares Sj of the same size adjacent to the square S (x ₀ , y ₀ ). J is 1 ≦ j ≦ 8. The average value of the pixel values in S and Sj is set to μ and μ _j , respectively, and the following equation is calculated.

This calculated value is used as the feature amount of the point (x ₀ , y ₀ ). Such feature amount calculation is executed for all points of the reference image data such as the US image data 19-1, for example, to obtain the feature image data η (x, y). The feature image data η (x, y) is the center of the small region E ₁ alignment point of maximum value. Length 1 is determined empirically.

  References include, for example, Satoshi Kanazawa and Kenichi Kanaya, Extracting image feature points for explanatory computer vision, IEICE Vol.87, No.12, 2004, and [13] L. Kitchen and A Rosenfeld, “Gray-level corner detection,” Pattern Recognit. Lett., Vol.1, no.2, pp.95-102, Dec. 1982. and [19] C. Harris and M. Stephens, “A combined corner and edge detector, "Proc. 4th Alvey Vision Conf., pp.147-151, Manchester, UK, Aug. 1988. and [2] SM Smith and JM Brady,“ SUSAN | A New Approach to Low Level Image Processing, "IJCV, 23 (1), pp.45-78 (1997) and [31] HP Moravec,“ Towards automatic visual obstacle avoidance, ”Int. Joint Conf. Art. Intell., Cambridge, MA, USA, p.584, Aug. 1977.

The area setting unit 15 that automatically sets the small areas E ₁ and E ₂ of the alignment candidates as described above uses, for example, characteristic image data set in the reference image data such as the US image data 19-1 before treatment. feature amount between the small area E ₁ of the alignment candidate is searched from the target image data of the US image data 19-2, etc. after a plurality of areas for example the treatment of upper take an extreme value with. The area setting unit 15, the small to image similarity areas around each area among the searched plurality of regions on the target image data constituting the small region E ₁ and the pair of the reference image data becomes the largest The region E ₂ is searched, and the small region E ₁ and the small region E ₂ are paired.

That is, the region setting unit 15 creates each feature image data in the reference image data such as the US image data 19-1 before the treatment and the target image data such as the US image data 19-2 after the treatment for the alignment. After that, a plurality of large ones of these feature image data having extreme values are detected, and the image similarity between them is calculated to determine the small regions E ₁ and E ₂ of the alignment candidates.

The region setting unit 15 first generates feature image data in two pieces of two-dimensional image data, and then selects the top N of the feature image data whose feature values have extreme values. And then selecting a pair of feature image data from the two sets of candidates, calculating an image similarity for a region around the feature image data, and taking the maximum value of the feature image data of the pair Data is determined as small areas E ₁ and E ₂ .

The region setting unit 15 for determining such a pair of small regions E ₁ and E ₂ is not limited to determining _one pair of small regions E ₁ and E ₂ , but selects a plurality of pairs and sets them in advance. An optimal pair is determined based on the constraint conditions. The constraint condition is, for example, that a line segment connecting points selected as a pair does not intersect. This method can be considered as random voting based on image similarity.

  References include MA Fischler and RC Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM, no.24, vol.6, pp.381-395, June 1981. and PJ Rousseeuw and AM Leroy, Robust Regression and Outlier Detection, Wiley, New York, USA, 1987.

  The region setting unit 15 performs image alignment of two volume data, for example, reference image data such as US image data 19-1 before treatment and target image data such as US image data 19-2 after treatment. Recognize and change the algorithm for determining the alignment region according to the image attribute. Recognition of image attributes is automatically obtained from information such as US image data 19-1, 19-2 stored in the image database 9. The recognition of the image attribute is obtained by receiving a manual operation of the operation unit 12 such as a keyboard by an operator such as a doctor. Recognition of image attributes is obtained by a combination of these automatic acquisitions and acquisitions in response to manual operations.

  The attribute information of image data is roughly classified into two types: information related to generation of image data such as modality and presence / absence of contrast, and information related to anatomical feature points used for alignment. For example, in the treatment determination of liver cancer by ultrasonic puncture, it is necessary to compare and refer to, for example, US image data before treatment, US image data after treatment, and US contrast image data at least. For this purpose, it is necessary to align normal US image data, contrast US image data, and normal US image data and contrast US image data. The combination of these image data can be known from the information stored in the image database 9.

For example, when the attribute information “alignment due to contrast defect” is known, the region setting unit 15 uses the following algorithm. That is, the region setting unit 15 detects an isolated large low density region associated with a contrast defect from the reference image data in the following procedure.
First, the area setting unit 15 binarizes image data. At this time, the threshold value of the binarization processing is, for example, the literature (Nobuyuki Otsu, automatic threshold selection method based on discrimination and least squares method, IEICE Transactions, J63-D-4 (1980-4), 349 ? 356.) Is used.
Next, the region setting unit 15 forms a large closed region by, for example, a closing process. Next, the region setting unit 15 fills the periphery with a search from the outer periphery.
Next, the region setting unit 15 performs a labeling process on the remaining isolated region, and then determines the largest isolated region.

Then, the area setting unit 15, a region for alignment with the center of gravity of the isolated region, for example the center of the small region E _1.
Next, the area setting unit 15 determines the area size with respect to the center of the determined small area E ₁ as described above, and the image similarity with the small area E ₁ set in the US image data 19-1. There exploring the most significant subregion E ₂ after which for example the treatment the target image data from the US image data 19-2.

  In the case of contrasted US image data, it is promising to use a contrast defect site as an alignment region. In this case, for example, as shown in FIG. If an operator such as a doctor operates the mouse or the like of the operation unit 12 and clicks the first alignment button 29, the contrast US image data can be aligned with each other based on the alignment defect region.

Positioning unit 16 includes a small area E ₁ of the reference image data in the US, such as image data 19-1 before being set, for example, treated by the area setting unit 15, after treatment US image data 19-2, etc. of the target image obtains an image similarity between the small regions E ₂ on the data, the image similarity is to align the largest so as to reference image data and target image data.
In this case, the alignment unit 16 estimates the mutual information amount or entropy between the reference image data and the target image data as the image similarity, performs an optimization process, and based on the mutual information amount or the extreme value of the entropy. A relative position between the reference image data and the target image data is determined. When the registration unit 16 estimates the mutual information amount between the reference image data and the target image data, if there is a registration candidate region in a region that does not coexist in the reference image data and the target image data, Mutual information or entropy is estimated excluding the region.

  The alignment unit 16 displays a second alignment button 30 on the display 11 as shown in FIG. When an operator such as a doctor operates the mouse or the like of the operation unit 12 and clicks the second alignment button 30, the alignment unit 16 sets the reference image data and the image data so that the image similarity is maximized as described above. Align with target image data.

  The mutual information estimation method is as follows.

For example, target regions defined in two volume data of reference image data such as pre-treatment US image data 19-1 and the like and target image data such as post-treatment US image data 19-2 are represented by R (x, y , Z), F (x, y, z), and a two-dimensional histogram generated from pixel values at the same position in the two target regions is histgram _RF (r, f).

At this time, the probability density functions P _R (r), P _F (f), and P _RF (r, f) taken by the pixel values of the target region are expressed by the following equation from the two-dimensional histogram histgram _RF (r, f) defined above. It is calculated by.

V is the total number of pixel positions where both the target regions R (x, y, z) and F (x, y, z) exist.
From these probability density functions, the mutual information T (F; R) is defined as follows.

  Here, the coordinate conversion of the target image will be described. The target image is coordinate-transformed using at least nine parameters including rotation and translation in the X, Y, and Z directions, and if necessary, nine parameters including three shear directions. If this coordinate transformation is α, the target image after coordinate transformation can be represented by α (F).

Further, the alignment optimization method will be described. The mutual information amount T (α (F); R) between the target image α (F) after the coordinate transformation defined above and the target region R (x, y, z). Α that maximizes

Is the coordinate transformation you want to find.
Since this cannot be solved analytically, it must be solved using some optimization technique. As this optimization method, for example, a downhill / symplex method or a Powell method is used.

  References include, for example, F. Maes, “Multimodality Image Registration by Maximization of Mutual Information”, IEEE Trans. Med. Imag., Vol. 16, pp. 187-198, Apr. 1997, Press WH, Teukolsky SA, Vetterling WT, et al., Masakazu Enkeiji, Haruhiko Okumura, Toshiro Sato, other translations: Recipe for numerical computation in C language, pp.295-299, Technical Review, Tokyo, (1993).

For example, referring to the joint histogram shown in FIG. 6, for example, the reference image data R such as the US image data 19-1 before treatment and the target image data F such as the US image data 19-2 after treatment are roughly aligned. I will explain it. The joint histogram J includes a point I _R (x, y) on the reference image data R and a point I _F (x, y) on the target image data F that matches the point I _R (x, y). Created. When the reference image data R and the target image data F are aligned and the images match, the mutual information amount T between the reference image data R and the target image data F increases, and the straight line L It is represented by Therefore, aligning the reference image data R and the target image data F is to increase the mutual information amount T so that it is represented by the straight line L.

  Such an alignment unit 16 estimates the mutual information amount T (α (F); R) from the target regions R and F by excluding the outside of the volume data region included in R and F. In this case, the alignment unit 16 generates a joint histogram between the reference image data such as the pre-treatment US image data 19-1 and the target image data such as the post-treatment US image data 19-2. When generating the joint histogram, it may be calculated every time from the geometric information of the original data such as US raster data original data and the spatial information of R and F, for example.

In addition, when the entire volume data is generated in advance as part of the preprocessing, the alignment unit 16 may generate an array in which the inside of the area is represented by 1 and the outside of the area is represented by zero at the same time. Further, the outside of the area may be set to a specific pixel value (for example, 255), and the luminance may be expressed by 0 to 254.
That is, CT image data and MR image data take a very small value because the periphery of the image data is air. On the other hand, the US image data used for diagnosis is generated as zero outside the visual field around the in-vivo information of a subject such as a patient, regardless of whether the original data is beam data or rasterized data. The Here, what must be taken into consideration are the image data of the air portion around the CT image data and the MR image data, and the very small value or zero portion existing outside the visual field in the US image data. When the image data of the air portion in the CT image data and the MR image data and the small value or zero portion of the US image data are included in the alignment data, it is outside the region that is not the in-vivo information of the subject such as the imaged patient. As a result, they are aligned with each other, and correct alignment is often impossible.

For example, referring to FIG. 7, for example, the case where the reference image data R such as the US image data 19-1 before treatment and the target image data F such as the US image data 19-2 after treatment are aligned will be schematically described. To do. The reference image data R is composed of the original volume data Ra and an area outside the area (out-of-view) Rb of the volume data Ra. The target image data F is also composed of the original volume data Fa and an area outside the area (out-of-view) Fb of the volume data Fa. For the alignment of the reference image data R and the target image data F, for example, the image similarity between the regions Sa and Sb is obtained, and the reference image data R and the target image data are set so that the image similarity is maximized. Align with F.
At this time, the out-of-area Rb of the volume data R in the area Sa and the out-of-area Fb of the volume data F in the area Sb are excluded, that is, processed outside the field of view. Then, the image similarity between the regions Sa and Sb excluding the regions Rb and Fb is obtained, and the reference image data R and the target image data F are positioned so that the image similarity is maximized. To be combined. When the out-of-view processing is performed in this manner, the original volume data Ra of the reference image data R and the original volume data Fa of the target image data F match as shown in FIG. On the other hand, if there is no out-of-view processing, the original volume data Ra of the reference image data R and the original volume data Fa of the target image data F are not aligned as shown in FIG.

  The alignment unit 16 compares the first size outside the original image data in the reference image data R with the second size outside the original image data in the target image data F, and the first size is the second size. Is larger than the reference image data and the target image data, the mutual information amount T (α (F); R), the entropy H (R) of the reference image data R, the entropy H ( F) is estimated.

  That is, generally, US image data, which is three-dimensional volume data generated by an ultrasonic diagnostic apparatus, occupies a quadrangular pyramidal space region unlike CT image data and MR image data. When the relative positional relationship between such US image data is examined, the number of voxels as a target changes due to a change in the spatial area common to the two US image data due to the relative positional relationship. For this reason, the target parameter changes.

In particular, when the target image is near the outside of the original US image data area (small value or zero part), if the probability density distribution is the same, the smaller area makes the histogram sparse, resulting in mutual information. T (α (F); R) increases.
When the target image data is near the outside of the region, it often moves outside the region due to the optimization and does not converge to the correct extreme value. However, it is appropriate to switch to the target image data that is the movement target, which is less likely to have data outside the area around it.

The alignment unit 16 sends a command to display the alignment result to the display unit 17 when the optimization is completed.
When the alignment unit 16 is not satisfied with the alignment result, it returns to the initial setting image (UNDO function). In this case, the alignment unit 16 displays a redo switch 31 on the display 11 as shown in FIG. When an operator such as a doctor operates the mouse or the like of the operation unit 12 and clicks the redo switch 31, the alignment unit 16 returns to the initial setting image.
The alignment unit 16 can perform alignment again from the alignment result.

Positioning unit 16 includes a small region E ₁ on the reference image data such as, for example, pre-treatment of the US image data 19-1 as described above, the small on the target image data 19-2 such US image data after treatment obtains an image similarity between the region E _2, image similarity when this case the image similarity is to align the largest so as to reference image data and target image data, the alignment is deviated largely If the ratio between the image criterion and the image similarity criterion at the end of optimization is significantly smaller than the value obtained empirically, it is determined that a local solution has occurred, and this state is used as the initial value. Perform optimization again.

  In general, the optimization ends when it is no longer possible to improve the image similarity criterion. However, there is a possibility that it has fallen into a local solution at the end of this time. Naturally, the image similarity criterion at this time is smaller than that of the optimal solution, and the ratio with the image similarity criterion when the position is greatly shifted is compared with that of the optimal solution that is empirically known. It can be determined by doing. If it is determined that a local solution has fallen, it can be expected to converge to the optimal solution by changing the parameters a little from the position and executing the optimization again. For example, in the case of the downhill / symplex method, the parameter is changed by increasing the position of the simplex to be initialized from the previous time.

  The alignment unit 16 calculates image similarity for a plurality of positions around the initial position set on the reference image data before starting the optimization, searches for a better initial position, and then performs optimization. Execute. In order to perform the process before starting the optimization, the alignment unit 16 displays a third alignment switch 32 on the display 11 as shown in FIG. 3, for example. If an operator such as a doctor operates the mouse or the like of the operation unit 12 and clicks the third alignment switch 32, the alignment unit 16 performs a process before starting optimization.

  For example, in the case of a plurality of volume data such as US image data collected continuously during a test on a subject such as a patient, it can be expected that the relative position between these volume data does not change much. On the other hand, generally, there is a great difference in the posture of a subject such as a patient to be imaged between image data having different photographing dates and times and image data having different modalities. In this case, the initial position may be set outside the range where the optimization succeeds (capture range).

  In order to solve this, before the optimization is started, a better initial position is roughly searched around the initial position set on the reference image data, thereby narrowing down within the capture range. For example, when positioning is performed with respect to the rotation of the three axes and the translation of the three axes, for example, with respect to six parameters, for example, the range of ± 30 degrees for the rotation is every 10 degrees, for example, for the rotation. For example, 7 ^ 6 provisional initial position images are created for every 10 pixels within a range of ± 30 pixels, the image similarity with the reference image data is calculated, and the position with the highest similarity is defined as the initial position. To do. This is especially true for translation. Regarding the rotation, at least the overlap of the regions does not change, but attention is paid to the fact that the amount of overlap changes in the parallel movement.

  The alignment unit 16 calculates, for each of the basic region set on the reference image data and the plurality of auxiliary regions, the image similarity between the corresponding regions on the target image data and integrates them. Align with the indicator. Integration methods include index product and weighted sum.

Let R _B be the basic region set in the reference image data, and Ri be the plurality of auxiliary regions. i is 1 to n. And region F _B corresponding to one area which is set on the target image data in the basic region. The position of this region F _B with reference to determine Fi. These regions between the image similarity index _{T B (F B; R B} ), T i (F i; R i): calculating a i = 1 to n. Each index T, T integrated from these is created.

  References include, for example, JPWPluim, JBAMaintz, MAViergever, “Image Registration by Maximization of Combined Mutual Information and Gradient Information”, IEEE Trans. Med. Imag., Vol. 19, pp. 809-814, August 2000. is there.

  When the alignment unit 16 performs alignment using the integrated index, the alignment unit 16 performs alignment using only the basic region, and uses the integrated index when it converges to some extent. This is because there is a high probability that the initial position of the auxiliary area is shifted.

The alignment unit 16 is provided between the reference image data R and the target image data F, for example, reference image data R such as US image data 19-1 before treatment and target image data such as US image data 19-2 after treatment. When the mutual information amount T between the reference image data R and the target image data F is estimated when the mutual information amount T with the F is estimated, the mutual information amount T is corrected according to the size of the non-overlapping region. In this case, as shown in the following equation (12), the registration unit 16 entropy H (R) of the reference image data R, entropy H (F) of the target image data, reference image data R, and target image data F The mutual information T is obtained based on the entropy H (R, F) of the joint histogram generated based on the above, and the entropy H (R, F) of the joint histogram J in the mutual information T Correction M according to the size of the non-overlapping area is performed.
T = H (R) + H (F) -H (R, F, M) (12)
That is, for example, the reference image data R such as the pre-treatment US image data 19-1 and the target image data F such as the post-treatment US image data 19-2 are given to the subject such as the affected part before and after the operation. When an ultrasound image is acquired by applying an ultrasonic probe, it is difficult to make the posture of the subject in the same position, for example, the degree of twist when the posture is in the horizontal direction, and the ultrasonic probe is applied to the subject. Since it is difficult to make the directions the same, a region G that does not overlap even if the reference image data R and the target image data F are aligned as shown in FIG.

  As a result, when the reference image data R and the target image data F are completely coincident with each other, the reference image data R and the target image data F have the same number of overlapping pixels. Less than the number of pixels. When the reference image data R and the target image data F are aligned, for example, a joint histogram J as shown in FIG. 6 is generated, but the number of pixels whose positions overlap between the reference image data R and the target image data F The decrease in the number of samples means that the number of samples when generating the joint histogram J (the number of pixels whose positions overlap between the reference image data R and the target image data F) decreases. Then, the entropy H (R, F) of the joint histogram J becomes smaller due to the decrease in the number of samples.

  In order to align the reference image data R and the target image data F, the mutual information amount T between the reference image data R and the target image data F is increased. However, as shown in FIG. 8, there is a region G that does not overlap even if the reference image data R and the target image data F are aligned, and the number of overlapping pixels between the reference image data R and the target image data F Decreases, the mutual information amount T decreases. For this reason, the reference image data R and the target image data F shift in a direction in which they are not aligned.

  However, the correction M is performed according to the size of the region that does not overlap with the entropy H (R, F) of the joint histogram J in the mutual information amount T. As a result, even if the region G that does not overlap even if the reference image data R and the target image data F are aligned, the reference image data R and the target image data F can be aligned.

  The display unit 17 displays two volume data of reference image data such as the US image data 19-1 before treatment and target image data such as the US image data 19-2 after treatment, which are aligned by the alignment unit 16. Displayed on the display 11. The US image data 19-1, US image data 19-2, and the like displayed on the display 11 are cross-sectional image data or three-dimensional cross-sectional image data of a subject such as a patient. The display unit 17 rotates at least the reference image data such as the US image data 19-1 before the treatment and the target image data such as the US image data 19-2 after the treatment, which are aligned by the alignment unit 16, at least. The movement is performed synchronously and displayed on the display 11.

  The display unit 17 includes a plurality of sets of image data individually aligned, for example, results of alignment of two volume data of US image data 19-1 before treatment and US image data 19-2 after treatment, Based on the result of alignment between the X-ray CT image data before treatment and the US image data 19-2 after treatment, the result of alignment between the US image data 19-2 after treatment and the US contrast image data, etc. Then, the relative positions of all the volume data included in the image data are obtained by integration.

For example, alignment is performed using four types of volume data A, B, C, and D, for example, volume data A and B, volume data B and C, and volume data C and D are aligned, and their coordinate conversion is performed. It is assumed that α _AB , α _BC , and α _CD are obtained. When these three coordinate transformations α _AB , α _BC , α _CD are obtained, the display unit 17 calculates the coordinate transformations α _AC , α _AD using these coordinate transformations α _AB , α _BC , α _CD . As a result, the volume data B, C, and D can be integratedly aligned based on the volume data A.
In order to perform integrated positioning of such volume data B, C, D, etc., the display unit 17 displays a positioning integration switch 33 on the display 11 as shown in FIG. When an operator such as a doctor operates the mouse of the operation unit 12 and clicks the alignment integration switch 33, the display unit 17 performs integrated alignment of the volume data B, C, D, and the like.

The display unit 17 synchronously changes the cross-sectional position and the three-dimensional line-of-sight direction by rotating and translating in the X, Y, and Z directions in the integrated N volume data. In this case, the display unit 17 performs selection of volume data for changing the cross-sectional position displayed in synchronization.
The display unit 17 includes, for example, a grid on the plurality of cross-sectional images after alignment displayed on the display 11, for example, on the cross-sectional images of the US image data 19-1 before treatment and the US image data 19-2 after treatment. Display the scale. FIG. 9 shows an example in which a grid-like scale 34 is displayed on each cross-sectional image of US image data 19-1 and US image data 19-2, for example.

The display unit 17 includes a point on the plurality of cross-sectional images after alignment displayed on the display 11, for example, a point on the US image data 19-1 before treatment, and the US image data 19-2 after treatment corresponding to this point. A cursor is displayed at each of the upper points. FIG. 10 shows an example in which, for example, “+” shaped cursors 35a and 35b are displayed at corresponding points on the cross-sectional images of the US image data 19-1 and the US image data 19-2.
The display unit 17 is not limited to the “+”-shaped cursors 35a and 35b, for example, to display the points on the US image data 19-2 after treatment corresponding to the points on the US image data 19-1 before treatment. Instead, for example, a cursor formed in a line segment, rectangle, circle, or the like may be displayed.

  The display unit 17 selects one image from a plurality of image data after alignment, for example, a plurality of reference image data, a plurality of target image data, and each cross-sectional image data, and superimposes the other image data on one image data. To display. The display unit 17 displays the target image data such as the US image data 19-2 after treatment superimposed on the reference image data such as the US image data 19-1 before treatment, for example. In the superimposition process, for example, a portion of the target image data that is equal to or greater than the threshold is extracted using a variable threshold, and the extracted image data is transparently superimposed on the reference image data by changing the color. In the superimposition process, for example, contour region data such as a missing portion is extracted from the target image data using a variable threshold, and the extracted contour region data such as the missing portion is superimposed on the reference image data. The superimposed result changes following the change of the cross-sectional position.

  The display unit 17 compares each field of view among a plurality of cross-sectional image data after alignment, for example, a plurality of reference image data, a plurality of target image data, each cross-sectional image data, in particular, X-ray CT image data, US image data, and the like. In this case, two pieces of image data in which one field of view is larger than the other field of view are selected, and a field contour occupied by image data having a small field of view is displayed in image data having a large field of view, for example, X-ray CT image data.

  The display unit 17 regards a plurality of volume data with different fields of view taken under the same conditions, for example, US image data 19-1 before treatment and US image data 19-2 after treatment as virtually one volume data. , X direction, Y direction, and Z direction are combined and displayed in one window in accordance with the cross-sectional position change by rotation and translation. FIG. 11 shows a display example in which US image data 19-1 and US image data 19-2 are combined. When there are a plurality of candidates as pixel value candidates at a certain coordinate, a pixel of one volume data is selected, or an average value is compounded.

  In this case, the display unit 17 detects a boundary region between a plurality of image data, for example, a boundary region 36 between reference image data such as US image data 19-1 and target image data such as US image data 19-2, The pixel value of the boundary region 36 is determined from a plurality of volume data, for example, US image data 19-1 and US image data 19-2. Then, the display unit 17 determines, for example, a region using the reference image data, a region using the target image data, and a boundary region thereof. In the boundary region, the closer to each region, the larger the weight between the two image data. Interpolate. As a result, the boundary area between the plurality of image data, for example, the seam that is the US image data 19-1, the US image data 19-2, and the boundary area 36 becomes inconspicuous.

  References include AHGee, GMTreece, RWPrager, CjCCash and L. Berman, “Rapid Registration for Wide Field of View Freehand Three-Dimensional Ultrasound”, IEEE Trans. Med. Imag., Vol. 22, pp. 1344-1357, November 2003 and G.Xiao, JMBrady, JANoble, M. Burcher, and R. English, “Non-rigid Registration of 3D- Free-Hand Ultrasound Images of the Breast”, IEEE Trans. Med .Imag., Vol.21, pp. 405-412, April 2002.

  The image storage unit 18 displays a plurality of aligned image data, for example, reference image data such as US image data 19-1 before treatment and target image data such as US image data 19-2 after treatment, in the image database 9. Save to. At this time, the image storage unit 18 stores a plurality of image data, and also stores the data names of these image data, presence / absence of mirror and reverse, integrated pixel size, image size reference data name, volume data relative position information, and the like. Save to. The image storage unit 18 displays an image storage button 37 on the display 11 as shown in FIG. When an operator such as a doctor operates the mouse of the operation unit 12 and clicks the image storage button 37, the image storage unit 18 stores the image data in the plurality of aligned image data image databases 9.

The image storage unit 18 includes a plurality of registered image data stored in the image database 9, for example, reference image data such as US image data 19-1 before treatment and US image data 19-2 after treatment. The target image data is read out, and necessary processing such as mirroring, reverse, and pixel size adjustment is executed on the reference image data and the target image data to reproduce the relative position at the time of storage. The image storage unit 18 displays a reproduction button 38 on the display 11, for example, as shown in FIG. When an operator such as a doctor operates the mouse of the operation unit 12 and clicks the reproduction button 38, the image storage unit 18 needs to perform mirror, reverse, pixel size adjustment, etc. on the reference image data and the target image data. Execute the process and reproduce the relative position at the time of storage.
The image storage unit 18 stores and reproduces a plurality of volume data such as the aligned US image data 19-1 before treatment and the US image data 19-2 after treatment as one virtual volume.

Next, an image data alignment operation performed by the apparatus configured as described above will be described.
The image alignment apparatus 1 takes in a plurality of image data such as US image data from a plurality of modalities, for example, the US diagnostic apparatus 3, the X-ray CT apparatus 4, the MR apparatus 5 and the like, via the network 2, and stores these image data. Store in the image database 9.
For example, in the treatment determination of liver cancer by ultrasonic puncture, it is necessary to compare and refer to, for example, US image data before treatment, US image data after treatment, and US contrast image data at least. Of these, the pre-treatment US image data and the post-treatment US image data have a difference in the position of the subject such as a patient at the time of photographing, a difference in the presence or absence of imaging, and further due to the progress of treatment or medical condition There is a change in the shape of the subject itself. These factors make it necessary to spatially align each US image data before and after treatment. In addition, when comparing and referring to US image data before treatment and US contrast image data after treatment, or X-ray CT image data of other modalities, it is necessary to spatially align these image data.

  When an image data read out by an operator such as a doctor operating the mouse of the operation unit 12 is instructed, the data reading unit 13 reads, for example, a patient or the like from the image data of each modality stored in the image database 9, for example. US image data 19-1 before the liver treatment and US image data 19-2 after the liver treatment are read out. These US image data 19-1 and 19-2 are, for example, liver cross-sectional images. The pre-treatment US image data 19-1 and post-treatment US image data 19-2 are displayed on the display 11 by the display unit 17 as shown in FIG. 2.

  Next, the data attribute changing unit 14 sets at least attributes relating to rotation and translation with respect to the pre-treatment US image data 19-1 and the post-treatment US image data 19-2 read by the data reading unit 13. Change and display on the display 11. The rotation, parallel movement, and the like of the pre-treatment US image data 19-1 and the post-treatment US image data 19-2 are instructed by an operator such as a doctor operating the mouse or the like of the operation unit 12, for example. For example, when an operator such as a doctor operates the mouse or the like of the operation unit 12, the data attribute changing unit 14 sets the luminance values in the US image data 19-1 before treatment and the US image data 19-2 after treatment. Adjust, adjust the pixel size, or change the spatial orientation / direction.

Next, in the US image data 19-1 before treatment and the US image data 19-2 after treatment, for example, as shown in FIG. 4, the setting of the small regions E ₁ and E ₂ of the alignment candidates is set by manual operation or automatically. It is done by the target.
First, manual setting of each of the small regions E ₁ and E ₂ of the alignment candidates will be described.
The region setting unit 15 receives a manual operation on the operation unit 12 by an operator such as a doctor and performs a pair of pre-treatment US image data 19-1 and post-treatment US image data 19- 2 is displayed on the display 11, and the set of US image data 19-1 before treatment and US image data 19-2 after treatment are centered on the display 11, respectively in the X direction, the Y direction, and the Z direction. The initial setting is performed by rotating the shaft as an axis and translating in the X, Y, and Z directions.

  US image data is often taken from an imaging direction that is limited by anatomical restrictions of a subject such as a patient. For example, in the case of liver cancer diagnosis, US image data is often taken from a specific imaging direction, such as under the arch or between the intercostals. Therefore, the relative position to the coordinate system perpendicular to the body axis in these specific photographing directions is preset and registered. For example, the shooting direction when shooting from a specific shooting direction such as under the bow or between the eyelids is preset. In such a case, for example, if an operator such as a doctor clicks the mouse of the operation unit 12 on the preset button 28 displayed on the display 11 as shown in FIG. 3, the region setting unit 15 is preset. The imaging directions of the pre-treatment US image data 19-1 and the post-treatment US image data 19-2 can be aligned to a rough position within a predetermined misalignment range in a single process.

  In addition, the region setting unit 15 displays each image display mode of the US image data 19-1 before treatment and the US image data 19-2 after treatment, for example, normal MPR display, maximum value display with thickness, and minimum value with thickness. It can be switched to display, MPR display with thickness, or volume rendering display.

Next, the region setting unit 15 receives a click operation on the mouse of the operation unit 12 by an operator such as a doctor, and initially sets reference image data and target image data, for example, US image data 19-1 before treatment shown in FIG. And the sub-regions E ₁ and E ₂ of the alignment candidates are set on each image data of the US image data 19-2 after treatment. Each of the small regions E ₁ and E ₂ of the alignment candidates can be set by a manual operation by a so-called operator such as a doctor. The area setting unit 15 displays each rectangular area E ₁ , E _{2 as an} alignment candidate by displaying a rectangular boundary with a solid line “□” as shown in FIG. 5, for example, and displaying this rectangular center with a cross “+”. To do. The small areas E ₁ and E ₂ of the alignment candidates are set, for example, having an anatomically characteristic form.

The region setting unit 15 receives an operation on the operation unit 12 by an operator such as a doctor so that the sizes of the small regions E ₁ and E ₂ of the alignment candidates are always constant at the actual size of the subject such as a patient. In addition, the sizes of the small areas E ₁ and E ₂ are adjusted according to the pixel size of the target image data. The region setting unit 15 receives an operation on the operation unit 12 by an operator such as a doctor, and manually adjusts the sizes of the small regions E ₁ and E ₂ without moving the positions of the small regions E ₁ and E ₂ as alignment candidates. To change. The region setting unit 15 receives an operation on the operation unit 12 by an operator such as a doctor and changes the positions of the small regions E ₁ and E ₂ without changing the sizes of the small regions E ₁ and E ₂ of the alignment candidates. To do.

On the other hand, automatic setting of the small areas E ₁ and E ₂ of the alignment candidates will be described.
The area setting unit 15 sets each sub-area of the registration candidate based on the image similarity between the reference image data and the target image data, for example, the US image data 19-1 shown in FIG. 2 and the US image data 19-2 after treatment. E ₁ and E ₂ are set automatically. For the automatic setting of each of the sub-regions E ₁ and E ₂ of the alignment candidates, the first method for searching for a square region that maximizes entropy as described above, or an index based on the difference in average value from surrounding regions is used. This is the second method used.

The area setting unit 15 automatically sets the small areas E ₁ and E ₂ as alignment candidates by the following method, for example. This region setting unit 15 performs rotation in the X direction, Y direction, and Z direction, respectively, for a set of US image data 19-1 and US image data 19-2 after treatment shown in FIG. After performing the parallel movement and manually determining the initial tomographic plane image data, the small areas E ₁ and E ₂ as the alignment candidates are automatically set. In this case, the region setting unit 15 first determines the center of the small region E ₁ having characteristic image data used for alignment from the pre-treatment US image data 19-1. Then, the area setting unit 15 determines the area size with respect to the center of the determined small region E _1. Then, the area setting unit 15 searches from the US image data 19-2 image similarity is the largest small area E _2, for example after treatment between the small regions E ₁ set in the US image data 19-1 .

Among them, the center of the determination of the small area E ₁ is aligned on the basis of the image similarity between US image data 19-2 after treatment with US image data 19-1 shown in FIG. 2, for example by the area setting unit 15 A method of automatically setting each of the candidate small regions E ₁ and E ₂ is used. This method uses a first method that searches for a square region that maximizes entropy, or a second method that uses an index based on a difference in average value from a surrounding region.

Next, the alignment unit 16, a small region _{E 1} on the set for example to pre-treatment US image data 19-1 by the region setting unit 15, a small area on the US image data 19-2 after treatment _{E 2} The image similarity between the reference image data and the target image data is adjusted so that the image similarity is maximized. In this case, the alignment unit 16 estimates the mutual information amount T between the pre-treatment US image data 19-1 and the post-treatment US image data 19-2 as an image similarity, and performs an optimization process. Based on the extreme value of the mutual information amount T, the relative position between the pre-treatment US image data 19-1 and the post-treatment US image data 19-2 is determined.

For example, referring to the joint histogram shown in FIG. 6, for example, the registration of the reference image data R of the US image data 19-1 before treatment and the target image data F of the US image data 19-2 after treatment will be described. The joint histogram J includes a point I _R (x, y) on the reference image data R, a point I _F (x, y) on the target image data F that matches the point I _R (x, y), and Created by. When the reference image data R and the target image data F are aligned and the images match, the mutual information amount T between the reference image data R and the target image data F increases, and the straight line L It is represented by Therefore, aligning the reference image data R and the target image data F is to increase the mutual information amount T so that it is represented by the straight line L.

And when the alignment part 16 estimates the mutual information amount T between the US image data 19-1 before treatment and the US image data 19-2 after treatment, the US image data 19-1 before treatment and If there is an alignment candidate area in an area that does not coexist with the US image data 19-2 after treatment, the mutual information amount T is estimated except for the area. The method for estimating the mutual information amount T is as described above. For example, as shown in FIG. 7, the reference image data R and the target image data F are aligned, for example, the image similarity between the areas Sa and Sb is obtained, and the reference image is maximized. When the data R and the target image data F are aligned, the outside area Rb of the volume data R in the area Sa and the outside area Fb of the volume data F in the area Sb are excluded. Then, the image similarity between the regions Sa and Sb excluding the regions Rb and Fb is obtained, and the reference image data R and the target image data F are positioned so that the image similarity is maximized. To be combined.
As a result, when the registration unit 16 determines the relative position between the pre-treatment US image data 19-1 and the post-treatment US image data 19-2, for example, as shown in FIG. The US image data 19-2 after treatment is translated in, for example, the respective movement directions f ₁ and f ₂ and rotated in the rotation direction f ₃ without moving the US image data 19-1. Thereby, the alignment unit 16 aligns the US image data 19-1 before treatment and the US image data 19-2 after treatment, for example, as shown in FIG. The alignment unit 16 sends a command for displaying the alignment result to the display unit 17 when the optimization is completed.

  Next, the display unit 17 displays, on the display 11, two volume data, for example, the US image data 19-1 before treatment and the US image data 19-2 after treatment, which are aligned by the alignment unit 16. For example, as shown in FIG. 9, the display unit 17 includes, on the plurality of cross-sectional images after alignment displayed on the display 11, the US image data 19-1 before treatment and the US image data 19-2 after treatment. For example, a grid-like scale 34 is displayed on each cross-sectional image. The display unit 17 displays, for example, a point on the US image data 19-1 before treatment displayed on the display 11 and a point on the US image data 19-2 after treatment corresponding to this point. As shown in FIG. 10, the cursors 35a and 35b having a “+” shape are displayed. Further, the display unit 17 synthesizes and displays the US image data 19-1 and the US image data 19-2 as shown in FIG. In this case, the display unit 17 detects a boundary region 36 between the US image data 19-1 and the US image data 19-2, and the pixel value of the boundary region 36 is set to a plurality of volume data, for example, US image data 19-1. And the US image data 19-2, and the boundary region 36 interpolates between the two US image data 19-1 and the US image data 19-2 with a larger weight as it is closer to each region.

  Next, the image storage unit 18 stores, for example, the pre-treatment US image data 19-1 and the post-treatment US image data 19-2 in the image database 9. At this time, the image storage unit 18 stores the pre-treatment US image data 19-1 and the post-treatment US image data 19-2 in the image database 9, and the pre-treatment US image data 19-1. The data name with the US image data 19-2 after treatment, presence or absence of mirror or reverse, integrated pixel size, image size reference data name, volume data relative position information, etc. are stored in the image database 9. Note that the image data is stored, for example, when an operator such as a doctor clicks the mouse or the like of the operation unit 12 on an image storage button 37 displayed on the display 11 as shown in FIG.

  When an operator such as a doctor operates the mouse or the like of the operation unit 12 and clicks the reproduction button 38, the image storage unit 18 stores a plurality of aligned image data stored in the image database 9, for example, US before treatment. The reference image data such as the image data 19-1 and the target image data such as the US image data 19-2 after the treatment are read out, and the reference image data and the target image data are subjected to mirror, reverse, pixel size adjustment, etc. Perform the necessary processing and reproduce the relative position at the time of storage.

  On the other hand, for example, the reference image data R of the US image data 19-1 before the treatment and the target image data F such as the US image data 19-2 after the treatment are more than the subject such as the affected part before and after the operation. When acquiring an ultrasound image by applying an ultrasonic probe, it is difficult to make the posture of the subject to be in the same state, for example, the degree of twisting when the posture is in the horizontal direction, and the direction in which the ultrasonic probe is applied to the subject. Since it is difficult to make them identical, a region G that does not overlap even if the reference image data R and the target image data F are aligned as shown in FIG.

  In this case, the alignment unit 16 is configured such that the reference image data R between the reference image data R and the target image data F, for example, the reference image data R such as the US image data 19-1 before treatment and the US image data 19-2 after treatment, When the mutual information amount T between the target image data F and the target image data F is estimated, if a region where the reference image data R and the target image data F do not overlap is generated, the mutual information amount T is corrected according to the size of the non-overlapping region. To do. At this time, the alignment unit 16 calculates the above equation (12) to obtain the mutual information amount T between the reference image data R and the target image data F, and the entropy H ( Correction M is performed according to the size of the region that does not overlap R, F). As a result, even if the region G that does not overlap even if the reference image data R and the target image data F are aligned, the reference image data R and the target image data F can be aligned.

As described above, according to the embodiment, for example, the reference image data R such as the US image data 19-1 before treatment and the target image data F such as the US image data 19-2 after treatment are anatomically analyzed. The sub-regions E ₁ and E ₂ of the alignment candidates are set where the characteristic image is present, the image similarity between the small regions E ₁ and E ₂ is obtained, and the image similarity is maximized. The reference image data R such as the US image data 19-1 before treatment and the target image data F such as the US image data 19-2 after treatment are aligned. As a result, alignment between US image data and between US image data and image data of other modalities can be accurately performed in a short time.

  For example, in the treatment determination of liver cancer by ultrasonic puncture, it is necessary to compare and refer to, for example, US image data before treatment, US image data after treatment, and US contrast image data at least. Of these, the pre-treatment US image data and the post-treatment US image data have a difference in the position of the subject such as a patient at the time of photographing, a difference in the presence or absence of imaging, and further due to the progress of treatment or medical condition There is a change in the shape of the subject itself. These factors make it necessary to spatially align each US image data before and after treatment. In addition, when comparing and referring to US image data before treatment and US contrast image data after treatment, or X-ray CT image data of other modalities, it is necessary to spatially align these image data. Such alignment between US image data or between US image data and image data of other modalities, for example, X-ray CT image data can be accurately performed in a short time.

However, it can be used in multi-modalities such as the US diagnostic apparatus 3, the X-ray CT apparatus 4, the MR apparatus 5, and the like.
The region setting unit 15 rotates the pre-treatment US image data 19-1 and post-treatment US image data 19-2 on the display 11 with the X, Y, and Z directions as the central axes, respectively. Since the initial setting is performed by parallel movement in each of the direction, the Y direction, and the Z direction, strict initial position setting is not required.
A subject such as an affected part may be lost in the image data due to deformation due to torsion of the body position or the like, and a common characteristic part is not required for alignment of many image data.

The image similarity between the small regions E ₁ and E _{2 of the} alignment candidates is obtained, and the reference image data R such as the US image data 19-1 before the treatment and the US after the treatment so that the image similarity becomes the largest. Since the target image data F such as the image data 19-2 is aligned, the arithmetic processing by the main control unit 6 such as the CPU at this time may be only a small amount of data in each of the small areas E ₁ and E ₂ . Speed can be greatly improved. That is, since the entire data of the pre-treatment US image data 19-1 and the post-treatment US image data 19-2 is not processed, the processing speed can be increased. For example, in the case of a small small area E ₁ that reflects an anatomical feature that has been determined empirically around the position specified on the reference image data R, for example, in the case of a blood vessel in the liver, If the cube is about 2 cm on a side and the size of the voxel is 0.5 mm ³ , an area of 40 × 40 × 40 voxels is set and used as a reference image. Since sets each subregion E _1, E ₂ alignment candidate in a small area, the spatial transformation used for alignment can be assumed to be rigid deformation.

An operator such as a doctor performs the same anatomical feature point in, for example, the US image data 19-1 before treatment and the US image data 19-2 after treatment by clicking the mouse of the operation unit 12 manually. It is possible to roughly select cross-sectional image data in which small regions E ₁ and E _{2 having} are recognized.
Furthermore, the region setting unit 15 is a registration candidate that is an anatomical identical point based on, for example, the image similarity between the pre-treatment US image data 19-1 and the post-treatment US image data 19-2 shown in FIG. Each of the small areas E ₁ and E ₂ can be automatically set.

The region setting unit 15 receives an operation on the operation unit 12 by an operator such as a doctor, and targets the size of each of the small regions E ₁ and E ₂ of the alignment candidates to be always constant at the actual size of the subject such as a patient. The sizes of the small areas E ₁ and E ₂ can be adjusted according to the pixel size of the image data. In addition, the region setting unit 15 receives an operation on the operation unit 12 by an operator such as a doctor, and does not move the position of each of the small regions E ₁ and E _{2 that} are candidates for alignment, and the size of these small regions E ₁ and E ₂ . Can be changed manually. The region setting unit 15 receives an operation on the operation unit 12 by an operator such as a doctor and changes the positions of the small regions E ₁ and E ₂ without changing the sizes of the small regions E ₁ and E ₂ of the alignment candidates. it can.

  Before starting the search, the area around the initial position is squeezed at appropriate intervals, and an option to optimize the position where the image similarity is maximized as the initial position is provided. Thus, in consideration of the case where the relative difference between the two three-dimensional image data to be aligned, for example, the US image data 19-1 before treatment and the US image data 19-2 after treatment is relatively large, With respect to the alignment parameters, for example, six parameters of three-axis rotation and three-axis translation about the position designated on the target image data such as US image data 19-2, a limited range For example, for rotation, the image similarity between the reference image data and the target image data is calculated every 10 degrees for a range of ± 30 degrees and for every 10 pixels for a range of ± 30 pixels for translation. In such a case, for example, 7 ^ 6 searches are performed, and the relative position with the highest similarity is set as the initial value. It is possible to narrow down the point that is disadvantageous to the displacement of the translation amount of the small comparison area within the capture range by roughly searching around the initial position.

  The image storage unit 18 displays a plurality of aligned image data, for example, reference image data such as US image data 19-1 before treatment and target image data such as US image data 19-2 after treatment, in the image database 9. Save to. At this time, the image storage unit 18 stores a plurality of image data, and also stores the data names of these image data, presence / absence of mirror and reverse, integrated pixel size, image size reference data name, volume data relative position information, and the like. Save to. In this state, when an operator such as a doctor operates the mouse of the operation unit 12 and clicks the reproduction button 38, the image storage unit 18 stores a plurality of aligned image data stored in the image database 9, for example, Read out the reference image data such as the US image data 19-1 before treatment and the target image data such as the US image data 19-2 after treatment, and perform mirror, reverse, and pixel on the reference image data and the target image data. Necessary processing such as size adjustment can be executed, and the relative position at the time of storage can be reproduced.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

1 is a block configuration diagram showing an embodiment of an image alignment apparatus according to the present invention. The figure which shows each US image data before and after the treatment displayed on the display in the apparatus. The figure which shows the example of a display of each US image data before and after treatment displayed on the display in the same apparatus, X-ray CT image data before treatment, and US contrast image data. The figure which shows each area | region of the alignment candidate set to each US image data before and after the treatment displayed on the display in the apparatus. The figure which shows an example of the shape of the small area | region of the alignment candidate set by the area | region setting part in the same apparatus. The figure which shows the joint histogram used for position alignment with the reference image data by the position alignment part in the same apparatus, and object image data. The conceptual diagram which shows the position alignment with the reference image data in which the position alignment part in the same apparatus exists, and object image data. The figure which shows the alignment when the area | region which the reference image data and the target image data do not overlap by the alignment part in the same apparatus arises. The figure which shows an example of US image data which displayed the scale by the display part in the same apparatus. The figure which shows an example of US image data which displayed the cursor by the display part in the same apparatus. The figure which shows the example of a display which synthesize | combined each US image data by the display part in the same apparatus. The figure which shows the alignment operation | movement with the US image data before a treatment by the position alignment part in the same apparatus, and the US image data after a treatment.

Explanation of symbols

1: image alignment device, 2: network, 3: US diagnostic device, 4: X-ray CT device, 5: MR device, 6: main control unit, 7: program memory, 8: data memory, 9: image database, 10: Transmission unit, 11: Display, 12: Operation unit, 13: Data reading unit, 14: Data attribute changing unit, 15: Area setting unit, 16: Positioning unit, 17: Display unit, 18: Image storage unit, 19-1: US image data before treatment, 19-2: US image data after treatment, 20: X-ray CT image data before treatment, 21: US contrast image data, 22: Image discrimination data, 23: Size adjustment Buttons, 24: scale, 25: first mirror operation unit setting button, 26: second mirror operation unit setting button, 27: reverse unit setting button, 28: preset button, 29: first position Adjustment button, 30: second alignment button, 31: redo switch, 32: third alignment switch, 33: alignment alignment switch, 34: scale, 35a, 35b: cursor, 36: boundary region, 37: Image save button, 38: Reproduction button, E ₁ , E ₂ : Small region of alignment candidates.

Claims (29)

Each region of the alignment candidate formed in the size required to select the feature point on the three-dimensional reference image data and on at least one three-dimensional target image data to be aligned with the reference image data, respectively An area setting section to be set;
An alignment unit that obtains an image similarity between the regions set by the region setting unit and aligns the reference image data and the target image data so that the image similarity is maximized;
A display unit for displaying the reference image data and the target image data aligned by the alignment unit on a display;
An image alignment apparatus comprising: A data reading unit for reading the reference image data and the target image data;
A data attribute changing unit for changing at least the attribute relating to rotation and translation with respect to the reference image data and the target image data read by the data reading unit and displaying them on the display;
The image alignment apparatus according to claim 1, further comprising:   The image alignment apparatus according to claim 2, wherein the data reading unit reads the reference image data and the target image data acquired by the same or different modalities.   The image alignment apparatus according to claim 2, wherein the data reading unit reads at least one of the reference image data and the target image data as ultrasonic image data.   The image alignment apparatus according to claim 2, wherein the data reading unit reads the reference image data and the target image data having different pixel sizes.   The image alignment apparatus according to claim 2, wherein the data attribute changing unit matches at least the pixel sizes of the reference image data and the target image data having different pixel sizes.   The image alignment apparatus according to claim 2, wherein the data attribute changing unit changes at least a spatial direction of the reference image data and the target image data. An image storage unit for storing the reference image data and the target image data aligned by the alignment unit;
The image alignment apparatus according to claim 1, further comprising:   The image registration device according to claim 1, wherein the region setting unit adjusts the size of the region of the alignment candidate according to a pixel size of the target image data.   The image position according to claim 9, wherein the region setting unit adjusts the size of the region so as to maximize entropy obtained from a frequency distribution of pixels of the reference image data and the target image data. Alignment device.   The image position according to claim 9, wherein the area setting unit adjusts the size of the area based on a difference between average values of pixel values of the area of the alignment candidate and a peripheral area of the area. Alignment device.   The image registration device according to claim 1, wherein the region setting unit displays a center and a boundary of the region of the registration candidate on the display.   The region setting unit is configured to set the regions of the alignment candidates on the reference image data and the target image data in response to a manual selection operation of the regions of the alignment candidates. The image alignment apparatus according to claim 1. The reference image data and the target image data are displayed on the display, and when the manual selection operation of the region of the alignment candidate is performed on the display screen,
The region setting unit receives the manual selection operation and sets the regions of the alignment candidates on the reference image data and the target image data, respectively.
The image alignment apparatus according to claim 13.   The image registration device according to claim 1, wherein the region setting unit automatically sets the regions of the alignment candidates based on an image similarity between the reference image data and the target image data.   The region setting unit determines a center of the region having a characteristic image used for the alignment from the reference image data, determines a region size with respect to the determined center of the region, and the reference image data The image registration device according to claim 15, wherein the region having the largest image similarity with the region set to is searched from the target image data.   The region setting unit searches the target image data for a plurality of upper regions where feature amounts between the regions of the alignment candidates having the characteristic image set in the reference image data have extreme values. A pair of the image similarity of the regions surrounding each region among the plurality of the searched regions on the target image data paired with the region on the reference image data. The image registration apparatus according to claim 16, wherein an area is searched.   The alignment unit estimates a mutual information amount or entropy between the reference image data and the target image data as the image similarity, performs an optimization process, and based on an extreme value of the mutual information amount or entropy The image alignment apparatus according to claim 1, wherein a relative position between the reference image data and the target image data is determined.   When the alignment unit estimates the mutual information amount between the reference image data and the target image data, the alignment candidate region is in a region that does not coexist in the reference image data and the target image data. The image registration apparatus according to claim 18, wherein if there is, the mutual information amount or the entropy is estimated by excluding the region.   The alignment unit compares a first size outside the original image data in the reference image data with a second size outside the original image data in the target image data, and the first size is 20. The image registration device according to claim 19, wherein if the size is larger than a second size, the reference image data and the target image data are exchanged to estimate the mutual information amount or the entropy. When the relative position between the reference image data and the target image data is shifted, the alignment unit again determines the mutual information amount or entropy between the reference image data and the target image data as the image similarity. 19. The image registration according to claim 18, wherein the relative position between the reference image data and the target image data is determined based on a mutual information amount or an extreme value of entropy by performing estimation and performing optimization processing. apparatus.   Before performing the optimization process, the alignment unit obtains each image similarity with respect to a plurality of regions around the region of the alignment candidate set on the reference image data, and determines the image similarity 19. The image registration apparatus according to claim 18, wherein the region corresponding to the highest image similarity is searched, and the optimization processing is performed on the searched region.   The alignment unit sets the main region that is the main alignment candidate set on the reference image data and a plurality of auxiliary regions, and corresponds to the main region and the plurality of auxiliary regions. Each area is set on the target image data, and each image similarity between each area on the target image data corresponding to the main area on the reference image data and the plurality of auxiliary areas is set respectively. 2. The image alignment apparatus according to claim 1, wherein the relative position between the reference image data and the target image data is determined by an index obtained by integrating the image similarity.   The alignment unit first uses only the main region, and then determines a relative position between the reference image data and the target image data using the index that is integrated when it has converged to some extent. The image alignment apparatus according to claim 23.   When the registration unit estimates the mutual information amount between the reference image data and the target image data, an area that does not overlap the reference image data and the target image data is generated. The image registration apparatus according to claim 18, wherein the mutual information is corrected according to a size of the image.   The registration unit includes the mutual information based on entropy of the reference image data, entropy of the target image data, and entropy of a joint histogram generated based on the reference image data and the target image data. 26. The image registration apparatus according to claim 25, wherein an amount is obtained, and correction according to the size of the non-overlapping region is performed on the entropy of the joint histogram.   2. The display unit according to claim 1, wherein at least rotation and translation of the reference image data and the target image data aligned by the alignment unit are synchronized and displayed on the display. Image alignment device. Each region of the alignment candidate formed in the size required to select the feature point on the three-dimensional reference image data and on at least one three-dimensional target image data to be aligned with the reference image data, respectively Set,
Obtaining the image similarity between the set areas, aligning the reference image data and the target image data so that the image similarity is maximized,
Displaying the aligned reference image data and the target image data on a display;
An image alignment method characterized by the above. Each region of the alignment candidate formed in the size required to select the feature point on the three-dimensional reference image data and on at least one three-dimensional target image data to be aligned with the reference image data, respectively Let me set
The image similarity between the set regions is obtained, and the reference image data and the target image data are aligned so that the image similarity is maximized,
Displaying the aligned reference image data and the target image data on a display;
An image alignment program characterized by that.

Images