JP2001222712A - Image processing apparatus, convolution integration circuit and method thereof - Google Patents

Abstract

(57) [Summary] In computer image processing, convolution integration processing is performed at high speed. An adder (43 to 5) is used for convolution integration processing.
A pipeline arithmetic unit in which 1) is connected in cascade is used. As a result, the convolution integral calculation can be performed in parallel, and high-speed convolution integral calculation can be performed. A multiplying circuit (60) for multiplying each of the adding units (43 to 51) by common display data and each element data of the filter;
With the configuration including the adders (61 to 64), the display data once referred to can be used in each element of the filter, and the number of times of data reference is reduced.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for performing a filtering process on image data, a convolution integration circuit and a method therefor, and more particularly, to an image processing apparatus for performing convolution integration on image data, and a convolution integration circuit. And its method.

In recent years, the progress of computer graphics has been remarkable. In particular, in a game device and a simulation device, it is required to generate a complex and colorful image in real time.

[0003]

2. Description of the Related Art In a game device or a simulation device, the positions of a plurality of polygons constituting a character or the like are determined in accordance with an operation input, and rendering (drawing) processing is performed on the displayed polygons among the polygons. Then, image data of the display screen is generated. After the image data is written to the frame buffer, the image data in the frame buffer is displayed on the display device.

[0004] On the other hand, an image output by image processing by a computer is a clear image in which all objects are normally focused without blurring. However, depending on the type, movement, position, and the like of each object, it may be more effective on image display to process the image by blurring the image or emphasizing the edge, and may be closer to reality.

[0005] For example, in order to give a perspective, not only a near object is made large and a distant object is made small by a simple perspective transformation, but also an object near a focal point is sharp according to the depth of field. Objects that are out of focus are preferably blurred. In the course of the game, it is more effective to always perform image processing such as blurring or edge enhancement on a specific object or polygon.

For such image processing, a convolution integration process is used. This process is a process used to provide the above-described effects such as blurring, and performs an operation represented by the following equation (1).

[0007]

[Equation 1]

Here, s (x, y) is a generated image, f (x, y) is an original image, and g (x, y) is a filter function. FIG. 16 shows an example of one-dimensional image processing. In this example, a one-dimensional original image f (x) in the x direction is
This shows that a blurred image s (x) is generated by being convolved by the box filter g (x). By such image processing, effects on various images including blurring can be performed.

However, such a convolution process requires a very large amount of computation and a large number of memory accesses for referring to data. Therefore, even if a very high-speed processor is used, it takes a long time. there were.

For example, for one pixel, 1
In order to perform the convolution integral processing using the filter of 5 × 15 pixels, it is necessary to calculate the following equation (2).

[0011]

[Equation 2]

When this is executed by a general-purpose processor such as a CPU or a DSP, 225 multiplications and 224 additions are required to obtain image data of a certain pixel. To simplify the processing, the filter function gi, j
Is a digital filter represented by “0” / “1”, multiplication can be omitted. However, this calculation is performed using a general VGA (640 × 4
When the process is performed on an image having the size of 80), 68, 812, and 800 times are obtained only by addition, and the amount of calculation is extremely large. Similarly, in this sequential calculation, the number of memory accesses for referring to data is also the same, and cannot be ignored.

It has been proposed to provide hardware dedicated to convolution integration in order to eliminate the amount of calculation and memory access by the sequential processing by the general-purpose processor (for example, Japanese Patent Application No. 9-296812). This proposal shows a circuit that performs convolution integration in parallel, in which multipliers are provided in parallel by the size of the filter (the number of pixels), and adders that add the outputs of the multipliers are provided. .

[0014]

However, such a conventional proposal has the following problems.

(1) A parallel wiring path for inputting image data to the multipliers arranged in parallel and a parallel wiring path for connecting a number of multipliers arranged in parallel and an adder are required. . For this reason, in the case of being constituted by an LSI chip, there is a problem that a wiring space becomes large and a chip size becomes large, and it is difficult to form an LSI integrated with another image processing circuit. In particular, when color image data is targeted, it is necessary to provide parallel wiring paths for the number of bits of each color, which is difficult to realize.

(2) Normally, image data is stored in a memory and is sequentially read in accordance with an address. Therefore, in the above-described parallel operation method, image data of a filter size is read out of a target area. It is necessary to repeat by the number of pixels. For example, an area of 6 × 6 pixels is
In order to obtain 4 × 4 image data by performing convolution integration with a filter size of 3 × 3, it takes 9 cycles for reading and 1 cycle for calculation, and this must be repeated for 16 cycles. For this reason, the calculation time is reduced, but the number of memory accesses is increased, and the processing time cannot be shortened as a whole.

Accordingly, it is an object of the present invention to provide an image processing apparatus, a convolution integration circuit, and a method for shortening the wiring path even when the convolution integration operation is parallelized.

Another object of the present invention is to provide an image processing apparatus, a convolution integration circuit, and a method for shortening the processing time even if the convolution integration operation is parallelized.

Still another object of the present invention is to provide an image processing apparatus, a convolution integrator circuit, and a method for reducing the number of times of referencing a memory even when convolution integral processing is parallelized.

Still another object of the present invention is to provide an image processing apparatus, a convolution integration circuit, and a method for preventing an increase in the circuit scale even when the convolution integration processing is parallelized.

[0021]

According to an aspect of the present invention, there is provided an image processing apparatus comprising: a generating unit configured to generate display data from each image data; A convolution integration circuit that performs convolution integration by a filter to create the processed screen display data. The convolution integrator circuit includes a pipeline arithmetic unit provided in correspondence with each element of the filter and cascading a plurality of addition units to which the display data is supplied in common. , A multiplication circuit for multiplying the display data by the element data of the filter, and an adder for adding an input and the multiplication result.

A convolution integrator according to one embodiment of the present invention includes a pipeline arithmetic unit provided in correspondence with each element of a filter and cascading a plurality of addition units to which the display data is commonly supplied, Each of the adding units includes a multiplying circuit that multiplies the display data and element data of the filter, and an adder that adds an input and the multiplication result.

The image processing method according to one aspect of the present invention includes a generating step of generating display data from each image data, and convolving the generated display data with a filter having characteristics designated by a plurality of elements. And a convolution integration step for creating the processed screen display data. The convolution integration step includes a step of supplying the display data to a pipeline arithmetic unit in which a plurality of addition units provided in correspondence with the respective elements of the filter are cascade-connected. After the display data is multiplied by the element data of the filter, an addition step of adding an input and the multiplication result is provided.

In this embodiment of the present invention, a pipeline arithmetic unit in which adders are connected in cascade is used for the convolution integral processing. The pipeline operation unit is suitable for parallel processing, and can execute convolution integral operations in parallel, thereby enabling high-speed convolution integral operations. Next, by simply using a pipeline arithmetic unit in which adders are cascaded, the operation speed is increased, but the number of references to the memory is not reduced. In the present invention, the addition unit and the pipeline arithmetic unit are configured to efficiently use the read display data, and the number of times of referring to the memory is significantly reduced.
This is to improve the total calculation speed. That is, by configuring each addition unit of the pipeline arithmetic unit with a multiplication circuit that multiplies the common display data and each element data of the filter and an adder, the display data once referred to by each element of the filter. It can be used to reduce the number of data references.

In an image processing apparatus according to another aspect of the present invention, the pipeline arithmetic unit further includes a selector for controlling an input of data in the pipeline to a next-stage addition unit.

According to another embodiment of the convolution integrator of the present invention,
The pipeline arithmetic unit further includes a selector that controls an input of data in the pipeline to a next-stage addition unit.

In the image processing method according to another aspect of the present invention, the convolution integration step further includes a step of controlling the input of the data in the pipeline to the next addition unit by a selector.

In this aspect, by providing a selector for controlling the flow of data in the pipeline,
Even if the addition units are cascaded, addition results can be selected and added. Therefore, even if the display data is commonly supplied, a necessary convolution integration result can be obtained.

In an image processing apparatus according to another aspect of the present invention, the pipeline operation unit further includes a selector for controlling a feedback input of data in the pipeline to the addition unit.

A convolution integrator according to another aspect of the present invention is
The pipeline arithmetic unit further includes a selector that controls a feedback input of data in the pipeline to the addition unit.

In the image processing method according to another aspect of the present invention, the convolution integration step further includes a step of controlling a feedback input of the data in the pipeline to the addition unit by a selector.

In this embodiment, a feedback route to the addition unit is provided, and the addition unit is used for the data holding circuit. This eliminates the need for a circuit for the reference area, so that the circuit scale can be significantly reduced.

An image processing apparatus according to still another aspect of the present invention comprises:
The pipeline arithmetic unit further includes a holding circuit that temporarily holds data in the pipeline.

[0034] In a convolution integrator according to still another aspect of the present invention, the pipeline operation unit further includes a holding circuit for temporarily holding data in the pipeline.

In another image processing method according to the present invention, the convolution integration step further includes a step of temporarily holding data in the pipeline by a holding circuit.
Since the circuit for holding data in the pipeline is provided, data can be held in the pipeline even if the reference area is larger than the filter size. Thereby, the number of times of referring to the memory can be reduced.

In an image processing apparatus according to another aspect of the present invention, the holding circuits are provided in a number corresponding to the size of the filter and the size of the display data processing area.

According to another embodiment of the convolution integrator of the present invention,
The holding circuits are provided in a number corresponding to the size of the filter and the size of the processing area of the display data.

In this embodiment, since the number of holding circuits is determined by the filter size and the reference area, convolution integration can be performed with a minimum number of holding circuits. Thus, the circuit scale can be minimized, and the pipeline operation of convolution integration can be performed at high speed.

An image processing apparatus according to still another aspect of the present invention comprises:
The convolution integration circuit further includes a mask circuit for invalidating an operation of a multiplication circuit of a part of the addition units of the pipeline arithmetic unit in response to reading of the display data.

A convolution integrator according to still another aspect of the present invention includes a mask circuit for invalidating an operation of a multiplying circuit of an addition unit of a part of the pipeline arithmetic unit in response to reading of the display data. Have more.

According to still another aspect of the present invention, there is provided an image processing method comprising:
The convolution integration step further includes a step of disabling an operation of a multiplication circuit of a part of the addition units of the pipeline arithmetic unit in response to reading of the display data by a mask circuit.

In this embodiment, since the mask circuit is provided,
Even if display data is commonly supplied to each addition unit, it is possible to selectively perform an operation with a necessary filter element. Therefore, there is no need to input display data in parallel, the number of parallel paths can be reduced, and the convolution integration process can be executed by one memory reference.

[0043]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described with reference to an image processing apparatus,
The image processing operation, the convolution integration circuit, and other embodiments will be described in this order.

Image Processing Apparatus FIG. 1 is a block diagram of an image processing apparatus according to an embodiment of the present invention.

FIG. 1 shows a game device or a simulation device. Rendering processing unit (processor,
A renderer 34 is an image processing device, and is provided in a game device or a simulation device. The game device includes a renderer 34, a geometry processing unit (geometry operation unit) 32, a main CPU 30, a program ROM 36 for storing a game program, and a work RAM 38 used for executing the game program. Connected via.

The main CPU 30 executes a game program in the ROM 36 and performs necessary image processing in response to an operation input from an operator (not shown). In this example, polygon data constituting an object necessary for image processing, viewpoint information of a display screen, and the like are generated. The CPU 30 writes the polygon data and the like in the work RAM 38,
It causes the geometry processing unit 32 and the renderer 34 to execute image processing.

The geometry processing unit 32 performs a geometry process on the polygon data, such as a matrix operation for converting the arrangement of polygons in the three-dimensional coordinate space and a perspective conversion to two-dimensional coordinates on the display screen. Of course, this geometry processing may be performed by the main CPU 30.

The renderer 34 is provided with the geometry processing unit 32
The image data is generated in pixel units from the coordinates of the vertices of the polygon given by, and is written into the frame buffer 17. The Z value buffer 15 stores the Z value of the pixel at the forefront of the display screen. The components of the geometry processing unit 32 and the renderer 34 will be described later with reference to FIG.

The filter buffer 16 stores filter data for performing image processing such as blurring processing in pixel units or a predetermined plurality of pixel units. The creation of the filter data is performed by the renderer 3 as described later.
4 does. The convolution integration circuit 18 performs a filtering process set in the filter buffer 16 on the image data written in the frame buffer 17 and writes the image data in the frame buffer 17. The details of the convolution integration circuit 18 will be described later with reference to FIG.

In this embodiment, the frame buffer 1
After the writing of the image data including the RGB color data is completed in 7, the convolution integrator circuit 18 reads out the filter data of the filter buffer 16 in pixel units or a predetermined number of pixel units, and , Filter. For example, blur processing.

The display control unit 19 includes the frame buffer 17
Is read and supplied to the display device 20 for display. If the frame frequency is 60 Hz, 1
Every / 60 seconds, the image is written into the frame buffer 17 and the convolution integration circuit 18 performs the filtering process.

Next, the image processing operation of FIG. 1 will be described. FIG.
Is a flowchart of the image processing of FIG. 1, FIG. 3 is an explanatory diagram of vertex data of the polygon, and FIG. 4 is an explanatory diagram of pixel data of the polygon.

(S1) By executing the above-described game program, the CPU 30 writes the polygon vertex data and the register function for designating the processing mode in the RAM 38. As shown in FIG. 3, the polygon data includes vertex data constituting the polygon, for example, three-dimensional coordinates (x, y, z) of each vertex, color data (R, G, B, transparency a), The texture coordinates (Tx, Ty) of the texture buffer memory 14 for storing texture data as a material of polygons, a blur value (filter value) given from the CPU 30 for each polygon, and a velocity vector are included. or,
In some cases, normal vector data (Nx, Ny, Nz) is provided.

In this embodiment, since the blur processing is taken as an example of the filter processing, a blur value and a velocity vector are given. In the geometry processing unit 32, the data load circuit 3 sequentially reads such polygon data from the RAM 38 and outputs the polygon data to the coordinate conversion circuit 4.

(S2) The coordinate conversion circuit 4 performs coordinate conversion from the coordinate system in the polygon buffer of FIG. 3 to a coordinate system in a three-dimensional space. That is, the vertex data of FIG.
The arrangement (conversion) of the object in the three-dimensional space is performed according to the matrix information provided from U30. Specifically, instructions such as parallel movement and rotational movement of the polygon are given from the CPU 30 as matrix information, and the vertex coordinates and the velocity vector are coordinate-converted in accordance with the instructions. Further, the coordinate conversion circuit 4 sets a viewport in a three-dimensional space according to the viewpoint information.

(S3) Next, the clipping circuit 5 removes vertices outside the viewport and generates a new vertex at the boundary of the viewport. As a result, all polygons defined by vertices are made to fit within the viewport area. This process is a general clipping process.

Then, the perspective transformation circuit 6 performs the perspective transformation from the three-dimensional coordinates to the two-dimensional coordinates on the display screen for the vertex data in the viewport area. That is, perspective transformation is performed on the three-dimensional coordinates and the velocity vector of the vertex. Also, a Z value representing the depth in the display screen is generated at the same time.

The above is the processing in the geometry processing unit 32. These processings are performed for each polygon by each circuit under pipeline control. Of course, it can also be realized by software instead of a circuit.

(S4) Next, the processing of the renderer 34 will be described. First, the filling circuit 7 calculates pixel data in the polygon area defined by the vertices from the vertex data converted into the two-dimensional coordinate system. As shown in FIG. 4, the pixel data includes two-dimensional coordinates (x,
y), Z values, texture coordinates, normal vectors, color data, unique given blur values, velocity vectors, etc. In FIG. 4, one polygon is pixel 1,
The example which consists of 2, 3 is shown. The coordinates (x, y) of the pixel are obtained from the coordinate values of the vertex data by an interpolation method. Other attribute data can also be obtained by calculation by interpolation from the vertex data. Subsequent processes are performed one after another according to pipeline control in pixel units.

(S5) The Z value buffer memory 15 stores the Z value of the foreground (front) pixel at each pixel position on the display screen. The Z value comparison circuit 8 performs hidden surface processing by comparing the Z value of the pixel at the same position in the Z value buffer memory 15 with the Z value of the pixel being processed. For example, if the Z value of the pixel being processed is smaller than the Z value of the Z value buffer 15, it means that the pixel being processed is located further forward. Therefore, in order to display the pixel being processed on the display screen, the image data (R, G, B data, etc.) of the pixel being processed is written into the pixel position of the frame buffer 17. Therefore, the image data in the frame buffer 17 is the image data of the pixel whose Z value has been written in the Z value buffer memory 15.

(S6) Next, the texture generation circuit 9
The texture data in the texture buffer 14 is read in accordance with the texture coordinates, which is one attribute of the pixel in FIG. 4, and the texture color of the corresponding pixel is calculated. This is because such a calculation is performed because the data in the texture buffer 14 and the positions of the pixels in the display screen are not always in one-to-one correspondence. In the texture buffer 14, texture data is directly downloaded from the program ROM 36 and stored. The texture buffer 14 stores the work RA
It can also be provided in a storage area in M38.

(S7) Next, the luminance calculation circuit 10 calculates luminance information at the pixel being processed according to the influence from the light source. This luminance information includes, for example, diffused light (diffused color) according to the light emitted to the object,
Specularly reflected light (specular color) emitted from the object itself is included.

(S8) The blur value generation circuit 11 calculates the following value from the pixel data as a value indicating the degree of influence of the pixel under processing on the surrounding pixels.
Then, this is written into the filter buffer 16 for each pixel or for a predetermined number of pixels.

(1) Difference between Z value and depth of field (focus blur value) (2) Velocity vector (blur value by motion bra) (3) Luminance (blur value as light source) (4) Translucent Blur value (blur value behind translucent surface) (5) Blur value specific to polygon (specified blur value) Rather than writing these values individually,
It is desirable to use a weight value that directly indicates the degree of influence on surrounding pixels from one set of data. Also, depending on the trade-off between processing weight and display speed,
It is also possible to determine whether to store in the buffer 16.

(S9) The color modulation circuit 12 converts the color data of the pixel according to the luminance information from the texture color read from the texture buffer 14 and the diffuse light and the specular reflection light obtained by the luminance calculation circuit 10. Ask. The texture color is, for example, color information of the material when the material exists in a place that is 100% bright. Pixel color data is obtained by the following equation.

Color data = (texture color) ×
(Diffuse light) + (specular reflection light) That is, if the pixel is in a bright place, the texture color is expressed as it is, and the reflection light as a mirror surface is added thereto. Further, the color modulation circuit 12 performs modulation in consideration of the influence of fog or the like. For pixels having a large Z value in the fog, the color of the fog is blended according to the magnitude of the Z value.

(S10) The blend circuit 13 stores the color data of the pixel being processed and the frame buffer 1
7 and the already written color data,
The color data after blending is written to the frame buffer 17. For example, according to the transparency a of the pixel, the color data of the translucent pixel is blended with the color data of the pixel on the back side (already written in the frame buffer 17). This blended color data is subjected to image processing (here, blur processing)
, The boundary portion remains sharp.

In this way, the color data of each pixel is written to the frame buffer 17.

(S11) Next, the convolution integration circuit 18
Performs convolution integration of the color data of the frame buffer 17 with the filter value (blur value) of the filter buffer 16 and writes the result to the frame buffer 17.

(S12) The display controller 19 reads out the processed color data from the frame buffer 17 and displays it on the display device 20.

Next, the convolution integrator 18 in FIG. 1 will be described. FIG. 5 is a block diagram of a convolution integrator according to an embodiment of the present invention, FIG. 6 is an explanatory diagram of its reference region, filter, and processing result, FIG. 7 is a block diagram of its addition unit, and FIG. FIG. 9 is an explanatory diagram of image data, FIG. 9 is a configuration diagram of the mask circuit, FIG.
FIG.

First, for simplicity of description, as shown in FIG. 6, a 3 × 3 filter g (x, x,
The case of applying y) will be described. That is, to obtain a processing result (generated image) s (x, y) of 4 × 4, 6 × 6
3 × 3 filter g with reference to the original image f (x, y)
Multiply (x, y).

In this case, the convolution integrator 18 is arranged as shown in FIG.
Shown in As shown in FIG. 5, reference numerals 41 and 42 denote selectors, respectively, which control the flow of data in the pipeline by a select signal. Reference numerals 43 to 51 denote addition units, which perform calculations for convolution integration. 52, 53
Is a flip-flop (FF) that temporarily holds data in the pipeline.

Reference numeral 54 denotes a mask circuit, which generates mask signals Mask0 to Mask8 for masking an invalid operation region from the input filter data G00 to G22. 55 is
The normalization circuit normalizes the result obtained by the addition.

The addition units 43 to 51 are the number of the element data G00 to G22 (see FIG. 6) of the 3 × 3 filter, that is, 9
And are basically cascaded. Each of the adding units 43 to 51 is supplied with corresponding filter element data G00 to G22 from the mask circuit 14 described later. The image data F00 to F55 are supplied in parallel from the source (frame buffer 17) to the addition units 43 to 51.

The addition units 43 to 51 are basically
The supplied filter element data is multiplied by the image data, and the multiplication result is added to the output of the preceding addition unit. That is, the addition unit 43 multiplies the filter element data G00 by the input image data and adds the result to the input. Hereinafter, similarly, the addition units 44 to 51 multiply the element data G10 to G22 of the filter by the image data and add the result to the input.

Accordingly, by supplying the image data to each of the adding units 43 to 51, filtered image data S00 to S33 can be obtained by pipeline operation. FIG. 7 is a circuit diagram of such an addition unit, and FIG. 8 is an explanatory diagram of image data supplied from a source.

As shown in FIG. 8, color data is used as image data, R, G, and B data each have 8 bits, and 1-bit weight data (W) for normalization described later is provided. . Therefore, the original color data of one pixel is 25 bits.

As shown in FIG. 7, each of the adding units 43 to
51 includes a multiplier 60, adders 61 to 64, and a holding flip-flop 65. The multiplier 60 converts, for example, the element data (Filter) of each filter into 1/0.
In the case of a bit digital filter, it is configured by an AND gate. The AND gate 60 stores the image data Src
Is passed / not passed according to the element data.

The adders 61 to 64 are provided in accordance with R, G, B, and W, and the R, G, B, and W from the AND gate 60 are provided.
The input R, G, B, and W data from the preceding adder are added to the data. The addition result is the flip-flop 65
Is temporarily held and output.

Returning to FIG. 5, in this embodiment, furthermore, the image data S00 to S33 subjected to the filtering process can be calculated only by reading (referencing) each of the image data F00 to F55 once. are doing. That is, in order to perform cascade connection and pipeline operation, it is necessary to distribute and supply corresponding image data to an addition unit that multiplies element data of each filter. If this is provided and distributed by providing a parallel path, the number of wiring paths increases and the number of times of reading also increases.

In this embodiment, in order to prevent this, when the corresponding image data is input in synchronization with the sequentially read image data F00 to F55, the arithmetic operation is performed in each of the adding units 43 to 51. We are trying to make it work.

For this reason, first, a mask circuit 14 for generating a mask signal for validating / invalidating the operation of the adding unit is provided. Second, flip-flops 52 and 55 are provided to hold the data in the pipeline,
Further, selectors 41 and 42 are provided, and the addition unit is used for the holding circuit.

First, the mask circuit 14 is connected to the circuit shown in FIGS.
Will be described. As shown in FIG. 9, an X counter 70 that counts a read clock CL of image data,
An X signal generating circuit 71 for generating a gate signal when the X counter 70 has a predetermined value, a Y counter 72 for counting up the count of the X counter 70, and a Y signal when the Y counter 72 has a predetermined value. And a Y signal generation circuit 73 for generating

The Y shift register 74 is constituted by a 3-bit shift register, and shifts the input Y signal by counting up the X counter 70. Each bit of the Y shift register 74 includes AND gates 75, 76, 77
Has been entered. The outputs of the AND gates 75, 76, 77 are input to the three X shift registers 78, 79, 80. Each of the X shift registers 78, 79, and 80 is constituted by a 3-bit shift register, and shifts an input signal by a clock CL.

Each bit of each of the X shift registers 78, 79 and 80 is input to nine AND gates 81 to 89, respectively. The AND gates 81 to 89 correspond to the number of elements of the 3 × 3 filter, and element data G22 to G22 respectively.
G00 is input and outputs mask signals Mask8 to Mask0, respectively. These mask signals Mask0 to Mask8 are supplied to each of the adding units 43 to 51 in FIG. 5 (the filter signal Filter in FIG. 7).
Is input.

This operation will be described with reference to FIG. X
The counter 72 counts the clock, changes from “0” to “7”, and counts up. Then, the X signal generation circuit 71 sets the gate signal to “1” when the X counter 72 indicates a value of “0” to “3”. Y counter 72
Counts the count-up clock of the X counter 70 and changes from “0” to “5”. Then, the Y signal generation circuit 74 sets the Y signal to “1” when the Y counter 73 indicates a value of “0” to “3”.

FIG. 10 shows the values of the Y and X counters and 3
The state of the three bit shift registers 78 to 80 is 3 ×
3 is displayed in a matrix. In FIG. 10, the upper stage of the 3 × 3 matrix is the shift register 7
8 indicates the state of 3 bits, the left end is Mask0, and the center is
Mask1 and the right end are Mask2. The middle stage of the 3 × 3 matrix shows the 3-bit state of the shift register 79. The left end is Mask3, the center is Mask4, and the right end is Mask5.
It is. Further, the lower stage of the 3 × 3 matrix shows the 3-bit state of the shift register 80, and the left end is the Mask
6, the center is Mask7 and the right end is Mask8.

The convolution operation of the circuit of FIG. 5 will be described with reference to the time chart of FIG. In FIG.
Src is the image data from the source (frame buffer 17), and the columns indicated by G00 to G22 indicate that the values held by the addition units 43 to 51 to which the respective elements of the filter are stored are the elements S00 of the generated image. To S33. X0 and X1 are elements of the generated image held by the FFs 52 and 53. Sel indicates the state of the selector signal.

When the selector signal is "1", the selector 4
According to 1, the data of X0 is added to the addition unit 43 of G00.
The selector 42 feeds back the data of X1 to the adding unit 46 of G01. On the other hand, when the selector signal is “0”, the selector 4
2, the data of X0 is added to the addition unit 46 of G01.
And the selector 41 outputs “0” data,
The data input to the addition unit 43 of G00 and X1 is
G02 is passed to the addition unit 49.

In the rows of G00 to G22, the portion surrounded by the solid line is the mask circuit 54 so that the operation is not performed.
Indicates that it is masked. The addition units 43 to 51 add the image data Src to the input data when the filter element Gij is “1” in the portion not surrounded by the solid line.

At the clock 0, when the image data F00 is supplied to each of the adding units 43 to 51, the mask signal 0
By (FIG. 10), the operations of the addition units 44 to 51 other than the addition unit 43 are invalidated, and only the addition unit 43 performs the operation of the element S00 (= G00 · F00) of the generated image.

Similarly, when the image data F10 is supplied to each of the addition units 43 to 51 at the clock 1, the operation of the addition units 45 to 51 other than the addition units 43 and 44 is performed by the mask signal 1 (FIG. 10). Will be disabled,
The adding unit 43 generates the generated image S10 (= G00 · F10), and the adding unit 44 generates the generated image element S00 (= G00 · F00).
+ G10 ・ F10) is calculated.

Further, when the image data F20 is supplied to each of the adding units 43 to 51 at the clock 2, when the mask signal 2 (FIG. 10) is used, the adding units 43, 44 and 45 are used.
Since the operations of the addition units 46 to 51 other than are invalidated, the addition unit 43 outputs the generated image S20 (= G00 · F2
0), the adding unit 44 determines the element S10 (= G00
-F10 + G10-F20) and the addition unit 45 calculate the element S00 (= G00-F00 + G10-F10 + G20-F20) of the generated image.

Further, when the image data F30 is supplied to each of the addition units 43 to 51 at clock 3, when the mask signal 3 (FIG. 10) is used, the addition units 43, 44, and 45 are output.
Since the operations of the addition units 46 to 51 other than are invalidated, the addition unit 43 outputs the generated image S30 (= G00 · F3
0), the addition unit 44 determines whether the element S20 (= G00
-F20 + G10-F30) and the addition unit 45 calculate the element S10 (= G00-F10 + G10-F20 + G20-F30) of the generated image. Element S00 of generated image (= G00 · F00 + G10 ·
F10 + G20 · F20) is held in the FF 52 and the selector 4
1 is fed back to the adding unit 43.

At the clock 4, when the image data F40 is supplied to each of the adding units 43 to 51, the mask signal 4
By (FIG. 10), the operations of the addition units 43 and 46 to 51 other than the addition units 44 and 45 are invalidated.
The adding unit 43 outputs the generated image S
00, and the addition unit 44 outputs the element S30 of the generated image.
(= G00 · F30 + G10 · F40), the addition unit 45 generates the element S20 (= G00 · F20 + G10 · F30 + G20 ·) of the generated image.
Perform the calculation of F40). Element S10 of generated image (= G00 · F10
+ G10 · F20 + G20 · F30) is held in the FF 52, and is fed back to the addition unit 43 by the selector 41.

At the clock 5, when the image data F50 is supplied to each of the addition units 43 to 51, the mask signal 5
By (FIG. 10), the operations of the addition units 43, 44, 46 to 51 other than the addition unit 45 are invalidated.
The addition units 43 and 44 hold the generated images S01 and S00 that have been fed back, and the addition unit 45 generates the elements S30 (= G00 · F30 + G10 · F40 + G20 · F50) of the generated images.
Is calculated. The element S20 of the generated image is held in the FF 52, and is fed back to the addition unit 43 by the selector 41.

In the clocks 6 and 7, no image data is supplied, and the full addition units 43 to 51 are masked. For this reason, the data held in each of the adding units 43 to 45 and the FF 52 are shifted to a subsequent circuit. As a result, the generated image S00 is held in the FF 52. That is, the preparation for outputting the generated images S00 to S30 to the adding unit 46 is completed. This means that when six image data in one row of the original image are input in series, the addition unit is used for the data shift circuit to reduce the scale of hardware.

At clock 8, the image data F01 is supplied to each of the adding units 43 to 51, and the mask data 8 (FIG. 1)
0), the operations of the addition units 44, 45, 47 to 51 other than the addition units 43, 46 are invalidated.
The addition unit 43 determines the element S01 (= G00 · F0) of the generated image.
1) is calculated, the adding unit 44 holds S30, the adding unit 45 holds S20, and the FF 52 holds S10. The addition unit 46 generates the element S00 (= G00
・ F00 + G10 ・ F10 + G20 ・ F20 + G01 ・ F01) is calculated.

Hereinafter, similarly, at clocks 9 to 13,
The addition units 43 to 48 perform calculations and feedback on the generated images S00 to S31. And clocks 14, 1
At 5, the generated images S00 and S01 return to the FFs 52 and 53.

At the clock 16, the image data F02 is supplied to each of the adding units 43 to 51, and by the mask signal 16 (FIG. 10), the adding units 44, 45, 47 to 48 other than the adding units 43, 46, and 49 are used. Since the operations of 50 to 51 are invalidated, the adding unit 43 calculates the element S02 (= G00 · F02) of the generated image, and the adding unit 44,
45 holds S31 and S21, and FF52 holds S11. The adder unit 46 generates the element S01 (= G00
・ F10 + G10 ・ F20 + G20 ・ F30 + G01 ・ F02), and the addition unit 49 outputs the element S00 (= G0) of the generated image.
0 ・ F00 + G10 ・ F10 + G20 ・ F20 + G01 ・ F01 + G11
・ F11 + G21 ・ F21 + G02 ・ F02) is calculated.

As described above, the generated image S00 (= G00.F00 + G10.F10 + G20.F20 + G01) is generated by the clock 18.
・ F01 + G11 ・ F11 + G21 ・ F21 + G02 ・ F02 + G12 ・
The calculation of (F12 + G22 · F22) is completed, normalized by the normalization circuit 55, and written into the frame buffer 17.

Hereinafter, the same applies to the generated images S10 to S33.
As can be seen from this figure, a convolution operation of applying a 3 × 3 filter to a 4 × 4 area can be performed in only 45 cycles. In addition, the number of references to the memory at that time is 36, which is the number of reference pixels, and there is no waste. That is, each image data is referred to only once.

Further, since the selectors 41 and 42 are provided to control the flow of data in the pipeline, the adder unit can be used for the data shift circuit in cycles not required for convolution integration, and the 6 × 6 In order to process data, it is only necessary to provide two FFs in addition to the addition unit, and the circuit scale can be significantly reduced. For example, in the case of the 25-bit color data of FIG. 6 described above, the data flowing through the pipeline is accumulated nine times, so that each of R, R
Three bits are added to G, B, and W, for a total of 37 bits.
Therefore, each FF is composed of a 37-bit FF, and it is only necessary to add a 74-bit FF in addition to the addition unit.

Further, since a mask circuit is provided to partially control the operation of the pipeline arithmetic unit, the addition unit can be used as a data shift circuit, and the circuit scale can be significantly reduced.

The meaning of the above-described normalized data W is that the number of additions of each of the RGB data is different depending on the element data of the filter. Therefore, the normalized data W is added, the added number is integrated, and the obtained RGB is integrated. By dividing the data by the normalized data, a normalized output is obtained.

This filtering process is performed in the frame buffer 1
7 is applied to a specific area or the entire area of the screen. Therefore, a similar circuit is applied to a 16 × 16 area by 15 ×
In the case of creating a case where 15 filters are applied, it can be realized by providing a 15 × 15 addition unit and 14 FFs. In this case, the calculation is 9
60 cycles, 900 memory accesses, 16 × 1
The filtering process for the area No. 6 can be completed. When the above operation is performed for all the areas of the VGA, the operations are 115, 20
This is 0 cycle, and the entire area can be filtered at a very high speed as compared with the case where the processing is performed by the CPU or the like.

FIG. 12 is a circuit diagram of a convolution circuit according to another embodiment of the present invention. This embodiment is different from the circuit of the embodiment of FIG.
1 and 42, and the mask circuit 54 is deleted.
Those that are the same as those shown by are indicated by the same symbols.

The addition units 43 to 51 are the same as those in FIG. Five FFs 52 are provided at the subsequent stage of the addition unit 43. Further, five FFs 53 are provided at the subsequent stage of the adding unit 48.

FIG. 13 is a time chart thereof.
This is shown in accordance with the time chart of FIG. In this embodiment, similarly to FIG. 11, the calculation of the data of one column of the original image of FIG. 6 is completed in eight cycles. Therefore, as described above, five FFs 52 and 53 are provided to delay the operation result. Further, for example, F40.
Although operations not required for convolution integration, such as G00, F50 and G00, are also performed, they may be invalidated at a stage subsequent to the normalization circuit 55 (not shown). That is, only the effective operation results need to be captured in the buffer.

In this embodiment, the processing is completed in 45 cycles together with the time chart of FIG. 11, but the operation time can be further reduced. That is, FIG.
In the example of 1, the addition unit is used for the data shift circuit,
To shift the required data, do not read the data,
A clock cycle at which no operation is performed (for example, 6, 7, 14,
15, 22, 23, 30, 31, 38, 39), but this can be omitted. As a circuit, five FFs 52 and 53 are constituted by three FFs.

As a result, the operation cycle is shortened by 10 cycles by eliminating the clock cycle of FIG.
It can be reduced to 6 cycles. However, compared to the configuration of FIG.
Four more FFs are required. Therefore, the calculation speed and
The required configuration can be selected according to the circuit scale.

FIG. 14 is a circuit diagram of a convolution operation circuit according to another embodiment of the present invention. This embodiment is 3 ×
5 is obtained by removing selectors 41 and 42 from the circuit of the embodiment of FIG. 5 that performs the filtering process 3 and the same components as those shown in FIG. 5 are denoted by the same symbols.

The addition units 43 to 51 are the same as those in FIG. Five FFs 52 are provided at the subsequent stage of the addition unit 43. Further, five FFs 53 are provided at the subsequent stage of the adding unit 48. Mask circuit 5
4 is the same as FIG.

FIG. 15 is a time chart thereof.
This is shown in accordance with the time chart of FIG. In this embodiment, similarly to FIG. 11, the calculation of the data of one column of the original image of FIG. 6 is completed in eight cycles. Therefore, as described above, five FFs 52 and 53 are provided to delay the operation result. Further, for example, F40.
Operations not required for convolution integration, such as G00, F50 and G00, are invalidated by the mask circuit 54 as in FIG. As a result, unnecessary data becomes “0”, and malfunction can be prevented. Also, instead of a mask circuit,
By providing a selector, a similar operation can be realized.

In this embodiment, the processing is completed in 45 cycles together with the time chart of FIG. 11, but the operation time can be further reduced. That is, FIG.
In the example of 1, the addition unit is used for the data shift circuit,
To shift the required data, do not read the data,
A clock cycle at which no operation is performed (for example, 6, 7, 14,
15, 22, 23, 30, 31, 38, 39), but this can be omitted. As a circuit, five FFs 52 and 53 are constituted by three FFs.

As a result, the operation cycle is reduced by 10 cycles, and can be reduced to 36 cycles. However, FIG.
As compared with the configuration of the above, four extra FFs are required. Therefore, a necessary configuration can be selected according to the calculation speed and the circuit scale.

In addition to the above-described embodiment, the present invention can be modified as follows.

(1) In the above embodiment, the filter processing is described as the image blur processing, but other image processing such as edge enhancement processing can be applied.

(2) In the above description, the filter characteristics are set for each pixel. However, the filter characteristics can be set for every unit or region of the entire image.

(3) The multiplier of the addition unit has been described as an AND gate.
Although described in terms of bits, the filter element data may be two bits or more, and another multiplier may be used.

Although the present invention has been described with reference to the embodiment, various modifications are possible within the scope of the present invention.
They are not excluded from the scope of the present invention.

[0123]

As described above, according to the present invention,
The following effects are obtained.

(1) Since a pipeline arithmetic unit in which adders are cascaded is used for the convolution integral processing, the convolution integral operation can be executed in parallel, and a high-speed convolution integral operation can be performed. In particular, the image processing can be executed at a high speed in time for displaying.

(2) The addition unit and the pipeline operation unit are configured so as to use the read display data efficiently, so that the number of times of referring to the memory can be greatly reduced and the total operation speed can be improved. That is, by configuring each addition unit of the pipeline arithmetic unit with a multiplication circuit that multiplies the common display data and each element data of the filter and an adder, the display data once referred to by each element of the filter. Make it available and reduce the number of data references.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an image processing apparatus according to an embodiment of the present invention.

FIG. 2 is a processing flowchart for the image processing of FIG. 1;

FIG. 3 is an explanatory diagram of polygon data according to an embodiment of the present invention.

FIG. 4 is an explanatory diagram of pixel data according to an embodiment of the present invention.

FIG. 5 is a circuit diagram of the convolution integrator of FIG. 1;

FIG. 6 is an explanatory diagram of the image processing of FIG. 5;

FIG. 7 is a configuration diagram of an addition unit of FIG. 5;

FIG. 8 is a configuration diagram of the image data of FIG. 5;

FIG. 9 is a configuration diagram of the mask circuit of FIG. 5;

FIG. 10 is an operation explanatory diagram of the mask circuit of FIG. 9;

FIG. 11 is a time chart of the circuit of FIG. 5;

FIG. 12 is a configuration diagram of a convolution integrator according to another embodiment of the present invention.

FIG. 13 is a time chart of the circuit of FIG. 12;

FIG. 14 is a configuration diagram of a convolution integrator according to another embodiment of the present invention.

FIG. 15 is a time chart of the circuit of FIG. 14;

FIG. 16 is an explanatory diagram of image filter processing for explaining a conventional technique.

[Explanation of symbols]

 18 Convolution integrator 41, 42 Selector 43-51 Adder unit 52, 53 FF 54 Mask circuit 55 Normalizer 60 Multiplier 61-64 Adder

Claims (17)

[Claims] An image processing apparatus for generating screen display data from image data, comprising: a generation unit that generates display data from each image data; and a display unit that generates the display data with characteristics specified by a plurality of elements. A convolution integrator that creates convoluted screen display data by convolving and integrating with a filter, wherein the convolution integrator is provided corresponding to each element of the filter, and the display data is shared A plurality of addition units supplied in a cascade connection with a pipeline arithmetic unit, wherein each of the addition units multiplies the display data and element data of the filter, an input, and the multiplication result. An image processing device comprising: an adder that adds 2. The image processing apparatus according to claim 1, wherein said pipeline operation unit further includes a selector for controlling an input of data in said pipeline to an addition unit at a next stage. apparatus. 3. The image processing apparatus according to claim 1, wherein the pipeline operation unit further includes a selector for controlling a feedback input of data in the pipeline to the addition unit. . 4. The image processing apparatus according to claim 1, wherein said pipeline arithmetic unit further includes a holding circuit for temporarily holding data in said pipeline. 5. The image processing apparatus according to claim 4, wherein the holding circuits are provided in a number corresponding to a size of the filter and a size of a processing area of the display data. 6. The image processing apparatus according to claim 1, wherein the convolution integrator circuit disables an operation of a part of the adder unit of the pipeline arithmetic unit in response to the reading of the display data. An image processing apparatus further comprising: 7. A convolution integrator circuit for convolving image data with a filter having characteristics specified by a plurality of elements to create processed image data, the convolution integration circuit being provided corresponding to each element of the filter. A pipeline arithmetic unit in which a plurality of addition units to which the image data is commonly supplied are connected in cascade, wherein each of the addition units includes a multiplication circuit that multiplies the image data by element data of the filter; And a adder for adding the result of the multiplication. 8. The convolution integration circuit according to claim 7, wherein said pipeline operation unit further includes a selector for controlling an input of data in said pipeline to an addition unit at a next stage. circuit. 9. The convolution integrator according to claim 7, wherein said pipeline operation unit further comprises a selector for controlling a feedback input of data in said pipeline to said addition unit. . 10. The convolution integrator according to claim 7, wherein said pipeline operation unit further includes a holding circuit for temporarily holding data in said pipeline. 11. The convolution integrator according to claim 10, wherein the holding circuits are provided in a number corresponding to the size of the filter and the size of the processing area of the display data. 12. The convolution integration circuit according to claim 7, further comprising a mask circuit for invalidating an operation of a part of the addition units of the pipeline arithmetic unit in response to the reading of the display data. Convolution circuit. 13. An image processing method for generating display data for screen from image data, comprising: a generating step of generating display data from each image data; and generating the display data having characteristics specified by a plurality of elements. A convolution integration step of performing convolution integration with a filter to create the processed screen display data, wherein the convolution integration step includes a plurality of addition units provided corresponding to each element of the filter. In the cascaded pipeline operation unit,
Supplying the display data, and, in each of the adding units, adding the input and the multiplication result after multiplying the display data and the element data of the filter. Image processing method. 14. The image processing method according to claim 13, wherein the convolution integration step further includes a step of controlling input of data in the pipeline to an addition unit at a next stage by a selector. Image processing method. 15. The image processing method according to claim 13, wherein the convolution integration step further comprises a step of controlling a feedback input of the data in the pipeline to the addition unit by a selector. Processing method. 16. The image processing method according to claim 13, wherein the convolution integration step further includes a step of temporarily holding data in the pipeline by a holding circuit. 17. The image processing method according to claim 13, wherein in the convolution integration step, the operation of a part of the addition units of the pipeline operation unit is invalidated by a mask circuit in response to reading of the display data. An image processing method, further comprising a step.