CN104065937B - Real-time high-speed image preprocessing method for CMOS image sensor - Google Patents

Abstract

A kind of real time high-speed image pre-processing method for cmos image sensor, comprise the Bayer data exported by the photographic department of cmos image sensor, the line number of the single frames array of described Bayer data is M, columns is N, its innovation is: adopt the piecemeal of pipeline system, the mode of operation of classifying type access data, by the mode of piecemeal, classifying type, data are broken the whole up into parts, adopt the relatively less register of quantity to process the data after point zero simultaneously.Advantageous Effects of the present invention is: by changing caching process mode, use and take buffer structure on the less sheet of hardware resource, achieve high speed processing more data cached, significantly improve integrated level and the chip processing speed of cmos image sensor.

Description

Real-time high-speed image preprocessing method for CMOS image sensor

technical field

The invention relates to a photoelectric sensor signal processing technology, in particular to a real-time high-speed image preprocessing method for a CMOS image sensor.

Background technique

With the improvement of technology level and the rapid development of informatization, CMOS image sensor has gradually replaced CCD sensor with its advantages of high integration, high speed, low power consumption and low cost, and is widely used in digital cameras, video cameras and security cameras, etc. multiple fields.

In the prior art, there is a correlation between the imaging quality of the CMOS image sensor produced based on the existing process and the process size, and the smaller the process size, the worse the imaging quality. Therefore, how to make the CMOS image sensor obtain High-quality imaging is a hot topic in current research.

The data format output by the photosensitive part of the CMOS image sensor is generally in the Bayer format (hereinafter referred to as Bayer data), and the Bayer data is generally not used directly. It needs to be preprocessed and then processed by the subsequent image processing algorithm. When processing, the existing technology generally adopts memory and register serial method for processing, and its principle is shown in Figure 2. For a frame of Bayer data in an M (row)×N (column) array, if the color interpolation algorithm needs to cache M Row data is processed, and M×N registers and M-1 RAM memories are required for processing. When Bayer data is input into the register, in the first clock cycle, the data is read into the A _M B _N register, When the next clock arrives, at the same time as new data is read, the data of the previous beat in the original _AMBN is serially shifted into _{AMBN - 1} . In the subsequent process, the data is sequentially transferred in the register and RAM memory. When all the data is read into the first-level registers, after the next clock cycle arrives, the data in the first-level registers will be sequentially transferred to the second-level registers for algorithm processing (in the whole process, RAM plays the role of caching data), and the It takes at least (2M-1)×N+M clock cycles to move all the M×N Bayer data into the second-level buffer. The problem with the aforementioned processing method is that if the subsequent algorithm requires a larger pixel array and more data, the existing preprocessing method will require a large number of registers and memories, and the serial transmission of data will also require more time. This will increase the area and power consumption of the chip, and will reduce the processing speed of the algorithm. At the same time, due to the continuous serial input of data row by row, the order of pixel types maintains the order of pixel types in the original Bayer format, which also leads to code implementation. increased difficulty.

Contents of the invention

Aiming at the problems in the background technology, the present invention proposes a kind of real-time high-speed image preprocessing method for CMOS image sensor, including the Bayer data output by the photosensitive part of CMOS image sensor, the row number of the single frame array of described Bayer data is M, the number of columns is N, and its innovation lies in:

Build a cache module: use K static memories to build a cache module, 10≤K<M, and K is an even number, and the storage depth of a single static memory is J (the storage depth referred to in this article is the storage capacity, and its unit is byte); K static memories are divided into L processing groups, L=K/2, and each processing group corresponds to two static memories;

Bayer data is classified: the single-row data in the single-frame array is recorded as a data row, and multiple data in a single data row are divided into two groups according to the odd and even relationship of the column number serial number corresponding to each data: the same data row Among them, a plurality of data with an odd number of columns forms a data group 1, and a plurality of data with an even number of columns forms a data group 2; each data row corresponds to a data group 1 and a data group 2, and the data group The number of data in data group 1 and data group 2 is N/2, N/2=J; the way of classifying Bayer data is recorded as classification rules;

When processing, proceed as follows:

1) According to the classification rules, classify the data in the 1st to L data rows in the single frame array to obtain 2L data groups;

2) According to the order of rows in the single-frame array, input the data groups in each data row into the static memory in the cache module in turn, each static memory corresponds to a data group, and two data groups in the same data row are input into the same processing In the two static memories of the group; a plurality of data in the single data group is serially input in the static memory; Let the 1st data row correspond to the 1st processing group, the 2nd data row corresponds to the 2nd processing group, ... L The data row corresponds to the Lth processing group;

3) After the data input of the Lth data row is completed, the static memory starts to output the cached multiple data in parallel;

While the static memory is outputting in parallel, multiple data in the L+1th data row are classified according to the classification rules, and then serially input into the corresponding static memory in the first processing group to replace the multiple data in the first data row Data; the above-mentioned processing method of replacing old data in the processing group with new data is recorded as update processing, and in the process of subsequent parallel output of static memory data, continue to use new data rows to update the remaining processing groups sequentially;

4) After all the L processing teams have completed the update processing, according to the method in step 3), use the new data rows to re-update the L processing teams in turn until the cache module completes the cache and output of all data rows deal with.

The principle of the present invention is: the solution of the present invention essentially forms a pipelined working mode for accessing data by block and type, and the data is broken into zero by means of block and type. Fewer registers are used to process the zero-divided data, which not only greatly reduces the processing delay, improves the processing speed, but also saves the chip area and improves the chip integration. Specifically, the present invention is implemented in this way: "Blocking" is to serially input the cache one by one in units of data lines, divide a line of data into two blocks for storage, and output the data in all blocks and output them in parallel. After completing the data storage of all memory for the first time, when updating data, serial input and parallel output must be carried out simultaneously; "category" means that the data in each data row is divided into two data groups according to the number of columns odd and even, of course, after Therefore, the ability to make such a classification depends on the inherent characteristics of Bayer data. A single data row in Bayer data is either arranged alternately between R and G, or alternately arranged between G and B (RGB is the three primary colors of red, green and blue), and The odd and even relations of R, G and B in different data rows are constant, and this characteristic of Bayer data has been fully utilized in the present invention, and it is classified by odd and even relations, so that the data rows are divided into zeros to cooperate with the present invention. The implementation of the invention scheme; "pipeline" is to cache and output the data in a certain data line, and then continue to input the new data line into the cache module for processing; The direct effect is that the number of registers required in the preprocessing process is greatly reduced. From the perspective of the processed data scale, in the single processing of the prior art, it is necessary to simultaneously process M×N data in the single frame array. Therefore it needs M-1 RAM memories and M * N registers, and in the present invention, owing to " subdivision ", " classification " is carried out to single-frame array, thereby make the register quantity and the single static memory that single cache needs The depth of the chip is greatly reduced, which is conducive to layout on a limited chip area. At the same time, due to the parallel output of data, the design difficulty and processing delay are reduced, and the processing efficiency is improved.

The means of controlling data transfer in the present invention is the same as that of the prior art, and its control sequence adopts the common clock CLK control. In order to make the sequence control more simple, the present invention has also made the following improvements: two static memories in a single processing group The correspondence with data set 1 and data set 2 within a single data row remains constant.

Based on the reduction of the physical requirements of the static memory brought by the aforementioned solutions, the present invention also proposes the following preferred solution to improve the size parameters of the device: the static memory is realized by an on-chip memory. Due to size limitation, the processing capacity of the on-chip memory is not as good as that of the peripheral cache device, but after adopting the scheme of the present invention, a small amount of on-chip memory can afford the cache work of a large amount of data, which is of great significance for improving the integration of CMOS image sensors .

Preferably, when M is 512, N is 512, and K is 10, then the storage depth J of the static memory is 256.

The beneficial technical effects of the present invention are: by changing the cache processing method, using an on-chip cache structure that occupies less hardware resources, high-speed processing of larger cache data is realized, and the integration degree and on-chip processing speed of the CMOS image sensor are greatly improved .

Description of drawings

Fig. 1, schematic diagram of the processing principle of the present invention (the number of static memory in the figure is 10);

Fig. 2. Schematic diagram of the processing principle of the prior art;

Figure 3. Schematic diagram of data arrangement in Bayer data;

Figure 4. Schematic diagram of the structure of the CMOS sensor system on a chip.

detailed description

A real-time high-speed image preprocessing method for a CMOS image sensor, including Bayer data output by the photosensitive part of the CMOS image sensor, the number of rows of the single-frame array of the Bayer data is M, and the number of columns is N. The innovation lies in :

Build a cache module: use K static memories to build a cache module, 10≤K<M, and K is an even number, and the storage depth of a single static memory is J; divide K static memories into L processing groups, L=K/2 , each processing group corresponds to two static memories;

Bayer data is classified: the single-row data in the single-frame array is recorded as a data row, and multiple data in a single data row are divided into two groups according to the odd and even relationship of the column number serial number corresponding to each data: the same data row Among them, a plurality of data with an odd number of columns forms a data group 1, and a plurality of data with an even number of columns forms a data group 2; each data row corresponds to a data group 1 and a data group 2, and the data The number of data in group 1 and data group 2 is N/2, N/2=J; the way of classifying Bayer data is recorded as classification rules;

When processing, proceed as follows:

1) According to the classification rules, classify the data in the 1st to L data rows in the single frame array to obtain 2L data groups;

2) According to the order of rows in the single-frame array, input the data groups in each data row into the static memory in the cache module in turn, each static memory corresponds to a data group, and two data groups in the same data row are input into the same processing In the two static memories of the group; a plurality of data in the single data group is serially input in the static memory; Let the 1st data row correspond to the 1st processing group, the 2nd data row corresponds to the 2nd processing group, ... L The data row corresponds to the Lth processing group;

3) After the data input of the Lth data row is completed, the static memory starts to output the cached multiple data in parallel;

While the static memory is outputting in parallel, multiple data in the L+1th data row are classified according to the classification rules, and then serially input into the corresponding static memory in the first processing group to replace the multiple data in the first data row Data; the above-mentioned processing method of replacing old data in the processing group with new data is recorded as update processing, and in the process of subsequent parallel output of static memory data, continue to use new data rows to update the remaining processing groups sequentially;

4) After all the L processing teams have completed the update processing, according to the method in step 3), use the new data rows to re-update the L processing teams in turn until the cache module completes the cache and output of all data rows deal with.

Further, the corresponding relationship between the two static memories in a single processing group and the data group 1 and data group 2 in a single data row remains constant.

Further, the static memory is realized by an on-chip memory.

Further, when M is 512, N is 512, and K is 10, then the storage depth J of the static memory is 256.

Taking the Bayer data of a frame of 512 (rows) × 512 (columns) array as an example, the solution of the present invention is quantitatively compared with the prior art:

Based on the solution of the present invention, when 10 static memories are used to process 512×512 array data, when 5 lines of data are cached for processing, 10 static memories with a depth of 256 words and 10 registers are required for data cache operations, It takes 262144 clock cycles to read in serially and read out all the data of a single frame in parallel.

According to the prior art, when processing 512×512 data in the prior art, when caching 5 lines of data for processing, 5 static memories with a depth of 512 words and 3072 registers are required for data caching operations, all The clock cycle required for data access is 524288.

The present invention and the prior art have greater advantages in terms of chip area and layout. Using the idea of blocking, 5 static memories with a depth of 512 words are divided into 10 static memories with a depth of 256 words, which is conducive to the development of the system on chip. The overall layout reduces design difficulty, and for the same process condition, the number of registers used by the present invention is significantly smaller than that of the prior art (both static memories have the same capacity), which can save chip area and power consumption.

Claims (4)

1. for a real time high-speed image pre-processing method for cmos image sensor, comprise the Bayer data exported by the photographic department of cmos image sensor, the line number of the single frames array of described Bayer data be M, columns is N, it is characterized in that:

Build cache module: adopt K static memory to build cache module, 10≤K < M, and K is even number, the storage depth of single static memory is J, and described storage depth is exactly memory capacity, and its unit is byte; K static memory is divided into L process group, L=K/2, all corresponding two static memories of each process group;

Bayer data are classified: the single file data in described single frames array are designated as data line, multiple data during individual data is capable are divided into two groups by the odd, even relation of the columns sequence number corresponding to each data: in same data line, columns sequence number is that multiple data of odd number form data group one, and columns sequence number is that multiple data of even number form data group two; Then each data line is all to there being a data group one and data group two, and the data amount check in data group one and data group two is N/2, N/2=J; Classifying rules is designated as to the mode that Bayer data are classified;

During process, carry out as follows:

1) by classifying rules, the data in 1 to L in single frames array data line are classified, obtains 2L data group;

2) by the row order in single frames array, by in the static memory in the data group in each data line successively input buffer module, the corresponding data group of each static memory, two data groups in same data line input in two static memories of same process group; Multiple data in individual data group input in static memory in a serial fashion; If the corresponding 1st process group of the 1st data line, the corresponding 2nd process group of the 2nd data line ... L data line corresponding L process group;

3) after the data of L data line have inputted, the outside parallel output of multiple data that static memory has started buffer memory;

While the outside parallel output of static memory, after the multiple data in L+1 data line being pressed classifying rules classification, serial input the 1st processes in static memory corresponding in group, in order to replace the multiple data in the 1st data line; The processing mode that aforesaid new data replaces legacy data in process group is designated as update process, in the process of follow-up parallel output static memory data, continues to carry out update process with new data line successively to all the other process groups;

4) after L process group is completed update process, by mode in step 3), with new data line, again successively update process is carried out to L process group, until cache module is completed buffer memory, output processing to all data lines.

2. the real time high-speed image pre-processing method for cmos image sensor according to claim 1, is characterized in that: two static memories in single process group and individual data capable in data group one and the corresponding relation of data group two keep constant.

3. the real time high-speed image pre-processing method for cmos image sensor according to claim 1, is characterized in that: described static memory adopts on-chip memory to realize.

4. the real time high-speed image pre-processing method for cmos image sensor according to claim 1, is characterized in that: when M get 512, N get 512 time, K gets 10, then the storage depth J of static memory is 256.