CN102665049A - Programmable visual chip-based visual image processing system - Google Patents

Abstract

The invention discloses a visual image processing system based on a programmable visual chip, which includes an image sensor and a multi-stage parallel digital processing circuit. The image sensor mainly includes a pixel array, an analog preprocessing circuit array and an analog-to-digital conversion circuit array, and the digital processing circuit mainly includes a pixel-level parallel processing unit array, a row-parallel processing unit array, an on-chip artificial neural network, and a reduced instruction processor dual-core system. The system can realize high-speed and high-quality image acquisition and multi-level parallel image processing, and can realize a variety of high-speed intelligent vision applications through programming. Compared with traditional image systems, it has the advantages of high speed, high integration, low power consumption, and low cost. The present invention proposes an embodiment to realize the above architecture and various high-speed intelligent visual image processing algorithms based on this embodiment, including high-speed motion detection, high-speed gesture recognition and fast face detection, and the processing speed can reach 1000 frames per second. Meet the high-speed real-time processing requirements.

Description

Visual Image Processing System Based on Programmable Visual Chip

technical field

The present invention relates to the technical field of programmable visual chips and image processing, in particular to a visual image processing system based on programmable visual chips, which has the advantages of high speed, high integration, low power consumption, and low cost, and can be applied to various embedded High-speed real-time visual image processing system realizes various intelligent visual image applications including high-speed target tracking, natural human-computer interaction, environmental monitoring, intelligent transportation, robot vision, etc.

Background technique

The traditional visual image processing system includes a discrete camera and a general-purpose processor (or digital signal processor (DSP)), the camera uses an image sensor to acquire images, and serially transmits a large amount of raw image data acquired to the general-purpose processor or DSP processing, there are severe bandwidth constraints due to serial transfer. On the other hand, using software to process images in a general-purpose processor or DSP is often serially processed pixel by pixel, and there is a bottleneck of serial processing. Due to the limitations of serial transmission and serial processing, traditional visual image systems generally can only reach a speed of 30 frames per second, which is far from meeting the high-speed real-time requirements. For example, some industrial control systems often require a speed of 1000 frames per second .

The emergence of the vision chip effectively meets the high-speed real-time requirements. The vision chip imitates the principle of the human visual system, integrates the image sensor and the image processing circuit in the same chip, and the image data acquired by the image sensor is transmitted to the image processing in parallel. In the circuit, the image processing circuit itself adopts pixel-level large-scale parallel architecture in hardware, and finally the image processing circuit outputs a small amount of image feature data or analysis and recognition results, thus well overcoming the problem of data serialization in traditional visual image processing systems. The bottleneck of transmission and serial processing, the real-time performance has been greatly improved, and many systems using visual chips can achieve a processing speed of more than 1000 frames per second.

Vision chips can be divided into dedicated vision chips and programmable vision chips. The latter has greater practical value because it can flexibly implement multiple applications through programming and deal with complex and changeable actual environments.

However, there are serious deficiencies in the research on the architecture of programmable vision chips at home and abroad, as shown in:

(1) Each pixel unit includes a photosensitive element, a readout circuit and a processing circuit. The chip area is large, which greatly limits the resolution and filling rate, and the original image quality is poor; and because the photosensitive element and the readout circuit are analog circuits , so the processing circuit often uses analog circuits, resulting in poor reliability and flexibility of image processing.

(2) These pixel units are arranged in a two-dimensional array and work in the single instruction multiple data (SIMD) mode, which can realize full-pixel parallel image acquisition and local processing, but cannot realize fast and flexible wide-area processing;

(3) The above-mentioned programmable vision chip architecture working in SIMD mode supports low-level image processing and some intermediate-level image processing, but lacks advanced image processing functions, especially the simple and intuitive fast feature recognition ability similar to human brain nerves, so it must A complete visual image system can only be formed with the help of an external general-purpose processor, which limits the application of visual chips in some embedded occasions that have strict requirements on volume, power consumption and cost.

Contents of the invention

(1) Technical problems to be solved

Aiming at the problems existing in the above programmable visual chip, the present invention provides a visual image processing based on a programmable visual chip that separates pixel units and processing circuits, is based on multi-level parallel digital processing, and has an on-chip artificial neural network. System, in order to achieve higher resolution and fill rate, combined with local area processing and wide area processing functions, supports flexible and fast low, medium, and high-level image processing and on-chip feedback control, and realizes a full-featured on-chip vision system, through a variety of Typical high-speed intelligent vision application algorithm, its processing speed can reach 1000 frames per second.

(2) Technical solution

To achieve the above object, the invention provides a visual image processing system based on a programmable visual chip, comprising:

an image sensor for collecting raw image data at high speed, and transmitting the collected raw image data in parallel to a multi-stage parallel digital processing circuit; and

The multi-stage parallel digital processing circuit is used for performing fast parallel processing on the original image data received from the image sensor, and outputting the processing result.

In the above solution, the image sensor includes:

The N×N pixel array 1 is used to collect raw image data at high speed, and output the collected raw image data to an N×1 parallel analog preprocessing array 3, where N is a natural number;

N×1 parallel analog preprocessing array 3, used to remove fixed noise in the original image data, improve the dynamic range of the original image data, and output to N×1 parallel analog-to-digital conversion array 4;

N×1 parallel analog-to-digital conversion array 4, used to convert each column of analog pixel data into high-precision digital pixel data, and output to the output pixel selection module 5;

The output pixel selection module 5 is used to receive the N digital pixel data of the N×1 row parallel analog-to-digital conversion array 4 as input in parallel, and select M pixel data therefrom as the output of the image sensor, so as to realize the pixel row selection Select, where M is a natural number and M<N; and

The image sensor control module 6 is used to control the work of the N×N pixel array 1, the N×1 row parallel analog preprocessing array 3, the N×1 row parallel analog-to-digital conversion array 4 and the output pixel selection module 5 according to the internal parameter registers timing to achieve dynamic control of the image sensor.

In the above solution, the N×N pixel array 1 includes N×N two-dimensionally arranged pixel units 2, wherein each pixel unit 2 includes a photosensitive element and a corresponding readout circuit; the N×1 row parallel simulation The preprocessing array 3 includes N one-dimensionally arranged analog preprocessing units, wherein each analog preprocessing unit includes a correlated double sampling (CDS) circuit for removing fixed noise and a controllable gain amplification circuit for improving the dynamic range (PGA); the N × 1 row parallel analog-to-digital conversion array 4 includes N one-dimensionally arranged analog-to-digital conversion units; the output pixel selection module 5 cooperates with the image sensor control module 6 to select pixel rows and columns to realize the Flexible region processing and/or subsampling for image sensors.

In the above solution, the data in the parameter register in the image sensor control module 6 can be read and written from the outside of the module through the on-chip bus interface, so as to realize the dynamic control of the image sensor.

In the above solution, the image sensor control module 6 controls the rolling exposure of the N×N pixel array 1, and selects one of the columns each time to output N analog pixel values in a row-parallel manner to the N×1 row-parallel analog preprocessing The array 3 is used for noise removal and dynamic range improvement through the N×1 parallel analog preprocessing array 3, and then enters the N×1 parallel analog-to-digital conversion array 4 for parallel conversion into high-precision digital pixel data, and finally passes the The output pixel selection module 5 outputs M digital pixel data as the final output of the image sensor, which is provided to the multi-stage parallel digital processing circuit.

In the above solution, the multi-stage parallel digital processing circuit includes:

The M×M pixel-level parallel processing unit array 7 is used to perform local linear processing suitable for pixel-level parallel processing on the digital pixel data received from the image sensor, and output the processing result to the M×1 row processing unit array 9, wherein M Is a natural number and M<N;

M×1 row processing unit array 9, used to accelerate the non-linear processing and wide-area processing suitable for row-parallel processing in low- and middle-level images, so as to realize the extraction of image features;

The processing array control module 11 is used to fetch the control of controlling the M×M pixel-level parallel processing unit array 7 and the M×1 row processing unit array 9 from its internal variable-length single instruction multiple data (SIMD) instruction memory. Instructions are decoded and output to the M×M pixel-level parallel processing unit array 7 and the M×1 row processing unit array 9;

An artificial neural network 12 can be configured on the chip to complete feature recognition or feature compression tasks in advanced image processing, and its input is the feature vector data extracted by the M×1 row processing unit array 9, and the output is the result of feature recognition;

The streamlined instruction processor dual-core subsystem 13 is used to realize thread-level parallel processing, perform irregular processing in advanced image processing other than normal feature recognition, and control the entire system;

random/sequential mixed I/O memory 14;

system thread flag 15;

The on-chip bus 16 is used to map the read and write control signals and logical address information from the RISC dual-core subsystem 13 to the strobe enable signals and physical address information required by other bus slave modules to drive These slave modules perform various operations.

In the above scheme, the M×M pixel-level parallel processing unit array 7 includes M×M two-dimensionally arranged pixel-level parallel processing units PE8, and all pixel-level parallel processing units PE8 work in single instruction multiple data (SIMD) mode , accept the same PE array control command, and perform the same operation, but the data operated comes from the local memory of each unit.

In the above scheme, each pixel-level parallel processing unit PE8 corresponds to one or more image pixels of the N×N pixel array 1 in one frame, when each pixel-level parallel processing unit PE8 corresponds to one pixel, because M<N, the entire processing unit array corresponds to an M×M sub-region image of the N×N pixel array 1 or an M×M sub-sampled image of the entire N×N pixel array 1, at this time the The M×M pixel-level parallel processing unit array 7 processes a frame of M×M sub-images or sub-sampled images in a fully parallel manner; when each pixel-level parallel processing unit PE8 corresponds to multiple pixels, the entire The M×M pixel-level parallel processing unit array 7 corresponds to the entire N×N pixel array 1 or a sub-region larger than M×M in the N×N pixel array 1. At this time, the entire frame of image is processed in parallel with some pixels. deal with.

In the above solution, the visual image processing system uses the image sensor control module 6 to dynamically switch the correspondence between the pixel-level parallel processing unit PE8 and the image pixels, thereby realizing multi-resolution visual image processing.

In the above scheme, the pixel-level parallel processing unit PE8 is used to complete basic arithmetic logic operations such as 1-bit summation, negation, summation, and summation, and the multi-bit arithmetic logic operations in low-level image processing are decomposed into The above-mentioned basic 1-bit operation is realized on the pixel-level parallel processing unit PE8; the data of the pixel-level parallel processing unit PE8 can be interactively transmitted with its upper, lower, left, and right adjacent processing units, and through multiple adjacent Processing unit data transfer, each pixel-level parallel processing unit PE8 can interact with other processing units at any position.

In the above solution, the pixel-level parallel processing unit PE8 includes a first operand selector 31, a second operand selector 32, a 1-bit arithmetic logic operation unit 33, a 1-bit temporary data register 34 and a bit-plane random access memory 35, Wherein: the first operand selector 31 selects one from the output of the bit plane memory 35 of this unit or adjacent processing units according to the control instruction output by the processing array control module 11 as the first operation of the 1-bit ALU 33 number; the second operand selector 32 selects one from the output of the 1-bit temporary register 34 of this unit or the 1-bit immediate number 0 and 1 according to the control instruction output by the processing array control module 11 as a 1-bit ALU The second operand of 33.

In the above scheme, the 1-bit ALU 33 includes: a full adder, a NOT gate, a two-input AND gate, a two-input OR gate, a carry register and an output result selector; wherein the The carry register is used to store the carry result generated by the addition operation, and the carry result is used for multi-bit arithmetic operations. The carry register can be cleared by the control instruction output by the processing array control module 11; The control instruction output by the processing array control module 11 selects one of the outputs calculated by the full adder, the NOT gate, the AND gate, and the OR gate as the result of the 1-bit arithmetic logic operation unit 33 .

In the above scheme, the bit-plane random access memory 35 is a small-capacity random access memory with a data bit width of 1 bit and supports simultaneous reading and writing. Output from 1-bit ALU 33, which reads data as one of the inputs to the first operand selector of this unit or an adjacent processing unit.

In the above solution, the control instruction output by the processing array control module 11 can select whether to write each output result data of the 1-bit ALU 33 into the bit-plane random access memory 35 or the 1-bit temporary register 34, Only one of them must be written at a time.

In the above scheme, the M × 1 row processing unit array 9 includes M one-dimensionally arranged row parallel processing units RP 10, and all row parallel processing units RP 10 work in single instruction multiple data (SIMD) mode and accept the same The RP array control instruction performs the same operation, but the operated data comes from the local registers of each unit; the parallel processing unit RP10 of each row is used to complete k-bit arithmetic operations, including addition, subtraction, absolute value, Data shifting and size comparison, data operations larger than k-bit can be decomposed into several serial operations smaller than k-bit to complete.

In the above scheme, each row of parallel processing units RP 10 corresponds to all pixel-level parallel processing units PE 8 of the same row in the M×M pixel-level parallel processing unit array 7, and each pixel-level parallel processing unit of this row The data of PE 8 can enter row parallel processing unit RP 10 one by one to be further manipulated.

In the above solution, each row parallel processing unit RP can perform data interaction with the row parallel processing unit RP above and below it, and some row parallel processing units RP can also communicate with the row parallel processing unit RP separated by S rows above and below it. For data interaction, these row parallel processing units RP are called skip row processing units, and row parallel processing units RP other than these skip row processing units are called ordinary row processing units; in the entire row processing unit array, starting from the first At the beginning of the row, place a jump row processing unit every S rows, and place normal row processing units in the remaining rows; where S is a natural number.

In the above solution, the skipping row processing unit can directly perform data interaction at a long distance without going through all row parallel processing units RP 10 for data interaction one by one, and can realize fast and flexible inter-row wide-area processing.

In the above scheme, the row parallel processing unit RP includes: a k-bit buffer shift register 41, which is used to realize the serial-parallel/parallel-serial data conversion with the M×M pixel-level parallel processing unit array 7, and serve as Array external on-chip bus to the data access interface of the M × 1 row processing unit array 9, can be updated by the read data of the register file of the RP unit to which it belongs; a k-bit first operand selector 42, with According to the control command output by the processing array control module 11, one is selected from the output of the register file of the unit or the adjacent row processing unit, and the output of the buffer shift register of the unit as the first one of the k-bit arithmetic operation unit 44. Operand: a k-bit second operand selector 43, used to select one as the k from the temporary register output of this unit or from the immediate value of the array control instruction according to the control instruction output by the processing array control module 11 -the second operand of the bit arithmetic operation unit 44; a k-bit arithmetic operation unit 44 is used to perform wide-area processing and nonlinear processing, and the wide-area processing includes k-bit addition, subtraction, absolute value, data shift Bit and size comparison; a condition selector 45, for the 1bit data output from the pixel-level parallel processing unit PE 8 output of the row where the unit is located, from the k-bit arithmetic operation unit according to the control instruction output by the processing array control module 11 Select one of the condition flag registers of 44 and the 1bit constant 1 as the conditional operation enabling signal, which will enable the k-bit tri-state buffer gate 46; a k-bit tri-state buffer gate 46 for receiving k- The output result of the bit arithmetic operation unit 44 determines whether the data of this operation is written into the register file 48 of the k-bit temporary register 47 or the k-bit bit width under the control of the condition enable signal output by the condition selector 45, to realize conditional operation; and a k-bit temporary register 47 and a k-bit wide register file 48.

In the above solution, the k-bit buffer shift register 41 can be shifted left and right by bit under the array control instruction, so as to realize the serial-parallel/parallel-serial data conversion with the M×M pixel-level parallel processing unit array 7 It can also be shifted up and down in parallel with all bits of the buffer shift register in the upper and lower units of the parallel processing unit RP 10 of the row under the control of the external signal of the array, so as to realize the array external on-chip bus to the M × 1 row processing unit Data access of the array 9; the output of the k-bit buffer shift register 41 is used as one of the inputs of the k-bit first operand selector 42, and its value can also be updated by the read data of the register file.

In the above scheme, when the k-bit first operand selector 42 selects from the output of the register file of the unit or the adjacent line processing unit and the output of the buffer shift register of the unit according to the control instruction, if the unit is a jump row processing unit, the selection range also includes skip row processing units separated by S rows.

In the above scheme, the k-bit arithmetic operation unit 44 also updates its internal "carry/borrow" and "result is zero" flag registers according to the result of each operation, so as to facilitate data operations and conditional operations greater than k-bit; Its flag register can be cleared by processing the control command output by the array control module.

In the above scheme, the register file 48 of the k-bit bit width is a small-capacity random access memory or a register file with a data bit width k-bit and supports simultaneous reading and writing, and its read-write address comes from the output of the processing array control module 11 Control command, its writing data is from the output of k-bit tri-state buffer gate 46, and its reading data is as one of the input of the k-bit first operand selector 42 of this unit or adjacent line processing unit; If this unit If it is a skip row processing unit, it also includes a skip row processing unit separated by S rows.

In the above scheme, the control instruction output by the processing array control module 11 is used to select to write each output result data of the k-bit arithmetic operation unit 44 into the k-bit temporary register 47 or the k-bit wide Register files 48, when the k-bit tri-state buffer gate 46 is enabled, must and can only be written to one of them.

In the above scheme, the condition selector 45 can directly come from the 1-bit data of the pixel-level parallel processing unit PE8 as a condition enable signal, without going through the serial-to-parallel conversion based on the k-bit buffer shift register 41, which is beneficial to realize Flexible and fast in-line wide-area processing.

In the above scheme, when the M×1 line processing unit array 9 completes a relatively complex algorithm and the storage space of the register file is not enough, the data can be stored in the M×M through the k-bit buffer shift register 41. In the pixel-level parallel processing unit array 7; when all operations of the M×1 row processing unit array 9 are completed, the resulting data can be written into the k-bit buffer shift register 41, and then read by the array external on-chip bus 16 Walk.

In the above solution, the location where the processing array control module 11 reads the instruction segment from the variable-length SIMD memory is dynamically configured by the on-chip bus 16, and generates a completion flag to report to the on-chip bus 16 after the execution of the segment of the instruction is completed.

In the above solution, in order to support the coordinated operation of the M×M pixel-level parallel processing unit array 7 and the M×1 row processing unit array 9, and reduce the required on-chip instruction storage space, the visual image processing system adopts variable Long SIMD instruction mechanism, in which a 2L-bit instruction word is stored in each address of the variable-length SIMD instruction memory, which can be distinguished according to the instruction word header. The 2L-bit ultra-long SIMD instruction that M×1 row processing unit array 9 works together, or the two L that control the M×M pixel-level parallel processing unit array 7 and the M×1 row processing unit array 9 work independently -bit ordinary SIMD instruction; the processing array control module 11 is embedded with a scheduling and decoding functional unit for variable-length SIMD instructions.

In the above solution, the on-chip configurable artificial neural network 12 includes: an input neuron vector register group 51, including T1 input neuron registers, wherein each input neuron register is used to store J1 specific point data, where T1<< M; neuron broadcaster 52, used to accept the data of the input neuron vector register group 51, and select one of them to broadcast to the parallel operation unit array 53 at a time, as the operation of each parallel operation unit in the parallel operation unit array 53 One of the numbers; the parallel computing unit array 53 includes T2 parallel computing units, T2≤T1, each parallel computing unit accepts the input neuron broadcast by the neuron broadcaster 52 as the first operand, and simultaneously receives the weights respectively T2 weight/threshold data on each address of the threshold memory 55 is used as the second operand, wherein the weight/threshold is J ratio specific point data, J>J1; output neuron vector register group 54, including T2 output neurons Registers, wherein each output neuron register stores J2 ratio specific point data; weight/threshold memory 55, wherein there are weights and threshold data required for the operation process, T2 J ratio specific point data are arranged on each address; neural network control module 56, used to control the parallel operation process of the entire on-chip configurable artificial neural network 12 according to the configured parameter information, and the memory address of the on-chip configurable artificial neural network 12 is given by the neural network control module 56 when the on-chip configurable artificial neural network 12 works normally; the bus read-write interface 57, The data in the input neuron vector register group 51, the output neuron vector register group 54, and the weight/threshold memory 55 in the on-chip configurable artificial neural network 12 are externally written and read; the segmentation in the parallel operation unit The mapping function of the linear mapping unit and the control parameters of the neural network control module 56 are also flexibly configured by the bus read-write interface 57 .

In the above solution, each of the parallel operation units includes a fixed-point multiplier, an accumulation register and a piecewise linear mapping unit, wherein the fixed-point multiplier and the accumulation register are used to complete the input neuron data and the corresponding weight factor/threshold The multiplication and accumulation operation, the accumulation register can be cleared by the neural network control module, the piecewise linear mapping unit is used to implement the activation transfer function, and its output is used to update the output neuron vector register set 54 .

In the above scheme, under the control of the neural network control module 56, the neuron broadcaster 52 broadcasts one input neuron to the parallel operation unit array 53 at a time, and fetches the neuron from the weight/threshold memory 55 at the same time. The weight/threshold data corresponding to the broadcasted input neuron is sent to the parallel operation unit array 53, multiplied by the multipliers of each parallel operation unit and then accumulated to the accumulation register, and then the segmented linear mapping is implemented in parallel after all are completed, and the The final result is normalized to T2 bits and sent to the output neuron vector register set 54 .

In the above scheme, the data written into the weight/threshold memory 55 and the data configured with the parallel computing unit and the neural network control module 56 are obtained according to the training results of the neural network, and the training process is performed in the simplified instruction processor dual-core subsystem 13 or implemented on a general-purpose processor outside the system.

In the above solution, the on-chip configurable artificial neural network 12 supports a maximum of T1 input neurons, a maximum of T2 output neurons, and T2≤T1, when the number of input neurons is less than T1, or the number of output neurons is less than T2, The corresponding data in the remaining input neuron registers, output neuron registers and weight/threshold memory will be automatically set to 0.

In the above solution, the data of the output neuron register is read out by the on-chip bus 16, and input to the input neuron register again, so as to realize the calculation of the multi-layer neural network.

In the above scheme, the RISC dual-core subsystem 13 includes No. 1 RISC core (RISC#1), No. 1 RISC private program/data memory, No. 2 RISC core (RISC#2), No. 2 RISC private program/data memory, communication mailbox between processor cores and processor arbitrator, wherein: No. 1 RISC processor core (RISC#1) and No. 2 RISC of the RISC dual-core subsystem 13 The processor core (RISC#2) has a private program/data memory of P-bit data width to achieve thread-level parallel processing, which is responsible for irregular processing in advanced image processing other than normal feature recognition and for the entire system control.

In the above scheme, the No. 1 RISC #1 and No. 2 RISC cores (RISC #2) use the inter-processor core communication mailbox for communication to achieve necessary thread synchronization and data exchange; the No. 1 reduced instruction processor core (RISC#1) and the No. 2 reduced instruction processor core (RISC#2) have access rights to the on-chip bus controlled by the processor arbiter, which arbitrates The processor supports two arbitration methods of fixed priority and first-come-first-served in hardware; the communication mailbox between processor cores is a synchronous bidirectional FIFO.

In the above scheme, the RISC dual-core subsystem 13 also processes the macroscopic image information or interest obtained by processing the M×M pixel-level parallel processing unit array 7 and the M×1 row processing unit array 9. The target range dynamically adjusts the data in the parameter register of the image sensor control module 6 to adapt to the changing application environment and meet the multi-resolution processing requirements brought by the system or the relative movement of the target in the environment.

In the above scheme, the random/sequence mixed I/O memory 14 is a dual-port memory, wherein one port is a P bit width, which can be accessed randomly by the on-chip bus, and the other port is a PS (PS<P) bit Wide, sequential read and write access is performed by off-chip devices, and the read and write are independent of each other; the enable signal for sequential read and write outside the chip can be automatically mapped to the physical address of the memory by the address generation module embedded in the memory; the physical The address can be cleared by external redirection.

In the above scheme, the system thread flag 15 is a W bit register, wherein some bits are controlled by the on-chip bus 16 inside the system to control the writing, while other bits are controlled by the external devices of the system to control the writing; both inside and outside the system are readable All bits of the flags register.

In the above scheme, the on-chip bus 16 maps the read and write control signals and logical address information from the RISC dual-core subsystem 13 to the strobe enable signals and physical address information required by other bus slave modules When, the device module includes an image sensor control module, a processing array control module, an on-chip artificial neural network, a random/sequential mixed I/O memory, and a system thread flag.

In the above solution, in the visual image processing system, the digital pixel data obtained by the image sensor is loaded into the M×M pixel-level parallel processing unit array 7 in a row-parallel manner, and the M×M pixel-level parallel processing unit array 7 is Under the cooperation of the processing unit array 7 and the M×1 row processing unit array 9, various low-level and intermediate-level image processing can be flexibly completed, and image features are extracted and sent to the on-chip artificial neural network 12 for feature recognition, and also processed by simplified instructions. The processor dual-core subsystem 13 performs further analysis and processing to obtain and output a small amount of final required result data.

(3) Beneficial effects

As can be seen from the foregoing technical solutions, the present invention has the following beneficial effects:

1. The visual image processing system based on the programmable visual chip provided by the present invention, the separation of the pixel unit array and the processing circuit completely solves the problem that the area of the traditional visual chip increases rapidly with the resolution, and the resolution and filling rate are too low to limit the quality of the original image. difficult problem, and the flexible correspondence between the processing unit PE and the pixel unit helps to realize flexible multi-resolution processing.

2. The visual image processing system based on the programmable visual chip provided by the present invention introduces an architecture based on multi-level parallel digital processing circuits and on-chip artificial neural networks, and can quickly and flexibly complete various low, medium and high-level images through programming Processing, truly realize the single-chip on-chip visual image processing system, enrich and expand the application of visual chips in various embedded occasions with strict restrictions on volume, power consumption and cost.

3. In the visual image processing system based on the programmable visual chip provided by the present invention, the programmable row processing unit array with skip row processing unit can realize flexible and fast wide-area processing function, and accelerate the speed of feature extraction.

4. The visual image processing system based on the programmable visual chip provided by the present invention has high-speed, flexible real-time visual image processing capability, and the processing speed can exceed 1000 frames per second.

Description of drawings

Fig. 1 is the structural representation of the visual image processing system based on programmable vision chip provided by the present invention;

Fig. 2 is a circuit diagram of a continuous row in the image sensor array in Fig. 1, including a row of rolling exposure high-speed four-tube pixel unit and its subsequent row parallel analog processing unit (including correlated double sampling circuit and controllable gain amplifier circuit) and based on cyclic redundancy The analog-to-digital conversion circuit unit of the redundant mechanism;

Fig. 3 is a circuit structure diagram of a processing unit PE in the pixel-level parallel processing unit array in Fig. 1;

Fig. 4 is a circuit structure diagram of the row processing unit RP in the row parallel processing unit array in Fig. 1;

Fig. 5 is a circuit structure diagram of an on-chip configurable artificial neural network in Fig. 1;

Fig. 6 is the 1000 frame/second high-speed target tracking algorithm flow chart based on the visual image processing system in Fig. 1;

Fig. 7 is a high-speed gesture recognition algorithm flow chart based on the visual image processing system in Fig. 1;

Fig. 8 is the binarized image of the four types of gestures required to be recognized by the algorithm in Fig. 7;

FIG. 9 is a schematic diagram of an algorithm for fast face detection based on the visual image processing system in FIG. 1 .

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

As shown in Figure 1, Figure 1 is a schematic structural diagram of a visual image processing system based on a programmable visual chip provided by the present invention. The system includes two parts: an image sensing module and a multi-stage parallel digital image processing module. Wherein the image sensor module includes an N×N pixel array, an N×1 parallel analog preprocessing array, an N×1 parallel analog-to-digital conversion unit (ADC) array, an output pixel selection module and an image sensor control module, in this embodiment , N=256, which can reach the standard resolution in applications such as machine vision. Multi-level parallel digital image processing module includes M×M pixel-level parallel processing unit (PE) array, M×1 line processing unit (RP) array, on-chip configurable artificial neural network, reduced instruction processor dual-core subsystem, random/sequential Mixed I/O memory 14, system thread flags and high-speed on-chip bus. In this embodiment, M=64, with the flexible selection of the area and resolution of the pixel array of interest by the output pixel selection module and the image sensor control module, multi-resolution processing can be realized in a smaller chip area.

As shown in Figure 2, the analog preprocessing unit and analog-to-digital conversion unit (ADC) of a row of pixel units and corresponding rows in this embodiment, wherein the pixel unit adopts a standard four-tube pixel structure, cooperates with the correlation in the analog preprocessing unit The double-sampling circuit CDS can eliminate reset noise and fixed pattern noise. In addition, the controllable gain amplifier circuit PGA in the analog preprocessing unit can flexibly change the size of the equivalent feedback capacitor to achieve different gains and improve the dynamic range and contrast of the image. , and finally the analog image data is converted into a digital signal by a small-area cyclic redundant digital-to-analog conversion ADC unit, and input to a pixel-level parallel processing unit (PE) array through an output pixel selection module for processing.

The image sensor module can dynamically configure the output pixel selection module and image sensor control module through the external on-chip bus to achieve flexible row and column selection of the pixel array, that is, to select different regions of interest under different sub-sampling resolutions, and input them to the PE The array and its subsequent processing modules are processed to realize multi-resolution processing. In addition, the image sensor module can also dynamically configure the parameter registers of the image sensor control module through the external on-chip bus to achieve different integral exposure times, different frame rates and different PGA gains, etc., and adjust in real time according to the needs of the application environment and algorithms How image sensors work.

FIG. 3 is a specific circuit structure diagram of the pixel-level parallel processing unit PE in this embodiment. Each PE unit includes a first operand selector, a second operand selector, a 1-bit arithmetic logic operation unit (ALU), a 1-bit temporary data register, and a bit-plane random access memory. The first operand selector is used to select the output data from the bit plane memory in this unit or the four adjacent (upper, lower, left, right) PE units as the first operand of the 1bit ALU of this unit, thus realizing the connection between adjacent PE units The data interaction, and the second operand is used to select the data in the 1bit temporary register of the unit or the constant 0, 1 as the second operand of the 1bit ALU of the unit. The 1-bit ALU in the unit can complete 1-bit AND, OR, non-logic operations, and 1-bit addition operations. More complex arithmetic operations such as multi-bit addition, subtraction, and multiplication can be realized by decomposing into multiple serial 1-bit additions. Among them, the ALU The carry register is used to store the carry flag. The input and output data of the bit plane memory are both 1 bit, and the multi-bit grayscale image data is stored in the memory bit by bit, thus occupying multiple addresses. In this embodiment, the bit-plane memory has a capacity of 64 bits, which can meet the data storage requirements of low-level and medium-level image processing in most applications. The results of PE unit ALU processing can be stored in temporary registers or in the bit plane memory.

The instructions of the PE array are stored in the processing array control module, and it works in parallel under the control of the processing array control module in the single instruction multiple data (SIMD) mode, and each clock cycle can execute one instruction.

The pixel-level parallel processing unit (PE) array is mainly used for local linear operations in low-level and mid-level image processing, including background subtraction, linear grayscale transformation, smoothing filtering, edge detection, threshold segmentation, binary morphology, etc. , these operations can be completed in parallel at pixel level at high speed, and the acceleration ratio compared to serial processing is O(M×M).

FIG. 4 is a specific circuit structure diagram of the row parallel processing unit RP in this embodiment. Each RP unit includes a k-bit buffer shift register, a k-bit first operand selector, a k-bit second operand selector, a k-bit arithmetic operation unit, a condition selector, a A k-bit tri-state buffer gate, a k-bit temporary register and a k-bit wide register file. In this embodiment, k=8, because the grayscale image data is generally 8 bits, so k=8 can achieve a better performance-area balance. Each buffer shift register in the RP unit forms a shift register array, which can support the left and right shift of the word parallel bit string to realize the data interaction between each RP unit and the PE unit of the same row, and can also support the up and down shift of the word string bit string Bits to realize the data read and write interaction between the bus outside the array and the RP array.

RP units are divided into jumping RP units and ordinary RP units. Each type of RP unit can interact with adjacent (upper and lower) RP units through the selection of data input by the first operand selector, but the first operand of the jumping RP unit The selector can also select RP unit data that is separated from this unit by S rows to realize "jumping" interaction. These jumping RP units are placed every S rows in the RP array, and all RP units form a skipping chain to speed up some Statistical wide-area operations. In this embodiment, S=8, because the optimal value of S can be derived from the square root of M through simple theoretical derivation. The second operand of the RP unit ALU comes from the temporary register of the unit or the immediate field in the RP array instruction, and its ALU can complete the maximum/minimum value, addition/subtraction, data shift and absolute value calculation of 8bit data through hardware , and generate a flag bit to facilitate the conditional operation of the next cycle. The condition selector selects the flag bit of the ALU or the 1bit data from the row of PE units (usually binary image data or some flag data) to control the tri-state buffer gate to achieve conditional writing, thus realizing the condition of the RP unit operate. Temporary registers and register files are used to store data during RP unit processing. Since the output of the RP array is generally no longer image data, but some features in the image, the required storage capacity is relatively small. The row PE units share the storage space of the bit-plane memory. In this embodiment, the storage capacity of the RP unit register file is 8bit×16.

The RP unit is mainly used for wide-area operations and nonlinear operations that are suitable for row-parallel processing in low- and intermediate-level image processing, including median filtering, gray-scale morphology algorithms, average gray-scale calculations, and shape feature extraction (such as area, circumference, etc.). length, target area rectangular bounding box), etc., the main steps of these operations can be done in parallel, and the speedup ratio is O(M) compared to serial processing.

The RP array also works in the single instruction multiple data (SIMD) mode, and its instructions also come from the processing array control module, and each clock cycle can execute one instruction. In addition, the RP array often requires the cooperation of the PE array. In order to support the cooperative work of the PE array and the RP array, and not waste instruction storage space when only one of the arrays is required to work alone, a variable-length super-length SIMD is adopted. Instruction word (Variable VLIW SIMD, VVS) mechanism, through the instruction header, it can be distinguished whether it is a 2L-bit ultra-long SIMD instruction word that controls the work of the PE array and the RP array or two consecutive L that control the work of the PE array or the RP array -bit common SIMD instruction word. The processing array control module fetches instructions under the control of the control parameters written in the external bus, and is responsible for explaining and scheduling instructions. In this embodiment, L=32, and the on-chip instruction storage space is 8KB, which meets the storage requirements of most typical applications of low-level and middle-level image processing algorithms.

As shown in FIG. 5 is the specific structure of the on-chip configurable artificial neural network in this embodiment. The artificial neural network includes an input neuron register set of J1 bit precision, a neuron broadcaster, a parallel operation unit array, an output neuron register set of J2 bit precision, a weight/threshold memory of J bit precision and a neural network control module. Each parallel operation unit further includes: a fixed-point multiplier, an accumulation register and a segmented linear mapping unit.

The image feature data extracted by the RP array is loaded into T1 input neuron registers by the on-chip bus. Under the control of the neural network control module, the neuron broadcaster broadcasts the data of each input neuron register to the parallel operation unit array in turn as an operation One of the numbers, and the other operand comes from the weight/threshold memory, each address in the memory stores T2 weight/threshold data, corresponding to T2 parallel computing units, and the address of the memory is given by the neural network control module out. The parallel operation unit multiplies the input neuron register data and the corresponding weight data and accumulates them in the accumulation register. After the T1 input neuron register data are broadcast and processed, the threshold confidence in the weight/threshold memory is subtracted, and finally input To the piecewise linear mapping unit to implement the transfer function, the result is the value of the output neuron register representing the recognition result, and finally read out through the on-chip bus. The artificial neural network completes the feature recognition task in a vector-level parallel manner, and can obtain an acceleration ratio of about O(T2) compared with serial processing.

The effective number of input neurons (representing the image feature dimension) can be less than T1, and the effective number of output neurons (representing the classification number of target recognition) can also be less than T2; when these two situations occur, it can be passed The parameter register in the neural network control module is configured to simplify the operation process, so that the remaining invalid neuron data does not participate in the operation to speed up the processing speed. The two "inflection points" of the piecewise linear mapping function are also configurable. In addition, the data of the output neuron register after the last operation can be read out and fed back as the data of the input neuron register before the next operation, so that any multi-layer neural network can be dynamically realized to complete complex recognition task. In conclusion, the on-chip artificial neural network has very good configurability.

The weight/threshold data of the artificial neural network is obtained by training. Since the training process is not included in the normal working process of the system, it does not affect the real-time performance of the system operation, and the training process itself is too complicated to be directly implemented by hardware. Therefore, the training process can be completed on the RISC dual-core subsystem or even on the general-purpose processor outside the system. After the training, the obtained weight and threshold data can be downloaded to the weight/threshold memory of the artificial neural network. The training can adopt both supervised learning and unsupervised learning, so multiple neural networks including backpropagation (BP) neural network, self-organizing map (SOM) neural network and vector quantization (LVQ) neural network can be realized. an artificial neural network.

In this embodiment, T1=16, T2=8, J1=8, J2=8, J=12, and the capacity of the weight/threshold memory is 12bit×256, such precision and capacity configuration can satisfy most applications The requirements of the target recognition algorithm, when the feature dimension is higher than 16, the recognition process can be completed by processing different feature subspaces multiple times.

In this embodiment, the RISC dual-core subsystem in FIG. 1 is mainly used to complete irregular and complex algorithms in advanced image processing (such as artificial neural network training, Hough transform, principal component analysis, etc.) , dynamic configuration and parallel work of other modules in the unified control system. The RISC dual-core subsystem includes two P-bit data width RISC cores and their own private program/data memories, inter-processor core communication mailboxes and processor arbitrators. Each RISC can independently access its own private program/data memory, but must apply for access to other resources on the chip through the processor arbiter, which supports first-come-first-serve and fixed-priority arbitration algorithms in hardware and can be flexibly Configuration changes. The communication mailbox between processor cores is essentially a dual-port synchronous FIFO for supporting thread synchronization between dual cores. The RISC dual-core subsystem has thread-level parallel processing capabilities, which can obtain a certain speed-up ratio compared with single-core single-thread processing, and reduce the time for complex and advanced processing. In this embodiment, P=32 and the FIFO capacity is 32bit×16.

In this embodiment, the random/sequential mixed I/O memory in FIG. 1 is used for data exchange inside and outside the system, and is a dual-port memory. In order to reduce the number of system pins as much as possible, one of the ports is P bits wide, which can be accessed randomly by the on-chip bus, and the other port is PS (PS<P) bits wide, which can be accessed sequentially by off-chip devices. And reading and writing are independent of each other; the enable signal for sequential reading and writing outside the chip can be automatically mapped to the physical address of the memory by the address generation module embedded in the memory; the physical address can be cleared by external redirection. The two ports of the memory facing the inside and outside of the system can work at different clock frequencies, which is beneficial to expand the application range of the system. In this example, P=32 and PS=8.

In the present embodiment, the system thread mark in Fig. 1 is a W bit register, wherein some bits are responsible for controlling writing by the on-chip bus inside the system, and other bits are then being responsible for controlling writing by external devices of the system; both inside and outside the system All bits of the flags register can be read. This register can be used as a flag for the interaction and synchronization of threads inside and outside the chip system. In this embodiment, W=4, and three bits are written by external control of the system, and one bit is written by internal control of the system.

In this embodiment, the on-chip bus in Fig. 1 maps the read and write control signals and logical address information from the RISC dual-core subsystem main device to other various bus slave modules (including image sensor control module, processing array control module, on-chip Artificial neural network, random/sequential mixed I/O memory, thread flag) required strobe enable signal and physical address information to drive these slave device modules to complete various operations. In this embodiment, the data bit width of the on-chip bus is 32 bits, and supports up to 16 slave devices.

In this embodiment, the workflow of the entire high-speed on-chip vision system (programmable vision chip) is as follows: the digital pixel data obtained by the image sensor is loaded into the processing unit PE array in a row-parallel manner, and the processing unit PE array and row With the cooperation of the processing unit array, various low-level and intermediate-level image processing can be flexibly completed, and the extracted image features will be sent to the on-chip artificial neural network for feature recognition. Sometimes it is necessary to simplify the instruction processor dual-core subsystem for further analysis and processing to obtain the final required image. A small amount of result data and output to the outside of the system.

At the same time, the RISC dual-core subsystem can also dynamically adjust the data in the parameter register of the image sensor control module according to the macroscopic image information obtained by processing the PE array and the RP array or the target range of interest, so as to automatically adapt to the changing application environment, and Meet the multi-resolution processing requirements brought by the system or the relative motion of the target in the environment.

The specific application of this embodiment will be described in detail below through three typical high-speed visual image processing algorithms developed and run on the programmable vision chip-based visual image processing system proposed in this embodiment.

(1) High-speed target tracking

As shown in FIG. 6 , it is a high-speed target tracking algorithm flow based on the visual image processing system of this embodiment. First, the image sensor array is used to capture several frames of images, and a background image is synthesized in the PE array according to certain rules, and then it starts to work normally. Each frame of image captured during normal operation is first smoothed and filtered in the PE array to denoise and subtract the background image to obtain a differential image, and then the RP array is used to count the approximate distribution of the gray value of the image to determine the best dynamic Threshold, and then divide the differential image with this threshold in the PE array to obtain a binary image, which is the area with obvious moving objects in the scene. Next, each connected region of the binary image is segmented by binary morphological geodesic transformation in the PE array, the shape features of each region are extracted by the RP array, and the features of the target to be tracked are compared one by one in the RISC dual-core subsystem. Comparison, when the Euclidean distance or Manhattan distance in the feature space is the smallest and less than a certain predefined distance, it can be identified as the target feature, and the area and center coordinates of the target are locked accordingly, and the information is written into the I/O memory output to off-chip. Finally, the background of the non-moving area is updated according to a certain algorithm model to eliminate the interference of the slow change of the environment on the tracking process. In this algorithm, if the target collides with other moving objects or moves to an occluder, several frames will disappear, and the RISC dual-core subsystem will automatically predict the current area coordinates of the output target based on the previous statistical motion of the target; but when When the target reappears, the algorithm immediately locks on to it again. The algorithm has strong adaptability and robustness, and can deal with target tracking when there are multiple irregular high-speed moving objects in complex dynamic scenes. The above-mentioned high-speed target tracking algorithm can reach a processing speed of 1000 frames per second. In addition, in a manually controllable environment with a relatively simple background, the "self-window capture" method proposed by Miaowei et al. in patent ZL200510086902.2 can also be applied, and the area where the tracked target is located can be manually specified when the target tracking starts. The algorithm can also reach a processing speed of 1000 frames per second.

(2) High-speed gesture recognition

As shown in FIG. 7 , it is a high-speed gesture recognition algorithm flow based on the visual image processing system of this embodiment. The gesture recognition algorithm proposed by the present invention supports the recognition of four types of gestures, and is mainly used in the PPT gesture control system based on natural human-computer interaction. Figure 8 lists the binary images of these four types of gestures after threshold segmentation and the corresponding control Function. In the gesture recognition algorithm, the five steps from background synthesis to threshold segmentation are the same as those in the high-speed target tracking algorithm, and then the binary morphology area trimming algorithm is used in the PE array to remove small stray areas and fill in small areas in large areas. The hole, the last large complete area is the area where the gesture to be recognized is located. Afterwards, the artificial neural network is used for recognition. The artificial neural network must be fully trained before it can be used for recognition. During training, the normalized density features of the gesture recognition area are first extracted, that is, the area is divided into several rows and columns on average, and statistics are made separately. The ratio of the number of activated pixels in each row and each column (that is, pixels with a value of 1 in the binary image) to the total area of the region, these ratios form a set of vectors, and are used in the neural network under the supervision of the system thread flag Learning (that is, by externally writing the thread flag register to indicate which type of gesture is currently being learned), the learning process can be completed on the RISC dual-core subsystem inside the system, or it can be completed on a general-purpose processor outside the system. After the learning is completed, it is the recognition process. We noticed two special cases in the gestures to be recognized (that is, the "blank" gesture with no area to be recognized and the special mouse movement gesture with only one finger). In order to speed up the feature recognition, the algorithm uses A cascade classifier based on simple region features combined with artificial neural networks, the classifier first extracts simple features of the region to be recognized (such as the total number of activated pixels, shape parameters, vertex coordinates, etc.) and tries to recognize the above-mentioned special gestures on the RISC core. If it is unsuccessful, further extract more complex and complete normalized dense features and use the artificial neural network for unified recognition, and finally output the recognized gesture category code and gesture vertex coordinates (vertex coordinates are only used for mouse movement gestures). Since the above two special gestures spend most of the time in a typical application process, the overall processing speed can be greatly improved, and the average frame rate of the system can reach more than 1000 frames. The high frame rate is conducive to the further use of RISC to check the recognition results and perform software-based time-domain low-pass filtering to suppress the interference caused by environmental noise and gesture jitter on the recognition results.

(3) Fast face detection

As shown in FIG. 9 , it is a flow of a fast face detection algorithm based on the visual image processing system of this embodiment, and the algorithm can be used for counting the flow of people in special occasions. When applying this algorithm, it is necessary for the RISC core to control the image sensor to output a 64×64 resolution image of the area to be monitored each time. This algorithm mainly uses the PPED feature vector mentioned in the article An Image Representation Algorithm Compatible With Neural-Associative-Processor-Based Hardware Recognition Systems published by Masakazu et al. in IEEE Transactions on NeuralNetworks in 2003 for face detection. PPED The extraction of feature vectors is mainly divided into 5×5 template edge detection and edge mark generation in four directions: horizontal, vertical, plus 45 degrees and minus 45 degrees, and combining and compressing edge marks in these four directions according to certain rules to form a 64-dimensional The two steps of the PPED vector are completed on the PE array and RP array, and then the artificial neural network is used to judge whether it is a human face. Before the judgment, the neural network must be fully trained with the template in the standard face database. Due to the high feature dimension, it can be divided into feature subspaces for training and recognition one by one, or when the real-time requirements are high and the accuracy rate does not need to be too high, the 64-dimensional PPED vector is further compressed into a 16-dimensional vector to increase processing speed. On the system in this embodiment, the complete 64-dimensional PPED vector is adopted, and the 256×64 region in the middle of each frame of 256×256 image is divided into ten subregions of 64×64 by this algorithm (because the 64×64 There must be a certain overlap between sub-regions to minimize missed detection) The processing time required for face detection is about 18ms, or the frame rate of the entire system can be higher than 50 frames per second, much higher than serial processing system.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (41)

1. A visual image processing system based on a programmable vision chip, characterized in that, comprising: an image sensor for collecting raw image data at high speed, and transmitting the collected raw image data in parallel to a multi-stage parallel digital processing circuit; and The multi-stage parallel digital processing circuit is used for performing fast parallel processing on the original image data received from the image sensor, and outputting the processing result. 2. the visual image processing system based on programmable vision chip according to claim 1, is characterized in that, described image sensor comprises: N×N pixel array (1), used for high-speed acquisition of original image data, and outputting the collected original image data to N×1 parallel analog preprocessing array (3), wherein N is a natural number; N×1 parallel analog preprocessing array (3), used to remove fixed noise in the original image data, improve the dynamic range of the original image data, and output to N×1 parallel analog-to-digital conversion array (4); N×1 row parallel analog-to-digital conversion array (4), used to convert each column of analog pixel data into high-precision digital pixel data, and output to the output pixel selection module (5); The output pixel selection module (5) is used to receive N digital pixel data of the N×1 row parallel analog-to-digital conversion array (4) in parallel as input, and select M pixel data therefrom as the output of the image sensor to realize selection of pixel rows, where M is a natural number and M<N; and An image sensor control module (6), used to control an N×N pixel array (1), an N×1 row parallel analog preprocessing array (3), and an N×1 row parallel analog-to-digital conversion array (4) according to internal parameter registers and output the working sequence of the pixel selection module (5), to realize the dynamic control of the image sensor. 3. the visual image processing system based on programmable vision chip according to claim 2, is characterized in that, The N×N pixel array (1) includes N×N two-dimensionally arranged pixel units (2), wherein each pixel unit (2) includes a photosensitive element and a corresponding readout circuit; The N×1 parallel analog preprocessing array (3) includes N one-dimensionally arranged analog preprocessing units, wherein each analog preprocessing unit includes a correlated double sampling (CDS) circuit for removing fixed noise and a A controllable gain amplifier circuit (PGA) for improving the dynamic range; The N×1 parallel analog-to-digital conversion array (4) includes N one-dimensionally arranged analog-to-digital conversion units; The output pixel selection module (5) cooperates with the image sensor control module (6) to select pixel rows and columns to realize flexible area processing and/or sub-sampling processing of the image sensor. 4. the visual image processing system based on programmable vision chip according to claim 2, is characterized in that, the parameter register in the described image sensor control module (6), the data wherein can be carried out from module outside by on-chip bus interface Read and write to realize the dynamic control of the image sensor. 5. the visual image processing system based on programmable vision chip according to claim 2, is characterized in that, described image sensor control module (6) controls described N * N pixel array (1) rolling exposure, and every time Select one of the columns to output N analog pixel values to the N×1 row parallel analog preprocessing array (3) in a row parallel manner, and perform noise removal and dynamic range through the N×1 row parallel analog preprocessing array (3) Then enter the N×1 row parallel analog-to-digital conversion array (4) and convert it into high-precision digital pixel data in parallel, and finally output M digital pixel data through the output pixel selection module (5) as the final output of the image sensor The output is provided to the multi-stage parallel digital processing circuit. 6. the visual image processing system based on programmable vision chip according to claim 1, is characterized in that, described multistage parallel digital processing circuit comprises: M×M pixel-level parallel processing unit array (7), used to perform local linear processing suitable for pixel-level parallel processing on the digital pixel data received from the image sensor, and output the processing result to the M×1 row processing unit array (9 ), wherein M is a natural number and M<N; M×1 row processing unit array (9), used to accelerate the nonlinear processing and wide-area processing suitable for row-parallel processing in low-level and mid-level images, so as to realize the extraction of image features; Processing array control module (11), used to fetch and control the M×M pixel-level parallel processing unit array (7) and the M×1 row processing unit from its internal variable-length single instruction multiple data (SIMD) instruction memory The control instruction of the array (9), and decoding and outputting to the M×M pixel-level parallel processing unit array (7) and the M×1 row processing unit array (9); On-chip configurable artificial neural network (12), used to complete feature recognition or feature compression tasks in advanced image processing, its input is the feature vector data extracted by the M × 1 line processing unit array (9), and the output is feature recognition the result of; A streamlined instruction processor dual-core subsystem (13), used to implement thread-level parallel processing, perform irregular processing in advanced image processing other than normal feature recognition, and control the entire system; random/sequential mixed I/O memory (14); system thread flag (15); On-chip bus (16), used for mapping read and write control signals and logical address information from the RISC dual-core subsystem (13) to gate enable signals and physical addresses required by other bus slave modules information to drive these slave modules to complete various operations. 7. The visual image processing system based on a programmable visual chip according to claim 6, characterized in that, the M×M pixel-level parallel processing unit array (7) includes M×M two-dimensionally arranged pixel-level parallel Processing unit PE (8), all pixel-level parallel processing units PE (8) work in single instruction multiple data (SIMD) mode, accept the same PE array control instruction, perform the same operation, but the data operated comes from each unit local storage. 8. the visual image processing system based on programmable vision chip according to claim 7, is characterized in that, described each pixel level parallel processing unit PE (8) is corresponding to described N * N pixel array ( 1) one or more image pixels, When each pixel-level parallel processing unit PE (8) corresponds to one pixel, since M<N, the entire processing unit array corresponds to an M×M sub-region image of the N×N pixel array (1) or the entire The M×M sub-sampled image of the N×N pixel array (1), at this time, the M×M pixel-level parallel processing unit array (7) processes a frame resolution M×M sub-image or Subsampled images for processing; When each pixel-level parallel processing unit PE (8) corresponds to a plurality of pixels, the entire M×M pixel-level parallel processing unit array (7) corresponds to the entire N×N pixel array (1) or N×N pixel For the sub-regions larger than M×M in the array (1), the entire frame of image is processed in parallel with some pixels. 9. the visual image processing system based on programmable visual chip according to claim 8, is characterized in that, this visual image processing system is to switch pixel level parallel processing unit PE (8) and PE (8) dynamically by image sensor control module (6) and The correspondence between image pixels, thereby realizing multi-resolution visual image processing. 10. the visual image processing system based on programmable vision chip according to claim 7, is characterized in that, The pixel-level parallel processing unit PE (8) is used to complete basic arithmetic logic operations such as 1-bit summation, negation, summation, and summation, and multi-bit arithmetic logic operations in low-level image processing are decomposed into the above-mentioned The basic 1-bit operation is realized on the pixel-level parallel processing unit PE (8); The data of the pixel-level parallel processing unit PE (8) can be interactively transmitted with its upper, lower, left, and right adjacent processing units. Through multiple data transfers of adjacent processing units, each of the pixel-level parallel processing units PE (8) It can interact with other processing units at any position. 11. the visual image processing system based on programmable vision chip according to claim 7, is characterized in that, described pixel level parallel processing unit PE (8) comprises first operand selector (31), second operand Selector (32), 1 bit ALU (33), 1 bit temporary data register (34) and bit plane random access memory (35), wherein: The first operand selector (31) selects one as a 1-bit ALU ( 33) the first operand; The second operand selector (32) selects one from the output of the 1-bit temporary register (34) of this unit or the 1-bit immediate number 0 and 1 according to the control instruction output by the processing array control module (11) as 1 bit Second operand of the ALU (33). 12. the visual image processing system based on the programmable vision chip according to claim 11, is characterized in that, described 1 bit arithmetic logic operation unit (33) comprises: a full adder, a NOT gate, a two-input AND gate, a two-input OR gate, a carry register, and an output result selector; where, The carry register is used to store the carry result generated by the addition operation, and the carry result is used for multi-bit arithmetic operations, and the carry register can be cleared by the control instruction output by the processing array control module (11); The output result selector selects one from the output calculated by the full adder, the NOT gate, the AND gate and the OR gate according to the control instruction output by the processing array control module (11) as the output of the 1-bit arithmetic logic operation unit (33) result. 13. the visual image processing system based on the programmable vision chip according to claim 11, is characterized in that, described bit plane random access memory (35) is the small-capacity random access memory that data bit width is 1 bit, supports reading and writing simultaneously , its read-write address comes from the control instruction that described processing array control module (11) outputs, and its write-in data comes from the output of 1 bit arithmetic logic operation unit (33), and its read-out data is used as the first of this unit or adjacent processing unit One of the inputs to an operand selector. 14. The visual image processing system based on a programmable visual chip according to claim 11, characterized in that, the control instruction output by the processing array control module (11) can select the 1-bit arithmetic logic operation unit (33) Whether the output result data is written into the bit-plane RAM (35) or the 1-bit temporary register (34) each time, only one of them must be written each time. 15. the visual image processing system based on the programmable visual chip according to claim 6, is characterized in that, described M * 1 row processing unit array (9) comprises the row parallel processing unit RP (10 of M one-dimensionally arranged ), all row parallel processing units RP (10) work in single instruction multiple data (SIMD) mode, accept the same RP array control instruction, and perform the same operation, but the operated data comes from the local register of each unit; The parallel processing unit RP (10) of each row is used to complete k-bit arithmetic operations, including addition, subtraction, absolute value, data shift, and comparison size, and data operations greater than k-bit can be decomposed into Several operations smaller than k-bit are performed serially. 16. the visual image processing system based on programmable vision chip according to claim 15, is characterized in that, described each row parallel processing unit RP (10) is corresponding to described M * M pixel level parallel processing unit array ( 7) For all the pixel-level parallel processing units PE(8) in the same row, the data of each pixel-level parallel processing unit PE(8) in the row can enter the row parallel processing unit RP(10) one by one for further operation. 17. The visual image processing system based on a programmable visual chip according to claim 15, wherein each row parallel processing unit RP can perform data interaction with the row parallel processing unit RP above and below it, some of which The row parallel processing unit RP can also perform data interaction with the row parallel processing units RP separated by S rows above and below it. These row parallel processing units RP are called skip row processing units, and the row parallel processing units other than these skip row processing units The unit RP is called a common row processing unit; in the entire row processing unit array, a jumping row processing unit is placed every S rows starting from the first row, and ordinary row processing units are placed in the remaining rows; where S is a natural number. 18. The visual image processing system based on programmable visual chip according to claim 17, characterized in that, the skip row processing unit can directly carry out data interaction at a long distance, without passing through all row parallel processing units RP (10) one by one ) for data interaction, which can realize fast and flexible inter-line wide-area processing. 19. The visual image processing system based on a programmable visual chip according to claim 15, wherein the row parallel processing unit RP comprises: A k-bit buffer shift register (41), used to realize the serial-to-parallel/parallel-to-serial data conversion with the M×M pixel-level parallel processing unit array (7), and as an array external on-chip bus to the M× The data access interface of the 1-line processing unit array (9) can be updated by the read data of the register file of the RP unit to which it belongs; A k-bit first operand selector (42), used for outputting from the register file of this unit or adjacent row processing units according to the control instruction output by the processing array control module (11), the buffer shift register of this unit Select one as the first operand of the k-bit arithmetic operation unit (44) in the output; A k-bit second operand selector (43), used for selecting one from the immediate data output from the temporary register of this unit or from the array control instruction according to the control instruction output by the processing array control module (11) as the The second operand of the k-bit arithmetic operation unit (44); A k-bit arithmetic operation unit (44) is used to perform wide-area processing and nonlinear processing, and the wide-area processing includes k-bit addition, subtraction, absolute value, data shift and size comparison; A condition selector (45), used for the 1-bit data output from the pixel-level parallel processing unit PE (8) of the row where the unit is located according to the control instruction output by the processing array control module (11), from the k-bit arithmetic operation Select one in the condition flag register of unit (44) and 1bit constant 1 as conditional operation enabling signal, and this signal will enable described k-bit tri-state buffer gate (46); A k-bit tri-state buffer gate (46) is used to receive the output result of the k-bit arithmetic operation unit (44), and determines whether to perform this operation under the control of the condition enable signal output by the condition selector (45). Write the data of k-bit temporary register (47) or the register file (48) of k-bit bit width, to realize conditional operation; And A k-bit temporary register (47) and a k-bit wide register file (48). 20. the visual image processing system based on programmable vision chip according to claim 19, is characterized in that, described k-bit buffer shift register (41) can carry out left and right shift by bit under array control instruction, with Realize serial-to-parallel/parallel-to-serial data conversion with the M×M pixel-level parallel processing unit array (7); also under the control of the external signal of the array, it can be connected with the upper and lower units of the row parallel processing unit RP (10) All bits of the buffer shift register are shifted up and down in parallel to realize the data access of the M × 1 row processing unit array (9) by the on-chip bus outside the array; the output of the k-bit buffer shift register (41) is used as k- bit One of the inputs to the first operand selector (42), whose value can also be updated by read data from the register file. 21. The visual image processing system based on a programmable visual chip according to claim 19, characterized in that, the k-bit first operand selector (42) is in accordance with the control instruction from this unit or the adjacent row processing unit When selecting between the register file output of the unit and the output of the buffer shift register of the unit, if the unit is a skip line processing unit, its selection range also includes the skip line processing unit separated by S lines. 22. The visual image processing system based on programmable visual chips according to claim 19, characterized in that, said k-bit arithmetic operation unit (44) also updates its internal "carry/borrow" according to each operation result " and the "result is zero" flag register, which is convenient for data operations and conditional operations larger than k-bit; its flag register can be cleared by the control command output by the processing array control module. 23. the visual image processing system based on programmable visual chip according to claim 19, is characterized in that, the register file (48) of described k-bit bit width is data bit width k-bit, supports simultaneous read and write Small-capacity random access memory or register file, its read-write address comes from the control command that described processing array control module (11) outputs, and its write-in data comes from the output of k-bit tri-state buffer gate (46), and its read-out data is used as One of the inputs of the k-bit first operand selector (42) of this unit or an adjacent row processing unit; if this unit is a skipping row processing unit, it also includes a skipping row processing unit separated by S rows. 24. the visual image processing system based on the programmable vision chip according to claim 23, is characterized in that, the control instruction that described processing array control module (11) outputs is used for selecting described k-bit arithmetic operation unit ( 44) each output result data is written into k-bit temporary register (47) or the register file (48) of k-bit bit width, must when described k-bit tri-state buffer gate (46) is enabled And only one of them can be written. 25. the visual image processing system based on programmable vision chip according to claim 19, is characterized in that, described condition selector (45) can directly use the 1bit data from pixel level parallel processing unit PE (8) as condition The energy signal does not need to be serial-to-parallel conversion based on the k-bit buffer shift register (41), which is conducive to the realization of flexible and fast intra-line wide-area processing. 26. The visual image processing system based on the programmable visual chip according to claim 19, characterized in that, When the M×1 line processing unit array (9) completes more complex algorithms and the storage space of the register file is not enough, the data can be stored in the M×M through the k-bit buffer shift register (41). In the pixel-level parallel processing unit array (7); When all operations of the M×1 row processing unit array (9) are completed, the resulting data can be written into the k-bit buffer shift register (41), and then read out by the on-chip bus (16) outside the array. 27. the visual image processing system based on programmable vision chip according to claim 6, is characterized in that, described processing array control module (11) reads the position of instruction segment from variable-length SIMD memorizer by on-chip bus (16) ) is dynamically configured, and when the execution of this section of instructions is completed, a completion flag is generated and reported to the on-chip bus (16). 28. The visual image processing system based on a programmable visual chip according to claim 6, characterized in that, in order to support both the M×M pixel-level parallel processing unit array (7) and the M×1 row processing unit The coordinated operation of the array (9) reduces the required on-chip instruction storage space. The visual image processing system adopts a variable-length SIMD instruction mechanism, wherein a 2L-bit instruction word is stored on each address of the variable-length SIMD instruction memory, according to The instruction header can distinguish whether this is a 2L-bit ultra-long SIMD instruction that controls the cooperative work of the M×M pixel-level parallel processing unit array (7) and the M×1 row processing unit array (9), or controls all Two L-bit ordinary SIMD instructions that the M*M pixel-level parallel processing unit array (7) and the M*1 row processing unit array (9) work independently; the processing array control module (11) is embedded with The scheduling and decoding functional unit of variable-length SIMD instructions. 29. The visual image processing system based on a programmable visual chip according to claim 6, wherein the configurable artificial neural network (12) on the chip comprises: Input neuron vector register group (51), including T1 input neuron registers, wherein each input neuron register is used to store J1 ratio specific point data, wherein T1<<M; Neuron broadcaster (52), used to accept the data of the input neuron vector register group (51), and select one of them to broadcast to the parallel operation unit array (53) at a time, as in the parallel operation unit array (53) One of the operands of each parallel operation unit; Parallel operation unit array (53), comprising T2 parallel operation units, T2≤T1, each parallel operation unit accepts the input neuron broadcast by the neuron broadcaster (52) as the first operand, and simultaneously receives the weight T2 weight/threshold data on each address of the/threshold memory (55) is used as the second operand, wherein the weight/threshold is J ratio specific point data, J>J1; Output neuron vector register group (54), including T2 output neuron registers, wherein each output neuron register stores J2 ratio specific point data; Weight/threshold memory (55), which stores the required weight and threshold data of the operation process, and there are T2 J ratio specific point data on each address; The neural network control module (56) is used to control the parallel computing process of the entire on-chip configurable artificial neural network (12) according to the configured parameter information, and the memory address of the on-chip configurable artificial neural network (12) is determined by the neural network control module when it is working normally. (56) gives; Bus read-write interface (57), for input neuron vector register group (51), output neuron vector register group (54), weight/threshold memory (55) in configurable artificial neural network (12) on-chip Data is externally written and read; the mapping function of the segmented linear mapping unit in the parallel computing unit and the control parameters of the neural network control module (56) are also flexibly configured by the bus read-write interface (57). 30. The visual image processing system based on a programmable visual chip according to claim 29, wherein each parallel operation unit includes a fixed-point multiplier, an accumulation register and a piecewise linear mapping unit, wherein the fixed-point The multiplier and the accumulation register are used to complete the multiplication and accumulation operation of the input neuron data and the corresponding weight factor/threshold, the accumulation register can be cleared by the neural network control module, and the piecewise linear mapping unit is used to realize the activation transfer function whose output is used to update the output neuron vector register set (54). 31. The visual image processing system based on programmable vision chips according to claim 29, characterized in that, under the control of the neural network control module (56), the neuron broadcaster (52) broadcasts An input neuron is sent to the parallel computing unit array (53), and the weight/threshold data corresponding to the broadcasted input neuron is taken out from the weight/threshold memory (55) to the parallel computing unit array (53) simultaneously. ), after being multiplied by the multipliers of each parallel operation unit, accumulate to the accumulating register, implement the piecewise linear mapping in parallel after all are completed, and send the final result into the described output neuron vector register group after being normalized into T2 bits ( 54). 32. The visual image processing system based on a programmable visual chip according to claim 31, characterized in that, the data written into the weight/threshold memory (55) and the configuration parallel operation unit and neural network control module (56) The data is obtained according to the training result of the neural network, and the training process is realized on the reduced instruction processor dual-core subsystem (13) or the general-purpose processor outside the system. 33. The visual image processing system based on a programmable vision chip according to claim 29, wherein the configurable artificial neural network (12) on the chip supports a maximum of T1 input neurons, a maximum of T2 output neurons, And T2≤T1, when the number of input neurons is less than T1, or the number of output neurons is less than T2, the corresponding data in the remaining input neuron registers, output neuron registers and weight/threshold memory will be automatically set to 0. 34. the visual image processing system based on programmable vision chip according to claim 29, is characterized in that, the data of described output neuron register is read out by on-chip bus (16), and input to input neuron register again, Realize the computation of multi-layer neural network. 35. The visual image processing system based on a programmable vision chip according to claim 6, characterized in that, the RISC dual-core subsystem (13) includes No. 1 RISC core (RISC#1), No. 1 RISC private program/data memory, No. 2 reduced instruction processor core (RISC#2), No. 2 RISC private program/data memory, inter-processor core communication mailbox and processor arbitrator, in which: No. 1 RISC #1 and No. 2 RISC cores (RISC #2) of the RISC dual-core subsystem (13) respectively have a private program/data memory of P bit data bit width , to achieve thread-level parallel processing, responsible for irregular processing in advanced image processing other than normal feature recognition and control of the entire system. 36. The visual image processing system based on the programmable visual chip according to claim 35, characterized in that, The No. 1 RICP core (RISC#1) and No. 2 RICP core (RISC#2) use the inter-processor core communication mailbox to communicate to realize necessary thread synchronization and data exchange; The access rights of the No. 1 RISC processor core (RISC#1) and the No. 2 RISC processor core (RISC#2) to the on-chip bus are controlled by the processor arbiter, which is implemented on the hardware Support fixed priority and first-come-first-served two arbitration methods; The inter-processor core communication mailbox is a synchronous bidirectional FIFO. 37. The visual image processing system based on a programmable visual chip according to claim 35, characterized in that, the RISC dual-core subsystem (13) is also based on the M×M pixel-level parallel processing unit array ( 7) dynamically adjust the data in the parameter register of the image sensor control module (6) by processing the obtained macroscopic image information or the target range of interest with the M×1 row processing unit array (9), to continuously adapt Changing application environments, and satisfying the multi-resolution processing requirements brought about by the relative motion of the system or the target in the environment. 38. The visual image processing system based on programmable visual chip according to claim 6, characterized in that, said random/sequential mixed I/O memory (14) is a dual-port memory, wherein one port is P bit width , which can be accessed randomly by the on-chip bus, and the other port is PS (PS<P) bit wide, and the sequential read and write access is performed by off-chip devices, and the read and write are independent of each other; the enabling of sequential read and write outside the chip The signal can be automatically mapped to the physical address of the memory by the address generation module embedded in the memory; the physical address can be cleared by external redirection. 39. the visual image processing system based on programmable vision chip according to claim 6, is characterized in that, described system thread mark (15) is W bit register, and wherein some bits are controlled by system internal on-chip bus (16) It is responsible for controlling the write, while other bits are controlled by the external device of the system; all bits of the flag register can be read inside and outside the system. 40. The visual image processing system based on a programmable visual chip according to claim 6, characterized in that, the on-chip bus (16) transmits the read and write control signals from the RISC dual-core subsystem (13) When the logical address information is mapped to the strobe enable signal and physical address information required by other bus slave device modules, the device modules include image sensor control module, processing array control module, on-chip artificial neural network, random/order hybrid I/O memory, and system thread flags. 41. The visual image processing system based on a programmable visual chip according to claim 6, characterized in that, in the visual image processing system, the digital pixel data obtained by the image sensor is loaded into the M In the ×M pixel-level parallel processing unit array (7), various low-level 1. Intermediate image processing, extracting image features and sending them to the on-chip artificial neural network (12) for feature recognition, and further analysis and processing by the reduced instruction processor dual-core subsystem (13), to obtain a small amount of final required result data and output .

Images