CN109086879B - Method for realizing dense connection neural network based on FPGA - Google Patents

Abstract

The invention discloses a method for realizing a dense connection neural network based on an FPGA (field programmable gate array), which comprises the following steps of: dividing the whole convolutional neural network into a plurality of dense connecting blocks; designing a convolution operation unit by using resources on the FPGA, and further designing an FPGA end convolution operation module; designing the data transceiving logic of the whole neural network, which comprises seven parts: input Feature Map, Send Buffer, convolution operation module, Receive Buffer, Output Feature Map, sense Block Buffer, Max Buffer; designing the sizes of storage areas required by an Input Feature Map, an Output Feature Map and a sense Block Buffer according to the size of Input and Output data of each layer of the Dense connection neural network, and designing the sizes of the storage areas required by a Send Buffer and a Receive Buffer according to the size of the Block and the parallelism of a convolution operation unit; and designing data transceiving logic according to the characteristics of each layer of the dense connection neural network. The method can reduce the width of each layer of the network on the premise of ensuring the accuracy of the algorithm, reduce the number of parameters, improve the data transmission efficiency and improve the running speed of the neural network.

Description

An implementation method of densely connected neural network based on FPGA

technical field

The invention belongs to the field of image processing, and in particular relates to an implementation method of a densely connected neural network based on FPGA.

Background technique

In the field of image processing, Convolutional Neural Network (CNN) has become the dominant machine learning method. With the continuous development of neural network and the deepening of network depth, network performance and recognition accuracy have been greatly improved. However, with the deepening of the network depth, the disappearance of input information and gradients has gradually become the dominant factor preventing the improvement of the accuracy of convolutional neural networks. Research has shown that convolutional neural networks can express more accurately and efficiently if shorter connections are used between layers close to the input and layers close to the output.

Densely connected neural networks connect all layers (with feature maps of the same size) together to ensure maximum information flow between network layers. Each layer concatenates the outputs of all previous layers as the input of this layer, and passes the output of this layer to all subsequent layers. Densely connected neural network alleviates the problem of gradient disappearance, enhances feature propagation, and realizes feature reuse. Since it does not need to relearn redundant feature maps, it requires fewer parameters than traditional convolutional neural networks. .

FPGA (Field-Programmable Gate Array), namely Field Programmable Gate Array, is a high-speed, high-density semi-custom circuit in the field of application-specific integrated circuits. The chips produced by FPGA manufacturers do not contain configuration information. Users can configure according to their actual needs, and realize the required functions by making full use of the resources provided on the chip.

The traditional convolutional neural network has too many parameters and complex network structure, which makes its running speed low. Therefore, a new network architecture is needed to reduce the number of parameters and make full use of the parallel operation characteristics of FPGA to improve the performance of the convolutional neural network. running speed.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method for implementing a densely connected neural network based on FPGA, by splicing the outputs of all the previous layers as the input of this layer, to realize the repeated use of features, and to reduce the reduction in the accuracy of the algorithm under the premise of ensuring The width of each layer of the network can reduce the number of parameters. By designing the data sending and receiving logic of each layer of the neural network, the reuse of data is strengthened, the number of data transmissions is reduced, the efficiency of data transmission is improved, and the parallel operation of FPGA is fully utilized to improve the neural network. The speed of the network.

In order to achieve the above-mentioned purpose, the solution of the present invention is:

An implementation method of a densely connected neural network based on FPGA, comprising the following steps:

Step 1. Divide the entire convolutional neural network into multiple densely connected blocks;

Step 2, using FPGA on-chip DSP resources, LUT and logic resources to design a convolution operation unit, and then design an FPGA side convolution operation module;

Step 3: Design the overall data sending and receiving logic of the neural network, including seven parts: Input Feature Map, Send Buffer, Convolution Operation Module, Receive Buffer, Output Feature Map, Dense BlockBuffer, Max Buffer;

Step 4: Design the size of the storage area required for Input FeatureMap, Output Feature Map, and Dense Block Buffer according to the size of the input and output data of each layer of the densely connected neural network, and design the Send Buffer, The size of the storage area required for the Receive Buffer;

Step 5, according to the characteristics of each layer of the densely connected neural network, specifically design its data sending and receiving logic.

After adopting the above scheme, the present invention can be realized by the hardware platform Xilinx ZYNQ-702N. The present invention designs a convolutional neural network based on the dense connection, and reuses the features of the front layer through the dense connection, under the premise of ensuring the accuracy of the algorithm Reduce the width of each layer of the network, thereby reducing the number of parameters. By designing the data sending and receiving logic of each layer of the neural network, the reuse of data is strengthened, the number of data transmissions is reduced, the efficiency of data transmission is improved, and the parallel operation of FPGA is fully utilized to improve the The efficiency of neural network operation.

Description of drawings

Figure 1 is a convolutional neural network structure based on dense connection design;

Figure 2 is a schematic diagram of the design of the convolution unit;

Figure 3 is the overall data transceiver logic of the neural network;

Fig. 4 is the block division schematic diagram of Feature Map;

Figure 5 is a schematic diagram of the implementation of adding Output Feature Map and padding operation;

Among them, (a) is a schematic diagram of a single Map plus padding, (b) is a schematic diagram of storing all the maps with padding in the Output Feature Map in order;

Figure 6 is a schematic diagram of each layer of dense connection block buffering to Dense Block Buffer.

Detailed ways

The technical solutions and beneficial effects of the present invention will be described in detail below with reference to the accompanying drawings.

The present invention provides a method for implementing a densely connected neural network based on FPGA, comprising the following steps:

Step 1: Divide the entire convolutional neural network into multiple densely connected blocks based on dense connections, and replace the adjacent layers with the same feature map size in the traditional convolutional neural network with densely connected blocks. As shown in Figure 1, its design rules Yes:

(1) The feature map size of each layer in each dense connection block is consistent;

(2) Each layer in each densely connected block splices the outputs of all previous layers as the input of this layer, and transmits the output of this layer to all subsequent layers in the block;

(3) The densely connected blocks are connected by a convolution pooling layer.

Step 2: Using the resources on the FPGA, design the convolution operation unit as shown in Figure 2, and then design the convolution operation module on the FPGA side. The convolution operation module places P convolution operation units in parallel on the FPGA side to improve the degree of parallelism , where P takes the maximum value that does not exceed the resource constraints on the FPGA side.

Convolution operation unit: Multiply the m*m convolution kernel with the corresponding position element in the m*m area of the feature map, add the obtained results, and then add the bias bias to obtain an output value output. The feature maps and weights that need to be calculated are sequentially sent to the convolution operation module for operation, that is, the multiply-accumulate operation is performed, and the calculation result is sent to the output buffer.

Step 3: Design the overall data sending and receiving logic of the neural network, including seven parts: Input Feature Map, Send Buffer, Convolution Operation Module, Receive Buffer, Output Feature Map, Dense BlockBuffer, and Max Buffer. As shown in Figure 3, the transceiver logic is as follows:

个Block，第一次的发送顺序为1¹，1²，1³，1⁴，2¹，2²，2³，2⁴，...，16¹，16²，16³，16⁴，其中i^j表示第j张Map的第i个Block，其后的发送顺序以此类推；(1) Divide the data in the Input Feature Map according to the size of the Block, and transmit P blocks each time to the Send Buffer in the order of the Feature Map, as shown in Figure 4: Set the size of the input image to 320*320, The number of input channels input_channels=4, P=64, then each Map can be sent each time

Blocks, the first sending sequence is 1 ¹ , 1 ² , 1 ³ , 1 ⁴ , 2 ¹ , 2 ² , 2 ³ , 2 ⁴ ,..., 16 ¹ , 16 ² , 16 ³ , 16 ⁴ , where i ^j represents the i-th Block of the j-th Map, and the subsequent sending order is analogous;

(2) Send the Map data in the Send Buffer to the convolution operation module through DMA transmission;

(3) The weight and offset corresponding to the sent Feature Map Block are sent to the Send Buffer, and P convolution kernels are sent each time;

(4) Send the weight and offset in the Send Buffer to the convolution operation module through DMA transmission, and start the convolution operation;

(5) Transfer the calculation result to Receive Buffer through DMA;

(6) After a batch of Feature Maps and weights are convolved, if max pooling is not required, the calculation results received in the Receive Buffer are sent to the Output Feature Map in the order of Feature Maps. Put the Block to the corresponding position of the corresponding Map;

If max pooling is required, the calculation results received in the Receive Buffer are sent to the Max Buffer in the order of the Feature Map, each received Block is placed in the corresponding position of the corresponding Map, and the data in the Max Buffer is max pooled and send it to the Output Feature Map;

If it is a densely connected block, each layer will splicing the outputs of all previous layers in the Dense Block Buffer and store it in the Input Feature Map as the input of this layer, and pass the output of this layer to the block through the Dense BlockBuffer. all subsequent layers. See Step 5 for the specific design of the dense connection block.

(7) Swap the Input Feature Map and Output Feature Map pointers, so as to realize the output of the previous layer as the input of the next layer.

Step 4: Design the storage area size required for Input FeatureMap, Output Feature Map, and Dense Block Buffer according to the size of the input and output data of each layer of the densely connected neural network. Design the size of the storage area required for Send Buffer and Receive Buffer according to the block size and the parallelism of the convolution operation unit.

(1) Input Feature Map:

max{input_featureMapSize _l ² *input_channels _l |l=0,1,...,n}

where input_featureMapSize _l is the input feature map size of the lth layer, input_channels _l is the number of input channels of the lth layer, and n is the number of layers of the entire convolutional neural network.

(2)Output Feature Map:

max{output_featureMapSize _l ² *output_channels _l |l=0,1,...,n}

where output_featureMapSize _l is the feature map size output by the first layer, output_channels ₁ is the number of output channels of the first layer, and n is the number of layers of the entire convolutional neural network.

(3) Dense Block Buffer:

where input_featureMapSize _i is the input feature map size of the first layer of the ith dense connection block, input_channels _i is the number of input channels, j is the jth layer of the ith dense connection block, and _ni is the layer of the ith dense connection block output_featureMapSize _j is the output feature map size of the jth layer of the ith dense connection block, output_channels _j is the number of output channels, and m is the number of dense connection blocks.

(4) Send Buffer, Receive Buffer: blockSize ² *P

where blockSize is the block size divided by the Input Feature Map, and P is the number of convolution operation units.

Step 5: Design the data sending and receiving logic according to the characteristics of each layer of the densely connected neural network, including the convolutional layer with the number of input channels input_channels<P, the convolutional layer with the number of input channels input_channels>P, the max pooling layer, and the dense connection. The specific design of the block.

(1) For the convolutional layer with the number of input channels input_channels<P, each time P blocks are sent to perform the convolution operation, and the obtained result is the Block of the corresponding output layer.

则将每次卷积运算结果对应的Block相加后暂存至Max Buffer，若发送的次数

则将卷积运算结果对应的Block相加后存至Output FeatureMap，其中input_channels必须为P的整数倍。(2) For the convolutional layer with the number of input channels input_channels>P, each time P blocks are sent for convolution operation, and the obtained result is temporarily stored in the Max Buffer.

Then add the blocks corresponding to the results of each convolution operation and temporarily store them in the Max Buffer.

Then add the blocks corresponding to the results of the convolution operation and store them in the Output FeatureMap, where input_channels must be an integer multiple of P.

(3) If the next layer is the max pooling layer, the calculation results received in the Receive Buffer of this layer are sent to the Max Buffer in the order of the Feature Map, and each received Block is placed in the corresponding position of the corresponding Map, and the All output data of this layer are sent to Max Buffer and then pooled (m*m max pooling, strides=m):

The data in the Max Buffer is divided into multiple blocks according to m*m, and the maximum value of the m*m pieces of data in each block is taken and sent to the corresponding position in the Output Feature Map.

(4) If padding needs to be added to the next layer, set a continuous storage area S of OutputFeature Map to zero before sending Map data to Send Buffer at this layer, S=(output_featureMapSize+padding) ² *output_channels, where output_featureMapSize is this The size of the layer output feature map, padding is the number of rows/columns that need to be added padding, and output_channels is the number of output channels of the layer. As shown in Figure 5, Figure (a) is a schematic diagram of adding padding to a single Map, and Figure (b) is a schematic diagram of storing all the Maps with padding in the Output FeatureMap in order. According to the storage order of the Output Feature Map shown in Figure 5(b), the operation of adding padding to each Output Feature Map is realized by storing the output Blocks received in the Send Buffer in the corresponding positions in the order of the Output Feature Map.

的存储区域用于拼接前面所有层的输出作为下一层的输入，其中input_channels₁为稠密连接块第一层的输入通道数，map_size为特征图尺寸，i为稠密连接块中的第i层，output_channels₁为第1层的输出通道数，m为稠密连接块的层数。(5) For dense connection blocks, when storing the calculation results of the i-th layer to the Output Feature Map, reserve

The storage area is used to splicing the outputs of all previous layers as the input of the next layer, where input_channels ₁ is the number of input channels of the first layer of the dense connection block, map_size is the feature map size, i is the ith layer in the dense connection block, output_channels ₁ is the number of output channels of the first layer, and m is the number of layers of densely connected blocks.

个数据及第i层的输出拼接后存入Input Feature Map作为本层输入。As shown in Figure 6, the feature map input of the densely connected block and the feature map output of each layer in the block are sequentially stored in the Dense Block Buffer.

The data and the output of the i-th layer are spliced and stored in the Input Feature Map as the input of this layer.

The above embodiments are only to illustrate the technical idea of the present invention, and cannot limit the protection scope of the present invention. Any changes made on the basis of the technical solution according to the technical idea proposed by the present invention all fall into the protection scope of the present invention. Inside.

Claims (3)

1. an implementation method of a densely connected neural network based on FPGA, is characterized in that comprising the steps: Step 1. Divide the entire convolutional neural network into multiple densely connected blocks; Step 2, design the convolution operation module on the FPGA side; Step 3: Design the overall data sending and receiving logic of the neural network, including seven parts: Input Feature Map, SendBuffer, Convolution Operation Module, Receive Buffer, Output Feature Map, Dense Block Buffer, MaxBuffer; The transceiver logic in step 3 is as follows: Logic 1: Divide the data in the Input Feature Map according to the size of the block, and transmit P blocks each time to the Send Buffer in the order of the Feature Map: set the size of the input image to 320*320, and the number of input channels input_channels=4 , P=64, then each Map can be sent every time

Blocks, the first sending sequence is 1 ¹ , 1 ² , 1 ³ , 1 ⁴ , 2 ¹ , 2 ² , 2 ³ , 2 ⁴ ,…, 16 ¹ , 16 ² , 16 ³ , 16 ⁴ , where i ^j represents the i-th Block of the j-th Map, and the subsequent sending order is analogous; Logic 2, send the Map data in the Send Buffer to the convolution operation module through DMA; Logic 3, transmit the weight and bias corresponding to the sent Feature Map Block to the Send Buffer, and transmit P convolution kernels each time; Logic 4: Send the weights and offsets in the Send Buffer to the convolution operation module through DMA, and start the convolution operation; Logic 5, transfer the calculation result to Receive Buffer through DMA; Logic 6: After a batch of Feature Maps and weights are convolved, if max pooling is not required, the calculation results received in the Receive Buffer are sent to the Output Feature Map in the order of Feature Maps. The Block is placed in the corresponding position of the corresponding Map; if max pooling is required, the calculation results received in the Receive Buffer are sent to the Max Buffer in the order of the Feature Map, and each received Block is placed in the corresponding position of the corresponding Map, The data in the Max Buffer is max-pooled and sent to the Output Feature Map; if it is a dense connection block, each layer will splicing the outputs of all the previous layers in the Dense Block Buffer and store it in the Input Feature Map as this. The input of the layer, and the output of this layer is passed to all subsequent layers in the block through the Dense Block Buffer; Logic 7: Swap the Input Feature Map and Output Feature Map pointers, so as to use the output of the previous layer as the input of the next layer; Step 4: According to the size of the input and output data of each layer of the densely connected neural network, design the storage area size required for the Input Feature Map, Output Feature Map, and Dense Block Buffer, and design the Send Buffer according to the block size and the parallelism of the convolution operation unit. , the size of the storage area required by the Receive Buffer; Step 5, design its data sending and receiving logic according to the characteristics of each layer of the densely connected neural network; In the step 5, the design data sending and receiving logic includes the design of the convolutional layer with the number of input channels input_channels<P, the convolutional layer with the number of input channels input_channels>P, the max pooling layer, and the design of the dense connection block, specifically: For the convolutional layer with the number of input channels input_channels<P, each time P blocks are sent to perform the convolution operation, and the obtained result is the Block of the corresponding output layer; For the convolutional layer with the number of input channels input_channels>P, each time P blocks are sent to perform the convolution operation, and the obtained result is temporarily stored in the Max Buffer.

Then add the blocks corresponding to the results of each convolution operation and temporarily store them in the Max Buffer.

Then add the blocks corresponding to the convolution operation results and store them in the Output Feature Map, where input_channels must be an integer multiple of P; If the next layer is the max pooling layer, the calculation results received in the Receive Buffer of this layer are sent to the Max Buffer in the order of Feature Map, each received Block is placed in the corresponding position of the corresponding Map, and the All output data is sent to Max Buffer and then pooled, m*m max pooling, strides=m: Divide the data in the Max Buffer into multiple blocks according to m*m, and take the maximum value of the m*m pieces of data in each block and send it to the corresponding position in the Output Feature Map; For densely connected blocks, when storing the calculation results of the i-th layer to the Output Feature Map, reserve

The storage area is used to splicing the outputs of all previous layers as the input of the next layer, i=1,2,...,m-1, where input_channels ₁ is the number of input channels of the first layer of the dense connection block, and map_size is the feature map size , i is the ith layer in the densely connected block, output_channels ₁ is the number of output channels of the lth layer, and m is the number of layers of the densely connected block. 2. the realization method of a kind of dense connection neural network based on FPGA as claimed in claim 1 is characterized in that: in described step 1, the design rule that whole convolutional neural network is divided into a plurality of dense connection blocks is: First, the feature map size of each layer in each dense connection block is consistent; Second, each layer in each densely connected block concatenates the outputs of all previous layers as the input of this layer, and transmits the output of this layer to all subsequent layers in the block; Third, the densely connected blocks are connected by convolutional pooling layers. 3. the realization method of a kind of dense connection neural network based on FPGA as claimed in claim 1, is characterized in that: the concrete content of described step 4 is: Input Feature Map: maxinput_featureMapSize ₁ ² *input_channels _l |l=0,1,…,n} where input_featureMapSize _l is the input feature map size of the lth layer, input_channels _l is the number of input channels of the lth layer, and n is the number of layers of the entire convolutional neural network; Output Feature Map: max{output_featureMapSize _l ² *output_channels _l |l=0,1,…,n} where output_featureMapSize _l is the feature map size output by the lth layer, output_channels _l is the number of output channels of the lth layer, and n is the number of layers of the entire convolutional neural network; Dense Block Buffer:

where input_featureMapSize _i is the input feature map size of the first layer of the ith dense connection block, input_channels _i is the number of input channels, j is the jth layer of the ith dense connection block, and _ni is the layer of the ith dense connection block number, output_featureMapSize _j is the output feature map size of the jth layer of the ith dense connection block, output_channels _j is the number of output channels, m is the number of dense connection blocks; Send Buffer, Receive Buffer: blockSize ² *P where blockSize is the block size divided by the Input Feature Map, and P is the number of convolution operation units.

Images