CN112529171B - In-memory computing accelerator and optimization method thereof - Google Patents
In-memory computing accelerator and optimization method thereof Download PDFInfo
- Publication number
- CN112529171B CN112529171B CN202011406904.6A CN202011406904A CN112529171B CN 112529171 B CN112529171 B CN 112529171B CN 202011406904 A CN202011406904 A CN 202011406904A CN 112529171 B CN112529171 B CN 112529171B
- Authority
- CN
- China
- Prior art keywords
- resolution
- processing unit
- adc
- dac
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Neurology (AREA)
- Memory System (AREA)
Abstract
The application discloses an in-memory computing accelerator and an optimization method thereof, and relates to the technical field of in-memory computing accelerators. The in-memory computing accelerator includes a plurality of processing units including: the storage units are distributed in a plurality of arrays, the output ports of each column of storage units are correspondingly provided with ADCs, and the ADCs are ADCs with configurable resolution; a plurality of resolution control modules; the resolution control module is used for controlling the resolution of the corresponding ADC. The application also discloses an optimization method of the in-memory computing accelerator, which comprises training and quantifying a neural network model; in the process of deploying the neural network model, determining the sparseness degree of the whole neural network; according to the sparseness of each layer of neural network, calculating the optimal resolution of the ADC in each processing unit; the resolution of the ADC in each processing unit is dynamically adjusted to an optimal resolution. The method and the device are used for improving the performance of the in-memory computing accelerator.
Description
Technical Field
The application relates to the technical field of in-memory computing accelerators, in particular to an in-memory computing accelerator and an optimization method thereof.
Background
In recent years, neural networks have achieved significant success in various practical applications, such as image classification and object detection, but these efforts have largely relied on complex neural network models with a large number of parameters and calculations. Deployment of these complex neural network models, which require extensive computation and data movement, to a Feng neumann architecture-based neural network accelerator (e.g., CPU, GPU, FPGA) would present a "memory wall" problem, i.e., the data movement speed is not followed by the data processing speed and the data movement energy consumption is much higher than the data processing energy consumption.
In-memory computing is an emerging computing architecture, and compared with the traditional neural network accelerator based on Feng Neumann architecture, which separates storage and computing, the in-memory computing integrates storage and computing, namely, the computing is completed inside a storage unit. In-memory computing is expected to solve the problem of "memory wall" existing in the Feng Neumann architecture. Because of the integration of storage and computation, in-memory computation can realize a neural network consisting of a large number of multiplication and addition operations with higher performance.
In existing in-memory computational accelerators, the power consumption of analog-to-digital converters (ADCs) for analog and digital signal conversion can be as high as 50% or more. It can be seen that the optimal design of the ADC is a major bottleneck for the current in-memory computational accelerator. The prior art generally reduces ADC power consumption by optimizing for the ADC internal circuitry alone, but the effect of this approach is not significant.
Disclosure of Invention
The embodiment of the application provides an in-memory computing accelerator and an optimization method thereof, which can dynamically optimize the resolution of an ADC (analog-to-digital converter) in the in-memory computing accelerator according to the sparsity difference of each layer of a quantized neural network, thereby reducing in-memory computing power consumption and improving in-memory computing capability.
To achieve the above object, in one aspect, an embodiment of the present application provides an in-memory computing accelerator, including a plurality of processing units, where the processing units include: the storage units are distributed in a plurality of arrays, and the output ports of the storage units in each column are correspondingly provided with ADCs, wherein the ADCs are ADC with configurable resolution; a plurality of resolution control modules; the resolution control module is used for controlling the resolution of the corresponding ADC.
Further, the in-memory computing accelerator further comprises a DAC (digital-to-analog converter) correspondingly arranged at an input port of each row of storage units, wherein the DAC is a DAC with configurable resolution; the resolution control module is used for controlling the resolution of the corresponding DAC.
On the other hand, the embodiment of the application provides an optimization method of the in-memory computing accelerator, which is characterized by comprising the following steps: training and quantifying a neural network model; in the process of deploying the neural network model, determining the sparseness degree of the whole neural network; according to the sparseness of each layer of neural network, calculating the optimal resolution of the ADC in each processing unit; the resolution of the ADC in each processing unit is dynamically adjusted to an optimal resolution.
Further, the step of calculating the optimal resolution of the ADC in each processing unit according to the sparseness of the neural network of each layer includes: calculating the optimal resolution of the ADC in each processing unit according to equation (1):
wherein RowNum is the number of rows in the memory cell array in each processing unit; w is a weight; density is the weight Density of each layer of the neural network; and storing the calculated optimal resolution of the ADC in each processing unit into a resolution control module of the corresponding processing unit.
Further, after dynamically adjusting the resolution of the ADC in each processing unit to the optimal resolution, the method further includes: calculating the optimal resolution of the DAC in each processing unit according to the input excitation precision of each layer of neural network; the resolution of the DAC in each processing unit is dynamically adjusted to optimize the DAC resolution.
Further, the step of calculating the optimal resolution of the DAC in each processing unit according to the accuracy of the input excitation of the neural network of each layer includes: calculating the optimal resolution of the DAC in each processing unit according to equation (2):
DAC optimum resolution=input excitation precision+1 equation (2)
And storing the calculated optimal resolution of the DAC in each processing unit into a resolution control module of the corresponding processing unit.
Compared with the prior art, the application has the following beneficial effects:
1. according to the method and the device, the ADC with configurable resolution and the resolution control module for controlling the resolution of the ADC are integrated in the in-memory computing accelerator, the resolution of the ADC can be dynamically optimized according to the sparseness degree of each layer of the neural network, the power consumption and the conversion time of the ADC in the in-memory computing accelerator are flexibly and comprehensively reduced, and therefore the power consumption of the in-memory computing accelerator is reduced, and the computing capability of the in-memory computing accelerator is improved.
2. The method and the device integrate the DAC with configurable resolution into the in-memory computing accelerator, dynamically optimize the resolution of the ADC and the resolution of the DAC through the resolution control module, reduce the power consumption and the conversion time of the ADC and the DAC in the in-memory computing accelerator, further reduce the power consumption of the in-memory computing accelerator and improve the computing capability of the in-memory computing accelerator.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of an in-memory computational accelerator based on quantifying sparsity of a neural network according to the present application;
FIG. 2 is a schematic diagram of a processing unit in an in-memory computing accelerator based on quantifying sparseness of a neural network according to the present application;
FIG. 3 is a flow chart of an in-memory calculation accelerator optimization method based on quantifying sparsity of a neural network according to one embodiment of the present application;
fig. 4 is a flowchart of an optimization method of an in-memory computing accelerator based on quantifying sparsity of a neural network according to another embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
In the existing in-memory computing accelerator, the power consumption ratio of the ADC for converting analog signals and digital signals can reach more than 50%, so that the optimal design of the ADC is a main bottleneck of the existing in-memory computing accelerator, however, the research on how to optimize the ADC in the in-memory computing accelerator is basically carried out by singly aiming at the realization of an internal circuit of the ADC, and the optimization of the ADC in the in-memory computing accelerator based on the characteristics of a neural network is omitted. However, related experimental studies have shown that the weights of the neural networks have a high degree of sparsity, particularly the quantized neural networks. Further, at the circuit design level, as the resolution of the ADC increases, the power consumption of the Flash (Flash) type ADC increases exponentially, while the power consumption and conversion time of the Successive Approximation (SAR) type ADC show a tendency to increase linearly.
The method is based on sparsity existing in a common neural network model, particularly a quantized neural network, and a large number of weights with values of 0 exist. During the computation, i.e. when the word line or row line in the memory array is activated, the memory cell storing "0" will not increase or decrease any voltage or current to the bit line, and the voltage or current range on the bit line will also be reduced, thereby reducing the requirements on the ADC resolution. Therefore, when the weight sparseness degree stored on the same bit line is larger, namely the weight with the value of 0 is more, the resolution of the ADC can be optimized to be smaller, so that the power consumption of the ADC and the conversion time of the ADC are reduced, the power consumption of the in-memory computing accelerator is further reduced, and the computing capability of the in-memory computing accelerator is improved.
Referring to fig. 1, an embodiment of the present application provides an in-memory computing accelerator, which includes a plurality of slicing modules, a pooling module, an accumulation module, an activation module, and a global buffer. Weights of all layers of the neural network are deployed to all the segmentation modules respectively. In each of the fragmentation modules there are multiple processing units (PEs) and fragmentation buffers, accumulators and output buffers.
Referring to fig. 2, the processing unit mainly includes a memory cell array, a plurality of ADCs and resolution control modules thereof, a bit line decoder, a word line decoder, an analog multiplexer, a shift register, and the like.
The storage unit array comprises a plurality of storage units distributed in an array mode, and an ADC (analog to digital converter) is correspondingly arranged at an output port of each column of storage units and is configurable in resolution. In the process of the neural network deployment, the optimized resolution of the ADC in each processing unit obtained through software calculation is stored in a resolution control module of each processing unit, the resolution of the ADC in each processing unit is adjusted to the optimized resolution, and in the process of the storage unit calculation, a bit line decoder and a word line decoder jointly control a storage unit array. After multiply-add calculation is completed in the storage unit, conversion from analog signals to digital signals is completed through the ADC. The resolution of the ADC during conversion is determined by the optimal resolution of the ADC stored in the resolution control module. In addition, the ADC may be owned by each bit line individually or shared by multiple bit lines through an analog multiplexer. Therefore, according to the embodiment of the application, the resolution of the ADC can be dynamically optimized according to the sparseness degree of each layer of the neural network, the power consumption of the ADC in the in-memory computing accelerator and the conversion time of the ADC are greatly reduced, so that the power consumption of the in-memory computing accelerator is reduced, and the computing capability of the in-memory computing accelerator is improved.
In some embodiments, the word line decoder in the processing unit further includes a digital-to-analog converter (DAC) disposed at the input port of each row of memory cells, the DAC also being a resolution configurable DAC, and the resolution of the DAC also being adjusted by the resolution control module. Therefore, the resolution control module can dynamically optimize the resolution of the ADC and the resolution of the DAC, power consumption and conversion time of the ADC and the DAC in the in-memory computing accelerator are reduced, power consumption of the in-memory computing accelerator is further reduced, and computing capacity of the in-memory computing accelerator is improved.
Referring to fig. 3, the embodiment of the present application also provides an optimization method for the in-memory computing accelerator, which includes the following steps:
and S1, training and quantifying a neural network model.
And step S2, determining the sparseness degree of the whole neural network in the neural network model deployment process.
And S3, calculating the optimal resolution of the ADC in each processing unit according to the sparseness degree of the neural network of each layer.
Step S31, calculating the optimal resolution of the ADC in each processing unit according to the formula (1);
wherein RowNum is the number of rows in the memory cell array in each processing unit; w is a weight; density is the weight Density of each layer of the neural network;
and step S32, storing the calculated optimal resolution of the ADC in each processing unit into a resolution control module of the corresponding processing unit.
And S4, dynamically adjusting the resolution of the ADC in each processing unit to the optimal resolution.
It should be noted that, in step S2, the module for determining the sparseness of the entire neural network is a sparseness calculation module, and the module may be integrated into the in-memory calculation accelerator or may be disposed outside the in-memory calculation accelerator. In step S4, the resolution of the ADC in the corresponding processing unit is dynamically adjusted to the optimal resolution by the resolution control module of each processing unit. Therefore, the optimization method of the embodiment of the application is based on the quantized sparse characteristics of the neural network, and the ADC resolution is dynamically optimized according to the sparseness of each layer of the neural network. The optimization method greatly reduces the power consumption of the ADC and the conversion time of the ADC in the in-memory computing accelerator, thereby reducing the power consumption of the in-memory computing accelerator and improving the computing capability of the in-memory computing accelerator.
Referring to fig. 4, in some embodiments, the optimization method of the embodiments of the present application further includes:
and S5, calculating the optimal resolution of the DAC in each processing unit according to the input excitation precision of the neural network of each layer.
Step S51, calculating the optimal resolution of the DAC in each processing unit according to the formula (2):
DAC optimum resolution=input excitation precision+1 equation (2)
And step S52, storing the calculated optimal resolution of the DAC in each processing unit into a resolution control module of the corresponding processing unit.
And S6, dynamically adjusting the resolution of the DAC in each processing unit to optimize the resolution of the DAC.
It should be noted that in step S6, the resolution control module of each processing unit dynamically adjusts the DAC resolution in the corresponding processing unit to the optimal resolution. Therefore, the resolution control module dynamically optimizes the resolution of the ADC and the resolution of the DAC, reduces the power consumption and conversion time of the ADC and the DAC in the in-memory computing accelerator, further reduces the power consumption of the in-memory computing accelerator, and improves the computing capability of the in-memory computing accelerator.
The foregoing is merely a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present disclosure should be covered in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (3)
1. A method of optimizing an in-memory computational accelerator, the in-memory computational accelerator comprising a plurality of processing units, the processing units comprising: the storage units are distributed in a plurality of arrays, and the output ports of the storage units in each column are correspondingly provided with ADCs, wherein the ADCs are ADC with configurable resolution; a plurality of resolution control modules; the resolution control module is used for controlling the resolution of the corresponding ADC; the device also comprises a DAC correspondingly arranged at the input port of each row of storage units, wherein the DAC is a DAC with configurable resolution; the resolution control module is used for controlling the resolution of the corresponding DAC; the optimization method comprises the following steps:
training and quantifying a neural network model;
in the process of deploying the neural network model, determining the sparseness degree of the whole neural network;
according to the sparseness of each layer of neural network, calculating the optimal resolution of the ADC in each processing unit;
dynamically adjusting the resolution of the ADC in each processing unit to an optimal resolution;
according to the sparseness degree of the neural network of each layer, the step of calculating the optimal resolution of the ADC in each processing unit comprises the following steps:
calculating the optimal resolution of the ADC in each processing unit according to equation (1);
wherein RowNum is the number of rows in the memory cell array in each processing unit;
w is a weight;
density is the weight Density of each layer of the neural network;
and storing the calculated optimal resolution of the ADC in each processing unit into a resolution control module of the corresponding processing unit.
2. The optimization method according to claim 1, characterized in that,
the step of dynamically adjusting the resolution of the ADC in each processing unit to an optimal resolution further comprises:
calculating the optimal resolution of the DAC in each processing unit according to the input excitation precision of each layer of neural network;
the resolution of the DAC in each processing unit is dynamically adjusted to optimize the DAC resolution.
3. The optimization method according to claim 2, wherein the step of calculating the optimal resolution of the DAC in each processing unit based on the accuracy of the input excitation of the neural network of each layer comprises:
calculating the optimal resolution of the DAC in each processing unit according to formula (2);
DAC optimum resolution=input excitation precision+1 equation (2)
And storing the calculated optimal resolution of the DAC in each processing unit into a resolution control module of the corresponding processing unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011406904.6A CN112529171B (en) | 2020-12-04 | 2020-12-04 | In-memory computing accelerator and optimization method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011406904.6A CN112529171B (en) | 2020-12-04 | 2020-12-04 | In-memory computing accelerator and optimization method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112529171A CN112529171A (en) | 2021-03-19 |
| CN112529171B true CN112529171B (en) | 2024-01-05 |
Family
ID=74997550
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011406904.6A Active CN112529171B (en) | 2020-12-04 | 2020-12-04 | In-memory computing accelerator and optimization method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112529171B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP4367665B1 (en) * | 2021-07-05 | 2025-05-07 | Silicon Storage Technology Inc. | Programmable output blocks for analog neural memory in a deep learning artificial neural network |
| CN114300014B (en) * | 2021-12-30 | 2024-08-02 | 厦门半导体工业技术研发有限公司 | In-memory data processing circuit and resistive random access memory |
| US12198766B2 (en) | 2023-02-22 | 2025-01-14 | Macronix International Co., Ltd. | Artificial neural network operation circuit and in-memory computation device thereof |
| CN116402106B (en) * | 2023-06-07 | 2023-10-24 | 深圳市九天睿芯科技有限公司 | Neural network acceleration method, neural network accelerator, chip and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104506195A (en) * | 2014-12-25 | 2015-04-08 | 北京兆易创新科技股份有限公司 | SAR ADC (successive approximation register analog-to-digital converter) with resolution configurable |
| US9906232B1 (en) * | 2017-03-10 | 2018-02-27 | Xilinx, Inc. | Resolution programmable SAR ADC |
| US10009035B1 (en) * | 2017-04-24 | 2018-06-26 | Huawei Technologies Co., Ltd. | Dynamic control of ADC resolution |
| CN111026700A (en) * | 2019-11-21 | 2020-04-17 | 清华大学 | Memory computing architecture for realizing acceleration and acceleration method thereof |
| CN111079919A (en) * | 2019-11-21 | 2020-04-28 | 清华大学 | Memory computing architecture supporting weight sparsity and data output method thereof |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8369458B2 (en) * | 2007-12-20 | 2013-02-05 | Ralink Technology Corporation | Wireless receiving system with an adaptively configurable analog to digital converter |
| JP2013021599A (en) * | 2011-07-13 | 2013-01-31 | Renesas Electronics Corp | Data processing system |
| CN111630527B (en) * | 2017-11-14 | 2024-12-31 | 技术研发基金会有限公司 | Analog-to-digital converters using memory in neural networks |
-
2020
- 2020-12-04 CN CN202011406904.6A patent/CN112529171B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104506195A (en) * | 2014-12-25 | 2015-04-08 | 北京兆易创新科技股份有限公司 | SAR ADC (successive approximation register analog-to-digital converter) with resolution configurable |
| US9906232B1 (en) * | 2017-03-10 | 2018-02-27 | Xilinx, Inc. | Resolution programmable SAR ADC |
| US10009035B1 (en) * | 2017-04-24 | 2018-06-26 | Huawei Technologies Co., Ltd. | Dynamic control of ADC resolution |
| CN110546887A (en) * | 2017-04-24 | 2019-12-06 | 华为技术有限公司 | Dynamic Control of ADC Resolution |
| CN111026700A (en) * | 2019-11-21 | 2020-04-17 | 清华大学 | Memory computing architecture for realizing acceleration and acceleration method thereof |
| CN111079919A (en) * | 2019-11-21 | 2020-04-28 | 清华大学 | Memory computing architecture supporting weight sparsity and data output method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112529171A (en) | 2021-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112529171B (en) | In-memory computing accelerator and optimization method thereof | |
| Fick et al. | Analog matrix processor for edge AI real-time video analytics | |
| US10241971B2 (en) | Hierarchical computations on sparse matrix rows via a memristor array | |
| Wang et al. | Low power convolutional neural networks on a chip | |
| EP3754561A1 (en) | Reconfigurable memory compression techniques for deep neural networks | |
| CN112329910B (en) | Deep convolution neural network compression method for structure pruning combined quantization | |
| CN111026700B (en) | Memory computing architecture for realizing acceleration and acceleration method thereof | |
| WO2021036905A1 (en) | Data processing method and apparatus, computer equipment, and storage medium | |
| WO2021036904A1 (en) | Data processing method, apparatus, computer device, and storage medium | |
| CN115018062B (en) | A convolutional neural network accelerator based on FPGA | |
| WO2021036908A1 (en) | Data processing method and apparatus, computer equipment and storage medium | |
| CN111079919B (en) | An in-memory computing architecture supporting weight sparse and its data output method | |
| KR20240046492A (en) | Sparsity-aware in-memory computing | |
| CN110442323A (en) | Carry out the architecture and method of floating number or fixed-point number multiply-add operation | |
| KR20240025523A (en) | Computation in memory (CIM) architecture and data flow supporting depth-specific convolutional neural network (CNN) | |
| US12217184B2 (en) | Low-power, high-performance artificial neural network training accelerator and acceleration method | |
| CN112181895A (en) | Reconfigurable Architectures, Accelerators, Circuit Deployment, and Computational Dataflow Methods | |
| CN115664422B (en) | Distributed successive approximation type analog-to-digital converter and operation method thereof | |
| CN119152906B (en) | A high-density near-memory computing and in-memory computing hybrid architecture and computing method based on eDRAM | |
| Zhang et al. | YOLOv3-tiny object detection SOC based on FPGA platform | |
| CN113516235B (en) | Deformable convolution accelerator and deformable convolution acceleration method | |
| CN113723044A (en) | Data sparsity-based extra row activation and storage integrated accelerator design | |
| CN221200393U (en) | Small chip device and artificial intelligent accelerator device | |
| Ji et al. | Compacc: Efficient hardware realization for processing compressed neural networks using accumulator arrays | |
| Chang et al. | BFP-CIM: Runtime Energy-Accuracy Scalable Computing-in-Memory-Based DNN Accelerator Using Dynamic Block-Floating-Point Arithmetic |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |