WO2018121118A1 - Calculating apparatus and method - Google Patents

Abstract

Provided in the present disclosure are a calculating apparatus and method. The calculating apparatus comprises a storage device and a neural network processor electrically connected to the storage device. The neural network processor and the storage device exchange data therebetween, and execute neural network calculations. The calculating apparatus and method in the present disclosure can effectively satisfy data transmission and storage requirements during operation, and have the advantages of high memory utilization, low power consumption, low cost and high data transmission rate.

Description

Computing device and method Technical field

The present disclosure relates to the field of neural network computing, and more particularly to a neural network computing apparatus and method.

Background technique

At present, the field of artificial intelligence is developing rapidly, and machine learning is also affecting all aspects of people's lives. As an important part of the machine learning field, research on neural networks is also a hot topic in both industry and academia. Due to the huge amount of data in neural network computing, how to deal with neural network algorithm execution becomes an important issue that we need to solve. Therefore, a dedicated neural network computing device has emerged.

At present, the types of dynamic random access memory DRAMs used in the architecture of neural network computing devices are mostly high performance graphics memory (Graphics Double Data Rate Version 4, GDDR4) or high performance graphics memory (Graphics Double Data Rate Version 5). , GDDR5). However, in the neural network computing device, due to the consideration of bandwidth, performance, power consumption and area, GDDR4 or GDDR5 can not fully meet the needs of the neural network computing device, and the technological development has entered a bottleneck period. Adding 1GB of bandwidth per second will result in more power consumption, which is not a sensible, efficient or cost-effective option for designers and consumers. At the same time, GDDR4 or GDDR5 still has serious problems that are difficult to reduce the area. Therefore, GDDR4 or GDDR5 will gradually hinder the continuous growth of the performance of neural network computing devices.

Summary of the invention

(1) Technical problems to be solved

In view of the above technical problems, the present disclosure provides a neural network computing apparatus and method for solving bottlenecks in terms of bandwidth, power consumption, area, and the like faced by the neural network computing device proposed above.

(2) Technical plan

According to an aspect of the present disclosure, a neural network computing device is provided, including: a storage device; and a neural network processor electrically connected to the storage device, and the data exchange between the neural network processor and the storage device And perform neural network calculations.

According to another aspect of the present disclosure, a chip is provided, including the neural network meter Calculation device.

According to still another aspect of the present disclosure, a chip package structure including the chip is provided.

According to still another aspect of the present disclosure, a board is provided, including the chip package structure.

According to still another aspect of the present disclosure, an electronic device including the board is provided.

According to still another aspect of the present disclosure, a neural network calculation method is provided, comprising: storing data by a storage device; and performing data exchange with the storage device by a neural network processor and performing neural network calculation.

(3) Beneficial effects

It can be seen from the above technical solutions that the neural network computing device and method of the present disclosure have at least one of the following beneficial effects:

(1) In the process of neural network operation, high-bandwidth memory, as the memory of the neural network computing device, can exchange data of input data and operation parameters between the buffer and the memory more quickly, which greatly shortens the I/O time. .

(2) The high-bandwidth memory with stacked storage structure and the neural network processor with HBM memory control module can greatly improve the storage bandwidth, and the bandwidth can be increased to more than twice that of the prior art, and the computing performance is greatly improved.

(3) Since the high-bandwidth memory is a stacked (stacked) structure that does not occupy a horizontal plane space, the area of the neural network computing device can be greatly reduced, and the area of the neural network computing device can be reduced to about 5% of the prior art.

(4) The power consumption of the neural network computing device is reduced, and the data transmission bandwidth and the transmission speed are improved by the micro-bumping process and the TSV process wiring interconnection.

(5) HMC's memory provides high data transmission bandwidth, its data transmission bandwidth can exceed 15 times the bandwidth of DDR3; reduce the overall power consumption, HMC memory technology, compared to the commonly used memory technology such as DDR3/DDR4, for each Bit storage can save more than 70% of power consumption.

(6) The HMC memory module includes a plurality of cascaded HMC memory units, which can flexibly select the number of HMC memory units according to the actual memory size required in the neural network operation process, thereby reducing the waste of functional components.

(7) Multiple HMC memory units cascaded for unified addressing, the multiple HMC memory units The cascading mode and the specific implementation are transparent to the neural network processor, and the neural network processor is convenient to read and write the memory device position, thereby reducing the delay of the memory component.

(8) The HMC memory unit adopts a stacked structure, which can reduce the memory footprint by more than 90% compared with the existing RDIMM technology, and at the same time greatly reduce the overall volume of the neural network computing device. Since the HMC can perform massive parallel processing, the latency of the memory components is small.

(9) The neural network computing device has a plurality of neural network processors, and the plurality of neural network processors are interconnected and communicate information with each other to avoid data consistency problems during operation; and the neural network computing device Support multi-core processor architecture, can make full use of the parallelism in the neural network operation process, accelerate the operation of the neural network.

DRAWINGS

1 is a block diagram showing the overall structure of a high bandwidth memory based neural network computing device in accordance with an embodiment of the present disclosure.

2 is a cross-sectional view of the neural network computing device of FIG. 1.

3 is an overall architectural diagram of a neural network processor with a HBM memory control module in accordance with an embodiment of the present disclosure.

4 is a cross-sectional view of a high bandwidth memory based neural network computing device in accordance with another embodiment of the present disclosure.

FIG. 5 is a schematic diagram of an overall structure of a neural network computing device based on an HMC memory according to an embodiment of the present disclosure.

FIG. 6 is a schematic diagram of an HMC memory unit of a HMC memory-based neural network computing device according to an embodiment of the present disclosure.

FIG. 7 is a schematic diagram of capacity expansion of an HMC memory of a neural network computing device based on an HMC memory according to an actual need according to an embodiment of the present disclosure.

8 is a flow chart of a high bandwidth memory based neural network calculation method in accordance with an embodiment of the present disclosure.

FIG. 9 is a flowchart of a HMC memory-based neural network calculation method according to an embodiment of the present disclosure.

[Explanation of main component symbols of the present disclosure]

1-HMC memory module; 2-neural network processor; 3-external access unit; 4-other modules; 11-mixed memory cube; 12-logic base layer; 101, 201, 401-package substrate; 102, 202-intermediate layer; 103, 203, 403-logic die; 104-high bandwidth memory; 105, 205, 402- Neural network processor; 204-HBM memory control module; 206, 406-through hole; 207, 407-micro solder ball; 208, 405-dynamic random access memory DRAM; 301-storage interface; 302-package structure; Control unit; 304, 404-HBM memory control module; 305-buffer; 306-buffer control module; 307-neural processing unit.

detailed description

The present disclosure will be further described in detail below with reference to the specific embodiments thereof and the accompanying drawings.

The present disclosure provides a neural network computing device, including:

At least one storage device;

At least one neural network processor is electrically connected to the storage device, the neural network processor exchanges data with the storage device, and performs neural network calculation.

In a specific embodiment of the present disclosure, the storage device is a high bandwidth memory (correspondingly, the neural network computing device is a high bandwidth memory based neural network computing device), and each high bandwidth memory includes a stacked (stacked) A plurality of accumulated memories; at least one neural network processor electrically coupled to the high bandwidth memory, the neural network processor and the high bandwidth memory exchange data, and perform neural network calculations.

The High-Bandwidth Memory (HBM) is a new type of low-power memory with excellent communication data path, low power consumption and small area. An embodiment of the present disclosure provides a neural network computing device based on a high-bandwidth memory. Referring to FIG. 1 , the neural network computing device includes: a package substrate 101 (Package Substrate), an interposer 102 (Interposer), and a logic die 103 ( Logic Die), a high bandwidth memory 104 (Stacked Memory) and a neural network processor 105. among them,

The package substrate 101 is configured to carry the other components of the neural network computing device and is electrically connected to the host device, such as a computer, a mobile phone, and various embedded devices.

The interposer 102 is formed on the package substrate 101 and carries the logic die 103 and the neural network processor 105.

The logic die 103 is formed on the interposer 102, and the logic die 103 is used to connect the interposer 102 and the high bandwidth memory 104 to implement a layer encapsulation of the high bandwidth memory 104.

The high bandwidth memory 104 is formed on a logic die 103 that includes a plurality of memories stacked (stacked) in a direction perpendicular to the package substrate 101.

The neural network processor 105 is also formed on the interposer 102 for performing neural network calculations, which can complete the entire neural network calculation, and can also perform basic operations of neural network calculation such as convolution, and the neural network processor 105 passes through the interposer 102. The data is exchanged with the logic die 103 and with the high bandwidth memory 104.

Referring to FIG. 2, a cross-sectional view of the neural network computing device of the present embodiment in a direction perpendicular to the package substrate is shown. The neural network computing device is a 2.5D (2.5-dimensional) memory architecture. The high-bandwidth memory includes four dynamic random access memories (DRAMs) 208, and the DRAM 208 is stacked and stacked by a microbumps process. Micro solder balls 207 are formed between the adjacent DRAMs 208, and a high-bandwidth memory is formed. The bottommost DRAM 208 is formed on the logic die 203 by a micro-bumping process, the logic die 203 and the neural network processor 205 are formed on the interposer 202 by a micro-bumping process, and the interposer 202 is formed by a flip chip soldering process. On the package substrate 201. Through holes 206 are formed in the DRAM 208 by using a through-silicon via (TSVs) process, and through holes are formed in the logic die 203 and the interposer 202 by using a through-silicon via process, and the wires are arranged by using the through holes and the micro solder balls. The DRAM 208 is electrically connected to the logic die 203, and the logic die 203 is electrically connected to the neural network processor 205 through the wires of the vias of the interposer 202 to realize interconnection between the high bandwidth memory and the neural network processor 205, and is processed in the neural network. Under the control of the HBM memory control module 204 of the 205, data is transferred between the neural network processor 205 and the high bandwidth memory.

The existing GDDR memory has a channel width of 32 bits and a 16 channel memory bus width of 512 bits. In the present embodiment, the high bandwidth memory may include four DRAMs, each having two 128-bit channels, and the high bandwidth memory providing a bit width of 1024 bits, which is twice the bit width of the above GDDR memory.

The above description is only illustrative of the disclosure, but the content of the disclosure is not limited thereto. For example, the logical network of the neural network computing device may be multiple, and correspondingly, the high-bandwidth memory may also be multiple. The DRAM of each high-bandwidth memory can also be greater than four, and the number of the above components can be set according to actual needs.

For example, a neural network computing device can include four logical dies and four high-bandwidth memories, each high-bandwidth memory including four DRAMs, each having two 128-bit channels. Each high-bandwidth memory can provide 1024 bits of bit width, and four high-bandwidth memories can provide 4096 bits of bit width, which is eight times the bit width of the above GDDR memory.

Each high-bandwidth memory can also include eight DRAMs, each with two 128-bit channels, each high-bandwidth memory can provide 2048 bits of bit width, and four high-bandwidth memories can provide 8192 bits of bit width. GDDR memory is sixteen times wider than the bit width.

Referring to FIG. 3, an overall architecture diagram of a neural network processor with a HBM memory control module of the present embodiment is shown. The neural network processor includes: a storage interface 301 (Memory Interface), a control unit 303 (Control Processor), a HBM memory control module 304 (HBM Controller), a buffer 305 (BUFFER), a buffer control module 306 (Buffer Controller), and neural processing. The unit 307 (NFU), wherein the control processor 303, the HBM memory control module 304, the buffer 305, the buffer control module 306, and the neural processing unit 307 are integrally packaged to form a package structure 302.

The storage interface 301, as an interface between the neural network processor and the high-bandwidth memory, is electrically connected to the DRAM of the logic die and the high-bandwidth memory through wires, and is used for receiving data transmitted by the high-bandwidth memory and transmitting data to the high-bandwidth memory.

The HBM memory control module 304 is configured to control data transmission between the high bandwidth memory and the buffer, including coordinating the data bandwidth of the high bandwidth memory and the buffer, and synchronizing the clock of the high bandwidth memory and the buffer. The HBM memory control module 304 synchronizes the clocks of the high bandwidth memory and the buffer, converts the bandwidth of the data of the high bandwidth memory received by the storage interface 301 into a bandwidth matched with the buffer, and transmits the bandwidth matched data to the buffer; The bandwidth of the buffer's data is converted to a bandwidth that matches the high bandwidth memory, and the bandwidth matched data is transferred to the high bandwidth memory via the storage interface 301.

The buffer 305 is an internal storage unit of the neural network processor for receiving bandwidth-matched data transmitted by the HBM memory control module 304 and transmitting the stored data to the HBM memory control module 304.

The buffer control module 306 is configured to control data interaction between the buffer 305 and the neural processing unit 307, transmit the data stored by the buffer 305 to the neural processing unit 307, the neural processing unit 307 performs neural network calculation, and the buffer control module 306 performs the neural network. The calculation result of the processing unit 307 is transmitted to the buffer 305.

The control unit 303 decodes the instruction to the HBM memory control module 304, the buffer 305, The buffer control module 306 and the neural processing unit 307 send control commands, coordinate and schedule the above modules to work together to implement the computing functions of the neural network processor.

It can be seen that the neural network computing device of the present disclosure, the high-bandwidth memory using the stacked storage structure and the neural network processor with the HBM memory control module can greatly improve the storage bandwidth, and the bandwidth can be increased to twice that of the prior art. In the above, the computing performance is greatly improved. In the process of neural network operation, the high-bandwidth memory is used as the memory of the neural network computing device, and the data exchange between the input data and the operation parameters can be performed between the buffer and the memory more quickly, which makes IO time is greatly reduced. Since the high-bandwidth memory is a stacked (stacked) structure that does not occupy horizontal planar space, the area of the neural network computing device can be greatly reduced, and the area of the neural network computing device can be reduced to about 5% of the prior art; The power consumption of the neural network computing device; and the DRAMs are interconnected by a micro-bumping process and a TSV process wiring, and the intermediate layer is used for data exchange between the neural network processor and the high-bandwidth memory, thereby further improving between different DRAMs and Transmission bandwidth and transmission speed between the neural network processor and the high bandwidth memory.

A high-bandwidth memory-based neural network computing device proposed by another embodiment of the present disclosure, referring to FIG. 4, the same features as those of the above embodiments are not repeatedly described. The neural network computing device is a 3D (3-dimensional) memory architecture, including a package substrate 401, a neural network processor 402, a logic die 403, and a high bandwidth memory stacked from bottom to top. The high-bandwidth memory includes four DRAMs 405. The DRAM 405 is stacked and stacked by a micro-bumping process. A micro solder ball 407 is formed between the adjacent DRAMs 405. The bottommost DRAM 405 of the high-bandwidth memory is formed by a micro-bumping process. On the logic die 403, the logic die 403 is formed on the neural network processor 402 by a micro-bumping process, and the neural network processor 402 is formed on the package substrate 401 by a micro-bumping process. A via hole 406 is formed in the DRAM 405 by a through-silicon via process, and a via hole is formed in the logic die 403 by using a through-silicon via process. The via hole and the micro solder ball are used to arrange the wire, the DRAM 405 and the logic die 403, and the neural network processor 402. The electrical connection enables vertical interconnection of the high bandwidth memory with the neural network processor 402. Under the control of the HBM memory control module 404 of the neural network processor 402, data is transferred between the neural network processor 402 and the high bandwidth memory.

It can be seen that the neural network computing device can further save the neural network compared to the 2.5D storage architecture because the high bandwidth memory is directly stacked (stacked) on the neural network processor. The area of the computing device is particularly advantageous for miniaturization of the neural network computing device; and the distance between the high bandwidth memory and the neural network processor is shorter, meaning that the wiring between the two is shorter, and the signal transmission quality and transmission speed can be further improved. .

In still another embodiment of the present disclosure, the storage device is an HMC memory module (corresponding, the neural network computing device is a neural network computing device based on a Hybrid Memory Cube (HMC) memory); at least one nerve The network processor is connected to the HMC memory module, and is configured to acquire data and instructions required for the neural network operation from the HMC memory module, perform a neural network partial operation, and write the operation result back to the HMC memory module.

The HMC-based neural network computing device and method can effectively meet the requirements of the neural network computing device for data transmission and storage in the operation process, and the computing device can provide unprecedented system performance and bandwidth, and at the same time, high memory utilization. Low power consumption, low cost and fast data transfer rate.

FIG. 5 is a schematic diagram of an overall structure of a neural network computing device based on an HMC memory according to an embodiment of the present disclosure. As shown in FIG. 5, the present disclosure is based on a HMC memory-based neural network computing device. The device includes an HMC memory module 1, a neural network processor 2, and an external access unit 3.

The HMC memory module 1 includes: a plurality of cascaded HMC memory units, each of which includes a mixed memory cube and a logical base layer.

The hybrid memory cube is connected by a plurality of memory die layers through through silicon vias (TSVs), and the logic substrate layer includes a logic control unit that controls the mixed memory cube for data read and write operations and an external processor or HMC device or external The link to which the access unit is connected.

The neural network processor is configured to perform a function of the neural network operation, acquire the required instructions and data of the neural network operation from the HMC memory module, perform a neural network partial operation, and calculate the intermediate value or the final result Writing back to the HMC memory module, while transmitting data preparation completion signals to other neural network processors through the data path between the neural network processors. The data includes first data and second data, the first data includes network parameters (such as weights, offsets, etc.) and a function table, and the second data includes network input data, and the data may further Includes the intermediate value after the operation.

The external access unit connects the HMC memory module and an external designated address, and reads the neural network instructions and data required for the neural network operation from the external designated address to the HMC memory. The module, as well as the neural network operation results from the HMC memory module to the output operation of the external specified address space.

Further, the cascaded multiple HMC memory units are uniformly addressed, and the cascading manner and specific implementation of the multiple HMC memory units are transparent to the neural network processor, and the neural network processor is cascaded. The memory address of the HMC memory unit reads and writes the memory location of the HMC memory module.

In addition, it is considered that the neural network processor may use a multi-core processor in a specific implementation process to improve the efficiency of operation. The HMC memory module of the present disclosure can be used by a multi-core neural network processor. In view of the fact that the neural network processor may generate data consistency problems during actual operation, multiple neural network processors are interconnected and information is transmitted between each other. Specifically, when a part of the neural network processors in the plurality of neural network processors are in an operational state, and the other neural network processors are in an operation result waiting for one of the partial neural network processors, the other neural network processing The device is in a wait state, and after one of the partial neural network processors transmits the operation result to the HMC memory module and transmits the data delivery signal to the other neural network processor, the other neural network processor is woken up. Read the corresponding data from the HMC memory module and perform the corresponding operation.

FIG. 6 is a schematic diagram of an HMC memory unit of a HMC memory-based neural network computing device according to an embodiment of the present disclosure. As shown in FIG. 6, the HMC includes a hybrid memory cube 11 and a logic substrate layer 12.

The mixed memory cube 11 has a memory composed of a plurality of memory bodies, wherein each memory body includes a plurality of memory grain layers. Preferably, the mixed memory cube has a memory composed of 16 memory banks, wherein each memory bank includes 16 memory grain layers. The top and bottom of the memory die layer are interconnected using a through silicon via (TSV) structure. The TSV interconnects a plurality of memory die layers, adding another dimension in addition to the rows and columns to form a three-dimensional structure of the die. The memory die layer is a conventional random dynamic memory (DRAM). For the access of the hybrid memory cube, the logical base layer is configured to directly select the desired memory die layer by vertically rising. In a memory die layer, rows and columns are used to organize DRAM. A read or write request to a specific location in the memory bank is performed by the memory body controller.

The logic base layer includes a logic control unit that controls the mixed memory cube for data read and write operations, and the logic control unit includes a plurality of memory body controllers, each of which is at a logical base layer There are corresponding memory body controllers for managing 3D access control of the memory. The 3D layering method allows memory accesses to be accessed not only in the direction of the rows and columns but also in parallel between multiple memory die layers.

The logical base layer further includes a link to the external HMC memory unit for connecting the plurality of HMC memory units to increase the total capacity of the HMC storage device. The link includes an external I/O link and internal routing and switching logic connected to an HMC memory module, the external I/O link including a plurality of logical links attached to the switching logic The bootstrap internal routing controls the data transfer of each memory bank or forwards data to other HMC memory cells and neural network processors. Preferably, the external I/O link comprises 4 or 8 logical links, each logical link comprising 16 or 8 serial I/O or SerDes bidirectional link groups. Among them, HMC memory cells with 4 logical links can transmit data at 10Gbps, 12.5Gbps and 15Gbps, and HMC memory cells with 8 logical links can transmit data at 10Gbps. The switching logic determines whether the HMC memory address passed through the external I/O link is on the HMC memory unit in the address strobing phase. If yes, convert the HMC memory address to the data format of the HMC internal hybrid memory cube address that can be recognized by the internal route; if not, the HMC memory address is forwarded to another HMC memory unit connected to the HMC memory unit. In the data read and write phase, if data read and write occurs on the HMC memory unit, the HMC memory unit is responsible for data conversion between the external I/O link and the internal route. If it does not occur on the HMC memory unit, the HMC memory The unit is responsible for forwarding the data on the two external I/O links connected to it to complete the data transmission.

FIG. 7 is a schematic diagram of capacity expansion of an HMC memory of a neural network computing device based on an HMC memory according to an actual need according to an embodiment of the present disclosure. The logical base layer structure in the HMC memory unit supports attaching the device to a neural network processor or another HMC memory unit. By connecting multiple HMC memory units together, the total capacity of the HMC storage device (memory module) can be increased without changing the structure of the HMC memory unit. Multiple HMC memory units can be connected to other modules 4 through topology connection. As shown in FIG. 7, by connecting two HMC memory units, the memory capacity of the neural network computing device can be doubled. At the same time, this design can cause the neural network computing device to dynamically change the HMC memory according to the needs of the actual application. The number of units enables the HMC memory module to be fully utilized, and at the same time, the configurability of the neural network computing device is greatly improved.

Another embodiment of the present disclosure further discloses a chip including the high bandwidth memory based neural network computing device of the above embodiment.

Another embodiment of the present disclosure also discloses a chip package structure including the above chip.

Another embodiment of the present disclosure also discloses a board that includes the above chip package structure.

Another embodiment of the present disclosure also discloses an electronic device including the above card.

Electronic devices include data processing devices, robots, computers, printers, scanners, tablets, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, cloud servers, cameras, cameras, projectors, watches, headphones, mobile Storage, wearable device vehicles, household appliances, and/or medical devices.

The vehicle includes an airplane, a ship, and/or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, a rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, a range hood; the medical device includes a nuclear magnetic resonance instrument, B-ultrasound and / or electrocardiograph.

The present disclosure also provides a neural network calculation method, including:

Electrically connecting at least one storage device to at least one neural network processor;

Data exchange between the at least one neural network processor and the at least one storage device and performing neural network calculations.

In a specific embodiment of the present disclosure, the neural network calculation method performs neural network calculation based on the neural network computing device of the high bandwidth memory, and referring to FIG. 8, includes:

Step S1: Write the operation parameters calculated by the neural network into the high bandwidth memory.

The high bandwidth memory is connected to an external storage device such as an external disk through an external access unit. The external access unit writes the operation parameters of the external specified address to the high-bandwidth memory, and the operation parameters include the weight, the offset table, and the function table.

Step S2: transmitting the input data calculated by the neural network from the high-bandwidth memory to the buffer of the neural network processor, which may specifically include:

Sub-step S21: the HBM memory control module addresses according to the start address of the input data. If the address hits, the data of the bit width starting from the start address is sequentially transmitted to the HBM through the storage interface according to the high bandwidth provided by the high-bandwidth memory. The memory control module until all input data is transferred to the HBM memory control module. For example, a high-bandwidth memory can provide 1024 bits of bits. Wide, the stored input data has a total of 4096 bits, then the high-bandwidth memory transmits 1024-bit input data to the HBM memory control module each time, and transfers the input data to the HBM memory control module after four transmissions.

Sub-step S22: The HBM memory control module converts the bit width of the input data into a bit width matched with the buffer, and transmits the bit-width matched input data to the buffer.

The input data can be the input neuron vector calculated by the neural network.

Step S3: The operation parameters calculated by the neural network are transmitted from the high-bandwidth memory to the buffer of the neural network processor, and the step may be similar to the step S2, and the step may specifically include:

Sub-step S31: the HBM memory control module addresses according to the start address of the operation parameter. If the address hits, the data of the bit width starting from the start address is sequentially transmitted to the HBM through the storage interface according to the high bandwidth provided by the high-bandwidth memory. The memory control module until all the operating parameters are transferred to the HBM memory control module.

Sub-step S32: The HBM memory control module converts the bit width of the operation parameter into a bit width matched with the buffer, and transmits the operation parameter of the bit width matching to the buffer.

Step S4: The buffer control module transmits the input data and the operation parameters stored in the buffer to the neural processing unit, and the neural processing unit processes the input data and the operation parameters to obtain the output data of the current neural network calculation, and the buffer control module outputs the data. Store to the cache.

Wherein, in the process of processing the input data and the operation parameter by the neural processing unit, if the neural network calculates the existence of the intermediate data, the buffer control module stores the intermediate data in the buffer, and the neural processing unit continues the operation, when the intermediate data is required to be input into the operation. When the buffer control module transmits the intermediate data back to the neural processing unit, the neural processing unit continues the operation using the intermediate data to obtain the output data of the neural network calculation, and the output data may be the output neuron vector.

Step S5: The output data in the buffer is transferred to the high bandwidth memory, and the output data is transmitted to the external storage device through the external access unit. Specifically, the HBM memory control module converts the bit width of the output data into a bit width matched with the high bandwidth memory, and transmits the bit width matched output data to the high bandwidth memory, and the external access unit stores the output data of the high bandwidth memory. Transfer to an external storage device.

At this point, the operation result of the current neural network calculation can be obtained. If the next neural network calculation is continued, the process returns to step S2 to obtain the operation result of the next neural network calculation.

It can be seen that the neural network calculation method of the embodiment, using the high-bandwidth memory of the stacked storage structure and the neural network processor with the HBM memory control module, can greatly improve the storage bandwidth, and the computing performance is greatly improved and improved. Signal transmission bandwidth and transmission speed.

In another specific embodiment of the present disclosure, the neural network calculation method performs neural network calculation based on the neural network computing device of the HMC memory module, and referring to FIG. 9, includes:

Step S1, the external access unit writes data and instructions of the external designated address into the HMC memory module;

Step S2, the neural network processor reads, from the HMC memory module, required data and instructions for performing a partial operation of the neural network;

Step S3, the neural network processor performs a partial operation of the neural network, and writes the intermediate value or the final value obtained by the operation back to the HMC memory module;

Step S4, the operation result is written from the HMC memory module to the external designated address via the external access unit;

In step S5, if there is an operation termination instruction in the operation instruction, the execution is terminated, otherwise, the process returns to step S1.

Further, in the step S1, the method includes the following steps: Step S11, the first data of the external designated address is written into the HMC memory module 1 through the external access unit 3. The first data includes a weight, an offset, a function table, and the like for performing a neural network operation; and in step S12, an externally designated address is written into the HMC memory module 1 through the external access unit 3. The instruction includes an operation instruction for performing a neural network operation at this time. If the operation is not performed after the current neural network operation, the instruction further includes an operation termination instruction; and in step S13, the second data of the external designated address is externally accessed. Unit 3 is written to the HMC memory module 1. The second data includes input data for performing neural network operations at this time.

In the step S2, the neural network processor reads the required data for performing the neural network partial operation from the HMC memory module, including: S21 sets the data preparation state in the neural network processor to be operated to be prepared; S22 The data preparation state is that the prepared neural network processor reads the required data for performing the neural network partial operation from the HMC memory module; after the S23 reading is completed, the data preparation state is the prepared data preparation state of the neural network processor. Set to not prepared.

In the step S3, it is further included that it is determined whether the neural network operation ends, and if it is finished, Going to step S4; otherwise, transmitting a data preparation completion signal to the other neural network processors, and setting the data preparation state in the other neural network processors to be prepared, returning to step S2.

The HMC memory-based neural network operation method, in which a part of the neural network processors in the plurality of neural network processors are in an operational state, and the other neural network processors are in an operation result of waiting for one of the partial neural network processors At the time, the other neural network processor is in a waiting state, after one of the partial neural network processors transmits the operation result to the HMC memory module and sends the data delivery signal to the other neural network processor. The other neural network processors are woken up, the corresponding data is read from the HMC memory module, and corresponding operations are performed.

The processes or methods depicted in the preceding figures may include hardware (eg, circuitry, dedicated logic, etc.), firmware, software (eg, software embodied on a non-transitory computer readable medium), or both The combined processing logic is executed. Although the processes or methods have been described above in some order, it should be understood that certain operations described can be performed in a different order. Moreover, some operations may be performed in parallel rather than sequentially.

In addition, each functional unit in various embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Each functional unit/module may be hardware, such as the hardware may be a circuit, including digital circuits, analog circuits, and the like. Physical implementations of hardware structures include, but are not limited to, physical devices including, but not limited to, transistors, memristors, and the like. The computing modules in the computing device can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, ASIC, and the like.

It should be noted that the implementations that are not shown or described in the drawings or the text of the specification are all known to those of ordinary skill in the art and are not described in detail. In addition, the above definitions of the various elements and methods are not limited to the specific structures, shapes or manners mentioned in the embodiments, and those skilled in the art can simply modify or replace them.

It should also be noted that an example of parameters containing specific values may be provided herein, but these parameters need not be exactly equal to the corresponding values, but may approximate the corresponding values within acceptable error tolerances or design constraints. In addition, the order of the above steps is not limited to the above, and may be varied or rearranged depending on the desired design, unless specifically described or necessarily occurring in sequence. And the above embodiments can be mixed with each other or mixed with other embodiments based on design and reliability considerations. The use of the technical features in different embodiments can be freely combined to form more embodiments.

The specific embodiments of the present invention have been described in detail with reference to the specific embodiments of the present disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and scope of the present disclosure are intended to be included within the scope of the present disclosure.

Claims (37)

A neural network computing device includes: Storage device The neural network processor is electrically connected to the storage device, and the neural network processor exchanges data with the storage device and performs neural network calculation. The neural network computing device of claim 1 wherein said storage device has a stacked structure. The neural network computing device according to claim 1 or 2, wherein The storage device is at least one high bandwidth memory, and each high bandwidth memory includes a plurality of stacked accumulation memories; The neural network processor is electrically connected to the high bandwidth memory, exchanges data between the neural network processor and the high bandwidth memory, and performs neural network calculation. The neural network computing device of claim 3, wherein the neural network processor comprises: a storage interface, an HBM memory control module, a buffer, a buffer control module, and a neural processing unit; The high bandwidth memory and the buffer exchange data through the storage interface and the HBM memory control module, and the HBM memory control module performs clock synchronization and bit width matching on the high bandwidth memory and the buffer; The buffer exchanges data with the neural processing unit through the buffer control module, and the neural processing unit performs neural network calculation. The neural network computing device according to claim 4, wherein The storage interface transmits data of the high bandwidth memory to the HBM memory control module, and transmits data of the HBM memory control module to the high bandwidth memory; The HBM memory control module synchronizes clocks of the high bandwidth memory and the buffer, converts a data bandwidth transmitted by the storage interface into a bandwidth matched with the buffer, and transmits bandwidth matched data to the cache Converting the data bandwidth of the buffer to a bandwidth that matches the high bandwidth memory and transmitting the bandwidth matched data to the storage interface. The neural network computing device of claim 4 or 5, wherein the neural network processor further comprises: And a control unit, configured to send a control instruction to the HBM memory control module, the buffer, the buffer control module, and the neural processing unit to implement a computing function of the neural network processor. The neural network computing device according to any one of claims 3 to 6, further comprising: a package substrate for carrying a high bandwidth memory; The stacked accumulated plurality of memories are a plurality of memories stacked cumulatively in a direction perpendicular to the package substrate. The neural network computing device of claim 7, further comprising: an interposer, and a logic die; The interposer is formed on the package substrate, The logic die and a neural network processor are formed on the interposer, The high bandwidth memory is formed on a logic die. The neural network computing device of claim 7 further comprising: a logic die; The neural network processor is formed on a package substrate, The logic die is formed on a neural network processor, The high bandwidth memory is formed on a logic die. A neural network computing device according to claim 8 or 9, wherein The plurality of memories are stacked and stacked by a micro-bumping process, and the bottommost memory of the high-bandwidth memory is formed on a logic die by a micro-bumping process, and the logic die and the neural network processor are formed by a micro-bumping process On the intermediation layer; The neural network computing device of claim 10 wherein said memory, logic die and interposer have vias opened by a through-silicon via process, and memory is routed through vias via logic die and neural network processor Electrical connection. A neural network computing device according to claim 8 or 9, wherein The plurality of memories are stacked by a micro-bumping process, and the bottommost memory of the high-bandwidth memory is formed on a logic die by a micro-bumping process, and the logic die is formed on the neural network processor by a micro-bumping process The neural network processor is formed on the package substrate by a micro projection welding process. A neural network computing device according to any one of claims 8 to 12, wherein The memory and logic die have vias opened by a through-silicon via process, and the memory passes The vias arranged in the via are electrically connected to the neural network processor via the logic die. The neural network computing device according to any one of claims 2 to 13, wherein the stacked accumulated memories are a plurality of stacked dynamic random access memories (DRAMs). A neural network computing device according to claim 1 or 2, wherein The storage device is an HMC memory module; At least one of the neural network processors is connected to the HMC memory module, and is configured to acquire data and instructions required for a neural network operation from the HMC memory module, perform a neural network partial operation, and write the operation result back to the The HMC memory module. The neural network computing device of claim 15 further comprising The external access unit is connected to the HMC memory module, and reads the neural network commands and data required for the neural network operation from the external designated address, inputs the data into the HMC memory module, and uses the neural network operation result from the HMC memory. The module outputs to an external specified address space. The neural network computing device according to claim 15 or 16, wherein the HMC memory module comprises: a plurality of cascaded HMC memory units, the HMC memory unit having a stacked structure; Each HMC memory unit includes: a mixed memory cube having a memory composed of a plurality of memory bodies, wherein each memory body includes: a plurality of memory die layers, a top layer of the memory die layer The bottom is interconnected by a through silicon via structure to form a three dimensional structure of the die. The neural network computing device according to claim 17, wherein the cascaded plurality of HMC memory units are uniformly addressed, and the cascade manner and implementation of the plurality of HMC memory units are transparent to the neural network processor. The neural network processor reads and writes a memory body location of the HMC memory module by using a memory address of the plurality of HMC memory units that are cascaded. The neural network computing device according to claim 17 or 18, wherein the HMC memory unit further comprises a logic base layer, the logic base layer comprising: a logic control unit for controlling the mixed memory cube for data read and write operations, The logic control unit includes a plurality of memory body controllers, and performs read and write requests for specific locations in the memory block of the HMC memory unit through the memory body controller. The neural network computing device of claim 19, the logical base layer further comprising: a link; The link includes an external I/O link and internal routing and switching logic connected to an HMC memory unit, the external I/O link including a plurality of logical links attached to the switching logic The internal routing is controlled to control the data transfer of each memory bank, and the link is used to connect multiple HMC memory cells to increase the total capacity of the HMC storage device. A chip comprising the neural network computing device of any one of claims 1 to 20. A chip package structure comprising the chip of claim 21. A board comprising the chip package structure of claim 22. An electronic device comprising the card of claim 23. A neural network calculation method includes: Storing data by a storage device; Data exchange is performed by the neural network processor with the storage device and neural network calculations are performed. The neural network calculation method according to claim 25, wherein said storage means adopts a stacked structure. The neural network calculation method according to claim 26, wherein the data exchange between the neural network processor and the storage device and the execution of the neural network calculation comprises: Transmitting the input data calculated by the neural network from the storage device to the buffer of the neural network processor; wherein the storage device is at least one high bandwidth memory; Transmitting the operational parameters calculated by the neural network from the high bandwidth memory to the buffer of the neural network processor; Transmitting the input data and operation parameters stored in the buffer to the neural processing unit, the neural processing unit processes the input data and the operation parameters, and obtains the output data calculated by the neural network and stores the output data to the buffer; Transfer the output data of the buffer to the high bandwidth memory. The neural network calculation method according to claim 27, wherein the transmitting the input data of the current neural network calculation from the high bandwidth memory to the buffer of the neural network processor comprises: The input data is sequentially transmitted to the HBM through the storage interface according to the bit width of the high bandwidth memory. Store the control module until all input data is transferred to the HBM memory control module; The HBM memory control module converts the bit width of the input data into a bit width that matches the buffer and transmits the bit width matched input data to the buffer. The neural network calculation method according to claim 27 or 28, wherein the transmission of the operation parameters of the current neural network calculation from the high bandwidth memory to the buffer of the neural network processor comprises: The operation parameters are sequentially transmitted to the HBM memory control module via the storage interface according to the bit width of the high bandwidth memory until all the operation parameters are transmitted to the HBM memory control module; The HBM memory control module converts the bit width of the operation parameter into a bit width that matches the buffer, and transmits the bit width matching operation parameter to the buffer. The neural network calculation method according to any one of claims 27 to 29, wherein transmitting the output data of the buffer to the high bandwidth memory comprises: The HBM memory control module converts the bit width of the buffer output data to a bit width that matches the high bandwidth memory; The bit width matched output data is transferred to the high bandwidth memory. The neural network calculation method according to any one of claims 26 to 30, wherein Before transmitting the input data calculated by the neural network from the high bandwidth memory to the buffer of the neural network processor, the operating parameter calculated by the neural network is written into the high bandwidth memory; After the output data of the buffer is transferred to the high bandwidth memory, the method further includes: transmitting the output data to the external storage device through the external access unit. The neural network calculation method according to any one of claims 26 to 31, wherein the data exchange between the neural network processor and the storage device and the execution of the neural network calculation comprises: The external access unit writes data and instructions of the external designated address to the storage device; wherein the storage device is an HMC memory module; At least one of the neural network processors reads data and instructions required for performing a neural network partial operation from the HMC memory module; The neural network processor performs a neural network partial operation, and writes the calculated intermediate value or final value back to the HMC memory module. The neural network calculation method according to claim 32, wherein said neural network Data exchange between the processor and the storage device, and performing neural network calculations further includes: The operation result is written from the HMC memory module to the external designated address via the external access unit; If there is an operation termination instruction in the operation instruction, the execution is terminated. Otherwise, the external access unit returns to the step of writing the data and instruction of the external designated address into the HMC memory module. The neural network calculation method according to claim 32 or 33, wherein the neural network processor reads the required data for performing the neural network partial operation from the HMC memory module, including: Setting the data preparation state in the neural network processor to be operated to be prepared; The data preparation state is that the prepared neural network processor reads the required data for performing the neural network partial operation from the HMC memory module; After the reading is completed, the data preparation state of the prepared neural network processor is set to unprepared. The neural network calculation method according to any one of claims 32 to 34, wherein the neural network processor performs a neural network partial operation, and writes the calculated intermediate value or final value back to the HMC memory module. Also includes: Determine whether the neural network operation ends. If it is finished, write the operation result from the HMC memory module to the external designated address via the external access unit; otherwise, send the data preparation completion signal to other neural network processors, and other neural network processors The data preparation status in is set to ready, returning to the data reading step. The neural network calculation method according to claim 35, wherein a part of the neural network processors in the plurality of neural network processors are in an operational state, and the other neural network processors are in one of waiting for the partial neural network processors The operation result of the other neural network processor is first in a waiting state, and one of the partial neural network processors transmits the operation result to the HMC memory module and sends a data delivery signal to the other neural network processor. Thereafter, the other neural network processors are woken up, the corresponding data is read from the HMC memory module, and corresponding operations are performed. The neural network calculation method according to any one of claims 32 to 36, wherein the data includes a weight, an offset, and a function table for performing a neural network operation.

Images