CN115827221A - BAM file parallel reading method, system and medium - Google Patents

Abstract

The invention discloses a method, a system and a medium for reading BAM files in parallel, which sequentially read compressed data blocks from a BAM file to be analyzed, group the compressed data blocks according to a preset rule and generate a queue to be decompressed consisting of grouped compressed blocks; the reading thread takes the packet compression block as a unit and parallelly acquires data from the queue to be decompressed; the reading thread decompresses the packet compression block and analyzes the corresponding BAM data until the data analysis of the whole packet compression block is completed to generate an analysis data block; the method and the device have the advantages that the decompression and analysis work of the grouped data blocks is processed simultaneously through the multiple threads, the reading efficiency of the whole BAM file is improved, the read compressed data blocks are grouped according to the preset rule, the size of data in a decompression queue is adjusted, the thread scheduling time ratio is reduced, and the parallel efficiency is improved.

Description

A BAM file parallel reading method, system and medium

technical field

The invention relates to the field of biological information, in particular to a BAM file parallel reading method, system and medium for realizing the reading of BAM files.

Background technique

In bioinformatics, especially in sequencing data analysis, the SAM (Sequence Alignment/Map) format is commonly used to record the data format of the alignment (mapping) results of short fragment sequences and reference sequences. The file size of SAM format is usually It is very inconvenient to store more than 100G, so the corresponding BAM (Binary SAM) format came into being. The BAM format is a binary compression format of the SAM format. It retains exactly the same content information as the SAM format. The size is usually reduced by more than 4 times. The BAM format has improved the storage problem of the SAM format to a certain extent, but the size of the BAM file is still large, and the reading efficiency of the BAM file is still a very important link in the gene sequencing work.

At present, the industry generally uses htslib or samtools library to read BAM files. The implementation process is shown in Figure 1, including the following steps:

Step A: read the compressed data block from the BAM file, and pass it to the decompression queue;

Step B: multiple decompression threads obtain the compressed data from the decompression queue in parallel for decompression, and transfer the decompressed data block to the decompression data queue;

Step C: sequentially acquire the decompressed data blocks in the decompressed data queue, and analyze them.

In the above-mentioned prior art, there are the following problems for the BAM file reading process:

1. At present, only data decompression is processed in parallel, and data reading and data analysis are executed sequentially, resulting in low efficiency;

2. In a standard BAM file, the size of the data block is about 65KB. The compressed data block read each time is too small, which makes the decompression thread scheduling time too much. Although it is parallel processing, the parallel efficiency is reduced;

3. The operations of reading and parsing data blocks from the decompression queue are performed sequentially. Since a large number of other tasks are required after parsing the data, such as compression, calculation acceleration, etc., subsequent tasks will be stuck in the data parsing step.

To sum up, in the field of biological information with a huge amount of data, the efficiency of the existing BAM file reading method needs to be improved.

Contents of the invention

Technical problem to be solved by the present invention: Aiming at the above-mentioned problems in the prior art, a method, system and medium for parallel reading of BAM files are provided. The present invention better parallelizes the reading process of BAM files and improves the efficiency of reading BAM files.

In order to solve the problems of the technologies described above, the technical solution adopted in the present invention is:

A kind of BAM file parallel reading method, implementation step comprises:

1) sequentially read the compressed data blocks from the BAM file to be parsed, group the above compressed data blocks according to preset rules, and generate a queue to be decompressed composed of grouped compressed blocks;

2) Multiple reading threads obtain data from the queue to be decompressed in parallel in units of compressed blocks;

3) The reading thread decompresses the obtained packet compression block, and parses the corresponding BAM data, until the data analysis of the entire packet compression block is completed to generate a corresponding analysis data block;

4) Merge and analyze the data blocks to complete the reading of the BAM file.

Optionally, the preset rule in step 1) is setting the number of compressed data blocks.

Optionally, the preset rule in step 1) is to preset the size of the packet compression block, and fill the compressed data block into the packet compression block for grouping.

Optionally, the size of the preset packet compression block is dynamically adjusted according to the performance of the processing device.

The present invention also provides a BAM file parallel reading system, including computer equipment, the computer equipment is programmed or configured to perform the steps of the above-mentioned BAM file parallel reading method, or stored on the memory of the computer equipment is programmed or configured to A computer program for executing the above BAM file parallel reading method.

The present invention also provides a computer-readable storage medium, on which a computer program programmed or configured to execute the above-mentioned method for reading BAM files in parallel is stored.

Compared with the prior art, the present invention has the following advantages: the present invention puts the data decompression and data parsing processing in the same reading thread, and multiple threads simultaneously process the decompression and parsing of the grouped data blocks, improving Improve the efficiency of reading the entire BAM file. At the same time, the invention reasonably adjusts the size of data in the decompression queue by grouping the read compressed data blocks according to preset rules, reduces the proportion of thread scheduling time, and improves the parallel efficiency of reading threads.

Description of drawings

FIG. 1 is a schematic flow chart of reading a BAM file in the prior art.

FIG. 2 is a schematic flow chart of parallel reading of BAM files in an embodiment of the invention.

Detailed ways

As shown in Figure 2, the implementation steps of the BAM file parallel reading method of the present embodiment include:

1) sequentially read the compressed data blocks from the BAM file to be parsed, group the above compressed data blocks according to preset rules, and generate a queue to be decompressed composed of grouped compressed blocks;

2) Multiple reading threads obtain data from the queue to be decompressed in parallel in units of compressed blocks;

3) The reading thread decompresses the obtained packet compression block, and parses the corresponding BAM data, until the data analysis of the entire packet compression block is completed to generate a corresponding analysis data block;

4) Merge and analyze the data blocks to complete the reading of the BAM file.

In this embodiment, the data decompression and data parsing are placed in the same reading thread, and multiple threads simultaneously process the decompression and parsing of the grouped data blocks, thereby improving the efficiency of reading the entire BAM file. At the same time, this embodiment reasonably adjusts the size of data in the decompression queue by grouping the read compressed data blocks according to preset rules, reduces the proportion of thread scheduling time, and improves the parallel efficiency of reading threads.

The preset rule in step 1) in this embodiment may be grouping according to the number of the compressed data blocks, for example, grouping every 100 compressed data blocks into groups.

The preset rule in step 1) of this embodiment may also be the size of the preset packet compression block, and the compressed data block is filled into the packet compression block for grouping, for example, the preset packet compression block size is 200MB.

The size of the preset packet compression block in step 1) of this embodiment is dynamically adjusted according to the performance of the processing device, for example, the size of the packet compression block can be adjusted according to the CPU performance.

In addition, this embodiment also provides a BAM file parallel reading system, including a computer device, the computer device is programmed or configured to execute the steps of the above-mentioned BAM file parallel reading method, or the memory of the computer device stores the programmed Or a computer program configured to perform the above BAM file parallel reading method.

In addition, this embodiment also provides a computer-readable storage medium, on which a computer program programmed or configured to execute the above method for reading BAM files in parallel is stored.

The above descriptions are only preferred implementations of the present invention, and the protection scope of the present invention is not limited to the above-mentioned embodiments, and all technical solutions under the idea of the present invention belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, some improvements and modifications without departing from the principles of the present invention should also be regarded as the protection scope of the present invention.

Claims (6)

1. A BAM file parallel reading method is characterized by comprising the following implementation steps:

1) Sequentially reading compressed data blocks from a BAM file to be analyzed, grouping the compressed data blocks according to a preset rule, and generating a queue to be decompressed consisting of grouped compressed blocks;

2) A plurality of reading threads take a packet compression block as a unit and acquire data from a queue to be decompressed in parallel;

3) The reading thread decompresses the obtained packet compression block and analyzes the corresponding BAM data until the data of the whole packet compression block is analyzed to generate a corresponding analysis data block;

4) And merging and analyzing the data blocks to finish reading the BAM file.

2. The method for reading BAM files in parallel as claimed in claim 1, wherein the predetermined rule of step 1) is to set the number of the compressed data blocks.

3. The method of claim 1, wherein the predetermined rule in step 1) is a predetermined size of a packet compression block, and the compressed data block is filled into the packet compression block for grouping.

4. The BAM file parallel reading method according to claim 3, wherein the size of the preset packet compression block is dynamically adjusted according to the performance of a processing device.

5. A BAM file parallel reading system comprising a computer device, characterized in that the computer device is programmed or configured to perform the steps of the BAM file parallel reading method of any one of claims 1 to 4, or that a memory of the computer device has stored thereon a computer program programmed or configured to perform the BAM file parallel reading method of any one of claims 1 to 4.

6. A computer-readable storage medium having stored thereon a computer program programmed or configured to perform the BAM file parallel reading method of any one of claims 1 to 4.

Images