CN112162956B - A skip-reading pre-reading method, device, equipment and storage medium - Google Patents
A skip-reading pre-reading method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112162956B CN112162956B CN202010954943.3A CN202010954943A CN112162956B CN 112162956 B CN112162956 B CN 112162956B CN 202010954943 A CN202010954943 A CN 202010954943A CN 112162956 B CN112162956 B CN 112162956B
- Authority
- CN
- China
- Prior art keywords
- reading
- read
- data
- skip
- length
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明公开了一种跳读预读方法、装置、设备及存储介质，该方法包括：接收读请求，如果本地已经存储有所述读请求对应数据，则由本地读取所述读请求对应数据，否则，由目标文件中读取所述读请求对应数据；在读取所述读请求对应数据后，判断当前的读模式是否为跳读模式，如果是，则基于所述跳读模式预测当前时刻之后需读取数据的预读数据位置及预读数据长度，由所述目标文件中预读所述预读数据位置及所述预读数据长度对应的数据，并将预读的数据存储在本地，否则，确定无需进行预读操作；预读操作为从所述目标文件中预读相应数据的操作。因此本申请在以跳读模式实现读操作的场景中，能够通过数据预读的方式一定程序上提高数据读取速度，提升读性能。
The present invention discloses a skip read pre-reading method, device, equipment and storage medium, the method comprising: receiving a read request, if the data corresponding to the read request has been stored locally, then reading the data corresponding to the read request locally, otherwise, reading the data corresponding to the read request from a target file; after reading the data corresponding to the read request, determining whether the current read mode is a skip read mode, if so, predicting the pre-read data position and pre-read data length of the data to be read after the current moment based on the skip read mode, pre-reading the data corresponding to the pre-read data position and the pre-read data length from the target file, and storing the pre-read data locally, otherwise, determining that no pre-reading operation is required; the pre-reading operation is an operation of pre-reading corresponding data from the target file. Therefore, in the scenario where the read operation is implemented in the skip read mode, the present application can improve the data reading speed and the reading performance to a certain extent by pre-reading data.
Description
Technical Field
The present invention relates to the field of distributed file system technologies, and in particular, to a skip pre-reading method, device, apparatus, and storage medium.
Background
For a distributed file system (object storage), files are generally segmented according to blocks with a certain size, index information is stored in fixed positions of different blocks, when the files are accessed, a storage server of the distributed file system generally accesses the files to read required data after receiving a read request, and then the required data is returned to a sending end of the read request, but the inventor finds that the speed of acquiring the corresponding data is slower and the reading performance is poorer because the files are required to be accessed each time the corresponding data is acquired in the mode.
Disclosure of Invention
The invention aims to provide a skip pre-reading method, a skip pre-reading device, skip pre-reading equipment and a storage medium, wherein in a scene of realizing read operation in a skip mode, the data reading speed can be improved in a certain program in a data pre-reading mode, so that the reading performance is improved.
In order to achieve the above object, the present invention provides the following technical solutions:
A skip pre-reading method, comprising:
receiving a read request, if the read request corresponding data is stored locally, reading the read request corresponding data locally, otherwise, reading the read request corresponding data from a target file;
After the data corresponding to the read request is read, judging whether the current read mode is a skip mode, if so, predicting the position and the length of the read data to be read after the current moment based on the skip mode, and pre-reading the data corresponding to the position and the length of the read data in the target file, and storing the pre-read data in a local area, otherwise, determining that the pre-read operation is not needed, wherein the pre-read operation is the operation of pre-reading the corresponding data from the target file.
Preferably, determining whether the current read mode is a skip mode includes:
If the data length and the step length of the data read by the current read operation and the last read operation are the same, determining that the current read mode is a skip read mode, wherein the step length of any read operation is the data length between the read starting position of any read operation and the read ending position of the last read operation of any read operation, and the read operation is the operation of reading the data corresponding to any read request after receiving any read request.
Preferably, predicting the pre-read data position and the pre-read data length of the data to be read after the current time based on the skip mode includes:
If the first pre-reading operation is needed to be performed currently, determining that the pre-reading data position is the position after the reading end position of the last reading operation is shifted by the step length, wherein the pre-reading data length is the data length of the last reading operation;
If the n-th pre-reading operation in the plurality of continuous pre-reading operations is needed to be performed currently, determining that the pre-reading data positions are respectively shifted by 1 to n times from the reading end position of the last pre-reading operation by the step length, wherein the pre-reading data length corresponding to each pre-reading data position is the data length of the 1-th pre-reading operation in the plurality of continuous pre-reading operations.
Preferably, after predicting the position of the read-ahead data to be read after the current time based on the skip mode, the method further includes:
If the object used for realizing the storage of the target file is larger than the period of the skip mode, reading the object where each pre-read data position is located in a whole object reading mode, and storing the pre-read data locally, otherwise, executing the step of pre-reading the pre-read data position and the data corresponding to the pre-read data length in the target file.
Preferably, the method further comprises:
After each pre-reading operation and reading operation, updating the skip record information recorded in the skip record structure, wherein the skip record information comprises a step length, a data length and a read end position.
Preferably, before updating the skip record information recorded in the skip record structure, the method further comprises:
Opening the target file, creating a handle, and creating the skip record structure in the handle;
The method further comprises the steps of:
And closing the target file and the handle, and clearing the skip record structure.
Preferably, after the data corresponding to the read request is read locally, the method further includes:
And clearing the data corresponding to the read request stored locally.
A skip pre-reading device, comprising:
The reading module is used for receiving a reading request, if the reading request corresponding data is stored locally, the reading request corresponding data is read locally, otherwise, the reading request corresponding data is read from the target file;
The first pre-reading module is used for judging whether the current reading mode is a skip reading mode after reading the data corresponding to the reading request, if so, predicting the pre-reading data position and the pre-reading data length of the data to be read after the current moment based on the skip reading mode, pre-reading the data corresponding to the pre-reading data position and the pre-reading data length from the target file, and storing the pre-read data locally, otherwise, determining that the pre-reading operation is not needed, wherein the pre-reading operation is the operation of pre-reading the corresponding data from the target file.
A skip pre-read device, comprising:
a memory for storing a computer program;
A processor for implementing the steps of the skip pre-read method as described in any of the preceding claims when executing said computer program.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the skip pre-read method as described in any of the preceding claims.
The application provides a skip read pre-reading method, device, equipment and storage medium, wherein the method comprises the steps of receiving a read request, if the read request corresponding data is stored locally, locally reading the read request corresponding data, otherwise, reading the read request corresponding data from a target file, judging whether a current read mode is a skip read mode after reading the read request corresponding data, if so, predicting the pre-reading data position and the pre-reading data length of the data to be read after the current moment based on the skip read mode, pre-reading the pre-reading data position and the pre-reading data length corresponding data in the target file, and storing the pre-reading data locally, otherwise, determining that pre-reading operation is not needed, wherein the pre-reading operation is the operation of pre-reading the corresponding data from the target file. According to the application, after the corresponding data of the read request is read each time, whether the current read mode is the skip mode is judged, and when the judgment result is yes, the corresponding data is read from the target file based on the skip mode and then stored locally, so that when the corresponding read request of the data is received subsequently, the corresponding data can be read locally, and the data reading speed can be greatly improved compared with the data reading realized by the access file through the local realization of the data reading.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a skip pre-reading method according to an embodiment of the present invention;
fig. 2 is a general flow diagram of a skip pre-reading method according to an embodiment of the present invention in a specific application scenario;
FIG. 3 is a diagram illustrating a skip mode identification in a skip pre-reading method according to an embodiment of the present invention;
fig. 4 is an exemplary diagram of read-ahead initiation in a skip-read-ahead method according to an embodiment of the present invention;
Fig. 5 is a flowchart of a skip pre-reading method according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, a flowchart of a skip pre-reading method according to an embodiment of the present invention may include:
And S11, receiving a read request, if the read request corresponding data is stored locally, reading the read request corresponding data locally, otherwise, reading the read request corresponding data from the target file.
The skip pre-reading method provided by the embodiment of the application can be applied to a distributed file system, the target file is any file such as video, streaming media and the like which need to be accessed in the distributed file system, the execution main body of the skip pre-reading method can be a corresponding skip pre-reading device, and the skip pre-reading device can be arranged in a storage server of the distributed file system, so that the execution main body of the skip pre-reading method can be the storage server.
The storage server receives the read request, if the data corresponding to the read request is already stored locally, the data corresponding to the read request can be read locally, and the data corresponding to the read request is returned to the sending end of the read request, otherwise, the data corresponding to the read request is read from the target file, and the data corresponding to the read request is returned to the sending end of the read request. The method comprises the steps of receiving a storage server, wherein the storage server receives a reading request from the storage server, and the reading request comprises an identifier of data to be read, so that corresponding data can be positioned based on the identifier, and further data reading is achieved.
And S12, after the corresponding data of the read request is read, judging whether the current read mode is a skip mode, if so, executing the step S13, and if not, executing the step S14.
The read mode is a mode for reading data, and can be realized based on the read operation characteristic of the skip mode when judging whether the current mode is the skip mode, can be realized based on a preset mark, and can be realized in other modes set according to actual needs. Specifically, when the read operation characteristic based on the skip mode is implemented, since the read operation of the skip mode is generally the same in length of data read each time and the step length is the same when the data is read each time, whether the corresponding data is read in the skip mode or not can be judged based on the read operation and the last read operation, wherein the step length of any read operation is the data length between the start position of the data read by any read operation in the target file and the end position of the data read by the last read operation in the target file, whether the read operation is required to be implemented in the skip mode or not can be designated in advance when the read operation is implemented based on the preset mark, and the corresponding mark is set, so that whether the read operation is required to be implemented in the skip mode or not can be known based on the mark.
S13, predicting the pre-read data position and the pre-read data length of the data to be read after the current moment based on the skip mode, pre-reading the data corresponding to the pre-read data position and the pre-read data length in the target file, and storing the pre-read data locally.
If the current skip mode is determined, the data length and the step length of each time of data reading in the skip mode are the same, so that the position and the length of the data which needs to be read in the next time or even in the subsequent time can be determined based on the same, the corresponding data are pre-read based on the same and then stored locally, and after the corresponding read requests of the data are received, the corresponding data of the read requests can be directly read locally and returned, and compared with the data read from the target file, the speed of data reading can be greatly improved.
And S14, determining that the pre-reading operation is not needed, wherein the pre-reading operation is an operation of pre-reading corresponding data from the target file.
In the embodiment of the application, the reading and the pre-reading refer to different actions, the reading is the reading of the data corresponding to the read request from the local or target file, the pre-reading is the data to be read after the predicted current moment from the target file, the corresponding reading operation is the operation of reading the corresponding data, namely the operation of reading the data corresponding to the read request from the local or target file, and the pre-reading operation is the operation of pre-reading the corresponding data, namely the operation of pre-reading the data to be read after the predicted current moment from the target file.
According to the application, after the corresponding data of the read request is read each time, whether the current read mode is the skip mode is judged, and when the judgment result is yes, the corresponding data is read from the target file based on the skip mode and then stored locally, so that when the corresponding read request of the data is received subsequently, the corresponding data can be read locally, and the data reading speed can be greatly improved compared with the data reading realized by the access file through the local realization of the data reading.
The skip pre-reading method provided by the embodiment of the invention judges whether the current reading mode is a skip mode or not, and can comprise the following steps:
If the data length and the step length of the data read by the current read operation and the last read operation are the same, determining that the current read mode is a skip read mode, wherein the step length of any read operation is the data length between the read starting position of any read operation and the read ending position of the last read operation of any read operation, and the read operation is the operation of reading the data corresponding to any read request after receiving any read request.
When determining whether the read operation performed at the current time and within a period of time before the current time is the read operation of the skip mode, whether the current read mode is the skip mode can be determined based on whether the data length and the step length of the current read operation and the data length and the step length of the last read operation are the same, so that the judgment of whether the current read mode is the skip mode is simply, conveniently and effectively realized through the characteristics of the two read operations. The read start position is the start position of the read data or the pre-read data in the target file, and the read end position is the end position of the read data or the pre-read data in the target file.
The skip pre-reading method provided by the embodiment of the invention predicts the pre-reading data position and the pre-reading data length of the data to be read after the current moment based on the skip mode, and comprises the following steps:
if the first pre-reading operation is needed to be performed currently, determining that the pre-reading data position is the position after the reading end position of the last reading operation is shifted by the step length, wherein the pre-reading data length is the data length of the last reading operation;
If the n-th pre-reading operation in the continuous pre-reading operations is needed to be performed currently, determining that the pre-reading data positions are respectively shifted by 1 to n times of step length from the reading end position of the last pre-reading operation, wherein the pre-reading data length corresponding to each pre-reading data position is the data length of the 1 st pre-reading operation in the continuous pre-reading operations.
If the current skip mode is judged for the first time, in order to avoid invalid read data (namely, the pre-read data is not the data which needs to be read by the sending end of the read request) caused by the change of the read operation in the later period, the data in the period of one skip mode can be pre-read first in the application, and if the current skip mode is judged for a plurality of times continuously, the duration time of the skip mode is longer, the possibility of the change of the later period read operation is smaller, so that the application can pre-read the data in the period of a plurality of skip modes, and therefore, under the condition that the possibility of the later period still being in the skip mode is larger, more pre-read data are provided for reading, and the high hit rate of the pre-read data is ensured to a certain extent when the pre-reading is realized.
The method comprises the steps of reading data in a skip mode, wherein one cycle of the skip mode is the sum of the length and the step length of the data read by a certain read operation, the position of the pre-read data is the initial position of the data to be pre-read in a target file, the length of the pre-read data is the length of the data to be pre-read in the target file, n is a positive integer which is not 1, so that when the later stage is still in the skip mode, the data in a plurality of cycles possibly read in the future are read, namely, the position of the pre-read data is the position of the last pre-read data after one step length is shifted backwards (backward is the direction from the initial position to the end position of the data), the position of the pre-read data after two step lengths is shifted backwards, and the position of the pre-read data after n step lengths is shifted backwards, and the data corresponding to the pre-read data length is read at each pre-read data position, so that the effective pre-reading of the data is realized.
In addition, the first pre-reading operation may be a first pre-reading operation which is not performed with any pre-reading operation before the current moment, or may be a first pre-reading operation when there is a reading operation between the last pre-reading operation and the current pre-reading operation when the pre-reading operation is not continuously performed, and of course, other defining modes of the first pre-reading operation meeting the requirements may be set according to the actual needs, which are all within the protection scope of the present invention.
The skip pre-reading method provided by the embodiment of the invention can further comprise the following steps after predicting the pre-reading data position of the data to be read after the current moment based on the skip mode:
If the object used for realizing the object file storage is larger than the period of the skip mode, the object where each pre-read data position is located is pre-read in a whole object reading mode, and the pre-read data is stored locally, otherwise, the step of pre-reading the pre-read data position and the data corresponding to the pre-read data length in the object file is executed.
The distributed file system can be specifically realized in an object storage mode, the target file is stored in an object storage mode, if the size of the object is larger than the period of the skip mode, the object needs to be accessed for multiple times, so that in order to reduce the access times of the object, in the embodiment of the application, the object where the data needing to be pre-read is located, namely, the whole object where the data needing to be pre-read is located, can be read in a whole object mode, and when the size of the object is not larger than the period of the skip mode, the pre-reading of the data can be realized in the mode of the embodiment. Therefore, for skip dense requests (a plurality of small blocks of objects skip), the pre-reading can avoid the pressure of small blocks IO to a network disk, thereby reducing the pressure of a read scene network and the disk, further reducing the access times of access objects and improving the reading performance.
The skip pre-reading method provided by the embodiment of the invention can further comprise the following steps:
after each pre-reading operation and reading operation, updating the skip record information recorded in the skip record structure, wherein the skip record information comprises a step length, a data length and a read end position.
It should be noted that, in order to facilitate obtaining corresponding information quickly when judging whether the read operation is in the skip mode, determining the position of the pre-read data, the length of the pre-read data, and the like, the application may be provided with a skip record structure in advance, so that the step length, the read end position, the data length, and other information of each pre-read operation or read operation are recorded in the skip record structure, thereby facilitating quick implementation of corresponding steps.
The skip pre-reading method provided by the embodiment of the invention can further comprise the following steps before the skip record information recorded in the skip record structure is updated:
opening the target file, creating a handle, and creating a skip record structure in the handle;
The skip pre-reading method can further comprise the following steps:
closing the target file and the handle, and clearing the skip record structure.
It should be noted that, the creating of the skip record structure may specifically be creating a handle corresponding to the target file when the target file is opened, so as to create the skip record structure in the handle, thereby facilitating storage and inquiry of the skip record structure. Where a Handle (Handle) is an identifier that is used to identify an object or item, it may be used to describe a form, file, etc., it is noted that the Handle cannot be constant.
The skip pre-reading method provided by the embodiment of the invention can further comprise the following steps after the data corresponding to the read request is read locally:
and clearing the data corresponding to the read request stored locally.
It should be noted that, in order to ensure that the local storage server can have sufficient storage space, and realize the storage of the data that is not accessed, in the embodiment of the present application, after the local data of the storage server is read each time, the accessed data is cleared, so as to realize the cyclic and effective use of the local storage space.
In a specific application scenario, the general steps of a skip pre-reading method provided in the embodiment of the present invention may be shown in fig. 2, and specifically include:
1. Creating a handle after opening (open) the target file;
2. When a handle is created, creating a skip record structure and recording the current skip condition, wherein the skip record structure is stored in the handle;
3. updating skip record information in a skip record structure when each reading operation is performed, identifying a skip mode based on the skip record information, and calculating a skip data position so as to realize pre-reading;
4. and closing the handle when the file is closed, and simultaneously clearing the skip record structure.
An example diagram when implementing skip mode identification may be as shown in fig. 3 (where skip information is skip record information, and a pre-read position is a pre-read data position), which may specifically include:
1. the skip record information mainly includes skip_len (step size), last_read (read end position of last read operation), last_read_len (data length of data read by last read operation or pre-read operation, which may be simply referred to as read length), skip_num (skip number), readadhead _pos (read end position of last pre-read operation, i.e. end position of last pre-read operation);
2. after the first reading request comes, updating the current reading length of the last_read_len record;
3. after the second reading request comes, recording skip_len and last_read_len;
4. after the third reading request comes, if the skip_len and last_read_len are found to be the same as the last time, the current reading mode is considered to be a skip mode, and the skip_num count is updated to be 1;
5. When the skip_num count is greater than 1, the current read mode is considered to be the skip mode, the pre-read data position is calculated, and pre-reading is initiated.
An exemplary diagram when implementing pre-reading of data may be as shown in fig. 4, and may specifically include:
1. After determining a skip mode, firstly, preliminarily calculating a pre-reading data position and a pre-reading data length, recording new pre-reading information (comprising readadhead _pos), further determining that the pre-reading data position is a reading end position (readahead _pos) of the last pre-reading operation, if the pre-reading data position is the first pre-reading operation, the pre-reading length is a reading length (last_ readahead _len) of the last pre-reading operation, if the pre-reading data position is the continuous pre-reading operation, the pre-reading data length is doubled by readahead _len each time (the pre-reading data length of the last pre-reading operation), the maximum length is 200M, the minimum length is 1 cycle (skip_len+len, len is the reading data length of the single read operation), and the pre-reading data position is aligned to the cycle;
2. the prereading position is refined, and two different prereading IO are adopted according to the cycle of the skip mode and the size of an object (obj):
1) The period is less than the obj size, more than two read requests are initiated by each object on average, and the whole object initiates pre-reading at the moment, namely the pre-reading data position preliminarily calculated in the step 1 is the pre-reading data position to be initiated at the time;
2) And (3) the cycle is greater than the obj size, each object initiates less than 1 read request on average, a plurality of skip read prereads are adopted at the moment, and the position to be actually read is selected according to the step length and the read length according to the preread data position preliminarily calculated in the step (1), so that the preread position is split into a plurality of preread skip fragments.
3. And initiating pre-reading according to the calculated pre-reading data position.
The application records the characteristic of the read operation (namely IO characteristic) to determine the skip mode, and simultaneously carries out whole object pre-reading or skip pre-reading according to the skip period and the size of the bottom object, thereby improving the pre-reading and reading performance and reducing the network disk consumption for small-step skip scenes.
The embodiment of the invention also provides a skip pre-reading device, as shown in fig. 5, which may include:
a reading module 11, configured to receive a read request, if the read request corresponding data is already stored locally, read the read request corresponding data locally, otherwise, read the read request corresponding data from the target file;
The first pre-reading module 12 is configured to determine whether the current reading mode is a skip reading mode after reading the data corresponding to the reading request, if so, predict a pre-reading data position and a pre-reading data length of the data to be read after the current time based on the skip reading mode, pre-read the data corresponding to the pre-reading data position and the pre-reading data length in the target file, and store the pre-read data locally, otherwise, determine that the pre-reading operation is not needed, where the pre-reading operation is an operation of pre-reading the corresponding data from the target file.
The skip pre-reading device provided by the embodiment of the invention, the first pre-reading module may include:
the first determining unit is used for determining that the current reading mode is a skip reading mode if the data length and the step length of the data read by the current reading operation and the last reading operation are the same, wherein the step length of any reading operation is the data length between the reading starting position of any reading operation and the reading ending position of the last reading operation of any reading operation, and the reading operation is the operation of reading the data corresponding to any reading request after any reading request is received.
The skip pre-reading device provided by the embodiment of the invention, the first pre-reading module may include:
And the second determining unit is used for determining that the read-out data position is the position after the read-out end position of the last read operation deviates from the step length if the read-out operation is required to be performed at present, determining that the read-out data position is the position after the read-out end position of the last read operation deviates from the step length by 1 to n times if the read-out operation is required to be performed at present for the nth read-out operation in a plurality of continuous read-out operations, and determining that the read-out data length corresponding to each read-out data position is the data length of the 1 st read-out operation in the plurality of continuous read-out operations.
The skip pre-reading device provided by the embodiment of the invention can further comprise:
And the second pre-reading module is used for pre-reading the object where each pre-reading data position is located in a whole object reading mode after predicting the pre-reading data position of the data to be read after the current moment based on the skip mode, if the object used for realizing the storage of the target file is larger than the period of the skip mode, and storing the pre-read data in a local mode, otherwise, executing the step of pre-reading the pre-reading data position and the data corresponding to the pre-reading data length in the target file.
The skip pre-reading device provided by the embodiment of the invention can further comprise:
and the updating module is used for updating the skip record information recorded in the skip record structure after each pre-reading operation and each reading operation, wherein the skip record information comprises a step length, a data length and a reading end position.
The skip pre-reading device provided by the embodiment of the invention can further comprise:
the opening module is used for opening the target file, creating a handle and creating a skip record structure in the handle before updating the skip record information recorded in the skip record structure;
and the closing module is used for closing the target file and the handle and clearing the skip record structure.
The skip pre-reading device provided by the embodiment of the invention can further comprise:
and the clearing module is used for clearing the locally stored read request corresponding data after the locally read request corresponding data.
The embodiment of the invention also provides skip pre-reading equipment, which can comprise:
a memory for storing a computer program;
a processor for implementing the steps of any one of the skip pre-read methods as described above when executing a computer program.
The embodiment of the invention also provides a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and when the computer program is executed by a processor, the steps of the skip pre-reading method can be realized.
It should be noted that, for the explanation of the relevant parts in the skip pre-reading device, the skip pre-reading device and the storage medium provided in the embodiments of the present invention, please refer to the detailed explanation of the corresponding parts in the skip pre-reading method provided in the embodiments of the present invention, and the detailed explanation is omitted here. In addition, the parts of the above technical solutions provided in the embodiments of the present invention, which are consistent with the implementation principles of the corresponding technical solutions in the prior art, are not described in detail, so that redundant descriptions are avoided.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (7)
1. A skip pre-reading method, comprising:
receiving a read request, if the read request corresponding data is stored locally, reading the read request corresponding data locally, otherwise, reading the read request corresponding data from a target file;
After the data corresponding to the read request is read, judging whether the current read mode is a skip mode, if so, predicting the position and the length of the read data to be read after the current moment based on the skip mode, and pre-reading the data corresponding to the position and the length of the read data in the target file, and storing the pre-read data in a local area, otherwise, determining that the pre-read operation is not needed, wherein the pre-read operation is the operation of pre-reading the corresponding data from the target file;
Determining whether the current reading mode is a skip reading mode or not, wherein the current reading mode is determined to be the skip reading mode if the data length and the step length of the data read by the current reading operation and the last reading operation are the same, the step length of any reading operation is the data length between the reading starting position of any reading operation and the reading ending position of the last reading operation of any reading operation, and the reading operation is the operation of reading the data corresponding to any reading request after any reading request is received;
Predicting a read-ahead data position and a read-ahead data length of data to be read after the current moment based on the skip mode, wherein the read-ahead data position and the read-ahead data length comprise that if the read-ahead operation is required to be performed for the first time, the read-ahead data position is determined to be the position after the step length is shifted by the read-end position of the last read operation, and the read-ahead data length is determined to be the data length of the first read operation in the plurality of continuous read operations, wherein the read-ahead operation is the data length of the first read operation in the plurality of continuous read operations, and the read-ahead operation is performed for the first read operation without any read-ahead operation before the current moment or the read operation is performed for the first read operation without read-ahead operation between the last read operation and the current read operation;
and predicting the pre-read data position of the data to be read after the current moment based on the skip mode, and further comprising the steps of pre-reading the object where each pre-read data position is located in a whole object reading mode if the object stored in the target file is larger than the period of the skip mode, and storing the pre-read data locally, or executing the step of pre-reading the pre-read data position and the data corresponding to the pre-read data length in the target file, wherein one period of the skip mode is the sum value of the data length and the step length read by a certain reading operation.
2. The method as recited in claim 1, further comprising:
After each pre-reading operation and reading operation, updating the skip record information recorded in the skip record structure, wherein the skip record information comprises a step length, a data length and a read end position.
3. The method of claim 2, further comprising, before updating the skip record information recorded in the skip record structure:
Opening the target file, creating a handle, and creating the skip record structure in the handle, wherein the method further comprises:
And closing the target file and the handle, and clearing the skip record structure.
4. A method according to claim 3, further comprising, after the reading of the read request corresponding data locally:
And clearing the data corresponding to the read request stored locally.
5. A skip pre-reading device, comprising:
The reading module is used for receiving a reading request, if the reading request corresponding data is stored locally, the reading request corresponding data is read locally, otherwise, the reading request corresponding data is read from the target file;
The first pre-reading module is used for judging whether the current reading mode is a skip reading mode after reading the data corresponding to the reading request, if so, predicting the pre-reading data position and the pre-reading data length of the data to be read after the current moment based on the skip reading mode, pre-reading the data corresponding to the pre-reading data position and the pre-reading data length from the target file, and storing the pre-read data locally, otherwise, determining that the pre-reading operation is not needed;
Wherein the first pre-reading module comprises:
The first determining unit is used for determining that the current reading mode is a skip reading mode if the data length and the step length of the data read by the current reading operation and the last reading operation are the same, wherein the step length of any reading operation is the data length between the reading starting position of any reading operation and the reading ending position of the last reading operation of any reading operation, and the reading operation is the operation of reading the data corresponding to any reading request after any reading request is received;
The second determining unit is used for determining that the read end position of the read operation is the position after the step length is shifted if the read operation is the first read operation, and the read data length is the data length of the last read operation; if the nth pre-reading operation in the continuous pre-reading operation is needed to be performed at present, determining that the pre-reading data position is respectively the position after the step length is shifted by 1 to n times from the reading end position of the last pre-reading operation, and the pre-reading data length corresponding to each pre-reading data position is the data length of the 1 st pre-reading operation in the continuous pre-reading operation, wherein the first pre-reading operation is the first pre-reading operation which is not performed with any pre-reading operation before the current moment or the first pre-reading operation which is performed with the reading operation between the last pre-reading operation and the current pre-reading operation when the pre-reading operation is not performed continuously;
The skip pre-reading device further comprises:
and the second pre-reading module is used for pre-reading the object where each pre-reading data position is located in a whole object reading mode after the pre-reading data position of the data to be read after the current moment is predicted based on the skip mode, if the object stored in the target file is larger than the period of the skip mode, and storing the pre-read data locally, otherwise, executing the step of pre-reading the pre-reading data position and the data corresponding to the pre-reading data length in the target file, wherein one period of the skip mode is the sum value of the data length and the step length read by a certain reading operation.
6. A skip pre-read device, comprising:
a memory for storing a computer program;
processor for implementing the steps of the skip pre-read method as claimed in any of claims 1 to 4 when executing said computer program.
7. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the skip pre-read method according to any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010954943.3A CN112162956B (en) | 2020-09-11 | 2020-09-11 | A skip-reading pre-reading method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010954943.3A CN112162956B (en) | 2020-09-11 | 2020-09-11 | A skip-reading pre-reading method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112162956A CN112162956A (en) | 2021-01-01 |
| CN112162956B true CN112162956B (en) | 2025-02-14 |
Family
ID=73858938
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010954943.3A Active CN112162956B (en) | 2020-09-11 | 2020-09-11 | A skip-reading pre-reading method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112162956B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817172A (en) * | 2021-01-18 | 2022-07-29 | 北京金山云网络技术有限公司 | Data reading method and device and electronic equipment |
| CN113626381B (en) * | 2021-06-30 | 2023-12-22 | 济南浪潮数据技术有限公司 | Optimization method and device based on interleaving read-ahead of distributed file system |
| CN114237518B (en) * | 2022-02-22 | 2022-05-24 | 苏州浪潮智能科技有限公司 | A data reading method, system, device and terminal |
| CN114661673A (en) * | 2022-03-30 | 2022-06-24 | 蚂蚁区块链科技(上海)有限公司 | File access method and device |
| CN117093554B (en) * | 2023-08-28 | 2025-05-02 | 济南浪潮数据技术有限公司 | A skip-reading pre-reading method and device based on distributed file system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106570108A (en) * | 2016-11-01 | 2017-04-19 | 中国科学院计算机网络信息中心 | Adaptive reading optimization method and system for mass data under cloud storage environment |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3598139B2 (en) * | 1994-12-28 | 2004-12-08 | 株式会社日立製作所 | Data processing device |
| US6789171B2 (en) * | 2002-05-31 | 2004-09-07 | Veritas Operating Corporation | Computer system implementing a multi-threaded stride prediction read ahead algorithm |
| US7333993B2 (en) * | 2003-11-25 | 2008-02-19 | Network Appliance, Inc. | Adaptive file readahead technique for multiple read streams |
| CN107491545A (en) * | 2017-08-25 | 2017-12-19 | 郑州云海信息技术有限公司 | The catalogue read method and client of a kind of distributed memory system |
| CN108874310A (en) * | 2018-05-28 | 2018-11-23 | 郑州云海信息技术有限公司 | The read buffer method, apparatus and computer readable storage medium of distributed storage |
| CN108959519A (en) * | 2018-06-28 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of method, apparatus and computer readable storage medium reading data |
| CN109947720A (en) * | 2019-04-12 | 2019-06-28 | 苏州浪潮智能科技有限公司 | A file pre-reading method, apparatus, device and readable storage medium |
-
2020
- 2020-09-11 CN CN202010954943.3A patent/CN112162956B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106570108A (en) * | 2016-11-01 | 2017-04-19 | 中国科学院计算机网络信息中心 | Adaptive reading optimization method and system for mass data under cloud storage environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112162956A (en) | 2021-01-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112162956B (en) | A skip-reading pre-reading method, device, equipment and storage medium | |
| US8082388B2 (en) | Optimizing operational requests of logical volumes | |
| US7500063B2 (en) | Method and apparatus for managing a cache memory in a mass-storage system | |
| US8775741B1 (en) | Using temporal access patterns for determining prefetch suitability | |
| US9684590B2 (en) | Storing corresponding data units in a common storage unit | |
| JP5621909B2 (en) | Information processing apparatus, storage system, and write control method | |
| CN100362462C (en) | Management Method of Disk Array Cache | |
| US20070106849A1 (en) | Method and system for adaptive intelligent prefetch | |
| KR20090026296A (en) | Predictive data loader | |
| JP4060506B2 (en) | Disk controller | |
| CN107168651B (en) | A small file aggregate storage processing method | |
| CN111506778B (en) | A storage system cache prefetch method, system and medium based on K-Truss graph | |
| JPH10124396A (en) | Buffer exchanging method | |
| US7000077B2 (en) | Device/host coordinated prefetching storage system | |
| CN106294211B (en) | A kind of detection method and device of multichannel sequential flow | |
| CN116069261A (en) | Data processing method, system, equipment and storage medium | |
| JPWO2013046342A1 (en) | Virtual tape device and control method of virtual tape device | |
| JP4104283B2 (en) | Storage subsystem and information processing system | |
| WO2024146330A1 (en) | Video storage method and video playback method | |
| US8069310B1 (en) | Methods and systems for incorporating sequential stream read requests into prefetch management for data storage having a cache memory | |
| CN1453710A (en) | Two-stage CD mirror server/client cache system | |
| CN116069752A (en) | Mixed prefetching method for distributed file system | |
| US7730260B2 (en) | Delete recycling of holographic data storage | |
| JP3166827B2 (en) | External storage device and cache memory control method | |
| JP3080521B2 (en) | Disk unit |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |