CN106294470B - Method for reading real-time incremental log information based on split log - Google Patents
Method for reading real-time incremental log information based on split log Download PDFInfo
- Publication number
- CN106294470B CN106294470B CN201510297384.2A CN201510297384A CN106294470B CN 106294470 B CN106294470 B CN 106294470B CN 201510297384 A CN201510297384 A CN 201510297384A CN 106294470 B CN106294470 B CN 106294470B
- Authority
- CN
- China
- Prior art keywords
- log
- file
- reading
- incremental
- log file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a real-time incremental reading method based on a segmentation log. The method comprises the following steps: scanning log files in a log directory, and setting a reading initial position of each file; starting a file listener to monitor log file changes in a log directory; when a log file new creation, modification or update event is monitored: judging whether the log file has a split file or not; and if the split file exists, reading incremental log information and setting the reading starting position as the current reading length.
Description
Technical Field
The invention relates to real-time incremental log information reading, in particular to a method for reading real-time incremental log information based on a split log.
Background
In a monitoring system, a log mode is often adopted to collect monitoring data of a monitored system. In order to ensure the accuracy of monitoring data and alarming, the accuracy and robustness of the log incremental content real-time reading scheme are important.
Real-time reading of log delta content involves file listening. File monitoring refers to using a file monitor to monitor changes of log files in real time, including events such as creation, modification and deletion of files. And responding to the monitored event to finish the reading of the incremental log content of the log after the log is cut.
The technical scheme in the prior art is as follows:
the common log file incremental reading scheme is as follows:
1. monitoring the file change condition in the log path, and recording the reading initial position of the file as 0 when a new log file is generated. Reading incremental log content from 0, and modifying a reading start position to be a current reading log length (position ═ length) after reading is finished;
2. when the log file is monitored to be modified, acquiring a value of a reading start position, if not, setting the value to be 0(position is 0), reading the incremental log content from the position, and after the reading is finished, modifying the reading start position to be the position plus a reading incremental log length (position + length);
3. when the file is monitored to be deleted, no processing is carried out.
Disadvantages of the prior art
The scheme is a common file increment reading scheme, the algorithm is intuitive and effective, and the method can be only suitable for reading the log increment content without segmentation.
However, in practice, in order to ensure the controllability of the log size, it is a general solution to write the log file in a manner of splitting the log size. As is well known to those skilled in the art, when the system is running, in order to monitor the running state of the system, log information is output to a log file, and the log file becomes very large for reading and management. Therefore, it is generally desirable that the log is stored in different files according to a certain period of time or file size, which involves log splitting. In order to read the incremental log and control the size of the log quickly, log files are saved in a monitoring system by adopting a round robin segmentation mode, for example. One skilled in the art will appreciate that other ways of splitting may be used.
For example, assume that the system outputs an original log of a.log, a log round size of 10MB, and a round log number of 3. Specifically, the log splitting process may be described as follows:
1. newly creating a log file a.log;
2. when the size of the a.log reaches 10M, the a.log is named as a.log.1, the a.log is newly built, and new log contents are continuously written into the a.log;
3. when the new a.log size reaches 10M, a.log.1 is named as a.log.2, a.log is named as a.log.1, and a.log is newly built;
4. when a.log reached 10M again, repeat step 3;
5. because the number of the rotation logs is set to be 3, when the a.log reaches 10M again after the a.log.3 is generated, the a.log.3 is deleted firstly, then the a.log.2 is named as a.log.3, the a.log.1 is named as a.log.2, the a.log is named as a.log.1, and a new a.log file is continuously written into a new log;
6. and triggering the segmentation event subsequently, and repeating the step 5.
After the log is split, at most, the following 4 log files are reserved:
log (newest), a.log.1 (next to newest), a.log.2, a.log.3 (old)
In this case, if there is an a.log file, when the file is split, the file increment reading scheme is as follows:
1. normally reading the a.log increment, and recording the reading initial position of the file as position;
2. if log splitting occurs at this time, it can be known from the above-described split-by-size process that a.log is renamed to a.log.1 (the file listener will mark the rename as a.log.1 new event and a.log delete event) first, and then a.log file is newly created, then three events occur in the whole process: newly building a.log.1, deleting a.log and newly building a.log;
3. responding to a new event of a.log.1, and reading a.log.1 file increment log from 0; log delete event is responded, and no processing is carried out; and reading the a.log incremental log from 0 in response to the a.log addition event.
From the above steps, we can see that the processing of deleting the a.log and adding the a.log is correct, but the adding event of the a.log.1 responds to reading the content of the whole log file, the increment reading is wrong, and the correct processing is to read the increment log from the position of the record to the end of the file. It can be seen that this scheme is not applicable to the incremental read based rotation log.
Therefore, a method is needed that enables real-time incremental reading based on split logs.
Disclosure of Invention
According to an embodiment of the present invention, a method for real-time incremental reading based on a split log is provided, which includes:
scanning log files in a log directory, and setting a reading initial position of each file;
starting a file listener to monitor log file changes in a log directory;
when a log file new creation, modification or update event is monitored:
judging whether the log file has a split file or not;
and if the split file exists, reading incremental log information and setting the reading starting position as the current reading length.
Preferably, further comprising: when a log file deletion event is monitored:
judging whether the log file exists or not; and
and if the log file does not exist, deleting the position of the file record.
Preferably, the reading the incremental log information further comprises:
acquiring a reading initial position of the log file;
reading incremental log information of the log file;
setting the read start to 0; and
transferring the log file to a backup path.
Preferably, further comprising: and if the split file does not exist, calculating the incremental log information length of the log file.
Preferably, further comprising: if the length of the incremental log information is greater than 0, then:
acquiring the reading starting position;
reading the incremental log information; and
the read start position is set to "the initial read position + the incremental log information length".
Preferably, the incremental log information length is the current file length minus the read start position.
Preferably, if the incremental log information length is not more than 0, the reading start position is set to 0.
Preferably, the sleep is performed for a predetermined time before determining whether the log file exists.
According to another aspect of the present invention, there is provided an electronic device for real-time incremental reading based on a split log, including:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to perform at least the following:
scanning log files in a log directory, and setting a reading initial position of each file;
starting a file listener to monitor log file changes in a log directory;
when a log file new creation, modification or update event is monitored:
judging whether the log file has a split file or not;
and if the split file exists, reading incremental log information and setting the reading starting position as the current reading length.
According to yet another aspect of the invention, there is provided a computer readable medium having stored thereon a computer program which when executed by a processor performs at least the following:
scanning log files in a log directory, and setting a reading initial position of each file;
starting a file listener to monitor log file changes in a log directory;
when a log file new creation, modification or update event is monitored:
judging whether the log file has a split file or not;
and if the split file exists, reading incremental log information and setting the reading starting position as the current reading length.
Other objects, features, and advantages will be apparent to those skilled in the art from the following detailed description of the disclosure and the accompanying drawings.
Drawings
The drawings illustrate embodiments of the invention and together with the description serve to explain the principles of the invention. In the drawings:
FIG. 1 illustrates a flow diagram of a method for real-time incremental content reading based on a split log in accordance with an embodiment of the present invention.
FIG. 2 illustrates a rotation-free split log incremental content read schematic according to an embodiment of the present invention.
FIG. 3 illustrates a diagram of a rotating split log incremental content read according to an embodiment of the invention.
Detailed Description
The embodiment of the invention discloses a method for real-time incremental reading based on a split log. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that embodiments of the invention may be practiced without these specific details.
In order to avoid the increment reading performance reduction and the reduction of the uncontrollable use of the server disk space caused by overlarge log files, the log files are segmented according to the size to store the monitoring logs are necessary, and the traditional log file increment reading scheme cannot meet the log collection task in the scene.
The invention aims to solve the problem of how to accurately and timely read incremental log content when the log content is recorded by using a rotation segmentation log, and the invention independently processes a plurality of monitoring events triggered when the log is segmented by adopting a monitoring mode of a body log file (a.log file in the previous example), and simultaneously adds a dual mechanism of filtering deletion events and checking the length of acquired incremental log information to ensure the accuracy and stability of reading the log file.
The rotation splitting log refers to a log file which is used for storing historical data after being split according to a certain rule, and is also called a rotation log. The a.log.1, a.log.2 and a.log.3 generated by the splitting method are all rotation splitting logs.
The technical scheme of the invention is improved on the common log incremental content reading scheme to form a new technical scheme, namely a log incremental content reading method based on rotation segmentation, as shown in fig. 1, a file monitor (for example, a FileAlterationMonitor) can monitor the change of a certain type of files under a certain path, and in the scheme, a body log file, namely a log file with a file name and a log end, is monitored.
FIG. 1 illustrates a flow diagram of a method for real-time incremental reading based on a split log according to an embodiment of the present invention. As shown in fig. 1, in step 101, all log files under the log directory are scanned, and the read start position of each file is set, for example, the read start position of each file is set to 0. In step 103, a file listener is turned on to listen for log file changes in the log directory. According to the embodiment of the invention, the log file events of new creation, modification or update and deletion are respectively processed in a targeted manner.
Newly creating log file events
When it is monitored that a new log file is created, i.e., a new event occurs when the log file changes, it is determined whether there is a rotation, i.e., whether there is a split file for the file, in step 105.
FIG. 2 illustrates a rotation-free split log incremental content read schematic according to an embodiment of the present invention. As shown in fig. 2, if it is judged in step 105 that there is no rotation, in step 107, an incremental log information length of the body log file is calculated, which is the current file length minus the read start position. Then, in step 109, it is determined whether the incremental log information length is greater than 0, and if so, in step 111, the reading start position is obtained, the incremental log information of the body log file, that is, the log content from the reading start position to the current file size, is read, and after the reading is completed, the reading start position is modified to "reading start position + incremental log information length". If not greater than 0, the reading start position is set to 0 in step 113.
FIG. 3 illustrates a diagram of a rotating split log incremental content read according to an embodiment of the invention. As shown in fig. 3, if it is judged in step 105 that there is a rotation, in step 115, a read start position is acquired, all the rotation log files are read in the order shown in fig. 3, and the read start position is set to 0 after the completion of the reading. The pivot log file is then transferred to the backup path at step 117. In step 119, the incremental log information of the body log file is read, and the reading start position is set to the current reading length.
Modifying or updating log file events
When it is monitored that the log file is modified or updated, that is, the log file is modified or updated when the log file changes, the processing method is the same as the above processing method of the new log file event, as shown in fig. 1.
Deleting log file events
When it is sensed that the log file is deleted, that is, the log file change condition is a deletion event, it is hibernated for a predetermined time (e.g., 0.5 seconds) in step 121. Then, in step 123, it is checked whether the log file exists. If the file does not exist, the location of the file record is deleted at step 125. Hibernation is an operation that is not processed immediately when a deletion event is listened to, but waits for a period of time. The reason for the hibernation is to confirm whether the deletion is real. Because the listener listens to two events, namely, delete and rebuild, when the log is renamed. For example, the deletion event here is obviously not a real deletion, and therefore it is necessary to determine whether the log file exists later. If yes, the segmentation operation is explained, otherwise, the real deletion is explained. It is important to sleep for a period of time in order to reserve time for the new file.
The decisions in the two dashed boxes in fig. 1 are used to filter and compatibly process false delete events triggered by rotation. When a file is deleted, if the location value of the file is not deleted, when a log file having the same name as the file is newly created and a log is written, the previously recorded location value will be used, i.e., the file content will be read from the location. In this case, when the length of the new file is smaller than the position, an abnormality occurs in reading. The area 2 part in the figure judges that the abnormity is compatible, the reading initial position is reset to 0, the reading is recovered to be normal next time, and the area 1 part judges and processes to avoid the abnormity problem.
Aiming at the problem that the existing log incremental content reading scheme can not accurately read log incremental content based on rotation segmentation, according to the technical scheme of the invention, the reliability of the real-time incremental reading method based on the segmentation log is ensured by decomposing the log rotation segmentation process and accurately extracting all monitoring events in the process, and simultaneously adding a dual mechanism of event deletion filtering and incremental log information length check to ensure that the log is read from old to new, the accurate reading of the log incremental content based on rotation segmentation is realized, and the robustness of the real-time incremental content reading method based on the segmentation log is greatly improved.
The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments of the present invention without departing from the spirit and scope of the invention. Accordingly, the invention is intended to embrace all such modifications and variations as fall within the scope of the invention as defined by the appended claims.
Claims (10)
1. A method for real-time incremental reading based on a split log comprises the following steps:
scanning a body log file under a log directory, and setting a reading initial position of the body log file;
starting a file listener to monitor changes of the body log file in a log directory;
when a new, modified or updated event of the body log file is monitored:
judging whether the body log file has a split file or not;
if the files are divided, acquiring a reading initial position, determining the oldest divided file, reading the incremental log information of each divided file according to the sequence from old to new to obtain the incremental log information of all the divided files, and then setting the reading initial position to be zero;
reading the incremental log information of the body log file, calculating the length of the incremental log information of the body log file, and setting the reading initial position of the body log file to be the length.
2. The method of claim 1, further comprising:
when the deletion event of the body log file is monitored:
judging whether the body log file exists or not; and
and if the body log file does not exist, deleting the position of the body log file in the log directory.
3. The method of claim 1, wherein the obtaining delta log information for all of the sliced files further comprises:
all the split files are transferred to the backup path.
4. The method of claim 1, further comprising: and if the file is not segmented, calculating the length of the incremental log information of the body log file.
5. The method of claim 4, further comprising:
if the length of the incremental log information is greater than 0, then:
and setting the reading starting position of the body log file as the reading starting position plus the increment log information length.
6. The method of claim 4, wherein the delta log information length is a current file length minus the read start position.
7. The method of claim 4, further comprising:
and if the length of the incremental log information is not more than 0, setting the reading starting position of the body log file to be 0.
8. The method of claim 2, wherein sleeping for a predetermined time is performed before determining whether the ontology log file exists.
9. An electronic device for real-time incremental reading based on a split log, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510297384.2A CN106294470B (en) | 2015-06-03 | 2015-06-03 | Method for reading real-time incremental log information based on split log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510297384.2A CN106294470B (en) | 2015-06-03 | 2015-06-03 | Method for reading real-time incremental log information based on split log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106294470A CN106294470A (en) | 2017-01-04 |
| CN106294470B true CN106294470B (en) | 2020-06-05 |
Family
ID=57656080
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510297384.2A Active CN106294470B (en) | 2015-06-03 | 2015-06-03 | Method for reading real-time incremental log information based on split log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106294470B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019445B (en) * | 2017-09-08 | 2021-07-06 | 北京京东尚科信息技术有限公司 | Data synchronization method and device, computing equipment and storage medium |
| CN108304305A (en) * | 2018-01-11 | 2018-07-20 | 北京潘达互娱科技有限公司 | The method and apparatus that journal file is read |
| CN110545296A (en) * | 2018-05-28 | 2019-12-06 | 阿里巴巴集团控股有限公司 | Log data acquisition method, device and equipment |
| CN112685370B (en) * | 2020-12-17 | 2022-08-05 | 福建新大陆软件工程有限公司 | Log collection method, device, equipment and medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1995016237A1 (en) * | 1993-12-10 | 1995-06-15 | Intelligence Quotient International Limited | Incremental backup system |
| CN104283719A (en) * | 2014-10-28 | 2015-01-14 | 北京国双科技有限公司 | Log processing method and device and server |
| CN104951474A (en) * | 2014-03-31 | 2015-09-30 | 阿里巴巴集团控股有限公司 | Method and device for acquiring MySQL binlog incremental logs |
-
2015
- 2015-06-03 CN CN201510297384.2A patent/CN106294470B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1995016237A1 (en) * | 1993-12-10 | 1995-06-15 | Intelligence Quotient International Limited | Incremental backup system |
| CN104951474A (en) * | 2014-03-31 | 2015-09-30 | 阿里巴巴集团控股有限公司 | Method and device for acquiring MySQL binlog incremental logs |
| CN104283719A (en) * | 2014-10-28 | 2015-01-14 | 北京国双科技有限公司 | Log processing method and device and server |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106294470A (en) | 2017-01-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200167238A1 (en) | Snapshot format for object-based storage | |
| US8290994B2 (en) | Obtaining file system view in block-level data storage systems | |
| CN106294470B (en) | Method for reading real-time incremental log information based on split log | |
| CN108510287B (en) | Judgment method, electronic device and computer-readable storage medium for customer return visit | |
| CN103049354B (en) | Data recovery method, data restore device and storage system | |
| CN105573859A (en) | Data recovery method and device of database | |
| CN106681863B (en) | Save the method and terminal device of electronic health record content of edit | |
| EP3089051B1 (en) | Database rollback using wal | |
| US10969970B2 (en) | Storage optimization of database in volatile and non-volatile storing unit | |
| CN111625402A (en) | Data recovery method and device, electronic equipment and computer readable storage medium | |
| US8271454B2 (en) | Circular log amnesia detection | |
| CN109614054B (en) | data reading method and system | |
| CN113485872A (en) | Fault processing method and device and distributed storage system | |
| CN112925759A (en) | Data file processing method and device, storage medium and electronic device | |
| CN118964775B (en) | Vue-based data management method and device, electronic equipment and storage medium | |
| CN114238018B (en) | Method, system and device for detecting integrity of log collection file and storage medium | |
| CN113625971B (en) | Data storage method, device, equipment and computer storage medium | |
| WO2017067397A1 (en) | Data recovery method and device | |
| CN113312309A (en) | Management method and device of snapshot chain and storage medium | |
| CN114461445A (en) | Data recovery method and apparatus, electronic device, and computer-readable storage medium | |
| CN113672167A (en) | Data consistency verification method, device and equipment of distributed storage system | |
| CN112650613A (en) | Error information processing method and device, electronic equipment and storage medium | |
| CN104268231B (en) | A kind of file access method, device and Intelligent File System | |
| CN108062323A (en) | A log reading method and device | |
| CN105573862A (en) | Method and equipment for recovering file systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |