KR101650316B1 - Apparatus and method for collecting and analysing HTML5 documents based a distributed parallel processing - Google Patents

Abstract

An apparatus and method for HTML5 document collection and analysis based on distributed parallel processing is provided. An HTML5 document collection and analysis apparatus based on the distributed parallel processing includes an injector module for storing root URL information in a first database, generating a list of URLs to be collected by receiving the root URL information from the first database, A generator module that stores the collection target URL list in a second database, a content provider that receives the collection target URL list from the second database, extracts content from a web page corresponding to the collection target URL list, A parser module for storing the parsing result information in the second database, a fetcher module for storing the parsing result information in the second database, a parsing module for receiving the parsing result information, A vulnerability analysis module for analyzing a vulnerability of the HTML code included in the content only when the document type of the web page is HTML5, Wherein the vulnerability analysis module divides the content into a plurality of sub-contents, extracts a keyword and an attribute for the sub-content, calculates a frequency of the keyword and the attribute, Analyze.

Description

Technical Field The present invention relates to an apparatus and method for collecting and analyzing HTML5 documents based on distributed parallel processing,

The present invention relates to an apparatus and method for HTML5 document collection and analysis based on distributed parallel processing.

With the advent of ubiquitous computing environments and the rapid development of the user-oriented Internet service market, the amount of data streams to be processed is rapidly increasing, and the types of data streams are becoming more diversified. Accordingly, researches related to distributed parallel processing of data streams to provide real-time data analysis and processing services for large-capacity data streams are actively under way.

Korean Patent Laid-Open Publication No. 2013-0095910 discloses an apparatus and method for managing a data stream distributed parallel processing service.

SUMMARY OF THE INVENTION The object of the present invention is to provide a method and system for collecting large-capacity HTML5 web documents and analyzing HTML5 security vulnerable tags and attributes based on distributed parallel processing, And to provide an HTML5 document collection and analysis apparatus based on distributed parallel processing capable of analyzing HTML5 security vulnerable tags and attributes targeting attacks.

Another problem to be solved by the present invention is to collect HTML5 web documents of large capacity and analyze HTML5 security vulnerable tags and attributes on the basis of distributed parallel processing. Especially, HTML5 security vulnerable tags targeting jacking or cross- The present invention provides an HTML5 document collection and analysis method based on distributed parallel processing capable of analyzing attributes.

The problems to be solved by the present invention are not limited to the above-mentioned problems, and other matters not mentioned can be clearly understood by those skilled in the art from the following description.

According to another aspect of the present invention, there is provided an HTML5 document collection and analysis apparatus based on distributed parallel processing, the apparatus comprising: an injector module for storing root URL information in a first database; A generator module for receiving the information, generating a list of URLs to be collected and storing the list of URLs to be collected in a second database, receiving the list of URLs to be collected from the second database, A fetcher module for extracting contents from the web page and storing the contents in the second database, parsing contents of the contents by receiving the contents from the second database to generate parsing result information, A parsing model for storing parsing result information in the second database A filter module that receives the parsing result information from the parsing module and determines whether the document type of the web page is HTML5, and a vulnerability module that, when the document type of the web page is HTML5, wherein the vulnerability analysis module divides the content into a plurality of sub-contents, extracts keywords and attributes for the sub-contents, and extracts keywords and attributes from the sub-contents, And analyzes the vulnerability of the content by calculating the frequency.

In some embodiments of the present invention, the vulnerability analysis module may extract the keyword and the attribute by arranging tags included in the sub-content in a tree structure.

In some embodiments of the present invention, the parser may further include an updater module for receiving the parsing result information from the second database and updating information stored in the first database.

In some embodiments of the present invention, the catcher module may generate content collection information about the content and further store the content collection information in the second database.

In some embodiments of the present invention, the updater module may update the information stored in the first database by receiving the content collection information from the second database.

In some embodiments of the present invention, the first database converts the Root URL information into a first format and stores the URL, the collection status, the collection time, the number of retries after collection, And may include information on the document format.

In some embodiments of the present invention, the second database converts the content into a second format and stores the converted content, and the second format includes information included in the first format and HTML content of the web page can do.

In some embodiments of the present invention, the second database may further store an outlink address in the form of parsing the contents, and a form in which the outlink is stored in units of text lines.

In some embodiments of the invention, it may further comprise a third database for storing information about the vulnerability.

In some embodiments of the present invention, the root URL information may be main URL information of a web page including a collection target URL.

According to another aspect of the present invention, there is provided an apparatus for collecting and analyzing HTML5 documents based on distributed parallel processing, the apparatus comprising: an injector module for extracting root URL information of a database and a first web page, A generator module for receiving the Root URL information to generate a list of URLs to be collected and storing the list of URLs to be collected in the database, extracting contents from a corresponding second web page by receiving the list of URLs to be collected, A fetcher module for storing the content in the database, a parsing module for receiving and parsing the content, generating parsing result information, and storing the parsing result information in the database, And a filter for determining whether the document type of the second web page is HTML5 and a vulnerability analysis module for analyzing a vulnerability of the HTML code included in the content only when the document type of the second web page is HTML5, Extracts a keyword and an attribute for the sub-content, calculates a frequency of the keyword and the attribute, and analyzes the vulnerability of the content.

In some embodiments of the present invention, the vulnerability analysis module may extract the keyword and the attribute by arranging tags included in the sub-content in a tree structure.

In some embodiments of the present invention, the catcher module may generate content collection information about the content and further store the content collection information in the database.

In some embodiments of the present invention, the database converts the Root URL information into a first format and stores the URL, the collection status, the collection time, the number of retries after collection, and the document format As shown in FIG.

In some embodiments of the present invention, the database converts and stores the content into a second format, and the second format includes information contained in the first format and HTML content of the second web page can do.

In some embodiments of the present invention, the database may further store an outlink address in the form of parsing the content, and a form in which the outlink is stored in units of text lines.

In some embodiments of the present invention, the root URL information may be the main URL information of the first web page including the collection target URL.

According to an aspect of the present invention, there is provided an HTML5 document collection and analysis method based on distributed parallel processing, the method comprising: generating a list of URLs to be collected based on root URL information; Extracts content from a web page, generates parsing result information by parsing the content of the content, and determines whether the document type of the web page is HTML5 based on the parsing result information, Analyzing a vulnerability of the HTML code included in the content only when the content is HTML5, the method comprising: splitting the content into a plurality of sub-contents, extracting keywords and attributes for the sub-contents, Keyword and the frequency of the attribute to analyze the vulnerability of the content.

In some embodiments of the present invention, analyzing the vulnerability may extract the keyword and the attribute by arranging the tags included in the sub content in a tree structure.

In some embodiments of the invention, it may further comprise storing the Root URL information in a database.

In some embodiments of the present invention, the method further comprises storing the collection URL list and the content in the database.

In some embodiments of the present invention, the root URL information may be main URL information of a web page including a collection target URL.

Other specific details of the invention are included in the detailed description and drawings.

According to the apparatus and method for HTML5 document collection and analysis based on the distributed parallel processing according to the present invention, it is possible to collect and analyze large-capacity HTML5 web documents based on distributed parallel processing, have. According to the present invention, HTML5 security vulnerable tags and attributes can be analyzed, targeting jacking or cross-site scripting (XSS) attacks.

1 is a block diagram of an HTML5 document collection and analysis apparatus based on distributed parallel processing according to an embodiment of the present invention.
2 shows a document format for collecting HTML5 documents.
3 is a diagram for explaining an analysis method based on distributed parallel processing.
4 is a diagram showing a tree structure for analyzing HTML5 security vulnerable tags and attributes.
FIG. 5 is an exemplary diagram illustrating a tree structure in which an attribute can be searched for an input tag among the HTML5 security vulnerable tags.
6 and 7 are tables showing HTML5 security vulnerability tags and attributes.
8 is a diagram for explaining a segment which is a logical storage unit.
9 is a block diagram of an HTML5 document collection and analysis apparatus based on distributed parallel processing according to another embodiment of the present invention.
10 is a block diagram of an HTML5 document collection and analysis apparatus based on distributed parallel processing according to another embodiment of the present invention.
11 is a block diagram of an HTML5 document collection and analysis apparatus based on distributed parallel processing according to another embodiment of the present invention.
12 is a flowchart sequentially illustrating an HTML5 document collection and analysis method based on distributed parallel processing according to an exemplary embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Each block may represent a portion of a module, segment, or code that includes one or more executable instructions for executing the specified logical function (s). It should also be noted that in some alternative implementations the functions mentioned in the blocks may occur out of order. For example, two blocks that are shown one after the other may actually be executed substantially concurrently, or the blocks may sometimes be performed in reverse order according to the corresponding function.

Although the first, second, etc. are used to describe various elements, components and / or sections, it is needless to say that these elements, components and / or sections are not limited by these terms. These terms are only used to distinguish one element, element or section from another element, element or section. Therefore, it goes without saying that the first element, the first element or the first section mentioned below may be the second element, the second element or the second section within the technical spirit of the present invention.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. It is noted that the terms "comprises" and / or "comprising" used in the specification are intended to be inclusive in a manner similar to the components, steps, operations, and / Or additions.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by one of ordinary skill in the art to which this invention belongs. Also, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

1 is a block diagram of an HTML5 document collection and analysis apparatus based on distributed parallel processing according to an embodiment of the present invention. 2 shows a document format for collecting HTML5 documents. 3 is a diagram for explaining an analysis method based on distributed parallel processing. 4 is a diagram showing a tree structure for analyzing HTML5 security vulnerable tags and attributes. FIG. 5 is an exemplary diagram illustrating a tree structure in which an attribute can be searched for an input tag among the HTML5 security vulnerable tags. 6 and 7 are tables showing HTML5 security vulnerability tags and attributes. 8 is a diagram for explaining a segment which is a logical storage unit.

Referring to FIG. 1, an apparatus 5 for collecting and analyzing HTML5 documents based on distributed parallel processing according to an embodiment of the present invention includes an injector module 100, a first database DB1, a generator module 110, 2 database (DB2), a fetcher module 120, a parsing module 130, a filter module 140, and a vulnerability analysis module 150.

A jacking attack is an attack technique that induces an attacker to click on another web document that they have planted regardless of the user's intention. An attacker can persuade a user to click on an iFrame containing a malicious web document. We mainly use frame-busting technique to prevent web contents from being loaded into iFrame as an client defense technique in existing HTML documents. However, the framebusting technique is a script-based defense technique. In the HTML5 document, the sandbox property of the iFrame is added, which makes it possible to invalidate the attribute of the framebusting script in the iFrame.

A cross-site scripting (XSS) attack is an attack technique in which an attacker inserts a malicious script into a web document to steal the user's information or perform an abnormal function of the web document. In order to insert a malicious script into an input tag or the like in an existing HTML document, a user must first input an input value. However, in the HTML5 document, attributes such as autofocus are added, and it is possible to execute an attacker's malicious script automatically without the input action of the user.

The HTML5 document collection and analysis apparatus 1 based on distributed parallel processing can improve the analysis processing speed by collecting HTML5 documents among a plurality of HTML documents and analyzing security vulnerabilities of HTML5 documents using distributed parallel processing . In particular, the HTML5 document collection and analysis apparatus 1 based on distributed parallel processing can be targeted for a jacking attack and a tag-based cross-site scripting attack, which are security weak points of HTML5 documents. However, the present invention is not limited thereto.

The injector module 100 stores the root URL information (RUI) in the first database DB1. Specifically, the injector module 100 may convert the root URL information (RUI) into the first format F1 and store it in the first database DB1. The first format F1 may include information on the URL of the web document to be collected, the collection status, the collection time, the number of retries after collection, and the document format. The format stored in the first database DB1 may be expressed as < URL, F1 >. The root URL means the main URL of the web page including the collection target URL (CTU). The collection target URL (CTU) means a plurality of URLs contained in one web page, and means a URL of a web page to be hyperlinked.

The generator module 110 receives the root URL information (RUI) from the first database DB1 to generate the collection target URL list CTUL and stores the collection target URL list CTUL in the second database DB2 . Specifically, the generator module 110 generates the collection target URL list (CTUL) by dividing the collection host 110 by the same host, and stores the list in the second database DB2. That is, when the generator module 110 is provided with a single Root URL information (RUI), it can generate a single collection target URL list CTUL.

The fetcher module 120 receives the collection target URL list CTUL from the second database DB2 and extracts the content C from the web page corresponding to the collection target URL list CTUL, 2 store it in the database (DB2). Specifically, the fetcher module 120 visits the web page corresponding to the collection target URL list CTUL to collect the contents of the document, and stores the contents of the second database (DB2) in the form of URL, F1, ). &Lt; / RTI > The content C refers to the HTML content of the document collected by visiting the web page corresponding to the collection target URL list CTUL.

The parsing module 130 receives the content C from the second database DB2 and generates the parsing result information PRI by parsing the content of the content C and outputs the parsing result information PRI to the second database DB2). Specifically, the parsing module 130 parses <URL, C> stored in the second database DB2 to extract an outlink. The parsing module 130 may store the extracted outlink in the second database DB2 in the form of <URL, F1>, <URL, PD>, <URL, PT>. PD (ParseData) means a parsed outlink address, and PT (ParseText) means that an outlink is stored in units of text lines.

The filter module 140 receives the parsing result information PRI from the parsing module 130 and determines whether the document type of the web page corresponding to the collection target URL list CTUL is HTML5. Specifically, referring to FIG. 2, in the previous HTML version, the HTML version is determined by referring to the dtd file. In HTML5, however, the dtd file does not exist and is defined only by <! DOCTYPEhtml>. Accordingly, the filter module 140 determines whether the HTML document is defined as <! DOCTYPEhtml> and whether the document type of the web page corresponding to the collection URL list (CTUL) is HTML5. The filter module 140 designates the document format of the parsing result information PRI as content-type: text / html5 when the document type of the web page corresponding to the collection target URL list CTUL is HTML5, (150).

The vulnerability analysis module 150 analyzes the vulnerability of the HTML code included in the content C when the document format of the parsing result information PRI is content-type: text / html5. Specifically, the vulnerability analysis module 150 splits the content C into a plurality of sub-content C_S and stores a keyword key and an attribute att in the sub-content C_S , And the frequency of the keyword (key) and the attribute (att) can be calculated.

Referring to FIG. 3, a method for analyzing the vulnerability of the HTML code included in the content C is illustrated in the vulnerability analysis module 150. a < URL, C > that contains the content C to be analyzed as an HTML5 document. a contains the HTML5 code of the document to be analyzed. Splits it into a plurality of sub contents C_S, and generates b using a mapper. b contains the document URL where the vulnerability was found, the name of the vulnerability, and the location of the vulnerability. c is a reducer, and b is sorted based on a specific keyword. d is an output file that contains information about the sorted URL, the name of the aligned vulnerability, and the location of the aligned vulnerabilities.

In particular, the vulnerability analysis module 150 can extract the keywords (key) and attributes (att) by arranging the tags included in the sub-content C_S into a tree structure. Referring to FIG. 4, an example of arranging the tags included in the sub-content C_S into a tree structure is shown. In order to analyze the vulnerability, the tags included in the sub content (C_S) can be divided into HTML5 vulnerable tags and attributes and JavaScript vulnerable tags, and then analyzed. FIG. 5 exemplarily shows a case where an autofocus attribute and an onfocus attribute are extracted for an input tag in a tree structure.

In FIG. 6, attributes and tags that may be used for jacking attacks are shown, and in FIG. 7, attributes and tags that may be used in a cross-site scripting attack are shown. By extracting such attributes and tags from the tree structure, it is possible to search for tags and attributes that are likely to be included in the web page. By using the distributed parallel processing method described above, the retrieval accuracy can be improved while improving the processing speed have.

When collecting and parsing the HTML code of a web page, it is difficult to process the contents of the web page and the connection information in one node in order to grasp the connection relation to another web page existing in the web page. Accordingly, the web page is stored and parsed based on the distributed parallel processing, and the second database DB2 can store the content C in a logical storage unit called a segment. In the segment, the contents of the web page and the connection information in the web page can be divided and stored (see Fig. 8).

Hereinafter, an HTML5 document collection and analysis apparatus based on distributed parallel processing according to another embodiment of the present invention will be described.

9 is a block diagram of an HTML5 document collection and analysis apparatus based on distributed parallel processing according to another embodiment of the present invention. For the sake of convenience of description, description of substantially the same parts as those of the HTML5 document collecting and analyzing apparatus based on the distributed parallel processing according to the embodiment of the present invention will be omitted.

9, an HTML5 document collection and analysis apparatus 2 based on distributed parallel processing according to another embodiment of the present invention includes an injector module 100, a first database DB1, a generator module 110, 2 database (DB2), a fetcher module 120, a parsing module 130, a filter module 140, a vulnerability analysis module 150, and an updater module 160.

The injector module 100, the first database DB1, the generator module 110, the second database DB2, the fetcher module 120, the parsing module 130, the filter module 140, the vulnerability analysis module 150, Is substantially the same as that described above.

The updater module 160 receives the parsing result information PRI from the second database DB2 and updates the information stored in the first database DB1. The updater module 160 refers to the parsing result information PRI stored in the second database DB2 and updates information included in the first format F1 stored in the first database DB1.

The fetcher module 120 may also generate content collection information (CCI) relating to the content C and store the content collection information (CCI) in the second database DB2. The content collection information (CCI) may include a collection time, location information, and the like collected from the content (C).

The updater module 160 may receive the content collection information CCI from the second database DB2 and update information included in the first format F1 stored in the first database DB1.

10 is a block diagram of an HTML5 document collection and analysis apparatus based on distributed parallel processing according to another embodiment of the present invention. For the sake of convenience of description, description of substantially the same parts as those of the HTML5 document collecting and analyzing apparatus based on the distributed parallel processing according to the embodiment of the present invention will be omitted.

10, the HTML5 document collection and analysis apparatus 3 based on the distributed parallel processing according to another embodiment of the present invention includes an injector module 100, a first database DB1, a generator module 110, A second database DB2, a fetcher module 120, a parsing module 130, a filter module 140, a vulnerability analysis module 150, and a third database DB3.

The injector module 100, the first database DB1, the generator module 110, the second database DB2, the fetcher module 120, the parsing module 130, the filter module 140, the vulnerability analysis module 150, Is substantially the same as that described above.

The third database (DB3) stores information (VI) related to the vulnerability. Specifically, the vulnerability analysis module 150 may generate a result of analyzing the vulnerability of the HTML code included in the content C as the information (VI) related to the vulnerability and store it in the third database DB3.

Vulnerability information (VI) may include information on the HTML5 security vulnerable tags and attributes. If new HTML5 security vulnerable tags and attributes are detected, the information (VI) on vulnerabilities stored in the third database (DB3) May be updated.

11 is a block diagram of an HTML5 document collection and analysis apparatus based on distributed parallel processing according to another embodiment of the present invention. For the sake of convenience of description, description of substantially the same parts as those of the HTML5 document collecting and analyzing apparatus based on the distributed parallel processing according to the embodiment of the present invention will be omitted.

11, the HTML5 document collection and analysis apparatus 4 based on the distributed parallel processing according to another embodiment of the present invention includes an injector module 100, a first database DB1, a generator module 110, A parser module 140, a vulnerability analysis module 150, an updater module 160, and a third database DB3, as shown in FIG.

The injector module 100, the first database DB1, the generator module 110, the second database DB2, the fetcher module 120, the parsing module 130, the filter module 140, the vulnerability analysis module 150, The updater module 160 and the third database DB3 are substantially the same as those described above.

Hereinafter, an HTML5 document collection and analysis method based on distributed parallel processing according to an embodiment of the present invention will be described.

12 is a flowchart sequentially illustrating an HTML5 document collection and analysis method based on distributed parallel processing according to an exemplary embodiment of the present invention.

12, in the HTML5 document collection and analysis based on the distributed parallel processing according to an embodiment of the present invention, a collection target URL list (CTUL) is first generated based on Root URL information (RUI) (S100). The collection URL list (CTUL) can be generated by dividing the same host (host) and stored in the database. When a single Root URL information (RUI) is provided, a single collection target URL list (CTUL) can be generated.

Subsequently, the content C is extracted from the web page corresponding to the collection target URL list CTUL (S110). The contents of the document can be collected by visiting a web page corresponding to the collection target URL list (CTUL), and stored in the database in the form of <URL, F1>, <URL, C>.

Subsequently, the contents of the contents C are parsed to generate parsing result information PRI (S120). The content C refers to the HTML content of the document collected by visiting the web page corresponding to the collection target URL list CTUL.

Then, based on the parsing result information PRI, it is determined whether the document type of the web page is HTML5 (S130). Parses <URL, C> stored in the database to extract the outlink. The extracted outlink can be stored in the database in the form of <URL, F1>, <URL, PD>, <URL, PT>. PD (ParseData) means a parsed outlink address, and PT (ParseText) means that an outlink is stored in units of text lines.

Then, when the document type of the web page is HTML5, the vulnerability of the HTML code included in the content (C) is analyzed (S140). When analyzing the vulnerability of the HTML code included in the content C, the content C is split into a plurality of sub-contents C_S and a keyword key and an attribute att ), And analyze the vulnerability of the content (C) by calculating the frequency of the keyword (key) and the attribute (att). At this time, the tags included in the sub content C_S can be arranged in a tree structure to extract the key and the attribute att.

The steps of a method or algorithm described in connection with the embodiments of the invention may be embodied directly in hardware, software modules, or a combination of the two, executed by a processor. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any form of computer readable media known in the art Lt; / RTI &gt; An exemplary recording medium is coupled to a processor, which is capable of reading information from, and writing information to, the recording medium. Alternatively, the recording medium may be integral with the processor. The processor and the recording medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and the recording medium may reside as discrete components in a user terminal.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

100: Injector module 110: Generator module
120: fetcher module 130: parser module
140: Filter module 150: Vulnerability analysis module

Claims (22)

An injector module for storing root URL information in a first database;
A generator module for receiving the Root URL information from the first database to generate a collection target URL list and storing the collection target URL list in a second database;
A fetcher module for receiving the collection target URL list from the second database, extracting contents from a web page corresponding to the collection target URL list, and storing the contents in the second database;
A parsing module for receiving the contents from the second database, generating parsing result information by parsing contents of the contents, and storing the parsing result information in the second database;
A filter module for receiving the parsing result information from the parsing module and determining whether the document type of the web page is HTML5; And
And a vulnerability analysis module for analyzing a vulnerability of the HTML code included in the content only when the document type of the web page is HTML5,
Wherein the vulnerability analysis module divides the content into a plurality of sub contents, extracts keywords and attributes by arranging the tags included in the sub contents in a tree structure, calculates frequency of the keywords and the attributes, An HTML5 document collection and analysis system based on distributed parallel processing that analyzes vulnerability of contents. delete The method according to claim 1,
And an updater module for receiving the parsing result information from the second database and updating the information stored in the first database. The method of claim 3,
Wherein the finder module generates content collection information about the content and further stores the content collection information in the second database. 5. The method of claim 4,
Wherein the updater module receives the content collection information from the second database and updates the information stored in the first database. The method according to claim 1,
Wherein the first database converts the Root URL information into a first format and stores the first format,
Wherein the first format includes information on a URL, a collection status, a collection time, a number of retries after collection, and a document format. The method according to claim 6,
The second database converts the content into a second format and stores the converted content,
Wherein the second format includes information included in the first format and HTML content of the web page. 8. The method of claim 7,
Wherein the second database further stores an outlink address in the form of parsing the content and a form in which the outlink is stored in units of text lines. The method according to claim 1,
And a third database for storing information on the vulnerability. The HTML5 document collection and analysis apparatus based on distributed parallel processing. The method according to claim 1,
Wherein the Root URL information is main URL information of a Web page including a URL to be collected, based on a distributed parallel processing. Database;
An injector module for extracting root URL information of a first web page and storing the extracted root URL information in the database;
A generator module for receiving the Root URL information to generate a list of URLs to be collected and storing the list of URLs to be collected in the database;
A fetcher module for receiving the collection URL list, extracting contents from a corresponding second web page, and storing the contents in the database;
A parsing module for receiving and parsing the content, generating parsing result information, and storing the parsing result information in the database;
A filter module for receiving the parsing result information and determining whether the document type of the second web page is HTML5; And
And a vulnerability analysis module for analyzing a vulnerability of the HTML code included in the content only when the document type of the second web page is HTML5,
Wherein the vulnerability analysis module divides the content into a plurality of sub contents, extracts keywords and attributes by arranging the tags included in the sub contents in a tree structure, calculates frequency of the keywords and the attributes, An HTML5 document collection and analysis system based on distributed parallel processing that analyzes vulnerability of contents. delete 12. The method of claim 11,
Wherein the finder module generates content collection information about the content and further stores the content collection information in the database. 12. The method of claim 11,
Wherein the database converts the Root URL information into a first format and stores the first format,
Wherein the first format includes information on a URL, a collection status, a collection time, a number of retries after collection, and a document format. 15. The method of claim 14,
Wherein the database converts the content into a second format and stores the converted content,
Wherein the second format includes information included in the first format and HTML content of the second web page. 16. The method of claim 15,
Wherein the database further stores an outlink address in the form of parsing the content and a form in which the outlink is stored in a unit of a text line, based on a distributed parallel processing. 12. The method of claim 11,
Wherein the root URL information is main URL information of the first web page including a URL to be collected, based on distributed parallel processing. Generates a collection URL list based on the root URL information,
Extracts a content from a web page corresponding to the collection target URL list,
Parsing the content of the content to generate parsing result information,
Determining whether the document type of the web page is HTML5 based on the parsing result information,
And analyzing a vulnerability of the HTML code included in the content only when the document type of the web page is HTML5,
Extracting a keyword and an attribute by arranging the tags included in the sub content in a tree structure, calculating frequency of the keyword and the attribute, and analyzing the vulnerability of the content Distributed Parallel Processing based HTML5 Document Collection and Analysis. delete 19. The method of claim 18,
Further comprising storing the Root URL information in a database. 21. The method of claim 20,
Further comprising storing the collection URL list and the content in the database. 19. The method of claim 18,
Wherein the Root URL information is main URL information of a Web page including a collection target URL, based on distributed parallel processing.

Images