Disclosure of Invention
In order to solve at least one of the technical problems in the prior art to a certain extent, the invention aims to provide an audit analysis method, system, device and medium based on operator combination operation.
The technical scheme adopted by the invention is as follows:
an audit analysis method based on operator combination operation comprises the following steps:
acquiring an audit database, and performing visual configuration on the process resources in the audit database to obtain a process resource operator;
moving the history resource operator to a canvas interface at the front end, and moving a preset packaged history operation operator to the canvas interface;
on a canvas interface, connecting the process resource operator and the process operation operator by using a directed line to generate a directed task flow chart;
the front end encapsulates the directed task flow chart into a protocol data packet by adopting a JSON protocol and sends the protocol data packet to an operation background;
and the operation background analyzes the protocol data packet, an execution engine is adopted to operate the analyzed protocol data packet, and an operation result is pushed to the front-end interface.
Further, the audit database comprises at least one of an Oracle database, a MySql database, a SqlServer database, a Damomeng database, a Hive database, a ClickHouse database, a Doris database, a Habase database or a Mongdb database;
the visually configuring the process resources in the audit database to obtain a process resource operator includes:
and packaging the process resources in the audit database into preset icons, distributing IDs to the preset patterns, and generating process resource operators.
Further, the course operation operator comprises a basic operator and a business operator;
the basic operator is an operation operator which is encapsulated by using SQL statements and taking the minimum function of query analysis in a database as a unit; the minimization function comprises SUM, COUNT, MAX, MIN, AVG, intersection, union, difference, filtration, association, left connection, right connection, full connection, extraction and interception;
the business operator is an arithmetic operator formed by encapsulating business functions and business models based on commonality.
Furthermore, different operators are distinguished by adopting different icons, and the internal structures corresponding to the operators comprise unique identifiers ID, types, input parameters, processing programs, output results, pre-operators ID and post-operators ID.
Further, the protocol data packet comprises an operator resource type, an operator front-back node relation and a protocol body specifically processed by the operator;
the operation background analyzes the protocol data packet, and the operation background comprises the following steps:
analyzing the protocol data packet by adopting a JSON mapping object entity mode, acquiring an operator executive program, and sending the operator executive program to an execution engine;
and if the format of the protocol data packet is detected to have errors, feeding back error information to the front end.
Furthermore, in the operation process of the execution engine, a websocket channel is established between the front end and the operation background, and the execution result of each course operation operator is transmitted to the front end through the websocket channel;
and after the execution engine operation is completed, closing the websocket channel.
Further, the audit analysis method based on operator combination operation further comprises the following steps:
and storing the result of each operation and generating a result snapshot so as to facilitate subsequent retrieval.
The other technical scheme adopted by the invention is as follows:
an audit analysis system based on operator combination operation, comprising:
the data encapsulation module is used for acquiring an audit database, visually configuring the process resources in the audit database and acquiring process resource operators;
the operator calling module is used for moving the process resource operator to a canvas interface at the front end and moving a preset and packaged process operation operator to the canvas interface;
the history construction module is used for connecting the history resource operator and the history operation operator by adopting a directed line on a canvas interface to generate a directed task flow chart;
the system comprises a process operation module, a processing module and a processing module, wherein the process operation module is used for encapsulating a directed task flow chart into a protocol data packet by adopting a JSON protocol at the front end and sending the protocol data packet to an operation background;
and a result feedback module. And the operation background is used for analyzing the protocol data packet, the execution engine is used for operating the analyzed protocol data packet, and the operation result is pushed to the front-end interface.
The other technical scheme adopted by the invention is as follows:
an audit analysis device based on operator combination operation comprises:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement the method described above.
The other technical scheme adopted by the invention is as follows:
a storage medium having stored therein a processor-executable program for performing the method as described above when executed by a processor.
The invention has the beneficial effects that: the invention encapsulates the SQL sentences audited and mined into operation components according to the atoms of the minimum execution unit, and auditors directly call and combine operators according to the analysis thought so as to realize the whole analysis thought, simplify the SQL compiling difficulty and improve the reusability.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided only for convenience of illustration, the order between the steps is not limited at all, and the execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.
In the description of the present invention, it should be understood that the orientation or positional relationship referred to in the description of the orientation, such as the upper, lower, front, rear, left, right, etc., is based on the orientation or positional relationship shown in the drawings, and is only for convenience of description and simplification of description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus, should not be construed as limiting the present invention.
In the description of the present invention, the meaning of a plurality of means is one or more, the meaning of a plurality of means is two or more, and larger, smaller, larger, etc. are understood as excluding the number, and larger, smaller, inner, etc. are understood as including the number. If the first and second are described for the purpose of distinguishing technical features, they are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.
In the description of the present invention, unless otherwise explicitly limited, terms such as arrangement, installation, connection and the like should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above terms in the present invention in combination with the specific contents of the technical solutions.
As shown in fig. 1, this embodiment provides an audit analysis method based on operator combination operation, including the following steps:
and S1, acquiring an audit database, and performing visual configuration on the process resources in the audit database to obtain a process resource operator.
Configuring process resources on a visual guide type form interface, and configuring different process resources aiming at different audit projects; the types of databases used in different audit scenarios may be different, and thus, a plurality of databases need to be configured compatibly. In this embodiment, a database may be configured such as: relational databases such as Oracle, MySql, SqlServer, dreams, etc. may be configured with big data repositories such as: hive, ClickHouse, Doris, etc., may configure columnar databases such as: habase, Mongodb, etc.
The configuration of the history resource can be detailed to a certain table and a view in a specific database, and a program automatically generates a history resource operator by taking a database table as a unit according to a configured data source, namely, one operator corresponds to one database table or view, and can drag and pull the operator to perform data preview and operation on a canvas. According to the data analysis processing thought mined by audit analysis and the data hierarchical architecture of the big data warehouse, four types of process resources can be divided, such as: the system comprises an original database, an audit factor library, an intermediate result set and an external data set.
In some embodiments, such as with the MySql database, when the tb _ person table in the MySql database is configured, the system automatically generates an lineage resource operator with an ID of "d 9154a5a-224b-462e-88f1-a7a2fd32ec 00", and the dragging operation on the canvas is based on the tb _ person table in the MySql database.
And S2, moving the history resource operator to a canvas interface at the front end, and moving the preset and packaged history operation operator to the canvas interface.
The course arithmetic operators are divided into basic operators and business operators, wherein the basic operators are the arithmetic operators which are packaged by using query analysis of SQL statements in a database according to a minimum function as a unit. For example, the function: SUM, COUNT, MAX, MIN, AVG, etc., mathematical operations: intersection, union, difference, etc., key grammars: filtering, associating, left connecting, right connecting, full connecting, extracting, intercepting and the like, wherein each SQL atomic operation is encapsulated into an operator. The business operator refers to an arithmetic operator formed by packaging a plurality of common business functions and business models, experience accumulation is convenient for subsequent repeated use, for example: the operators for less input of concealed services, the operators for more input of fictitious services, and the like.
The operator is represented as an icon which can be dragged and dragged to the calendar canvas on the interface, and operators with different functions are distinguished by different icons; the internal structure of the system is composed of attributes such as unique identification ID, type, input parameter, processing program, output result, prepositive operator ID, postpositive operator ID and the like, and the prepositive operator of the arithmetic operator is required to be a resource operator or an operation result set.
And S3, connecting the history resource operator and the history operation operator by adopting a directed line on the canvas interface to generate a directed task flow chart.
Selecting a proper course operator according to an analysis thought, selecting a resource operator firstly by dragging and pulling, then selecting a plurality of operational operators to a course canvas, and compiling the operators by using directed lines to form a complete directed task flow chart.
And S4, the front end encapsulates the directed task flow graph into a protocol data packet by adopting a JSON protocol, and sends the protocol data packet to the operation background.
The page visualization operation can be recognized and executed by a background program, a processing flow constructed by dragging operator components on a canvas needs to be encapsulated into a complete protocol data packet by using a JSON protocol according to an algorithm of directed graph breadth traversal, and the complete protocol data packet is transmitted to a background. The protocol body information must include operator resource types, operator front-back node relationships, operator specific processing protocol bodies, and the like.
A specific lineage JSON protocol for performing filtering operations on lineage resources is shown as follows:
and S5, analyzing the protocol data packet by the operation background, operating the analyzed protocol data packet by using the execution engine, and pushing the operation result to the front-end interface.
And analyzing the streaming protocol data packet submitted to the background by the front end in a JSON mapping object entity mode, and feeding back to the front end in time if the format of the data packet is wrong. The main node adopts a main and standby high-reliability framework and distributes the analyzed operator executive program to a corresponding execution engine for execution.
The object parsed by each operator node JSON protocol is as follows:
the object parsed by the JSON protocol is processed in the whole process as follows:
and searching a specific database and a table in the history resource operator resource pool according to the ID by the analyzed resource operator, packaging the protocol content into an executable SQL statement of the corresponding database according to the history resource type, and distributing the executable SQL statement to the corresponding execution engine.
In some optional embodiments, the protocol parsing node and the engine execution node adopt a master-slave architecture: one master node and multiple slaves, namely one master node of the protocol analysis, manage the protocol analysis and direct the protocol analysis to call a specific execution engine, and the execution engine is executed by different execution engines aiming at different calendar resources and operator types, such as a MySql engine, an Oracle engine, a Hive engine, a Habase engine and the like, by a load balancing framework of a plurality of nodes. The engine is a multi-node multi-process protocol execution and quickly pushes results to a front-end interface.
According to the previous example, the statement "select from tb _ person where >18limit 20" that requires an operation on the tb _ person table in the MySQL database was obtained as described above; the execution engine executes the statement and pushes a result and an execution log to the client for real-time presentation through the websocket.
In some optional embodiments, each flow job may open a websocket channel to keep communication with a background execution engine, an execution result of each operator node may push a message through the channel and an ID of an operator, and the channel is closed when the whole flow is completed, so that the task pushing and the result displaying are realized with as few resources as possible.
In some alternative embodiments, the history result is presented using a default paging table, and the presentation is different according to the data structure type, such as: the map with longitude and latitude data or address data can be marked and displayed on a map, the related relation is displayed by adopting a relational graph, and a chart display mode can be selected according to work reports and visual data trends.
In some optional embodiments, the process analysis is to store the result after performing a task analysis on current data under the current condition of the audit concept, and if the audit concept is to be implemented to act on continuous data, especially for continuously tracked audit items, the process analysis needs to be issued as an expert model. And configuring the audit process analysis task according to the dynamic needs of the service, automatically analyzing and mining at regular intervals to form an expert model, and automatically pushing an analysis result to remind related auditors. The user converts the existing history analysis task into an expert model, and the program automatically executes the history analysis according to the set execution period and generates a result snapshot for each history analysis.
In summary, compared with the prior art, the method of the embodiment has the following beneficial effects:
(1) and (3) packaging the SQL sentences which are audited and excavated into operation components according to atoms of the minimum execution unit, and building a building block-like combined operator by auditors according to the analysis idea to realize the whole analysis idea, so that the SQL compiling difficulty is simplified and the reusability is improved. The problem that auditing service personnel can also autonomously analyze and mine auditing big data on the premise of not knowing the technology is solved, and a passive learning result is changed into an active obtaining clue; the investment of personnel technical analysis personnel is reduced, and the labor cost is reduced. The database is closely adaptive, once modeling is used for many times, and the database cannot be modified due to different databases of audited units. And the large data processing capacity can support TB-level data analysis and mining on line. The data result is diversified, and the result can form an operator of an intermediate result set, an output expert model, chart display, map display, relation diagram display and the like.
(2) The user can carry out various basic queries, existing business analysis, result set collision and the like, meanwhile, the queries, the analyses and the collisions can be used as basic units to be combined and butted in a unified mode to form various complex data analysis processes, and the complex data analysis processes can be stored as analysis templates to form more complex units.
This embodiment also provides an audit analytic system based on operator combination operation, including:
the data encapsulation module is used for acquiring an audit database, visually configuring the process resources in the audit database and acquiring process resource operators;
the operator calling module is used for moving the process resource operator to a canvas interface at the front end and moving a preset and packaged process operation operator to the canvas interface;
the history construction module is used for connecting the history resource operator and the history operation operator by adopting a directed line on a canvas interface to generate a directed task flow chart;
the system comprises a process operation module, a processing module and a processing module, wherein the process operation module is used for encapsulating a directed task flow chart into a protocol data packet by adopting a JSON protocol at the front end and sending the protocol data packet to an operation background;
and a result feedback module. And the operation background is used for analyzing the protocol data packet, the execution engine is used for operating the analyzed protocol data packet, and the operation result is pushed to the front-end interface.
The audit analysis system based on operator combination operation can execute the audit analysis method based on operator combination operation provided by the method embodiment of the invention, can execute any combination implementation steps of the method embodiment, and has corresponding functions and beneficial effects of the method.
This embodiment also provides an audit analytical equipment based on operator composite operation, includes:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement the method of fig. 1.
The audit analysis device based on operator combination operation can execute the audit analysis method based on operator combination operation provided by the embodiment of the method of the invention, can execute any combination implementation steps of the embodiment of the method, and has corresponding functions and beneficial effects of the method.
The embodiment of the application also discloses a computer program product or a computer program, which comprises computer instructions, and the computer instructions are stored in a computer readable storage medium. The computer instructions may be read by a processor of a computer device from a computer-readable storage medium, and executed by the processor to cause the computer device to perform the method illustrated in fig. 1.
The embodiment also provides a storage medium, which stores an instruction or a program capable of executing the auditing analysis method based on operator combination operation provided by the method embodiment of the invention, and when the instruction or the program is run, the method embodiment can be executed by any combination implementation steps, and the method has corresponding functions and beneficial effects.
In alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flow charts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are performed independently.
Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and/or features may be integrated in a single physical device and/or software module, or one or more functions and/or features may be implemented in a separate physical device or software module. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary for an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be understood within the ordinary skill of an engineer, given the nature, function, and internal relationship of the modules. Accordingly, those skilled in the art can, using ordinary skill, practice the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative of and not intended to limit the scope of the invention, which is defined by the appended claims and their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the foregoing description of the specification, reference to the description of "one embodiment/example," "another embodiment/example," or "certain embodiments/examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.