CN111427708B - Message consumption method, device, computer equipment and storage medium - Google Patents
Message consumption method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111427708B CN111427708B CN202010221369.0A CN202010221369A CN111427708B CN 111427708 B CN111427708 B CN 111427708B CN 202010221369 A CN202010221369 A CN 202010221369A CN 111427708 B CN111427708 B CN 111427708B
- Authority
- CN
- China
- Prior art keywords
- message
- messages
- consumption
- file
- consumed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请涉及一种消息消费方法、装置、计算机设备和存储介质。所述方法包括：根据数据源的名称从预存消费规则配置文件中获取相应消息消费规则；所述消息消费规则包括需要消费的消息的起止时间，所述起止时间根据消息到达消息中间件的预估时间设定；根据所述消息消费规则，从消息中间件中查找在所述起止时间内到达所述消息中间件的消息，并对其进行消费；将已消费的所述消息存储到目标文件，并设置所述起止时间为所述目标文件的名称。采用本方法能够提供漏接消息的排查效率。
The present application relates to a message consumption method, device, computer equipment and storage medium. The method comprises: obtaining corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of a data source; the message consumption rules include the start and end times of the messages to be consumed, and the start and end times are set according to the estimated time when the messages arrive at the message middleware; according to the message consumption rules, searching the message middleware for messages that arrive at the message middleware within the start and end times, and consuming them; storing the consumed messages in a target file, and setting the start and end times as the name of the target file. The use of this method can improve the efficiency of troubleshooting missed messages.
Description
Technical Field
The present application relates to the field of communications technologies, and in particular, to a message consumption method, a device, a computer device, and a storage medium.
Background
The message middleware comprises a message production end and a message consumption end, wherein the message production end is used for providing messages, the message consumption end is used for connecting with the message production end and outputting messages, and the messages are text data of numbers, letters and symbols. The message middleware utilizes a high-efficiency and reliable message transmission mechanism to carry out platform-independent data communication, and has corresponding message consumption ends for each platform. Message middleware performs integration of a distributed system based on data communication, which can extend inter-process communication in a distributed environment by providing a message passing and message queuing model.
However, when a message missed occurs, IT operation and maintenance personnel need to check the history message (i.e. check whether the message is missed because the sender does not send a message or because the message receiver does not receive a message), usually, the IT operation and maintenance personnel need to consume the first message of the message middleware until the last message is consumed, store the consumed message in a text file, and then search the text file for the missed message. Because the message consumption end can not be interrupted halfway, the message quantity in the message middleware is huge, the text file used for storing the message is huge, and can reach tens of Gbytes to hundreds of Gbytes, the specific message to be checked is not different from a sea fishing needle when being searched in the large file, and the requirement on the hardware performance of a computer is very high when the message is searched in the large file, so the existing checking and missed connection efficiency is very low.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a message consumption method, apparatus, computer device, and storage medium capable of improving message exclusion efficiency.
A method of message consumption, the method comprising:
The message consumption rule comprises the start and stop time of the message to be consumed, wherein the start and stop time is set according to the estimated time of the message reaching the message middleware;
Searching the message reaching the message middleware in the start-stop time from the message middleware according to the message consumption rule, and consuming the message;
Storing the consumed information into a target file, and setting the start-stop time as the name of the target file.
In one embodiment, the step of obtaining the corresponding message consumption rule from the pre-stored consumption rule configuration file according to the name of the data source includes obtaining an application program interface for entering the message, an application program interface for outputting the message and an executable file for processing the input and output of the message from the pre-stored consumption rule configuration file according to the name of the data source, wherein the application program interface for entering the message, the application program interface for outputting the message and the executable file for processing the input and output of the message are configured with the corresponding message consumption rule, and the application program interface for outputting the message is configured with the start and stop time of the message to be consumed.
In one embodiment, the application program interface of the message output configures a naming mode of a file storing the message, a single file size storing the message and a directory of a file storing the message, and the step of storing the consumed message to a target file and setting the start-stop time to be the name of the target file includes storing the consumed message to the target file according to the naming mode of the file storing the message, the single file size storing the message and the directory of the file storing the message, and the naming mode of the file storing the message includes setting the start-stop time to be the name of the target file.
In one embodiment, the application program interface for entering the message is configured with a communication protocol of the message to be consumed, the message reaching the message middleware in the start-stop time is searched from the message middleware according to the message consumption rule, and the consumption step of the message comprises the steps of communicating with the corresponding message middleware according to the communication protocol of the message to be consumed, and searching the message reaching the message middleware in the start-stop time from the message middleware according to the message consumption rule, and consuming the message.
In one embodiment, the step of searching the message which reaches the message middleware within the start-stop time from the message middleware according to the message consumption rule and consuming the message comprises the steps of searching the message starting to consume and the message ending to consume from the message middleware according to the message consumption rule, sequentially sequencing the messages in the message middleware according to the sequence of the time of reaching the message middleware, and sequentially consuming the messages between the message starting to consume and the message ending to consume.
In one embodiment, after the step of sequentially consuming the messages between the location of the message to begin consumption and the location of the message to end consumption, the method includes recording the size of each of the consumed messages and/or counting the number of consumed messages.
In one embodiment, the step of storing the consumed message in a target file and setting the start-stop time as the name of the target file includes storing the consumed message in a text file according to a preset file size or a preset number of messages, and setting the start-stop time as the name of the text file.
A message consumption device, the device comprising:
The message consumption rule acquisition module is used for acquiring corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of a data source, wherein the message consumption rules comprise the start and stop time of a message to be consumed, and the start and stop time is set according to the estimated time of the message reaching the message middleware;
the consumption module is used for searching the message reaching the message middleware in the start-stop time from the message middleware according to the message consumption rule and consuming the message;
And the storage module is used for storing the consumed information into a target file and setting the start and stop time as the name of the target file.
A computer device comprising a memory storing a computer program and a processor which when executing the computer program performs the steps of:
The message consumption rule comprises the start and stop time of the message to be consumed, wherein the start and stop time is set according to the estimated time of the message reaching the message middleware;
Searching the message reaching the message middleware in the start-stop time from the message middleware according to the message consumption rule, and consuming the message;
Storing the consumed information into a target file, and setting the start-stop time as the name of the target file.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
The message consumption rule comprises the start and stop time of the message to be consumed, wherein the start and stop time is set according to the estimated time of the message reaching the message middleware;
Searching the message reaching the message middleware in the start-stop time from the message middleware according to the message consumption rule, and consuming the message;
Storing the consumed information into a target file, and setting the start-stop time as the name of the target file.
According to the message consumption method, the device, the computer equipment and the storage medium, the corresponding message consumption rules are set for each data source through the pre-stored consumption rule configuration file, so that message consumption can be integrated in one system, development workload of message consumption end tools configured according to different data sources is reduced, meanwhile, consumed messages are stored in the target file according to start and stop time, the investigation range of missed messages can be reduced, and the investigation efficiency of missed messages is improved.
Drawings
FIG. 1 is an application environment diagram of a message consumption method in one embodiment;
FIG. 2 is a flow diagram of a message consumption method in one embodiment;
FIG. 3 is a flow chart illustrating steps for looking up a message for consumption during a start-stop time in one embodiment;
FIG. 4 is a block diagram of a message consuming device in one embodiment;
FIG. 5 is a block diagram of a message consumption system in one embodiment;
fig. 6 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
The message consumption method provided by the application can be applied to an application environment shown in figure 1. Wherein the first platform terminal 102 and the second platform terminal 106 communicate with the message middleware server 104 over a network. The first platform terminal 102 obtains corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of a data source, wherein the message consumption rules comprise start and stop time of a message to be consumed, the start and stop time is set according to estimated time of the message reaching the message middleware server 104, the message reaching the message middleware server 104 within the start and stop time is searched from the message middleware server 104 according to the message consumption rules, the message which is consumed, the consumed message is stored in a target file, the start and stop time is set to be the name of the target file, and the second platform terminal 106 sends the message to the message middleware server 104. In this manner, the missed message transmitted from the second platform terminal 106 to the first platform terminal 102 can be examined.
The first platform terminal 102 and the second platform terminal 106 may be, but not limited to, various personal computers, notebook computers, smartphones, tablet computers, and portable wearable devices, and the server 104 may be implemented by a stand-alone server or a server cluster composed of a plurality of servers.
In one embodiment, as shown in fig. 2, a message consumption method is provided, which is illustrated by taking the application of the method to the first platform terminal 102 in fig. 1 as an example, and includes the following steps:
Step S110, obtaining corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of the data source, wherein the message consumption rules comprise the start and stop time of the message to be consumed, and the start and stop time is set according to the estimated time of the message reaching the message middleware.
The data source comprises data in message middleware such as ActiveMQ, rabbitMQ, kafka, rocketMQ, and the name of the data source is the name of the message middleware. The pre-stored consumption rule configuration file is provided with corresponding message consumption rules aiming at each data source, wherein the message consumption rules comprise an application program interface for entering a message, an application program interface for outputting the message and an executable file for processing the input and the output of the message, and the executable file for processing the input and the output of the message is a jar packet relied on by a self-defined message source. A message is text data consisting of a number, letter or symbol.
The start-stop time of the consumed message is set according to the estimated time of the message reaching the message middleware, and the message middleware receives the message and numbers and stores the received message. The message middleware may receive multiple messages within one second, that is, multiple messages may arrive at the message middleware at the same time, where the time when the multiple messages arrive at the message middleware is the same, for example, when the message A, B, C arrives at the message middleware S at the Beijing time 12:30:15, the message A, B, C arrives at the message middleware S at the Beijing time 12:30:15, the message middleware stores the message A, B, C according to the Beijing time 12:30:15, of course, the message A, B, C has corresponding numbers, if the message middleware stores 1000 messages in advance, the previously stored messages have been assigned numbers 1-1000, then the message A, B, C is continuously numbered according to the assigned numbers, the numbers of the message A, B, C may be 1001, 1002, 1003, the order of the message A, B, C may also be the message B, A, C, the corresponding message B, A, C may be 1001, 1002, 1003, the message A, B, C may also be in other order, the other order of the corresponding numbers of the message A, B, C may be 1001, 1002, 1003, and so on.
The estimated time is the time when the message middleware informs the message consumer (the first platform terminal 102) of the arrival of the message, and the start-stop time is a time period including the estimated time because the actual arrival time of the message in the message middleware deviates from the estimated time. By searching the message in the starting and ending time period, the message can be ensured to be searched under the condition that the estimated time has deviation, and the message is judged to be missed by the message middleware if the message does not exist in the message middleware and cannot be searched.
And step S120, searching the message reaching the message middleware in the start-stop time from the message middleware according to the message consumption rule, and consuming the message.
The message consumption process is a process of taking a message from a message middleware, wherein the message middleware can save a log of message consumption, and the message consumption process can be restored through the log.
Specifically, in the message consumption process of the message middleware, the terminal controls the starting time and the ending time of message consumption through a timer, counts the consumed messages through a counter, and acquires the position of the message which starts to be consumed and the position of the message which ends to be consumed through a message position acquisition module.
In one embodiment, the timer controls the start-stop time of the acquired messages, where each message has a time of reaching the message middleware, and the start-stop time of the acquired messages is set according to the time of reaching the message middleware, for example, if the time of missing messages (the message to be examined) is about 9:00, the start-stop time of the acquired messages may be set to 9:00-10:00, and the missing messages may be examined from the time period of 9:00-10:00. Wherein the counter controls the number of consumed messages and the byte size, for example, if the byte is too large for viewing for an entire message to be obtained, it is split into message segments and the byte size of each message segment is set as needed. The message position acquisition module controls the position of the acquired message in the middleware, for example, a plurality of messages exist in the message middleware, each message has a corresponding position in the message middleware, and the message position acquisition module can calculate the position of the message to be acquired in the message middleware by combining the start and stop time.
Step S130, storing the consumed information into a target file, and setting the start-stop time as the name of the target file.
The target file can be a text file, and the message consumed by the user can be conveniently checked through the text file. Of course, the target file may be a table file or other file.
In the message consumption method, the corresponding message consumption rules are set for each data source through the pre-stored consumption rule configuration file, so that the consumption of the message is integrated in one system, the development workload of configuring message consumption end tools according to different data sources is reduced, meanwhile, the consumed message is stored in the target file according to the start-stop time, the investigation range of missed messages can be reduced, and the investigation efficiency of missed messages is improved.
In one embodiment, the step S110 includes obtaining an application program interface for entering a message, an application program interface for outputting the message, and an executable file for processing input and output of the message from a pre-stored consumption rule configuration file according to a name of a data source, where the application program interface for entering the message, the application program interface for outputting the message, and the executable file for processing input and output of the message configure corresponding message consumption rules, and where the application program interface for outputting the message configures start and stop times of the message to be consumed.
The application program interface of the message entry is mainly used for defining a set of API (Application Programming Interface, application program interface) of the data source entry process. An application program interface is a set of definitions, programs, and protocols that enable the communication of computer software with each other through an API. When messages of various message middleware are consumed, the messages need to be communicated with the message middleware according to corresponding APIs. According to the embodiment, the consumption rule configuration file is prestored, and according to the type of the message middleware, the API is correspondingly configured, so that communication with a plurality of message middleware can be realized.
For example, a message-entering application program interface is used to obtain the TCP connection of an incoming message. TCP (Transmission Control Protocol ) is a connection-oriented, reliable, byte-stream based transport layer communication protocol.
Wherein the message output application program interface is used to define a set of data output APIs (Application Programming Interface, application program interface). In this embodiment, when outputting a message consumed from a message middleware, the consumed message is stored by pre-storing the start and stop time of the message, the naming mode of the message, the size of the target file and the directory of the target file output, which are set in the consumption rule configuration file.
In one embodiment, the application program interface of the message output configures a naming scheme for the files holding the message, a single file size holding the message, and a directory for the files holding the message. The step S130 includes storing the consumed message in a target file according to the naming mode of the message to be consumed, the size of a single file storing the message, and the directory of the file storing the message, where the naming mode of the file storing the message includes setting the start-stop time as the name of the target file.
The file of the message is named after storing the message consumed from the message middleware, when the message used for storing the consumption is a text file, the start-stop time is set as the name of the text file, for example, the message time consumed from the message middleware is from 55 minutes 0 seconds at the time of the month 30 in 2016 to 55 minutes 0 seconds at the time of the month 30 in 2016, and the message stored for the consumption is named as '20161230075500-20161230125500' when the message used for storing the consumption is a text file. Of course, when a plurality of files of the start-stop times are included, the files of the start-stop times may be numbered in the order of storage, for example, the message time consumed from the message middleware is 55 minutes 0 seconds at 7 th month 30 of 2016 to 55 minutes 0 seconds at 12 th month 30 of 2016, and the consumed message is stored in 2 text files, which are named "20161230075500-20161230125500-01" and "20161230075500-20161230125500-02", respectively.
Wherein storing the single file size of the message includes storing a capacity size of the single file of the message or storing a number of messages of the single file of the message. When there are more messages acquired in the start-stop time, the messages consumed in the start-stop time need to be saved in a plurality of files, for example, the capacity of a single file can be set to 2 mbytes, each file can save 2 mbytes of messages, or the number of messages stored in a single file is 100, each file can save 100 messages.
The directory storing the file of the message is a storage position storing the file of the message, and the application program interface outputting the message is configured with the directory storing the file of the message, so that the message can be conveniently searched.
In one embodiment, the application program interface of the message entering configures a communication protocol of the message to be consumed, and the step 120 includes communicating with a corresponding message middleware according to the communication protocol of the message to be consumed, searching the message reaching the message middleware within the start-stop time from the message middleware according to the message consumption rule, and consuming the message.
Wherein the communication protocol comprises a TCP/IP protocol, namely a network communication protocol, and can be connected with different message middleware according to different communication protocols so as to consume the messages in the message middleware.
In one embodiment, the step S120 includes a step S121 of inquiring the position of the message which starts to be consumed and the position of the message which ends to be consumed from the message middleware according to the message consumption rule, wherein the messages are sequentially ordered in the message middleware according to the sequence of the time when the messages arrive in the message middleware, and a step S122 of sequentially consuming the messages between the position of the message which starts to be consumed and the position of the message which ends to be consumed.
The messages in the message middleware have corresponding numbers, and the unique corresponding message can be found in the message middleware according to the numbers, and the messages corresponding to the start-stop time may be multiple, that is, the messages reaching the message middleware at the same time are multiple, so that the position of the message for starting consumption and the position of the message for ending consumption need to be obtained according to the start-stop time.
For example, the starting time of the message to be consumed is Beijing time 12:30:15, the message A, B, C arrives at the message middleware S at Beijing time 12:30:15, the message A, B, C arrives at the message middleware S at Beijing time 12:30:15, the message middleware stores the message A, B, C according to Beijing time 12:30:15, of course, the message A, B, C has corresponding numbers, if 1000 messages are stored in advance, the message stored in advance has been assigned numbers 1-1000, the message A, B, C continues to be numbered according to the assigned numbers, the message A, B, C numbers can be 1001, 1002 and 1003, and the position of the message A with the smallest starting time (Beijing time 12:30:15) number is taken as the position of the message to start consuming. For example, the end time of the message to be consumed is Beijing time 13:30:15, when the message L, I arrives at the message middleware S at the Beijing time 13:30:15, the message L, I arrives at the message middleware S at the Beijing time 13:30:15, the message middleware stores the message L, I according to the Beijing time 13:30:15, of course, the message L, I has corresponding numbers, if the message middleware stores 2000 messages in advance, the previously stored messages have been assigned numbers 1-2000, the message L, I continues to be numbered according to the assigned numbers, the message L, I numbers can be 2001 and 2002, and the position of the message I with the largest end time (Beijing time 13:30:15) number is taken as the position of the message for ending the consumption.
In one embodiment, after the step of sequentially consuming the messages between the location of the message to begin consumption and the location of the message to end consumption, the method includes recording the size of each of the consumed messages and/or counting the number of consumed messages. The size of each consumed message refers to the size of the consumed message, the consumed message is conveniently divided into a plurality of files to be stored according to the size of the message, and similarly, the statistics of the number of the consumed messages is also convenient to be divided into a plurality of files to be stored. For example, if the target file for storing the consumed message is 2 mbytes, the total size of the message stored in the target file is not more than 2 mbytes, or if the target file for storing the consumed message is capable of storing 100 messages, the total number of the messages stored in the target file is not more than 100.
In one embodiment, the step 130 includes storing the consumed messages in a text file according to a preset file size or a preset number of messages, where the text file stores the messages of the preset file size or the messages of the preset number of messages, and setting the start-stop time as the name of the text file.
It should be understood that, although the steps in the flowcharts of fig. 2-3 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 2-3 may include multiple steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor does the order in which the steps or stages are performed necessarily performed in sequence, but may be performed alternately or alternately with at least a portion of the steps or stages in other steps or other steps.
In one embodiment, as shown in FIG. 4, a message consumption device is provided that includes a message consumption rule acquisition module 210, a consumption module 220, and a storage module 230. Wherein:
The message consumption rule obtaining module 210 is configured to obtain a corresponding message consumption rule from a pre-stored consumption rule configuration file according to a name of a data source, where the message consumption rule includes a start-stop time of a message to be consumed, and the start-stop time is set according to an estimated time when the message arrives at the message middleware.
And the consumption module 220 is configured to search, from the message middleware, for a message that arrives at the message middleware within the start-stop time according to the message consumption rule, and consume the message.
The storage module 230 is configured to store the consumed message in a target file, and set the start-stop time as a name of the target file.
In one embodiment, the message consumption rule obtaining module 210 is further configured to obtain, from a pre-stored consumption rule configuration file, an application program interface for entering a message, an application program interface for outputting a message, and an executable file for processing input and output of the message according to a name of a data source, where the application program interface for entering the message, the application program interface for outputting the message, and the executable file for processing input and output of the message configure corresponding message consumption rules, and where the application program interface for outputting the message configures start and stop times of the message to be consumed.
In one embodiment, the application program interface of the message output configures a naming scheme for the files holding the message, a single file size holding the message, and a directory for the files holding the message. The storage module 230 is further configured to store the consumed message to a target file according to a naming manner of the message to be consumed, a single file size for storing the message, and a directory for storing a file of the message, where the naming manner of the file for storing the message includes setting the start-stop time to be a name of the target file.
In one embodiment, the application program interface of the message entering is configured with a communication protocol of the message to be consumed, and the consumption module 220 comprises a communication unit for communicating with a corresponding message middleware according to the communication protocol of the message to be consumed, and a consumption unit for searching the message reaching the message middleware in the starting and ending time from the message middleware according to the message consumption rule and consuming the message.
In one embodiment, the consumption module 220 includes a message location query unit configured to query, according to the message consumption rule, a message location of a message from a message middleware for starting consumption and a message location of a message from ending consumption, where the messages are sequentially ordered in the message middleware according to a time sequence of arrival at the message middleware, and a consumption unit configured to sequentially consume messages between the message location of the message from starting consumption and the message location of the message from ending consumption.
In one embodiment, the consumption module 220 further includes a recording unit, configured to record a size of each of the consumed messages and/or count a number of the consumed messages.
In one embodiment, the storage module 230 is further configured to store the consumed message to a text file according to a preset file size or a preset number of messages, where the text file stores the message of the preset file size or the message of the preset number of messages, and set the start-stop time as the name of the text file.
Specific limitations regarding the message consumption device may be found in the above limitations regarding the message consumption method, and will not be described in detail herein. The various modules in the message consumption device described above may be implemented in whole or in part in software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, as shown in FIG. 5, a message consumption system is provided that includes an input application program interface, an output application program interface, an adapter, a processor, and a consumption rules profile. The input application program interface comprises a communication protocol, a message type and a message configuration, the output application program interface comprises a message consumer and a message output configuration, the adapter comprises a message adapter, a message filter and a Java dependency package expanding unit, and the processor comprises a timer, a counter and a message position acquisition module.
The input application program interface is mainly used for defining a set of data source access processing API (Application Programming Interface, application program interface). An application program interface is a set of definitions, programs, and protocols that enable the communication of computer software with each other through an API. When messages of various message middleware are consumed, the messages are required to be communicated with the message middleware according to corresponding APIs, and the message middleware can be communicated with a plurality of message middleware through corresponding configuration APIs according to the type of the message middleware through a consumption rule configuration file. For example, the communication protocol is used to obtain a TCP (Transmission Control Protocol ) connection of an incoming message, TCP being a connection-oriented, reliable, byte-stream based transport layer communication protocol. The input application program interface is used for acquiring the type of the incoming message, such as the message type comprising kafka, rocketmq, activemq, rabbitmq, newdefinemq, and acquiring the configuration of the incoming message, such as the TCP connection string, the connection parameters, the extension dependent jar package and other data in the consumption rule configuration file. Wherein the jar package is a java executable file.
The output application program interface is mainly used for defining a set of APIs (Application Programming Interface, application program interface) for data output. In this embodiment, when outputting the consumed message, the output message is stored by using the start and stop time, naming mode, size of the single file and output directory of the message in the consumption rule configuration file. For example, the consumer of the message refers to the consumption thread created by the message processor module, and the output configuration of the message is that the starting time of the message to be consumed, the ending time of the message to be consumed, the naming mode of the file for storing the message, the catalog of the file for storing the message and the single file size for storing the message are all the output configuration of the message.
The consumption rule configuration file is mainly used for configuring message consumption rules. In the consumption rule configuration file, setting corresponding API for message source entering for various message middleware, defining rule of message output when outputting message, and configuring corresponding jar package depending on message processing. The jar package is a java executable file, which is designed according to a message source, and executes operations of consuming the message according to a consumption rule and storing the message according to the consumption rule. For example, the message consumption rules are used to configure a list including a message source type, tcp connection string, connection parameters, start time of message to be consumed, end time of message to be consumed, naming of file to hold the message, directory to hold the file of the message, and single file size to hold the message.
Wherein the adapter is used mainly for calling the message adapter and initializing the message filter. For example, the adapter calls a message adapter, the message adapter calls a Java dependent package extension unit, and the Java dependent package extension unit calls a jar package that is dependent on processing the message, the jar package being a Java executable file. The adapter is used to initialize the message filter, i.e. assign a value to each attribute of the message filter, for example:
dataSourceType=’kafka’
dataBeginTime=’20190601120500’
dataEndTime=’20190601180500’
dataName=’premium’
Wherein the adapter initializes the message filter, i.e., completes the settings of the message source type dataSourceType, the message start-stop time (dataBeginTime, dataEndTime), and the data naming dateName, the text file of the outgoing message can be named by the values of the various attributes set by the adapter.
Wherein the processor is configured to perform the process of consuming and storing the message according to the consumption rule profile. The processor comprises a timer, a counter and a message position acquisition module. The timer controls the start and stop time of the acquired messages, wherein each message has a time reaching the message middleware, the start and stop time of the acquired messages is set according to the time of the message reaching the message middleware, for example, the start and stop time of the acquired messages can be set to 9:00-10:00 when the time of missed messages is known to be about 9:00-10:00, the number of the missed messages can be checked from the time period of 9:00-10:00, the counter controls the size of the acquired messages according to the number of bytes, if the bytes are too large for the acquired whole message, the acquired whole message is divided into a plurality of message sections according to the requirement and the byte size of each message section is set, and the message position acquisition module pos controls the position of the acquired message in the middleware, wherein the plurality of messages exist in the message middleware, each message has a corresponding position in the message middleware, and the position of the acquired message in the message middleware can be calculated by combining the start and stop time of the message.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 6. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a message consumption method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structure shown in FIG. 6 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor when executing the computer program performing the steps of:
The message consumption rule comprises the start and stop time of the message to be consumed, wherein the start and stop time is set according to the estimated time of the message reaching the message middleware;
Searching the message reaching the message middleware in the start-stop time from the message middleware according to the message consumption rule, and consuming the message;
Storing the consumed information into a target file, and setting the start-stop time as the name of the target file.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:
The message consumption rule comprises the start and stop time of the message to be consumed, wherein the start and stop time is set according to the estimated time of the message reaching the message middleware;
Searching the message reaching the message middleware in the start-stop time from the message middleware according to the message consumption rule, and consuming the message;
Storing the consumed information into a target file, and setting the start-stop time as the name of the target file.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, or the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010221369.0A CN111427708B (en) | 2020-03-26 | 2020-03-26 | Message consumption method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010221369.0A CN111427708B (en) | 2020-03-26 | 2020-03-26 | Message consumption method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111427708A CN111427708A (en) | 2020-07-17 |
| CN111427708B true CN111427708B (en) | 2025-05-02 |
Family
ID=71548695
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010221369.0A Active CN111427708B (en) | 2020-03-26 | 2020-03-26 | Message consumption method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111427708B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112087501B (en) * | 2020-08-28 | 2023-10-24 | 北京明略昭辉科技有限公司 | Transmission method and system for maintaining data consistency |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109688200A (en) * | 2018-11-30 | 2019-04-26 | 北京奇艺世纪科技有限公司 | A kind of message treatment method, device and equipment |
| CN110909061A (en) * | 2019-11-27 | 2020-03-24 | 恩亿科(北京)数据科技有限公司 | Data source processing method and device, electronic equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7376957B1 (en) * | 2002-12-16 | 2008-05-20 | At&T Delaware Intellectual Property, Inc. | Method and system for recovering stranded outbound messages |
| CN110704212B (en) * | 2019-09-29 | 2022-04-22 | 广州荔支网络技术有限公司 | Message processing method and device |
-
2020
- 2020-03-26 CN CN202010221369.0A patent/CN111427708B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109688200A (en) * | 2018-11-30 | 2019-04-26 | 北京奇艺世纪科技有限公司 | A kind of message treatment method, device and equipment |
| CN110909061A (en) * | 2019-11-27 | 2020-03-24 | 恩亿科(北京)数据科技有限公司 | Data source processing method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111427708A (en) | 2020-07-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11222066B1 (en) | Processing data using containerized state-free indexing nodes in a containerized scalable environment | |
| US11269939B1 (en) | Iterative message-based data processing including streaming analytics | |
| US11106734B1 (en) | Query execution using containerized state-free search nodes in a containerized scalable environment | |
| US10984044B1 (en) | Identifying buckets for query execution using a catalog of buckets stored in a remote shared storage system | |
| US20190245694A1 (en) | Distributed logging of application events in a blockchain | |
| US11567993B1 (en) | Copying buckets from a remote shared storage system to memory associated with a search node for query execution | |
| US11562023B1 (en) | Merging buckets in a data intake and query system | |
| US11620336B1 (en) | Managing and storing buckets to a remote shared storage system based on a collective bucket size | |
| US12393631B2 (en) | Processing data using nodes in a scalable environment | |
| CN105898085B (en) | Harass recognition methods and the device of communications account | |
| WO2023087769A1 (en) | Method for deduplicating key field in real time on basis of distributed stream calculation engine flink | |
| US11789950B1 (en) | Dynamic storage and deferred analysis of data stream events | |
| US20170317961A1 (en) | Email notifications | |
| CN107450952B (en) | Application control method and device, storage medium and electronic equipment | |
| CN109842621A (en) | A kind of method and terminal reducing token storage quantity | |
| CN111580948A (en) | Task scheduling method, device and computer equipment | |
| CN106528243B (en) | Information processing method, information processing device, terminal device, and server | |
| CN109992369B (en) | Application processing method and apparatus, electronic device, and computer-readable storage medium | |
| CN111427708B (en) | Message consumption method, device, computer equipment and storage medium | |
| US10938773B2 (en) | Method and apparatus for synchronizing contact information and medium | |
| CN112732426B (en) | A method, system, device and medium for dynamically adjusting task priority | |
| US10694001B2 (en) | Method, apparatus and system for uploading a file | |
| CN109391658B (en) | Account data synchronization method and equipment, storage medium and terminal thereof | |
| CN113778996A (en) | Large data stream data processing method and device, electronic equipment and storage medium | |
| US20180217874A1 (en) | Resegmenting chunks of data for efficient load balancing across indexers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |