CN115757448B - Data processing method, device, electronic equipment and storage medium - Google Patents
Data processing method, device, electronic equipment and storage mediumInfo
- Publication number
- CN115757448B CN115757448B CN202211457910.3A CN202211457910A CN115757448B CN 115757448 B CN115757448 B CN 115757448B CN 202211457910 A CN202211457910 A CN 202211457910A CN 115757448 B CN115757448 B CN 115757448B
- Authority
- CN
- China
- Prior art keywords
- data
- target
- identifier
- aggregation
- pointer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明实施例公开了一种数据处理方法、装置、电子设备及存储介质，该方法包括：基于数据指针获取原始基础数据，并基于至少一个指标项从原始基础数据中确定出目标筛选数据；基于至少一个指标项中目标指标项的数据内容，确定至少一个聚合标识；基于至少一个聚合标识和相应的目标筛选数据，对数据库中的历史存储数据进行更新，得到目标数据。本发明实施例的技术方案，解决了现有技术中指标业务需求变更时，需要改动大量代码来实现功能,系统弹性差、可维护性差并且导致时间成本和人力成本高的问题，并且实现了根据聚合指针灵活的加载基础数据，进而达到了节约了人力及时间成本的技术效果。
The embodiment of the present invention discloses a data processing method, device, electronic device and storage medium, the method comprising: obtaining original basic data based on a data pointer, and determining target screening data from the original basic data based on at least one indicator item; determining at least one aggregation identifier based on the data content of the target indicator item in at least one indicator item; and updating the historical storage data in the database based on the at least one aggregation identifier and the corresponding target screening data to obtain the target data. The technical solution of the embodiment of the present invention solves the problem in the prior art that when the indicator business requirements change, a large amount of code needs to be modified to implement the function, the system has poor flexibility and maintainability, and leads to high time and labor costs, and realizes the flexible loading of basic data according to the aggregation pointer, thereby achieving the technical effect of saving labor and time costs.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a data processing method, a data processing device, electronic equipment and a storage medium.
Background
At present, after basic data corresponding to customer service consultation is acquired, an existing customer service monitoring system can determine an aggregation dimension according to service requirements, and then aggregate the aggregation dimension into a single index by executing a scheduling task.
The inventors found that when implementing the present technical solution based on the above-described mode, the following problems exist:
All data index aggregation logics are required to be customized and developed according to requirements, and the dimension solidification and the reusability are low. When the service requirement of the same index is changed, a large number of codes are required to be changed to realize the functions, and the system has poor elasticity and poor maintainability. The lack of a mechanism ensures flexible control of the aggregate computing task, resulting in the need for full computation for any errors and data changes in one computation, with the problems of wasted time and effort and high cost.
Disclosure of Invention
The invention provides a data processing method, a data processing device, electronic equipment and a storage medium, so as to achieve the technical effect of convenience in data processing.
In a first aspect, an embodiment of the present invention provides a data processing method, including:
acquiring original basic data based on a data pointer, and determining target screening data from the original basic data based on at least one index item;
Determining at least one aggregation identifier based on the data content of a target index item in the at least one index item;
And updating the historical storage data in the database based on the at least one aggregation identifier and the corresponding target screening data to obtain target data.
Further, the method for acquiring the original basic data based on the data pointers comprises the steps of determining a current data pointer from a pointer record table based on a received timing task, and acquiring the original basic data corresponding to the timing task from the basic data table based on the current data pointer.
Further, the determining of the target screening data from the original basic data based on the at least one index item comprises obtaining a target aggregation key consistent with the task identification of the timing task from an aggregation key table, wherein the target aggregation key comprises at least one index item, and the index item corresponds to a data dimension.
Further, the target screening data comprises data content corresponding to at least one target index item, the determining of at least one aggregation identifier based on the data content of the target index item in the at least one index item comprises determining at least one target index item with fixed data content from the at least one index item, determining at least one first identifier based on the data content corresponding to the at least one target index item, classifying the at least one first identifier, and determining at least one aggregation identifier, wherein each aggregation identifier corresponds to the at least one first identifier.
Further, updating the historical storage data in the database based on the at least one aggregation identifier and the corresponding target screening data to obtain target data, wherein the updating comprises determining the target data based on the target screening data corresponding to the current aggregation identifier and the historical storage data corresponding to the current aggregation identifier stored in the database if the database comprises the current aggregation identifier for each aggregation identifier.
Further, updating historical storage data in a database based on the at least one aggregation identifier and corresponding target screening data to obtain target data, wherein the method comprises the step of updating target screening data corresponding to the current aggregation identifier into the database as the target data if the current aggregation identifier is not included in the database for each aggregation identifier.
Further, the method further comprises the steps of determining a data pointer to be updated based on the obtained original basic data, and updating the data pointer to be updated in the pointer record table, so that when a timing task is received, the corresponding original basic data is obtained based on the data pointer in the pointer record table.
In a second aspect, an embodiment of the present invention further provides a data processing apparatus, including:
The target screening data determining module is used for acquiring original basic data based on the data pointer and determining target screening data from the original basic data based on at least one index item;
The aggregation identifier determining module is used for determining at least one aggregation identifier based on the data content of the target index item in the at least one index item;
And the target data determining module is used for updating the historical storage data in the database based on the at least one aggregation identifier and the corresponding target screening data to obtain target data.
In a third aspect, an embodiment of the present invention further provides an electronic device, including:
One or more processors;
Storage means for storing one or more programs,
The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data processing method according to any of the embodiments of the present invention.
In a fourth aspect, embodiments of the present invention also provide a storage medium containing computer-executable instructions which, when executed by a computer processor, are used to perform a data processing method according to any of the embodiments of the present invention.
According to the technical scheme, the original basic data are obtained based on the data pointers, the corresponding target screening data are determined from the original basic data according to the at least one index item, then at least one aggregation identifier can be determined according to the data content of the target index item in the at least one index item, after the aggregation identifier is determined, the historical storage data in the database are updated based on the at least one aggregation identifier and the corresponding target screening data, so that the target data are obtained, the problems that a large number of codes are required to be changed to realize functions when index service requirements are changed in the prior art are solved, and the problems of poor system elasticity, poor maintainability, high labor cost and high time cost are solved, and the technical effects of configuring various indexes, improving data processing efficiency and reducing cost are realized.
Drawings
In order to more clearly illustrate the technical solution of the exemplary embodiments of the present invention, a brief description is given below of the drawings required for describing the embodiments. It is obvious that the drawings presented are only drawings of some of the embodiments of the invention to be described, and not all the drawings, and that other drawings can be made according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of index generation according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a data acquisition method according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention;
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
Before the present technical solution is introduced, an application scenario may be illustrated. When the existing data processing method processes data, the data aggregation dimension is often aggregated into a single index based on a specific scheduling task after basic data stored in the system is acquired, for example, when indexes such as customer service satisfaction and the like are required to be aggregated, corresponding program codes are required to be developed and tested, so that the method can be put into use, and the problem of long development period and high time cost exists. Further, when the aggregation dimension changes, the corresponding program code needs to be redeveloped, which has the problems of higher cost and longer period.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention, where the embodiment is applicable to a case of determining at least one reference index based on a configuration item to obtain data corresponding to the corresponding reference index, and the method may be performed by a data processing apparatus, where the apparatus may be implemented in the form of software and/or hardware, and the hardware may be an electronic device for processing data, and the electronic device may perform the data processing method according to the embodiment of the present disclosure, so as to extract target screening data from basic data according to a task set by a user, and obtain corresponding target data according to an aggregation identifier.
It should be noted that, the apparatus for executing the data processing method provided by the embodiment of the present disclosure may be integrated in application software supporting data processing, where the application software may be installed in an electronic device, and optionally, the electronic device may be a mobile terminal or a PC terminal. The application software may be any type of software that needs data processing, and specific application software is not described here in detail, as long as the data processing can be implemented. Or integrated in the corresponding page, the user can realize data processing through the page integrated in the PC terminal.
As shown in fig. 1, the method includes:
s110, acquiring original basic data based on a data pointer, and determining target screening data from the original basic data based on at least one index item.
Where a data pointer may be understood as the location of data in a data table. The original basic data are all customer service data acquired based on a customer service monitoring system, and the customer service data comprise at least one of session data between users and customer service, comment data of the users on the customer service, evaluation star-level data of the users on the customers and satisfaction data. At the same time, the source of the original underlying data also includes at least one, which may be imported as different data. Optionally, the data entry dimension may include at least one of a channel dimension, an entry dimension, a skill set dimension, a customer service dimension, and a time period dimension. The channel dimension can be understood as a channel of a data source, for example, data from different application programs, pages and the like to jump to a designated page can be used as the channel dimension, the entry dimension can be understood as an entry into a functional interface, for example, when a user needs to consult customer service, the user can enter the customer service interface through different interfaces, the user can jump to the customer service interface through an interactive button in a My information page, the user can jump to the customer service interface through an interactive button in an after-sales service interface and the like, the mode of entering the customer service consultation interface can be used as an entry channel, and the customer service dimension can be a customer service category and the category can be two types of artificial customer service categories and intelligent customer service categories. Meanwhile, there is a certain difference in content of different artificial customer service or intelligent customer service, for example, the artificial customer service a service is mainly pre-sale information, and the artificial customer service B service is mainly after-sale information. For example, different customer service number ranges correspond to different tasks, customer service numbers 1-10 may be responsible for answering services, customer service numbers 11-20 may be responsible for after-sales services, etc., corresponding intelligent customer service may also have different directions of responsibility, and the time period dimension may be a statistical time period, for example, data is counted every half an hour, etc., and the time dimension mainly refers to the statistical time period. The index item is related to the target data to be obtained, for example, if the problem resolution of customer service is to be counted, the index item may include a problem resolution index. The target screening data may be understood as being based on statistical data determined from the raw base data according to the index items.
Specifically, the data identifier corresponding to the current pointer data is n, and at this time, the data identified as n from the data identifier may be considered as a starting point for acquiring the original base data. Furthermore, the original basic data can be filtered according to at least one index item, so that target screening data can be determined. For example, when the praise rate of customer service in a time period needs to be examined, taking ten to ten and a half in the morning as an example, data between ten and a half in the morning is extracted based on pointer data and is taken as original basic data, and since the praise rate index needs to be examined, the original basic data can be filtered based on the praise rate index and the customer service index, assuming that the extracted original basic data is 1000 pieces of data, and 100 pieces of data obtained by filtering the extracted original basic data based on the praise rate index and the customer service index are taken as target screening data.
Illustratively, as shown in FIG. 2, the sources of the bulk of the underlying data (raw underlying data) may be divided into different statistical dimensions. Correspondingly, data can be piled under the combined action of a plurality of conditions according to service requirements, and indexes are generated through logic operation. The logical operations may include summing, maximizing, minimizing, and the like. When different index service demands need to execute different grouping logics, the same bottom layer code is used, and different grouping logic demands can be realized only by configuring different aggregation keys.
On the basis of the scheme, the method for acquiring the original basic data based on the data pointers comprises the steps of determining a current data pointer from a pointer record table based on a received timing task, and acquiring the original basic data corresponding to the timing task from the basic data table based on the current data pointer.
The timing tasks can be tasks to be executed, which are preset by a user, and it can be understood that the user can set a plurality of timing tasks according to the requirements, the content to be executed of different timing tasks can be the same or different, the user can set according to the requirements, and the timing tasks can contain information such as task identification, data acquisition step length, data acquisition interval, task duration and the like. The pointer record table can be understood as a table for recording position information of the data pointer and history data record. The basic data table may be a table for recording basic data, and it should be noted that the basic data table may be a log of a system, or may be a storage space for recording all data processed by the system, or the like.
Specifically, according to the timing task set by the user, the position pointed by the data pointer at the moment is obtained from the pointer record table, and corresponding original basic data is obtained from the basic data table according to the current data pointer and the timing task set by the user.
For example, assuming that an operator needs to examine the data from ten a.m. to ten a.m., the operator may preset the duration of the task to ten a.m. and set the step length of the acquired data to 100 a.m. and set the interval of the acquired data to 3 minutes, when the time is ten a.m., the data is pulled every 3 minutes, and the data is pulled from the basic data every time in a manner of pulling 100 pieces of data until the end condition is satisfied, the specific method of pulling the data may refer to fig. 4, as shown in fig. 4, according to the data id corresponding to the data pointer, N pieces of data after the basic data table is loaded as the original basic data of the task statistics. The value of the acquired number N can be flexibly changed according to the service requirement, and if the number of the residual basic data is less than N, all the residual basic data are loaded. Further, if the purpose of pulling the data is to obtain the qualification rate of the customer service, the task identifier of the timing task may be set to be a qualification rate investigation, and 10 times of data needs to be obtained according to the task duration and the data obtaining interval, and 1000 pieces of data which are finally pulled may be obtained according to the data obtaining step length and the data obtaining times, and the 1000 pieces of data obtained by pulling are used as the original basic data corresponding to the timing task executed at this time.
According to the technical scheme provided by the embodiment of the disclosure, the current pointer data is determined from the pointer record table according to the received timing task, and the original basic data corresponding to the timing task is acquired, so that the obtained data and the task are ensured to correspond, and the data acquisition efficiency is improved.
On the basis of the scheme, the method for determining the target screening data from the original basic data based on the at least one index item comprises the steps of acquiring a target aggregation key consistent with the task identification of the timing task from an aggregation key table.
The aggregation key table may be a table for storing preset aggregation keys. The task identity may be an identity for identifying a currently timed task, and the task identity may also be used for characterizing what kind of index is needed to be used for screening the data. The target aggregation key comprises at least one index item, and the index item corresponds to the data dimension. The target aggregate key may be an aggregate key that is consistent in identity, obtained from an aggregate key table based on the task identity. It can be understood that different kinds of tasks need to acquire different target screening data, so that different kinds of tasks need to correspond to different aggregation keys, further, a user may need to examine multiple task items at the same time when examining a service, and therefore, the aggregation keys may include multiple index items.
Specifically, according to the task identifier contained in the timing task, a target aggregation key matched with the task identifier is obtained from the aggregation key table. For example, the aggregation key table may store the aggregation key and the task identifier correspondingly, that is, when the corresponding aggregation key needs to be searched according to the task identifier, the task identifier and the task identifier stored in the aggregation key table are matched one by one, and when the matching is successful, the corresponding aggregation key is determined to be the target aggregation key.
By way of example, with continued reference to fig. 3, a user may set corresponding parameters in a timed task according to need prior to processing data, which may include "consider satisfaction and valuation rate of customer service within ten to ten and a half am" and may set to acquire basic data once every three minutes, querying 100 pieces of data at a time. The initial position of the data pointer and the dimension of the original basic data to be queried can be obtained based on the set timing task, and then the corresponding original basic data is queried from the basic data table according to the query step length set by a user.
According to the technical scheme provided by the embodiment of the disclosure, the target aggregation key consistent with the task identifier of the timing task is obtained from the aggregation key table, so that a user can set the corresponding aggregation key according to the requirement, and the corresponding aggregation key is selected according to the timing task, the selection efficiency of the aggregation key is improved, and the selection accuracy is ensured.
S120, determining at least one aggregation identifier based on the data content of the target index item in the at least one index item.
The target index item may be an index item that needs to generate an aggregate identifier, and it is assumed that comprehensive data of customer service is to be examined, and because of different customer service, corresponding aggregate identifiers may be generated for different customer service, for example, a label of a customer may be used as the target index item, and then corresponding aggregate identifiers may be generated for "customer service 1", "customer service 2", "customer service 3", etc., where the number of aggregate identifiers may be one, two or more, for example, the number of aggregate identifiers may correspond to the number of customer service, and a method for generating aggregate identifiers may be an identifier obtained by processing customer service labels through a hash algorithm, or a corresponding aggregate identifier may be obtained through a snowflake algorithm. The data content may be basic data to be acquired in the target index item. Aggregate identity may be understood as identifying information generated by the content of different target index items.
Specifically, at least one aggregation identifier is determined based on the data content in the target index item in the at least one index item. For example, when an operator needs to examine two index items of the qualification rate and the problem solving rate of customer service, a corresponding aggregation identifier can be generated according to the qualification rate, the problem solving rate and a corresponding customer service identifier. Further, in a specific application, the operator may also select the number of customer services to be evaluated according to the requirement, for example, the "customer service 1" and the "customer service 3" need to be evaluated, and then the corresponding aggregation identifier may be determined according to the data content corresponding to the "customer service 1" and the "customer service 3".
On the basis of the technical scheme, the target screening data comprise data content corresponding to at least one target index item, and the determining of at least one aggregation identifier based on the data content of the target index item in the at least one index item comprises the steps of determining at least one target index item with fixed data content from the at least one index item, determining at least one first identifier based on the data content corresponding to the at least one target index item, classifying the at least one first identifier, and determining at least one aggregation identifier.
Wherein the first identifier is determined from the target index item. The target pointing target item is determined according to the data content of each target item. For example, the index item includes a customer service index item, and then the aggregate identifier, that is, the first identifier, may be determined according to specific information in the customer service index item. The number of aggregate identities may be one or more. It can be appreciated that when evaluating the data of one customer service, since the customer service may have original basic data of multiple dimensions, such as session data, evaluation data, etc., each aggregate identifier corresponds to at least one first identifier.
Specifically, it may be determined which index items have limited data content and change is known according to the data content corresponding to each index item, based on this, such index items may be used as target index items, and accordingly, a corresponding aggregation identifier may be generated according to the data content corresponding to the target index items. At this time, the number of the aggregation identifiers may be one or more, and at the same time, the aggregation identifiers may be the same or different, so that in order to perform overall processing on the data, the data with the same aggregation identifier may be categorized, that is, the data corresponding to one customer service or user is illustrated when the aggregation identifiers are the same. The categorization process is understood to be the association of the same data content of all aggregate identities with the same aggregate identity to obtain tree-like structured data.
According to the technical scheme provided by the embodiment of the disclosure, the corresponding first identifier is determined according to the data content in the target index item, and the first identifier is classified, so that the accuracy of data classification is ensured, and the data classification efficiency is improved.
S130, updating historical storage data in a database based on the at least one aggregation identifier and the corresponding target screening data to obtain target data.
The historical storage data may be data stored after the execution of the historical timing task. That is, the target screening data stored in the database before the current time is the history storage data. The target data may be understood as data that is stored in a database according to the need obtained after the current timing task is executed.
Specifically, historical storage data in a database is updated according to the at least one aggregation identifier and the corresponding target screening data, so that the corresponding target data is obtained. For example, after the obtained target screening data is subjected to aggregation treatment according to the aggregation identifier, corresponding index data is obtained, whether index data corresponding to the aggregation identifier exist or not is searched in a database according to the aggregation identifier, and if so, the data is updated, so that the corresponding target data is obtained.
Based on the scheme, the historical storage data in the database is updated based on the at least one aggregation identifier and the corresponding target screening data to obtain target data, wherein the method comprises the steps of determining the target data based on the target screening data corresponding to the current aggregation identifier and the historical storage data corresponding to the current aggregation identifier stored in the database if the database comprises the current aggregation identifier for each aggregation identifier.
Specifically, it can be described with reference to fig. 3, after statistics obtains the index data corresponding to the aggregation identifier, the database is queried according to the index data to determine whether there is history storage data corresponding to the aggregation identifier, if there is history storage data corresponding to the aggregation identifier in the database, the index data currently determined according to the aggregation identifier is updated and stored in the database.
Based on the scheme, the historical storage data in the database is updated based on the at least one aggregation identifier and the corresponding target screening data to obtain target data, wherein the method comprises the steps of updating target screening data corresponding to the current aggregation identifier into the database as the target data if the current aggregation identifier is not included in the database for each aggregation identifier.
Specifically, matching is performed in the database according to each aggregation identifier, if index data corresponding to the aggregation identifier does not exist in the database, new index data is created in the database based on the current aggregation identifier, index data obtained based on the current aggregation identifier and target screening data are correspondingly stored in the newly created data item, and then target data are obtained.
Illustratively, as shown in fig. 3, in the stage of aggregating data, the entity class of the multi-dimensional different data is distinguished by acquiring the current aggregation key, i.e. the aggregation task. The base data is grouped according to the multiple dimensions in the aggregate key. Dimensions typically include channels, portals, skill sets, customer service, time intervals, and the like. Traversing the grouped basic data, summing and summarizing satisfaction degree of different session evaluations, and generating the aggregation index data of the group. Finally, comparing the data with the existing aggregation index data in the database, and judging whether the index data exists or not. If the evaluation quantity is accumulated, the aggregation index data is updated.
On the basis of the scheme, the method further comprises the steps of determining a data pointer to be updated based on the obtained original basic data, and updating the data pointer to be updated in the pointer record table, so that when a timing task is received, the corresponding original basic data is obtained based on the data pointer in the pointer record table.
The data pointer to be updated may be a position in the base data table where the data pointer exists after the original base data is acquired.
Specifically, after data is collected in the basic data table according to the data collection step length set by the user, the data pointer after the data is collected is updated in the pointer record table and is used for recording the pointing position of the data pointer at the moment, and then when the data is captured next time, the original basic data can be accurately obtained according to the recorded content in the pointer record table. That is, when a timing task is received next time, the current pointer position may be acquired from the pointer record table according to the received timing task, and the corresponding data may be called.
In the example, as shown in fig. 3, in the pointer updating stage, after each time of aggregation index data generation, the position of the last piece of basic data loaded according to the current task in the basic data table is recorded in the basic data table, and the position of the last piece of current basic data in the basic data table is used as the position of the current data pointer, so that the data in the basic data table can be acquired when the task is executed next time. Furthermore, when the pointer record table records the position pointed by the data pointer, the aggregation index of this time can be correspondingly stored in the pointer record table, and then when the aggregation index data error is found, the original error data can be covered by the index callback.
According to the technical scheme, the original basic data are obtained based on the data pointers, the corresponding target screening data are determined from the original basic data according to the at least one index item, then at least one aggregation identifier can be determined according to the data content of the target index item in the at least one index item, after the aggregation identifier is determined, the historical storage data in the database are updated based on the at least one aggregation identifier and the corresponding target screening data, so that the target data are obtained, the problems that a large number of codes are required to be changed to realize functions when index service requirements are changed in the prior art are solved, and the problems of poor system elasticity, poor maintainability, high labor cost and high time cost are solved, and the technical effects of configuring various indexes, improving data processing efficiency and reducing cost are realized.
Fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention, and as shown in fig. 5, the apparatus includes a target screening data determining module 510, an aggregate identifier determining module 520, and a target data determining module 530.
The target screening data determining module 510 is configured to obtain original basic data based on a data pointer, and determine target screening data from the original basic data based on at least one index item;
an aggregate identifier determination module 520, configured to determine at least one aggregate identifier based on the data content of the target index item in the at least one index item;
The target data determining module 530 is configured to update the historical storage data in the database based on the at least one aggregation identifier and the corresponding target screening data, so as to obtain target data.
On the basis of the above technical solution, the target screening data determining module further includes:
A data pointer determining unit for determining a current data pointer from the pointer record table based on the received timing task;
And the original basic data determining unit is used for acquiring the original basic data corresponding to the timing task from the basic data table based on the current data pointer.
On the basis of the technical scheme, the target screening data determining module comprises:
and the aggregation key determining unit is used for acquiring a target aggregation key consistent with the task identifier of the timing task from the aggregation key table, wherein the target aggregation key comprises at least one index item, and the index item corresponds to the data dimension.
On the basis of the technical scheme, the aggregation identifier determining module is specifically used for determining at least one target index item with fixed data content from the at least one index item, determining at least one first identifier based on the data content corresponding to the at least one target index item, classifying the at least one first identifier, and determining at least one aggregation identifier, wherein each aggregation identifier corresponds to the at least one first identifier.
On the basis of the technical scheme, the target data determining module comprises:
And the target data updating unit is used for determining target data based on target screening data corresponding to the current aggregation identifier and historical storage data corresponding to the current aggregation identifier stored in the database if the database comprises the current aggregation identifier.
On the basis of the technical scheme, the target data determining module further comprises:
And the target data creation unit is used for updating target screening data corresponding to the current aggregation identifier into the database as the target data if the current aggregation identifier is not included in the database for each aggregation identifier.
On the basis of the technical scheme, the device further comprises:
And the data pointer updating module is used for determining a data pointer to be updated based on the obtained original basic data, and updating the data pointer to be updated in the pointer record table so as to obtain corresponding original basic data based on the data pointer in the pointer record table when a timing task is received.
According to the technical scheme, the original basic data are obtained based on the data pointers, the corresponding target screening data are determined from the original basic data according to the at least one index item, then at least one aggregation identifier can be determined according to the data content of the target index item in the at least one index item, after the aggregation identifier is determined, the historical storage data in the database are updated based on the at least one aggregation identifier and the corresponding target screening data, so that the target data are obtained, the problems that a large number of codes are required to be changed to realize functions when index service requirements are changed in the prior art are solved, and the problems of poor system elasticity, poor maintainability, high labor cost and high time cost are solved, and the technical effects of configuring various indexes, improving data processing efficiency and reducing cost are realized.
The data processing device provided by the embodiment of the invention can execute the data processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
It should be noted that the above-mentioned units and modules included in the apparatus are only divided according to the functional logic, but not limited to the above-mentioned division, so long as the corresponding functions can be implemented, and the specific names of the functional units are only used for distinguishing from each other, and are not used for limiting the protection scope of the embodiments of the present invention.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. Fig. 6 shows a block diagram of an exemplary electronic device 60 suitable for use in implementing the embodiments of the invention. The electronic device 60 shown in fig. 6 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.
As shown in fig. 6, the electronic device 60 is in the form of a general purpose computing device. The components of electronic device 60 may include, but are not limited to, one or more processors or processing units 601, a system memory 602, and a bus 603 that connects the different system components (including system memory 602 and processing units 601).
Bus 603 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
Electronic device 60 typically includes many types of computer system readable media. Such media can be any available media that is accessible by electronic device 60 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 602 may include computer system readable media in the form of volatile memory such as Random Access Memory (RAM) 604 and/or cache memory 605. Electronic device 60 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 606 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 6, commonly referred to as a "hard disk drive"). Although not shown in fig. 6, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In such cases, each drive may be coupled to bus 603 through one or more data medium interfaces. Memory 602 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of embodiments of the invention.
A program/utility 608 having a set (at least one) of program modules 607 may be stored in, for example, memory 602, such program modules 607 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 607 generally perform the functions and/or methods of the described embodiments of the invention.
The electronic device 60 may also communicate with one or more external devices 609 (e.g., keyboard, pointing device, display 610, etc.), one or more devices that enable a user to interact with the electronic device 60, and/or any device (e.g., network card, modem, etc.) that enables the electronic device 60 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 611. Also, the electronic device 60 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through a network adapter 612. As shown, the network adapter 612 communicates with other modules of the electronic device 60 over the bus 603. It should be appreciated that although not shown in FIG. 6, other hardware and/or software modules may be used in connection with electronic device 60, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
The processing unit 601 executes various functional applications and data processing by running programs stored in the system memory 602, for example, implementing the data processing method provided by the embodiment of the present invention.
Embodiments of the present invention also provide a storage medium containing computer-executable instructions for performing a data processing method when executed by a computer processor.
The method comprises the following steps:
acquiring original basic data based on a data pointer, and determining target screening data from the original basic data based on at least one index item;
Determining at least one aggregation identifier based on the data content of a target index item in the at least one index item;
And updating the historical storage data in the database based on the at least one aggregation identifier and the corresponding target screening data to obtain target data.
The computer storage media of embodiments of the invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for embodiments of the present invention may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211457910.3A CN115757448B (en) | 2022-11-21 | 2022-11-21 | Data processing method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211457910.3A CN115757448B (en) | 2022-11-21 | 2022-11-21 | Data processing method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115757448A CN115757448A (en) | 2023-03-07 |
| CN115757448B true CN115757448B (en) | 2025-09-16 |
Family
ID=85333885
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211457910.3A Active CN115757448B (en) | 2022-11-21 | 2022-11-21 | Data processing method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115757448B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817344A (en) * | 2021-01-18 | 2022-07-29 | 腾讯科技（深圳）有限公司 | A data acquisition method and device |
| CN115033273A (en) * | 2022-06-16 | 2022-09-09 | 北京京东振世信息技术有限公司 | Data processing method, device, equipment and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112199269B (en) * | 2019-07-08 | 2023-10-20 | 腾讯科技（深圳）有限公司 | Data processing method and related device |
| CN111460023B (en) * | 2020-04-29 | 2023-06-09 | 上海东普信息科技有限公司 | Method, device, equipment and storage medium for processing service data based on elastic search |
| CN114138785B (en) * | 2021-11-30 | 2024-07-30 | 中国平安财产保险股份有限公司 | Data retrieval method, device, equipment and storage medium suitable for large data volume |
| CN115080272A (en) * | 2022-06-28 | 2022-09-20 | 上海众源网络有限公司 | A business data processing method, electronic device and storage medium |
-
2022
- 2022-11-21 CN CN202211457910.3A patent/CN115757448B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817344A (en) * | 2021-01-18 | 2022-07-29 | 腾讯科技（深圳）有限公司 | A data acquisition method and device |
| CN115033273A (en) * | 2022-06-16 | 2022-09-09 | 北京京东振世信息技术有限公司 | Data processing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115757448A (en) | 2023-03-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10713589B1 (en) | Consistent sort-based record-level shuffling of machine learning data | |
| US11182691B1 (en) | Category-based sampling of machine learning data | |
| US7539753B2 (en) | Methods and apparatus for functional model-based data provenance in stream processing environments | |
| CN107729376B (en) | Insurance data auditing method and device, computer equipment and storage medium | |
| CN111045849A (en) | Identification method, device, server and storage medium for checking abnormal cause | |
| CN112035159B (en) | Configuration method, device, equipment and storage medium of audit model | |
| CN111666201A (en) | Regression testing method, device, medium and electronic equipment | |
| CN114579673B (en) | User portrait unique identification generation method | |
| CN113377637A (en) | Performance capacity diagnostic method and device | |
| CN112346952A (en) | Method, equipment and electronic equipment for regression testing of test cases | |
| CN110990445A (en) | Data processing method, device, equipment and medium | |
| CN113220530B (en) | Data quality monitoring method and platform | |
| CN114650167A (en) | Abnormity detection method, device, equipment and computer readable storage medium | |
| CN110955709A (en) | Data processing method and device and electronic equipment | |
| CN116383193A (en) | Data management method and device, electronic equipment and storage medium | |
| CN115345600B (en) | RPA flow generation method and device | |
| CN111414410A (en) | Data processing method, device, equipment and storage medium | |
| CN111784176A (en) | Data processing method, device, server and medium | |
| CN115757448B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN106940710B (en) | Information pushing method and device | |
| US10496524B2 (en) | Separating test coverage in software processes using shared memory | |
| CN118069491A (en) | Incremental coverage determination method, apparatus, device, storage medium, and program product | |
| CN112000619A (en) | Time sequence data storage method, device, equipment and readable storage medium | |
| CN115640325A (en) | Content recommendation method and device, readable medium and electronic equipment | |
| CN111222739A (en) | Task allocation method and task allocation system of nuclear power station |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |