CN111159213A - Data query method, device, system and storage medium - Google Patents
Data query method, device, system and storage medium Download PDFInfo
- Publication number
- CN111159213A CN111159213A CN201811317310.0A CN201811317310A CN111159213A CN 111159213 A CN111159213 A CN 111159213A CN 201811317310 A CN201811317310 A CN 201811317310A CN 111159213 A CN111159213 A CN 111159213A
- Authority
- CN
- China
- Prior art keywords
- data
- user
- data query
- processing platform
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000003860 storage Methods 0.000 title abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 64
- 238000007781 pre-processing Methods 0.000 claims description 40
- 230000002776 aggregation Effects 0.000 claims description 12
- 238000004220 aggregation Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 4
- 230000004044 response Effects 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 10
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data query method, a data query device, a data query system and a storage medium, which are used for improving data query efficiency and reducing data query response delay under a big data application scene. The data query system comprises a data query client, a first data processing platform and a second data processing platform, wherein the first data processing platform stores user data, and the second data processing platform stores user attribute information; the data query client is used for receiving a first data query request, and the data query request carries a first data query condition; searching user data meeting the first data query condition from the first data processing platform according to the first data query condition; searching user attribute information corresponding to the user identification from a second data processing platform according to the user identification carried in the inquired user data; and determining a data query result according to the searched user data and the corresponding user attribute information.
Description
Technical Field
The present invention relates to the field of database technologies, and in particular, to a data query method, apparatus, system, and storage medium.
Background
By big data is meant data that is of such a magnitude that the data involved is too big to be processed in a reasonable time by the mainstream software. When mass data is faced, the traditional relational database has the advantages of supporting integrity constraint, supporting affairs and the like, but the traditional relational database is not satisfactory in large-scale mass data.
Therefore, how to improve the efficiency of data query in a big data application scenario and reduce the delay of data query response becomes one of the technical problems to be solved urgently in the prior art.
Disclosure of Invention
The embodiment of the invention provides a data query method, a data query device, a data query system and a storage medium, which are used for improving data query efficiency and reducing data query response delay in a big data application scene.
In a first aspect, a data query system is provided, where the data query system includes a data query client, a first data processing platform, and a second data processing platform, where the first data processing platform stores user data, and the second data processing platform stores user attribute information; and
the data query client is used for receiving a first data query request, and the data query request carries a first data query condition; searching user data meeting the first data query condition from a first data processing platform according to the first data query condition; searching user attribute information corresponding to the user identification from the second data processing platform according to the user identification carried in the inquired user data; and determining a data query result according to the searched user data and the corresponding user attribute information.
Optionally, the first data processing platform is configured to obtain the user data according to the following procedure: the first data processing platform sends a second data query request to a basic database, wherein the second data query request carries a second data query condition; and receiving and storing the user data which is returned by the basic database and searched according to the second query condition.
Optionally, the first data processing platform further uses a received data preprocessing rule to preprocess the user data according to the received data preprocessing rule, where the data preprocessing rule includes at least one of the following: data grouping rules, data ordering rules, and data aggregation rules.
In a second aspect, a data query method is provided, including:
receiving a data query request, wherein the data query request carries a data query condition;
searching user data meeting the first data query condition from a first data processing platform according to the first data query condition; and
according to the user identification carried in the inquired user data, searching user attribute information corresponding to the user identification from the second data processing platform;
and determining a data query result according to the searched user data and the corresponding user attribute information.
In a third aspect, a data query method is provided, including:
sending a data query request to a basic database, wherein the data query request carries a data query condition;
and receiving and storing the user data which is returned by the basic database and searched according to the query condition.
Optionally, the data query method may further include:
receiving a data pre-processing rule, the data pre-processing rule comprising at least one of: data grouping rules, data sorting rules and data aggregation rules;
and preprocessing the user data according to the received data preprocessing rule.
In a fourth aspect, there is provided a data query apparatus, including:
the device comprises a receiving unit, a sending unit and a receiving unit, wherein the receiving unit is used for receiving a data query request which carries a data query condition;
the first searching unit is used for searching the user data meeting the first data query condition from a first data processing platform according to the first data query condition;
the second searching unit is used for searching user attribute information corresponding to the user identification from the second data processing platform according to the user identification carried in the inquired user data;
and the determining unit is used for determining a data query result according to the user data searched by the first searching unit and the corresponding user attribute information.
In a sixth aspect, there is provided another data query apparatus including:
the sending unit is used for sending a data query request to a basic database, wherein the data query request carries a data query condition;
and the first receiving unit is used for receiving and storing the user data which is returned by the basic database and searched according to the query condition.
Optionally, the second data query device further includes:
a second receiving unit, configured to receive a data preprocessing rule, where the data preprocessing rule includes at least one of: data grouping rules, data sorting rules and data aggregation rules;
and the preprocessing unit is used for preprocessing the user data according to the received data preprocessing rule.
In a seventh aspect, a computing device is provided, which includes at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor executes the steps of any of the above data query methods.
In an eighth aspect, there is provided a computer readable medium storing a computer program executable by a computing device, the program, when executed on the computing device, causing the computing device to perform the steps of any of the data query methods described above.
According to the data query method, the device, the system and the storage medium provided by the embodiment of the invention, the user data and the user attribute information are respectively stored in different databases, and the data storage quantity of the stored user data is reduced, so that the user data meeting the conditions can be quickly found according to the data query conditions when data query is carried out, further, the user attribute information is searched from the database for storing the user attribute information according to the user identification carried in the user data, and the user data searched from the two databases and the corresponding user attribute information are combined and provided for the user, so that the data query efficiency is improved, and the data query response delay is reduced.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a schematic flow chart of a first data query method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of an implementation of a second data query method according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a first data query device according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a second data query device according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a hardware structure of a computing apparatus for implementing the data query method according to the present invention.
Detailed Description
In order to improve data query efficiency and reduce data query response time delay in a big data application scene, embodiments of the present invention provide a data query method, apparatus, system and storage medium.
It should be noted that the terminal device in the present invention may be a terminal device such as a Personal Computer (PC), a tablet PC, a Personal Digital Assistant (PDA), a Personal Communication Service (PCs) phone, a notebook, a mobile phone, etc., or a computer having a mobile terminal, for example, a portable, pocket, hand-held, computer-embedded or vehicle-mounted mobile device, which can provide voice and/or data connectivity to a user, and exchange voice and/or data with a wireless access network.
Furthermore, the terms "first," "second," and the like in the description and in the claims, and in the drawings, in the embodiments of the invention are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein.
Reference herein to "a plurality or a number" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
The preferred embodiments of the present invention will be described below with reference to the accompanying drawings of the specification, it being understood that the preferred embodiments described herein are merely for illustrating and explaining the present invention, and are not intended to limit the present invention, and that the embodiments and features of the embodiments in the present invention may be combined with each other without conflict.
In a big data application scenario, the data volume of user data is large and is continuously updated, and user attribute information, such as user contact information, user identity information, user age, gender, bank account and the like, is small and stable in data volume, and the number of fields is large, but the influence on the data analysis result is small. If the user attribute information and the user data are not distinguished and are simultaneously stored in the same database, the memory of the database is overlarge, which is not beneficial to improving the data query efficiency and the subsequent data development of multiple modules.
In view of this, an embodiment of the present invention provides a data query system, which includes a data query client, a first data processing platform, and a second data processing platform, where the first data processing platform stores user data, and the second data processing platform stores user attribute information.
The data query client is used for receiving a first data query request, and the data query request carries a first data query condition; searching user data meeting the first data query condition from a first data processing platform according to the first data query condition; searching user attribute information corresponding to the user identification from the second data processing platform according to the user identification carried in the inquired user data; and determining a data query result according to the searched user data and the corresponding user attribute information.
In specific implementation, when a user needs to query data, the data processing client in the foreground can submit a data query request, wherein the data query request carries a data query condition, after receiving the data query request submitted by the user, the data processing client firstly queries all user data meeting the data query condition from the first data processing platform according to the data query condition carried in the data query request, and then queries corresponding user attribute information from the second data processing platform according to a user identifier contained in the queried user data. And merging the data inquired from the two data processing platforms according to the user identification and displaying the merged data to the user.
In the process, the user data and the user attribute information are stored separately, so that the memory of a data processing platform for storing the user data is reduced, the data query efficiency is improved, and the data query response delay is reduced
The first data processing platform is used for obtaining user data according to the following procedures: the first data processing platform sends a second data query request to the basic database, wherein the second data query request carries a second data query condition; and receiving and storing the user data which is returned by the basic database and searched according to the second query condition. All user data and user attribute information are stored in the basic database.
Further, in order to improve the data query efficiency, the first data processing platform may further receive a data preprocessing rule after obtaining the user data, and preprocess the user data according to the received data preprocessing rule, where the data preprocessing rule includes at least one of: data grouping rules, data ordering rules, and data aggregation rules.
In specific implementation, the first data processing platform preprocesses the searched user data according to the service function requirement provided by the user.
In one embodiment, the first data processing platform may be Kylin and the second data processing platform may be Redis.
For better understanding of the embodiment of the present invention, the following describes an implementation flow of the data query method provided in the embodiment of the present invention with reference to a user tax data processing flow.
The relevant field information of the user tax data is continuously updated, the user attribute information of the taxpayer is stable, the change is small, the number of the fields is large, and the influence on the analysis result of the tax big data is small. If all the user attribute information and the tax data of the taxpayer are stored in the Kylin without distinguishing, the memory occupied by the Kylin is overlarge, and the efficiency of data query is improved.
The Kylin can preprocess the tax data and store the preprocessing result according to the service function requirement. For example, the sum, average, etc. between different field data are pre-calculated, so that the user can greatly reduce the query response delay when querying.
In addition, Redis is used for storing user attribute information of the taxpayer, data are stored in a memory by the Redis, and the type of the stored data is similar to the key value pair of Hashmap, so that the query time is particularly fast.
In addition, in the specific implementation, in order to improve the accuracy of the query result, in the embodiment of the present invention, the query and filter fields may also be stored in Kylin, so as to implement an accurate screening result set.
The data stored in Kylin and Redis are connected through user identification, for example, for tax data, the data can be connected through taxpayers' identification and name.
The following describes an implementation process of the embodiment of the present invention with reference to a processing flow of tax data, which may include the following steps:
step one, establishing a data warehouse at hive, and storing user attribute information of taxpayers.
hive is a data warehouse tool based on Hadoop, can map structured data files into a database table, provides a simple sql query function, and can convert sql statements into MapReduce tasks for operation. Hive is a basic database that can be referred to in the embodiments of the present invention.
And step two, storing the tax data in hive.
And step three, exporting the user attribute information of the taxpayer to Redis from the hive, and storing the user attribute information according to a key-value (key value pair) format.
In specific implementation, a data query request is sent to the hive, wherein the data query request carries a data query condition, and the user attribute information of the taxpayer is screened out from the hive according to the data query condition.
And step four, extracting tax data from hive and synchronizing the tax data to Kylin.
In specific implementation, a data query request is sent to the hive, wherein the data query request carries a data query condition, and tax data meeting the condition is screened from the hive according to the data query condition.
Further, the user may submit a data preprocessing rule to the Kylin according to the business function requirement, where the data preprocessing rule includes at least one of the following: data grouping rules, data ordering rules, and data aggregation rules. And the Kylin preprocesses the tax data according to the received data preprocessing rule.
Specifically, in Kylin, a model is created, then a cube is created, wherein the cube has a fact table (fact table), dimensions and measures, the dimensions and the measures respectively represent a data grouping rule and a data aggregation rule, the dimensions and the measures are determined, and the preprocessed tax data is obtained by performing permutation, combination, pre-calculation and the like of various dimensions according to the determined dimensions and measures. Therefore, data dimensionality in the cube can be reduced through data aggregation, so that data query efficiency can be improved, and occupied memory space is reduced.
And step five, receiving a data query request submitted by a user, wherein the data query request carries data query conditions.
And step six, searching user data meeting the conditions from the Kylin according to the data query conditions carried in the data query request submitted by the user.
And step seven, searching corresponding user attribute information from the Redis according to the user identification contained in the searched user data.
And step eight, integrating the query results of the step six and the step seven for displaying.
The data query method provided by the embodiment of the invention stores the user data and the user attribute information in different databases respectively, and reduces the data storage amount of the stored user data, so that the user data meeting the conditions can be quickly searched according to the data query conditions when data query is carried out, further, the user attribute information is searched from the database storing the user attribute information according to the user identification carried in the user data, and the user data searched from the two databases and the corresponding user attribute information are combined and provided for the user, thereby improving the data query efficiency and reducing the data query response delay.
Based on the same inventive concept, the embodiment of the invention also provides a data query method and a data query device, and because the principle of solving the problems of the method and the device is similar to that of a data query system, the implementation of the method and the device can be referred to the implementation of the method, and repeated parts are not described again.
As shown in fig. 1, which is a schematic flow chart of an implementation of a first data query method provided in an embodiment of the present invention, the method includes the following steps:
s11, receiving a data query request, wherein the data query request carries data query conditions.
And S12, searching the user data meeting the first data query condition from the first data processing platform according to the first data query condition.
S13, according to the user identification carried in the inquired user data, searching the user attribute information corresponding to the user identification from the second data processing platform.
And S14, determining a data query result according to the searched user data and the corresponding user attribute information.
As shown in fig. 2, which is a schematic flow chart of an implementation of a second data query method according to an embodiment of the present invention, the method may include the following steps:
s21, sending a data query request to the basic database, wherein the data query request carries data query conditions.
And S22, receiving the user data which is returned by the basic database and searched according to the query condition, and storing the user data.
Optionally, the second data query method may further include the following steps:
the method comprises the following steps of firstly, receiving a data preprocessing rule, wherein the data preprocessing rule comprises at least one of the following items: data grouping rules, data sorting rules and data aggregation rules;
and secondly, preprocessing the user data according to the received data preprocessing rule.
As shown in fig. 3, which is a schematic structural diagram of a first data query apparatus provided in an embodiment of the present invention, including:
a receiving unit 31, configured to receive a data query request, where the data query request carries a data query condition;
a first searching unit 32, configured to search, according to the first data query condition, user data that meets the first data query condition from a first data processing platform;
the second searching unit 33 is configured to search, according to a user identifier carried in the queried user data, user attribute information corresponding to the user identifier from the second data processing platform;
the determining unit 34 is configured to determine a data query result according to the user data found by the first searching unit and the user attribute information corresponding to the user data.
As shown in fig. 4, a schematic structural diagram of a second data query device according to an embodiment of the present invention includes:
a sending unit 41, configured to send a data query request to a basic database, where the data query request carries a data query condition;
and the first receiving unit 42 is configured to receive and store the user data that is returned by the basic database and found according to the query condition.
Optionally, the second data query device provided in the embodiment of the present invention may further include:
a second receiving unit, configured to receive a data preprocessing rule, where the data preprocessing rule includes at least one of: data grouping rules, data sorting rules and data aggregation rules;
and the preprocessing unit is used for preprocessing the user data according to the received data preprocessing rule.
For convenience of description, the above parts are separately described as modules (or units) according to functional division. Of course, the functionality of the various modules (or units) may be implemented in the same or in multiple pieces of software or hardware in practicing the invention.
Having described the data query method and apparatus according to an exemplary embodiment of the present invention, a computing apparatus according to another exemplary embodiment of the present invention is described next.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
In some possible embodiments, a computing device according to the present invention may include at least one processor, and at least one memory. Wherein the memory stores program code which, when executed by the processor, causes the processor to perform the steps of the data query method according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute step S11 shown in fig. 1, receive a data query request, where the data query request carries data query conditions, and step S12, search for user data meeting the first data query conditions from the first data processing platform according to the first data query conditions; step S13, according to the user identification carried in the inquired user data, searching the user attribute information corresponding to the user identification from the second data processing platform; and step S14, determining a data query result according to the searched user data and the corresponding user attribute information. Or, step S21 shown in fig. 2 is executed, a data query request is sent to a basic database, where the data query request carries a data query condition, and step S22 is executed, where user data searched according to the second query condition and returned by the basic database is received and stored.
The computing device 50 according to this embodiment of the invention is described below with reference to fig. 5. The computing device 50 shown in fig. 5 is only an example and should not bring any limitations to the functionality or scope of use of embodiments of the present invention.
As shown in fig. 5, the computing apparatus 50 is in the form of a general purpose computing device. Components of computing device 50 may include, but are not limited to: the at least one processor 51, the at least one memory 52, and a bus 53 connecting the various system components (including the memory 52 and the processor 51).
Bus 53 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.
The memory 52 may include readable media in the form of volatile memory, such as Random Access Memory (RAM)521 and/or cache memory 522, and may further include Read Only Memory (ROM) 523.
The computing apparatus 50 may also communicate with one or more external devices 54 (e.g., keyboard, pointing device, etc.), with one or more devices that enable a user to interact with the computing apparatus 50, and/or with any devices (e.g., router, modem, etc.) that enable the computing apparatus 50 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 55. Also, the computing device 50 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) through the network adapter 56. As shown, the network adapter 56 communicates with other modules for the computing device 50 over the bus 53. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with computing device 50, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
In some possible embodiments, various aspects of the data query method provided by the present invention may also be implemented in the form of a program product, which includes program code for causing a computer device to execute the steps in the data query method according to various exemplary embodiments of the present invention described above in this specification when the program product runs on the computer device, for example, the computer device may execute step S11 shown in fig. 1, receive a data query request, where the data query request carries a data query condition, and step S12, search for user data meeting the first data query condition from a first data processing platform according to the first data query condition; step S13, according to the user identification carried in the inquired user data, searching the user attribute information corresponding to the user identification from the second data processing platform; and step S14, determining a data query result according to the searched user data and the corresponding user attribute information. Or, step S21 shown in fig. 2 is executed, a data query request is sent to a basic database, where the data query request carries a data query condition, and step S22 is executed, where user data searched according to the second query condition and returned by the basic database is received and stored.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The program product for data query of embodiments of the present invention may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a computing device. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device over any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., over the internet using an internet service provider).
It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functions of two or more of the units described above may be embodied in one unit, according to embodiments of the invention. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.
Moreover, while the operations of the method of the invention are depicted in the drawings in a particular order, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (11)
1. A data query system is characterized by comprising a data query client, a first data processing platform and a second data processing platform, wherein user data are stored in the first data processing platform, and user attribute information is stored in the second data processing platform; and
the data query client is used for receiving a first data query request, and the data query request carries a first data query condition; searching user data meeting the first data query condition from a first data processing platform according to the first data query condition; searching user attribute information corresponding to the user identification from the second data processing platform according to the user identification carried in the inquired user data; and determining a data query result according to the searched user data and the corresponding user attribute information.
2. The system of claim 1,
the first data processing platform is configured to obtain the user data according to the following procedures: the first data processing platform sends a second data query request to a basic database, wherein the second data query request carries a second data query condition; and receiving and storing the user data which is returned by the basic database and searched according to the second query condition.
3. The system of claim 2,
the first data processing platform is further configured to pre-process the user data according to a received data pre-processing rule by using a received data pre-processing rule, where the data pre-processing rule includes at least one of: data grouping rules, data ordering rules, and data aggregation rules.
4. A method for querying data, comprising:
receiving a data query request, wherein the data query request carries a data query condition;
searching user data meeting the first data query condition from a first data processing platform according to the first data query condition; and
according to the user identification carried in the inquired user data, searching user attribute information corresponding to the user identification from the second data processing platform;
and determining a data query result according to the searched user data and the corresponding user attribute information.
5. A method for querying data, comprising:
sending a data query request to a basic database, wherein the data query request carries a data query condition;
and receiving and storing the user data which is returned by the basic database and searched according to the query condition.
6. The method of claim 5, further comprising:
receiving a data pre-processing rule, the data pre-processing rule comprising at least one of: data grouping rules, data sorting rules and data aggregation rules;
and preprocessing the user data according to the received data preprocessing rule.
7. A data query apparatus, comprising:
the device comprises a receiving unit, a sending unit and a receiving unit, wherein the receiving unit is used for receiving a data query request which carries a data query condition;
the first searching unit is used for searching the user data meeting the first data query condition from a first data processing platform according to the first data query condition;
the second searching unit is used for searching user attribute information corresponding to the user identification from the second data processing platform according to the user identification carried in the inquired user data;
and the determining unit is used for determining a data query result according to the user data searched by the first searching unit and the corresponding user attribute information.
8. A data query apparatus, comprising:
the sending unit is used for sending a data query request to a basic database, wherein the data query request carries a data query condition;
and the first receiving unit is used for receiving and storing the user data which is returned by the basic database and searched according to the query condition.
9. The apparatus of claim 8, further comprising:
a second receiving unit, configured to receive a data preprocessing rule, where the data preprocessing rule includes at least one of: data grouping rules, data sorting rules and data aggregation rules;
and the preprocessing unit is used for preprocessing the user data according to the received data preprocessing rule.
10. A computing device comprising at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method of claim 4, 5 or 6.
11. A computer-readable medium, in which a computer program is stored which is executable by a computing means, when the program is run on the computing means, causing the computing means to perform the steps of the method of any one of claims 4, 5 or 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811317310.0A CN111159213A (en) | 2018-11-07 | 2018-11-07 | Data query method, device, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811317310.0A CN111159213A (en) | 2018-11-07 | 2018-11-07 | Data query method, device, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111159213A true CN111159213A (en) | 2020-05-15 |
Family
ID=70555276
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811317310.0A Pending CN111159213A (en) | 2018-11-07 | 2018-11-07 | Data query method, device, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111159213A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111694708A (en) * | 2020-05-28 | 2020-09-22 | 新浪网技术(中国)有限公司 | Data query method and device, electronic equipment and storage medium |
| CN111767312A (en) * | 2020-06-12 | 2020-10-13 | 上海传英信息技术有限公司 | A data query method, server and computer-readable storage medium |
| CN112100502A (en) * | 2020-09-24 | 2020-12-18 | 北京百度网讯科技有限公司 | Information query method, apparatus, computing device and medium for database |
| CN112580947A (en) * | 2020-12-08 | 2021-03-30 | 北京索为云网科技有限公司 | Information processing method, device and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103778239A (en) * | 2014-01-28 | 2014-05-07 | 北京京东尚科信息技术有限公司 | Multi-database data management method and system |
| US20140358963A1 (en) * | 2013-06-03 | 2014-12-04 | International Business Machines Corporation | Information retrieval from a database system |
| CN104462362A (en) * | 2014-12-08 | 2015-03-25 | 曙光信息产业(北京)有限公司 | Data storage, query and loading methods and devices |
| CN105718455A (en) * | 2014-12-01 | 2016-06-29 | 阿里巴巴集团控股有限公司 | Data query method and apparatus |
| CN107391506A (en) * | 2016-05-16 | 2017-11-24 | 华为软件技术有限公司 | Method and apparatus for inquiring about data |
| CN108460041A (en) * | 2017-02-20 | 2018-08-28 | 腾讯科技（深圳）有限公司 | The treating method and apparatus of data |
-
2018
- 2018-11-07 CN CN201811317310.0A patent/CN111159213A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140358963A1 (en) * | 2013-06-03 | 2014-12-04 | International Business Machines Corporation | Information retrieval from a database system |
| CN103778239A (en) * | 2014-01-28 | 2014-05-07 | 北京京东尚科信息技术有限公司 | Multi-database data management method and system |
| CN105718455A (en) * | 2014-12-01 | 2016-06-29 | 阿里巴巴集团控股有限公司 | Data query method and apparatus |
| CN104462362A (en) * | 2014-12-08 | 2015-03-25 | 曙光信息产业(北京)有限公司 | Data storage, query and loading methods and devices |
| CN107391506A (en) * | 2016-05-16 | 2017-11-24 | 华为软件技术有限公司 | Method and apparatus for inquiring about data |
| CN108460041A (en) * | 2017-02-20 | 2018-08-28 | 腾讯科技（深圳）有限公司 | The treating method and apparatus of data |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111694708A (en) * | 2020-05-28 | 2020-09-22 | 新浪网技术(中国)有限公司 | Data query method and device, electronic equipment and storage medium |
| CN111767312A (en) * | 2020-06-12 | 2020-10-13 | 上海传英信息技术有限公司 | A data query method, server and computer-readable storage medium |
| CN112100502A (en) * | 2020-09-24 | 2020-12-18 | 北京百度网讯科技有限公司 | Information query method, apparatus, computing device and medium for database |
| CN112580947A (en) * | 2020-12-08 | 2021-03-30 | 北京索为云网科技有限公司 | Information processing method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220335338A1 (en) | Feature processing tradeoff management | |
| US10452691B2 (en) | Method and apparatus for generating search results using inverted index | |
| US10318882B2 (en) | Optimized training of linear machine learning models | |
| US10339465B2 (en) | Optimized decision tree based models | |
| US11182691B1 (en) | Category-based sampling of machine learning data | |
| US9672474B2 (en) | Concurrent binning of machine learning data | |
| US11100420B2 (en) | Input processing for machine learning | |
| CN109933514B (en) | Data testing method and device | |
| CN103620601A (en) | Joining tables in a mapreduce procedure | |
| CN112800095A (en) | Data processing method, device, equipment and storage medium | |
| US20150379425A1 (en) | Consistent filtering of machine learning data | |
| CN113722600A (en) | Data query method, device, equipment and product applied to big data | |
| CN111159213A (en) | Data query method, device, system and storage medium | |
| US20180129708A1 (en) | Query processing management in a database management system | |
| US20150269234A1 (en) | User Defined Functions Including Requests for Analytics by External Analytic Engines | |
| CN113204621A (en) | Document storage method, document retrieval method, device, equipment and storage medium | |
| CN113010542A (en) | Service data processing method and device, computer equipment and storage medium | |
| US20150120697A1 (en) | System and method for analysis of a database proxy | |
| US20230153286A1 (en) | Method and system for hybrid query based on cloud analysis scene, and storage medium | |
| CN113360517A (en) | Data processing method and device, electronic equipment and storage medium | |
| US11544260B2 (en) | Transaction processing method and system, and server | |
| CN119782590A (en) | Question and answer processing method, device, electronic device and storage medium based on large model | |
| CN112527880B (en) | Method, device, equipment and medium for collecting metadata information of big data cluster | |
| US9965558B2 (en) | Cross-channel social search | |
| US11023485B2 (en) | Cube construction for an OLAP system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200515 |