CN110765206A - Data synchronization system, method, device, equipment and storage medium - Google Patents
Data synchronization system, method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN110765206A CN110765206A CN201910999995.XA CN201910999995A CN110765206A CN 110765206 A CN110765206 A CN 110765206A CN 201910999995 A CN201910999995 A CN 201910999995A CN 110765206 A CN110765206 A CN 110765206A
- Authority
- CN
- China
- Prior art keywords
- data
- synchronization
- source
- source data
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明适用计算机技术领域，提供了数据同步系统、方法、装置、设备及存储介质，同步管理平台接收源数据库到目标数据库的数据同步任务，将所述数据同步任务对应的同步任务配置推送至数据同步节点，数据同步节点根据所述同步任务配置和所述源数据库的日志，从所述源数据库中读取需要同步的源数据，将所述源数据放入到消息队列服务器，当检测到订阅者订阅的所述源数据时，从所述消息队列服务器获取所述源数据，将所述源数据写入到订阅者设置的目标数据库，从而通过消息队列服务器实现不同平台系统间的数据同步，减少与目标数据库和源数据库之间的交互，保证数据库中数据的安全。
The invention is applicable to the field of computer technology, and provides a data synchronization system, method, device, equipment and storage medium. The synchronization management platform receives the data synchronization task from the source database to the target database, and pushes the synchronization task configuration corresponding to the data synchronization task to the data synchronization task. The synchronization node, the data synchronization node reads the source data that needs to be synchronized from the source database according to the synchronization task configuration and the log of the source database, and puts the source data into the message queue server. When the source data subscribed by the subscriber, the source data is obtained from the message queue server, and the source data is written into the target database set by the subscriber, thereby realizing data synchronization between different platform systems through the message queue server, Reduce the interaction with the target database and the source database to ensure the security of the data in the database.
Description
技术领域technical field
本发明属于计算机技术领域，尤其涉及一种数据同步系统、方法、装置、设备及存储介质。The invention belongs to the field of computer technology, and in particular relates to a data synchronization system, method, device, device and storage medium.
背景技术Background technique
由于电商平台版本更新迭代较快，且数据量较大，在新老版本切换时，用于存储数据的关联数据库之间需要进行同步，而普通的数据同步方法，需要对数据库表进行直接操作，并且不能实时地将数据同步到不同的项目数据库当中，且每次同步时都需要频繁地跟数据库进行交互，影响数据库数据的存储效率，并且同步的过程中计算机程序代码是从上往下执行的，如果计算机程序代码中间发生故障会影响整个数据同步的过程。Due to the rapid update iteration of the e-commerce platform and the large amount of data, when the new and old versions are switched, the associated databases used to store data need to be synchronized, and the ordinary data synchronization method requires direct operations on the database tables. , and the data cannot be synchronized to different project databases in real time, and each synchronization requires frequent interaction with the database, which affects the storage efficiency of database data, and the computer program code is executed from top to bottom during the synchronization process. Yes, if a fault occurs in the middle of the computer program code, it will affect the entire process of data synchronization.
发明内容SUMMARY OF THE INVENTION
本发明的目的在于提供一种数据同步系统、方法、装置、设备及存储介质，旨在解决由于现有电商平台同步时与数据库交互频繁，导致同步效率不高的问题。The purpose of the present invention is to provide a data synchronization system, method, device, device and storage medium, aiming to solve the problem of low synchronization efficiency due to frequent interaction with the database during synchronization of the existing e-commerce platform.
一方面，本发明提供了一种数据同步系统，所述系统包括同步管理平台、数据同步节点以及消息队列服务器，其中：In one aspect, the present invention provides a data synchronization system, the system includes a synchronization management platform, a data synchronization node and a message queue server, wherein:
所述同步管理平台，用于接收源数据库到目标数据库的数据同步任务，将所述数据同步任务对应的同步任务配置推送至所述数据同步节点；The synchronization management platform is configured to receive a data synchronization task from a source database to a target database, and push the synchronization task configuration corresponding to the data synchronization task to the data synchronization node;
所述数据同步节点，用于根据所述同步任务配置和所述源数据库的日志，从所述源数据库中读取需要同步的源数据，将所述源数据放入到所述消息队列服务器，当检测到订阅者订阅的所述源数据时，从所述消息队列服务器获取所述源数据，将所述源数据写入到订阅者设置的目标数据库。The data synchronization node is configured to read the source data to be synchronized from the source database according to the synchronization task configuration and the log of the source database, and put the source data into the message queue server, When the source data subscribed by the subscriber is detected, the source data is obtained from the message queue server, and the source data is written to the target database set by the subscriber.
另一方面，本发明提供了一种基于上述数据同步系统的数据同步方法，包括：On the other hand, the present invention provides a data synchronization method based on the above-mentioned data synchronization system, comprising:
根据所述同步管理平台发送的同步任务配置和源数据库的日志，数据同步节点从所述源数据库中读取需要同步的源数据，将所述源数据放入到消息队列服务器；According to the synchronization task configuration sent by the synchronization management platform and the log of the source database, the data synchronization node reads the source data to be synchronized from the source database, and puts the source data into the message queue server;
当检测到订阅者订阅的所述源数据时，数据同步节点从所述消息队列服务器获取所述源数据，将所述源数据写入到订阅者设置的目标数据库。When detecting the source data subscribed by the subscriber, the data synchronization node obtains the source data from the message queue server, and writes the source data to the target database set by the subscriber.
另一方面，本发明还提供了基于上述数据同步系统的数据同步装置，包括：On the other hand, the present invention also provides a data synchronization device based on the above-mentioned data synchronization system, including:
数据发布单元，用于根据所述同步管理平台发送的同步任务配置和源数据库的日志，数据同步节点从所述源数据库中读取需要同步的源数据，将所述源数据放入到消息队列服务器；以及The data publishing unit is used for the data synchronization node to read the source data to be synchronized from the source database according to the synchronization task configuration sent by the synchronization management platform and the log of the source database, and put the source data into the message queue server; and
数据写入单元，用于当检测到订阅者订阅的所述源数据时，数据同步节点从所述消息队列服务器获取所述源数据，将所述源数据写入到订阅者设置的目标数据库。The data writing unit is used for, when detecting the source data subscribed by the subscriber, the data synchronization node obtains the source data from the message queue server, and writes the source data to the target database set by the subscriber.
另一方面，本发明还提供了同步设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如上所述方法的步骤。In another aspect, the present invention also provides a synchronization device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implements the above when executing the computer program steps of the method described.
另一方面，本发明还提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时实现如上所述方法的步骤。In another aspect, the present invention also provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, implements the steps of the above-mentioned method.
本发明提供的数据同步系统包括同步管理平台和数据同步节点，同步管理平台接收源数据库到目标数据库的数据同步任务，将所述数据同步任务对应的同步任务配置推送至数据同步节点，数据同步节点根据所述同步任务配置和所述源数据库的日志，从所述源数据库中读取需要同步的源数据，将所述源数据放入到消息队列服务器，当检测到订阅者订阅的所述源数据时，从所述消息队列服务器获取所述源数据，将所述源数据写入到订阅者设置的目标数据库，从而通过消息队列服务器实现不同平台系统间的数据同步，减少与目标数据库和源数据库之间的交互，保证数据库中数据的安全。The data synchronization system provided by the present invention includes a synchronization management platform and a data synchronization node. The synchronization management platform receives a data synchronization task from a source database to a target database, and pushes the synchronization task configuration corresponding to the data synchronization task to the data synchronization node. The data synchronization node According to the synchronization task configuration and the log of the source database, the source data to be synchronized is read from the source database, and the source data is put into the message queue server. When the source subscribed by the subscriber is detected When data is generated, the source data is obtained from the message queue server, and the source data is written to the target database set by the subscriber, thereby realizing data synchronization between different platform systems through the message queue server, reducing the need for communication with the target database and source data. The interaction between databases ensures the security of data in the database.
附图说明Description of drawings
图1是本发明实施例一提供的数据同步系统的结构示意图；1 is a schematic structural diagram of a data synchronization system provided in Embodiment 1 of the present invention;
图2是本发明实施例一提供的数据同步系统的结构示意图；2 is a schematic structural diagram of a data synchronization system provided by Embodiment 1 of the present invention;
图3是本发明实施例二提供的数据同步方法的实现流程图；Fig. 3 is the realization flow chart of the data synchronization method provided by the second embodiment of the present invention;
图4是本发明实施例三提供的数据同步装置的结构示意图；以及4 is a schematic structural diagram of a data synchronization apparatus provided in Embodiment 3 of the present invention; and
图5是本发明实施例四提供的同步设备的结构示意图。FIG. 5 is a schematic structural diagram of a synchronization device according to Embodiment 4 of the present invention.
具体实施方式Detailed ways
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.
以下结合具体实施例对本发明的具体实现进行详细描述：The specific implementation of the present invention is described in detail below in conjunction with specific embodiments:
实施例一：Example 1:
图1示出了本发明实施例一提供的数据同步系统的结构，为了便于说明，仅示出了与本发明实施例相关的部分。FIG. 1 shows the structure of the data synchronization system provided by the first embodiment of the present invention. For the convenience of description, only the part related to the embodiment of the present invention is shown.
在本发明实施例中，数据同步系统1包括同步管理平台11、数据同步节点12以及消息队列服务器13，其中：In the embodiment of the present invention, the data synchronization system 1 includes a
同步管理平台11用于接收源数据库到目标数据库的数据同步任务，将数据同步任务对应的同步任务配置推送至数据同步节点12；The
数据同步节点12用于根据同步任务配置和源数据库的日志，从源数据库中读取需要同步的源数据，将源数据放入到消息队列服务器13，当检测到订阅者订阅的所述源数据时，从消息队列服务器13获取源数据，将源数据写入到订阅者设置的目标数据库。The
在本发明实施例中，用户通过同步管理平台11对每次的数据同步进行统一配置，对数据同步过程进行监控。同步管理平台11在接收源数据库到目标数据库的数据同步任务后，将数据同步任务对应的同步任务配置推送至数据同步节点12。其中，同步任务可以是电商平台商品价格、商品描述等数据同步任务，同步任务配置可以是待同步数据、执行同步的时间等。数据同步节点12根据同步任务配置和源数据库的日志，从源数据库中捕捉、读取需要同步的源数据，进一步地可对读取的源数据进行处理，将源数据进行标准化(转换成对应的可处理数据类型)，之后将源数据放入到消息队列服务器13。当检测到订阅者订阅的源数据时，数据同步节点12从消息队列服务器13获取源数据，将源数据写入到订阅者设置的目标数据库。在将源数据写入到订阅者设置的目标数据库之前，可通过数据同步节点12对源数据进行数据过滤，添加数据库的事务处理，还可以通过异构类型转换把要同步的源数据转换成目标数据库支持的数据格式，显著提高系统的稳定性和可扩展性，最后再把源数据中的数据字段映射到对应的数据表中，最终把源数据写入到目标数据库中。In the embodiment of the present invention, the user performs unified configuration for each data synchronization through the
优选地，如图2所示，数据同步系统1还包括本地缓存服务器14，用于在数据同步节点12将源数据放入到消息队列服务器13时，对数据同步节点12要放入到消息队列服务器13的源数据进行缓存，将缓存的源数据推送至消息队列服务器13，从而实现从源数据库到消息队列服务器的数据快速写入。Preferably, as shown in FIG. 2 , the data synchronization system 1 further includes a
优选地，数据同步系统1还包括外部缓存服务器15，用于在数据同步节点将源数据写入到订阅者设置的目标数据库时，对从消息队列服务器13获取的源数据进行缓存，将缓存的源数据写入到订阅者设置的目标数据库，从而实现从消息队列服务器到目标数据库的数据快速写入。Preferably, the data synchronization system 1 further includes an
优选地，数据同步节点12还用于对传送的文件流进行检测，以监测文件流当中是否有需要进行同步的文件，当检测到订阅者订阅的同步文件时，将同步文件发送到目标文件中。具体在，在同步时，数据同步节点12对同步文件进行识别，区分这些文件的格式，再通过文件传输发送一个新的文件流把要同步的文件发送到目标文件中，从而实现文件的同步。Preferably, the
在本发明实施例中，通过消息队列服务器实现不同平台系统间的数据同步，减少了与目标数据库和源数据库之间的交互，保证了数据库中数据的安全，通过外部缓存服务器可以支持大量的数据同时同步，另外，通过订阅发布机制保证了同步数据通过过程的实时性。In the embodiment of the present invention, the data synchronization between different platform systems is realized through the message queue server, the interaction with the target database and the source database is reduced, the security of the data in the database is ensured, and the external cache server can support a large amount of data Simultaneous synchronization, in addition, the real-time nature of the synchronization data passing process is guaranteed through the subscription publishing mechanism.
在本发明实施例中，数据同步系统的各组成部分可由相应的硬件或软件单元实现，各单元可以为独立的软、硬件单元，也可以集成为一个软、硬件单元，在此不用以限制本发明。In this embodiment of the present invention, each component of the data synchronization system may be implemented by corresponding hardware or software units, and each unit may be an independent software and hardware unit, or may be integrated into a software and hardware unit, which is not intended to be limited here. invention.
实施例二：Embodiment 2:
图3示出了本发明实施例二提供的数据同步方法的实现流程，为了便于说明，仅示出了与本发明实施例相关的部分，详述如下：FIG. 3 shows the implementation process of the data synchronization method provided by the second embodiment of the present invention. For the convenience of description, only the part related to the embodiment of the present invention is shown, and the details are as follows:
在步骤S301中，根据同步管理平台发送的同步任务配置和源数据库的日志，数据同步节点从源数据库中读取需要同步的源数据，将源数据放入到消息队列服务器。In step S301, according to the synchronization task configuration sent by the synchronization management platform and the log of the source database, the data synchronization node reads the source data to be synchronized from the source database, and puts the source data into the message queue server.
本发明实施例适用于实施例一中的数据同步系统，该数据同步系统包括同步管理平台、数据同步节点以及消息队列服务器。同步管理平台在接收源数据库到目标数据库的数据同步任务后，将数据同步任务对应的同步任务配置推送至数据同步节点，这样，数据同步节点可根据同步任务配置和源数据库的日志，从源数据库中读取需要同步的源数据，将源数据放入到消息队列服务器。The embodiment of the present invention is applicable to the data synchronization system in the first embodiment, and the data synchronization system includes a synchronization management platform, a data synchronization node and a message queue server. After receiving the data synchronization task from the source database to the target database, the synchronization management platform pushes the synchronization task configuration corresponding to the data synchronization task to the data synchronization node. Read the source data that needs to be synchronized, and put the source data into the message queue server.
用户通过同步管理平台对每次的数据同步进行统一配置，对数据同步过程进行监控。同步管理平台在接收源数据库到目标数据库的数据同步任务后，将数据同步任务对应的同步任务配置推送至数据同步节点。其中，同步任务可以是电商平台商品价格、商品描述等数据同步任务，同步任务配置可以是待同步数据、执行同步的时间等。数据同步节点根据同步任务配置和源数据库的日志，从源数据库中捕捉、读取需要同步的源数据，进一步地可对读取的源数据进行处理，将源数据进行标准化(转换成对应的可处理数据类型)，之后将源数据放入到消息队列服务器。The user configures each data synchronization through the synchronization management platform, and monitors the data synchronization process. After receiving the data synchronization task from the source database to the target database, the synchronization management platform pushes the synchronization task configuration corresponding to the data synchronization task to the data synchronization node. The synchronization task may be data synchronization tasks such as commodity prices and commodity descriptions on the e-commerce platform, and the synchronization task configuration may be the data to be synchronized, the time to perform synchronization, and the like. The data synchronization node captures and reads the source data that needs to be synchronized from the source database according to the synchronization task configuration and the log of the source database, and can further process the read source data and standardize the source data (convert it into corresponding process data type), and then put the source data into the message queue server.
在步骤S302中，当检测到订阅者订阅的源数据时，数据同步节点从消息队列服务器获取源数据，将源数据写入到订阅者设置的目标数据库。In step S302, when detecting the source data subscribed by the subscriber, the data synchronization node obtains the source data from the message queue server, and writes the source data to the target database set by the subscriber.
在本发明实施例中，在将源数据写入到订阅者设置的目标数据库之前，可通过数据同步节点对源数据进行数据过滤，添加数据库的事务处理，还可以通过异构类型转换把要同步的源数据转换成目标数据库支持的数据格式，显著提高系统的稳定性和可扩展性，最后再把源数据中的数据字段映射到对应的数据表中，最终把源数据写入到目标数据库中。In the embodiment of the present invention, before writing the source data to the target database set by the subscriber, the source data can be filtered through the data synchronization node, the transaction processing of the database can be added, and the data to be synchronized can also be converted through heterogeneous type conversion. The source data is converted into the data format supported by the target database, which significantly improves the stability and scalability of the system. Finally, the data fields in the source data are mapped to the corresponding data tables, and the source data is finally written into the target database. .
当数据同步系统还包括本地缓存服务器时，优选地，在将源数据放入到消息队列服务器时，将数据同步节点要放入到消息队列服务器的源数据缓存到本地缓存服务器，将缓存的源数据推送至消息队列服务器，从而实现从源数据库到消息队列服务器的数据快速写入。When the data synchronization system further includes a local cache server, preferably, when the source data is put into the message queue server, the source data to be put into the message queue server by the data synchronization node is cached in the local cache server, and the cached source data is cached in the local cache server. Data is pushed to the message queue server, enabling fast data writing from the source database to the message queue server.
当数据同步系统还包括外部缓存服务器时，优选地，在将源数据写入到订阅者设置的目标数据库时，将从消息队列服务器获取的源数据缓存到外部缓存服务器，将缓存的所述源数据写入到订阅者设置的目标数据库，从而实现从消息队列服务器到目标数据库的数据快速写入。When the data synchronization system further includes an external cache server, preferably, when the source data is written to the target database set by the subscriber, the source data obtained from the message queue server is cached to the external cache server, and the cached source data is stored in the external cache server. Data is written to the target database set by the subscriber, thereby realizing fast data writing from the message queue server to the target database.
进一步优选地，数据同步节点对传送的文件流进行检测，当检测到订阅者订阅的同步文件时，将同步文件发送到目标文件中，从而实现文件的同步。Further preferably, the data synchronization node detects the transmitted file stream, and when detecting the synchronization file subscribed by the subscriber, sends the synchronization file to the target file, thereby realizing the synchronization of the file.
实施例三：Embodiment three:
图4示出了本发明实施例三提供的数据同步装置的结构，为了便于说明，仅示出了与本发明实施例相关的部分。FIG. 4 shows the structure of the data synchronization apparatus provided by Embodiment 3 of the present invention. For the convenience of description, only the part related to the embodiment of the present invention is shown.
本发明实施例适用于实施例一中的数据同步系统，本发明实施例提供的数据同步装置4包括：The embodiment of the present invention is applicable to the data synchronization system in the first embodiment, and the data synchronization apparatus 4 provided by the embodiment of the present invention includes:
数据发布单元41，用于根据所述同步管理平台发送的同步任务配置和源数据库的日志，数据同步节点从所述源数据库中读取需要同步的源数据，将所述源数据放入到消息队列服务器；以及The data publishing unit 41 is used for, according to the synchronization task configuration and the log of the source database sent by the synchronization management platform, the data synchronization node reads the source data that needs to be synchronized from the source database, and puts the source data into the message queue servers; and
数据写入单元42，用于当检测到订阅者订阅的所述源数据时，数据同步节点从所述消息队列服务器获取所述源数据，将所述源数据写入到订阅者设置的目标数据库。The data writing unit 42 is configured to, when detecting the source data subscribed by the subscriber, the data synchronization node obtains the source data from the message queue server, and writes the source data to the target database set by the subscriber .
在本发明实施例中，数据同步装置的各单元可由相应的硬件或软件单元实现，各单元可以为独立的软、硬件单元，也可以集成为一个软、硬件单元，在此不用以限制本发明。各单元的具体实现方式可参考实施例一的对应描述，在此不再赘述。In this embodiment of the present invention, each unit of the data synchronization apparatus may be implemented by a corresponding hardware or software unit, and each unit may be an independent software and hardware unit, or may be integrated into a software and hardware unit, which is not intended to limit the present invention. . For the specific implementation manner of each unit, reference may be made to the corresponding description of Embodiment 1, which will not be repeated here.
实施例四：Embodiment 4:
图5示出了本发明实施例四提供的同步设备的结构，为了便于说明，仅示出了与本发明实施例相关的部分。FIG. 5 shows the structure of the synchronization device provided in Embodiment 4 of the present invention. For convenience of description, only the part related to the embodiment of the present invention is shown.
本发明实施例的同步设备5包括处理器50、存储器51以及存储在存储器51中并可在处理器50上运行的计算机程序52。该处理器50执行计算机程序52时实现上述数据同步方法实施例中的步骤，例如图3所示的步骤S301至S302。或者，处理器50执行计算机程序52时实现上述各装置实施例中各单元的功能，例如图4所示单元41至42的功能。The
在本发明实施例中，同步管理平台接收源数据库到目标数据库的数据同步任务，将所述数据同步任务对应的同步任务配置推送至数据同步节点，数据同步节点根据所述同步任务配置和所述源数据库的日志，从所述源数据库中读取需要同步的源数据，将所述源数据放入到消息队列服务器，当检测到订阅者订阅的所述源数据时，从所述消息队列服务器获取所述源数据，将所述源数据写入到订阅者设置的目标数据库，从而通过消息队列服务器实现不同平台系统间的数据同步，减少与目标数据库和源数据库之间的交互，保证数据库中数据的安全。In the embodiment of the present invention, the synchronization management platform receives the data synchronization task from the source database to the target database, and pushes the synchronization task configuration corresponding to the data synchronization task to the data synchronization node, and the data synchronization node is based on the synchronization task configuration and the The log of the source database, read the source data that needs to be synchronized from the source database, put the source data into the message queue server, when the source data subscribed by the subscriber is detected, from the message queue server Obtain the source data, write the source data into the target database set by the subscriber, thereby realizing data synchronization between different platform systems through the message queue server, reducing the interaction with the target database and the source database, and ensuring that the Data security.
本发明实施例的同步设备可以为计算设备。该同步设备5中处理器50执行计算机程序52时实现数据同步方法时实现的步骤可参考前述方法实施例的描述，在此不再赘述。The synchronization device in the embodiment of the present invention may be a computing device. For the steps implemented when the
实施例五：Embodiment 5:
在本发明实施例中，提供了一种计算机可读存储介质，该计算机可读存储介质存储有计算机程序，该计算机程序被处理器执行时实现上述数据同步方法实施例中的步骤，例如，图1所示的步骤S301至S302。或者，该计算机程序被处理器执行时实现上述各装置实施例中各单元的功能，例如图4所示单元41至42的功能。In an embodiment of the present invention, a computer-readable storage medium is provided, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps in the foregoing data synchronization method embodiments are implemented, for example, FIG. Steps S301 to S302 shown in 1. Alternatively, when the computer program is executed by the processor, the functions of the units in the above-mentioned apparatus embodiments, for example, the functions of the units 41 to 42 shown in FIG. 4 , are implemented.
在本发明实施例中，同步管理平台接收源数据库到目标数据库的数据同步任务，将所述数据同步任务对应的同步任务配置推送至数据同步节点，数据同步节点根据所述同步任务配置和所述源数据库的日志，从所述源数据库中读取需要同步的源数据，将所述源数据放入到消息队列服务器，当检测到订阅者订阅的所述源数据时，从所述消息队列服务器获取所述源数据，将所述源数据写入到订阅者设置的目标数据库，从而通过消息队列服务器实现不同平台系统间的数据同步，减少与目标数据库和源数据库之间的交互，保证数据库中数据的安全。In the embodiment of the present invention, the synchronization management platform receives the data synchronization task from the source database to the target database, and pushes the synchronization task configuration corresponding to the data synchronization task to the data synchronization node, and the data synchronization node is based on the synchronization task configuration and the The log of the source database, read the source data that needs to be synchronized from the source database, put the source data into the message queue server, when the source data subscribed by the subscriber is detected, from the message queue server Obtain the source data, write the source data into the target database set by the subscriber, thereby realizing data synchronization between different platform systems through the message queue server, reducing the interaction with the target database and the source database, and ensuring that the Data security.
本发明实施例的计算机可读存储介质可以包括能够携带计算机程序代码的任何实体或装置、记录介质，例如，ROM/RAM、磁盘、光盘、闪存等存储器。The computer-readable storage medium of the embodiments of the present invention may include any entity or device capable of carrying computer program codes, recording medium, for example, memory such as ROM/RAM, magnetic disk, optical disk, flash memory, and the like.
以上所述仅为本发明的较佳实施例而已，并不用以限制本发明，凡在本发明的精神和原则之内所作的任何修改、等同替换和改进等，均应包含在本发明的保护范围之内。The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.
Claims (11)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910999995.XA CN110765206A (en) | 2019-10-21 | 2019-10-21 | Data synchronization system, method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910999995.XA CN110765206A (en) | 2019-10-21 | 2019-10-21 | Data synchronization system, method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110765206A true CN110765206A (en) | 2020-02-07 |
Family
ID=69332528
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910999995.XA Pending CN110765206A (en) | 2019-10-21 | 2019-10-21 | Data synchronization system, method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110765206A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111400408A (en) * | 2020-04-13 | 2020-07-10 | 上海东普信息科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN111581244A (en) * | 2020-05-14 | 2020-08-25 | 厦门中控智慧信息技术有限公司 | Method, system and equipment for orderly synchronizing business transaction data of heterogeneous system |
| CN112380227A (en) * | 2020-11-12 | 2021-02-19 | 平安科技（深圳）有限公司 | Data synchronization method, device and equipment based on message queue and storage medium |
| CN112597123A (en) * | 2020-11-20 | 2021-04-02 | 贝壳技术有限公司 | Data multi-version dynamic switching method and device |
| CN112836236A (en) * | 2021-02-04 | 2021-05-25 | 航天广通科技(深圳)有限公司 | Method and system for subscribing database table resources of data sharing exchange platform |
| CN113377863A (en) * | 2020-03-10 | 2021-09-10 | 阿里巴巴集团控股有限公司 | Data synchronization method and device, electronic equipment and computer readable storage medium |
| CN114143375A (en) * | 2020-08-13 | 2022-03-04 | 支付宝实验室(新加坡)有限公司 | Method and system for distributing data |
| CN115329006A (en) * | 2022-08-31 | 2022-11-11 | 保利和悦生活科技服务有限公司 | Data synchronization method and system for network mall background and third-party interface |
| CN115630124A (en) * | 2022-12-06 | 2023-01-20 | 北京志翔科技股份有限公司 | Data interconnection method, device and equipment |
| CN119865507A (en) * | 2024-12-31 | 2025-04-22 | 中电信智能网络科技有限公司 | Data synchronization method, device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030037029A1 (en) * | 2001-08-15 | 2003-02-20 | Iti, Inc. | Synchronization of plural databases in a database replication system |
| US20100153351A1 (en) * | 2008-12-16 | 2010-06-17 | Yung Alex P | Techniques for real-time database processing |
| CN105095364A (en) * | 2015-06-26 | 2015-11-25 | 车智互联（北京）科技有限公司 | Data synchronizing system and method |
| CN108256002A (en) * | 2017-12-31 | 2018-07-06 | 广东欧珀移动通信有限公司 | Across computer room method of data synchronization, device, system and server |
| CN109582731A (en) * | 2018-10-18 | 2019-04-05 | 恒峰信息技术有限公司 | A kind of real time data synchronization method and system |
| CN110162571A (en) * | 2019-04-26 | 2019-08-23 | 厦门市美亚柏科信息股份有限公司 | A kind of system, method, storage medium that data among heterogeneous databases synchronize |
-
2019
- 2019-10-21 CN CN201910999995.XA patent/CN110765206A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030037029A1 (en) * | 2001-08-15 | 2003-02-20 | Iti, Inc. | Synchronization of plural databases in a database replication system |
| US20100153351A1 (en) * | 2008-12-16 | 2010-06-17 | Yung Alex P | Techniques for real-time database processing |
| CN105095364A (en) * | 2015-06-26 | 2015-11-25 | 车智互联（北京）科技有限公司 | Data synchronizing system and method |
| CN108256002A (en) * | 2017-12-31 | 2018-07-06 | 广东欧珀移动通信有限公司 | Across computer room method of data synchronization, device, system and server |
| CN109582731A (en) * | 2018-10-18 | 2019-04-05 | 恒峰信息技术有限公司 | A kind of real time data synchronization method and system |
| CN110162571A (en) * | 2019-04-26 | 2019-08-23 | 厦门市美亚柏科信息股份有限公司 | A kind of system, method, storage medium that data among heterogeneous databases synchronize |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377863A (en) * | 2020-03-10 | 2021-09-10 | 阿里巴巴集团控股有限公司 | Data synchronization method and device, electronic equipment and computer readable storage medium |
| CN113377863B (en) * | 2020-03-10 | 2022-04-29 | 阿里巴巴集团控股有限公司 | Data synchronization method and device, electronic equipment and computer readable storage medium |
| CN111400408A (en) * | 2020-04-13 | 2020-07-10 | 上海东普信息科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN111400408B (en) * | 2020-04-13 | 2023-06-09 | 上海东普信息科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN111581244B (en) * | 2020-05-14 | 2023-04-25 | 厦门熵基科技有限公司 | Method, system and equipment for orderly synchronizing service transaction data of heterogeneous system |
| CN111581244A (en) * | 2020-05-14 | 2020-08-25 | 厦门中控智慧信息技术有限公司 | Method, system and equipment for orderly synchronizing business transaction data of heterogeneous system |
| CN114143375A (en) * | 2020-08-13 | 2022-03-04 | 支付宝实验室(新加坡)有限公司 | Method and system for distributing data |
| CN112380227A (en) * | 2020-11-12 | 2021-02-19 | 平安科技（深圳）有限公司 | Data synchronization method, device and equipment based on message queue and storage medium |
| CN112380227B (en) * | 2020-11-12 | 2024-05-07 | 平安科技（深圳）有限公司 | Data synchronization method, device, equipment and storage medium based on message queue |
| CN112597123A (en) * | 2020-11-20 | 2021-04-02 | 贝壳技术有限公司 | Data multi-version dynamic switching method and device |
| CN112597123B (en) * | 2020-11-20 | 2023-01-06 | 贝壳技术有限公司 | Data multi-version dynamic switching method and device |
| CN112836236A (en) * | 2021-02-04 | 2021-05-25 | 航天广通科技(深圳)有限公司 | Method and system for subscribing database table resources of data sharing exchange platform |
| CN115329006B (en) * | 2022-08-31 | 2023-08-04 | 保利和悦生活科技服务有限公司 | A method and system for synchronizing data between the background of an online shopping mall and a third-party interface |
| CN115329006A (en) * | 2022-08-31 | 2022-11-11 | 保利和悦生活科技服务有限公司 | Data synchronization method and system for network mall background and third-party interface |
| CN115630124A (en) * | 2022-12-06 | 2023-01-20 | 北京志翔科技股份有限公司 | Data interconnection method, device and equipment |
| CN119865507A (en) * | 2024-12-31 | 2025-04-22 | 中电信智能网络科技有限公司 | Data synchronization method, device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110765206A (en) | Data synchronization system, method, device, equipment and storage medium | |
| US8548945B2 (en) | Database caching utilizing asynchronous log-based replication | |
| CN104252501B (en) | A kind of computing device and method executing database manipulation order | |
| CN102780603B (en) | Web traffic control method and device | |
| CN103559231B (en) | A kind of file system quota management method, apparatus and system | |
| US10270852B2 (en) | Data migration apparatus and system | |
| WO2012045245A1 (en) | Method and system for maintaining data consistency | |
| WO2020019724A1 (en) | Method for acquiring sensor data in server, acquisition system, and related apparatus | |
| CN111131079B (en) | Policy query method and device | |
| CN111831618A (en) | Data writing method, data reading method, apparatus, device and storage medium | |
| WO2018095187A1 (en) | Document online preview method and device | |
| CN115168440A (en) | Data reading and writing method, distributed storage system, device, equipment and storage medium | |
| CN111290871A (en) | Method and device for obtaining application crash information | |
| KR102247247B1 (en) | Method to manage database | |
| CN113051244B (en) | Data access method and device, data acquisition method and device | |
| CN117632445B (en) | Request processing method and device, task execution method and device | |
| US12282394B2 (en) | System and method for optimizing incremental backups of network attached storage file data | |
| US20240089339A1 (en) | Caching across multiple cloud environments | |
| CN112416871B (en) | A data access method, device and system | |
| US10938701B2 (en) | Efficient heartbeat with remote servers by NAS cluster nodes | |
| CN115695587A (en) | Service data processing system, method, device and storage medium | |
| US12061523B2 (en) | System and method for parallelization backup of a folder including a large set of files stored in a network attached storage | |
| US12287709B2 (en) | System and method for managing a backup of a large set of files using a file system analysis for data stored in a network attached storage | |
| US12153498B2 (en) | System and method for generating incremental backups for file based backups stored in a network attached storage | |
| US12061527B2 (en) | System and method for managing network bandwidth for medium and large file sizes stored in a network attached storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200207 |