CN110096548A - Data synchronous method, apparatus, computer equipment and computer storage medium - Google Patents
Data synchronous method, apparatus, computer equipment and computer storage medium Download PDFInfo
- Publication number
- CN110096548A CN110096548A CN201910218795.6A CN201910218795A CN110096548A CN 110096548 A CN110096548 A CN 110096548A CN 201910218795 A CN201910218795 A CN 201910218795A CN 110096548 A CN110096548 A CN 110096548A
- Authority
- CN
- China
- Prior art keywords
- data
- synchronized
- synchronization
- parameter
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请公开了一种数据同步的方法、装置、计算机设备及计算机存储介质，涉及云存储技术领域，可以合理安排数据同步调度任务的时间片，提高数据同步效率。所述方法包括：获取业务方配置的待同步数据的初始调度参数；根据所述待同步数据的初始调度参数将所述待同步数据分配到各个同步节点上进行数据同步；在根据所述初始调度参数进行数据同步的过程中，监控所述各个同步节点上待同步数据对应的同步状态参数；根据所述各个同步节点上待同步数据对应的同步状态参数，实时调整所述待同步数据的初始调度参数，并根据调整后的调度参数进行数据同步。
The present application discloses a data synchronization method, device, computer equipment, and computer storage medium, which relate to the technical field of cloud storage, and can reasonably arrange time slices for data synchronization scheduling tasks to improve data synchronization efficiency. The method includes: acquiring the initial scheduling parameters of the data to be synchronized configured by the business party; distributing the data to be synchronized to each synchronization node for data synchronization according to the initial scheduling parameters of the data to be synchronized; performing data synchronization according to the initial scheduling In the process of data synchronization by parameters, monitor the synchronization state parameters corresponding to the data to be synchronized on each synchronization node; according to the synchronization state parameters corresponding to the data to be synchronized on each synchronization node, adjust the initial scheduling of the data to be synchronized in real time parameters, and perform data synchronization according to the adjusted scheduling parameters.
Description
技术领域technical field
本发明涉及云存储技术领域，尤其是涉及数据同步的方法、装置、计算机设备及计算机存储介质。The present invention relates to the technical field of cloud storage, in particular to a data synchronization method, device, computer equipment and computer storage medium.
背景技术Background technique
计算机网络的飞速发展和信息化的推进使得数据成为很多企业的宝贵资源，使得数据采集、存储和处理任务与日俱增，基于网络的数据共享已经成为人们获取数据资源不可或缺的方式。The rapid development of computer networks and the advancement of informatization have made data a valuable resource for many companies, and the tasks of data collection, storage and processing are increasing day by day. Network-based data sharing has become an indispensable way for people to obtain data resources.
为了更好地实现数据共享需要对共享数据进行同步，数据同步对企业来说至关重要，比如企业的服务器端用于管理分属于企业不同部门的业务端，并且在服务器端存储有管理数据，那么为了方便服务器端对各个业务端进行管理，同时提高业务端的数据处理效率，需要保证服务器端与各个业务端之间的数据同步。In order to better realize data sharing, it is necessary to synchronize the shared data. Data synchronization is very important for enterprises. For example, the server side of the enterprise is used to manage the business ends belonging to different departments of the enterprise, and the management data is stored on the server side. Then, in order to facilitate the management of each business end on the server side and improve the data processing efficiency of the business end, it is necessary to ensure the data synchronization between the server end and each business end.
在数据同步过程中，为了解决业务场景维度、机器性能等限制维度的问题，往往需要用业务人员筛选出需要同步的数据表，并人为设置需要同步的数据表的数据同步任务，进而根据数据同步任务完成业务数据的同步。然而，人为设置数据同步任务的方式主观差异性较高，容易在数据同步的过程中出现调度时间片的不平衡，任务调度的并发多线程没有合理利用，导致数据同步的效率低下。In the process of data synchronization, in order to solve the problem of limited dimensions such as business scenario dimensions and machine performance, it is often necessary to use business personnel to filter out the data tables that need to be synchronized, and manually set the data synchronization tasks of the data tables that need to be synchronized, and then according to the data synchronization The task completes the synchronization of business data. However, the way of artificially setting data synchronization tasks has high subjective differences, and it is easy to cause an imbalance of scheduling time slices in the process of data synchronization, and the concurrent multi-threading of task scheduling is not properly utilized, resulting in low efficiency of data synchronization.
发明内容Contents of the invention
有鉴于此，本发明提供了一种数据同步的方法、装置、计算机设备及计算机存储介质，主要目的在于解决目前相关技术中由于数据同步任务调度的时间片不平衡导致的数据同步效率低下的问题。In view of this, the present invention provides a data synchronization method, device, computer equipment, and computer storage medium, the main purpose of which is to solve the problem of low data synchronization efficiency caused by unbalanced time slices of data synchronization task scheduling in current related technologies .
依据本发明一个方面，提供了一种数据同步的方法，该方法包括：According to one aspect of the present invention, a method for data synchronization is provided, the method comprising:
获取业务方配置的待同步数据的初始调度参数；Obtain the initial scheduling parameters of the data to be synchronized configured by the business side;
根据所述待同步数据的初始调度参数将所述待同步数据分配到各个同步节点上进行数据同步；Allocating the data to be synchronized to each synchronization node for data synchronization according to the initial scheduling parameters of the data to be synchronized;
在根据所述初始调度参数进行数据同步的过程中，监控所述各个同步节点上待同步数据对应的同步状态参数；During the process of data synchronization according to the initial scheduling parameters, monitor the synchronization status parameters corresponding to the data to be synchronized on each synchronization node;
根据所述各个同步节点上待同步数据对应的同步状态参数，实时调整所述待同步数据的初始调度参数，并根据调整后的调度参数进行数据同步。Adjusting the initial scheduling parameters of the data to be synchronized in real time according to the synchronization status parameters corresponding to the data to be synchronized on each synchronization node, and performing data synchronization according to the adjusted scheduling parameters.
进一步地，在所述获取业务方配置的待同步数据的初始调度参数之前，所述方法还包括：Further, before acquiring the initial scheduling parameters of the data to be synchronized configured by the business party, the method further includes:
根据待同步数据中数据量大小预先对待同步数据中每个数据表进行数据量级别分类，确定所述待同步数据中每个数据表对应的数据量级别；Classify each data table in the data to be synchronized according to the size of the data volume in the data to be synchronized in advance, and determine the data volume level corresponding to each data table in the data to be synchronized;
所述获取业务方配置的待同步数据的初始调度参数包括：The initial scheduling parameters for obtaining the data to be synchronized configured by the business party include:
根据所述待同步数据中每个数据表对应的数据量级别，获取业务方配置的待同步数据的初始调度参数。According to the data volume level corresponding to each data table in the data to be synchronized, the initial scheduling parameters of the data to be synchronized configured by the business party are acquired.
进一步地，所述初始调度参数包括同步时间参数、同步调度时间级别参数以及同步节点权重参数，所述根据所述待同步数据中每个数据表对应的数据量级别，获取业务方配置的待同步数据的初始调度参数包括：Further, the initial scheduling parameters include a synchronization time parameter, a synchronization scheduling time level parameter, and a synchronization node weight parameter, and according to the data volume level corresponding to each data table in the data to be synchronized, the data to be synchronized configured by the business party is obtained. The initial scheduling parameters for the data include:
根据所述待同步数据中每个数据表对应的数据量级别预估待同步数据完成同步任务所需要的时间，获取业务方配置的待同步数据的同步时间参数；Estimate the time required for the data to be synchronized to complete the synchronization task according to the data volume level corresponding to each data table in the data to be synchronized, and obtain the synchronization time parameters of the data to be synchronized configured by the business party;
根据所述待同步数据中每个数据表对应的数据量级别预估待同步数据在同步任务中调度的单位时间，获取业务方配置的待同步数据的同步调度时间级别参数；Estimate the unit time for scheduling the data to be synchronized in the synchronization task according to the data volume level corresponding to each data table in the data to be synchronized, and obtain the synchronization scheduling time level parameter of the data to be synchronized configured by the business party;
根据所述待同步数据中每个数据表对应的数据量级别预估待同步数据在同步任务中需要开启同步节点数量，获取业务方配置的待同步数据的同步节点权重参数。According to the data volume level corresponding to each data table in the data to be synchronized, estimate the number of synchronization nodes that need to be enabled in the synchronization task of the data to be synchronized, and obtain the synchronization node weight parameters of the data to be synchronized configured by the business party.
进一步地，所述根据所述各个同步节点上待同步数据对应的同步状态参数，实时调整所述待同步数据的初始调度参数，并根据调整后的调度参数进行数据同步包括：Further, the adjusting the initial scheduling parameters of the data to be synchronized in real time according to the synchronization status parameters corresponding to the data to be synchronized on each synchronization node, and performing data synchronization according to the adjusted scheduling parameters includes:
将所述各个节点上待同步数据在同步任务中调度的单位时间内完成同步任务的百分比与第一预设限值进行比对，得到第一比对结果；Comparing the percentage of the data to be synchronized on each node completing the synchronization task per unit time scheduled in the synchronization task with a first preset limit value, to obtain a first comparison result;
根据所述第一比对结果，调整所述待同步数据的同步调度时间级别参数。According to the first comparison result, adjust the synchronization scheduling time level parameter of the data to be synchronized.
进一步地，所述第一预设限值包括第一预设上限值以及第一预设下限值，所述根据所述第一比对结果，调整所述待同步数据的同步调度时间级别参数包括：Further, the first preset limit value includes a first preset upper limit value and a first preset lower limit value, and according to the first comparison result, the synchronization scheduling time level of the data to be synchronized is adjusted Parameters include:
如果所述各个节点上待同步数据在同步任务中调度的单位时间内完成同步任务的百分比小于第一预设下限值，则调高所述待同步数据的同步调度时间级别参数；If the percentage of the data to be synchronized on each node that completes the synchronization task within the unit time scheduled in the synchronization task is less than the first preset lower limit value, then increase the synchronization scheduling time level parameter of the data to be synchronized;
如果所述各个节点上待同步数据在同步任务中调度的单位时间内完成同步任务的百分比大于第一预设上限值，则调低所述待同步数据的同步调度时间级别参数。If the percentage of the data to be synchronized on each node that completes the synchronization task within the unit time scheduled in the synchronization task is greater than a first preset upper limit, lower the synchronization scheduling time level parameter of the data to be synchronized.
进一步地，所述根据所述各个同步节点上待同步数据对应的同步状态参数，实时调整所述待同步数据的初始调度参数，并根据调整后的调度参数进行数据同步包括：Further, the adjusting the initial scheduling parameters of the data to be synchronized in real time according to the synchronization status parameters corresponding to the data to be synchronized on each synchronization node, and performing data synchronization according to the adjusted scheduling parameters includes:
将所述各个节点上待同步数据在同步任务中处理器使用率百分比与第二预设限值进行比对，得到第二比对结果；Comparing the processor usage percentage of the data to be synchronized on each node in the synchronization task with a second preset limit value to obtain a second comparison result;
根据所述第二比对结果，调整所述待同步数据的同步节点权重参数。Adjusting the synchronization node weight parameter of the data to be synchronized according to the second comparison result.
进一步地，所述第二预设限值包括第二预设上限值以及第二预设下限值，所述根据所述第二比对结果，调整所述待同步数据的同步节点权重参数包括：Further, the second preset limit value includes a second preset upper limit value and a second preset lower limit value, and according to the second comparison result, the synchronization node weight parameter of the data to be synchronized is adjusted include:
如果所述各个节点上待同步数据在同步任务中处理器使用率百分比小于第二预设下限值，则调低所述待同步数据的同步节点权重参数；If the processor usage percentage of the data to be synchronized on each node in the synchronization task is less than a second preset lower limit value, lower the synchronization node weight parameter of the data to be synchronized;
如果所述各个节点上待同步数据在同步任务中处理器使用率百分比大于第二预设上限值，则调高所述待同步数据的同步节点权重参数。If the percentage of processor usage in the synchronization task of the data to be synchronized on each node is greater than a second preset upper limit, then increase the synchronization node weight parameter of the data to be synchronized.
进一步地，所述根据所述待同步数据的初始调度参数将所述待同步数据分配到各个同步节点上进行数据同步包括：Further, the distributing the data to be synchronized to each synchronization node for data synchronization according to the initial scheduling parameters of the data to be synchronized includes:
根据所述待同步数据的初始调度参数将同步节点划分为至少一个部署单元中，每个部署单元中包含至少一个同步节点，并且每个部署单元中的同步节点用于处理相同初始调度参数的待同步数据；According to the initial scheduling parameters of the data to be synchronized, the synchronization nodes are divided into at least one deployment unit, each deployment unit contains at least one synchronization node, and the synchronization nodes in each deployment unit are used to process the data to be synchronized with the same initial scheduling parameters Synchronous Data;
将相同初始调度参数的待同步数据分配到一个部署单元中的同步节点上进行数据同步。Allocate the data to be synchronized with the same initial scheduling parameters to the synchronization nodes in a deployment unit for data synchronization.
依据本发明另一个方面，提供了一种数据同步的装置，所述装置包括：According to another aspect of the present invention, a data synchronization device is provided, the device comprising:
获取单元，用于获取业务方配置的待同步数据的初始调度参数；An acquisition unit, configured to acquire the initial scheduling parameters of the data to be synchronized configured by the business party;
分配单元，用于根据所述待同步数据的初始调度参数将所述待同步数据分配到各个同步节点上进行数据同步；An allocation unit, configured to allocate the data to be synchronized to each synchronization node for data synchronization according to the initial scheduling parameters of the data to be synchronized;
监控单元，用于在根据所述初始调度参数进行数据同步的过程中，监控所述各个同步节点上待同步数据对应的同步状态参数；A monitoring unit, configured to monitor the synchronization status parameters corresponding to the data to be synchronized on each synchronization node during the data synchronization process according to the initial scheduling parameters;
调整单元，用于根据所述各个同步节点上待同步数据对应的同步状态参数，实时调整所述待同步数据的初始调度参数，并根据调整后的调度参数进行数据同步。The adjustment unit is configured to adjust the initial scheduling parameters of the data to be synchronized in real time according to the synchronization status parameters corresponding to the data to be synchronized on each synchronization node, and perform data synchronization according to the adjusted scheduling parameters.
进一步地，所述装置还包括：Further, the device also includes:
分类单元，用于在所述获取业务方配置的待同步数据的初始调度参数之前，根据待同步数据中数据量大小预先对待同步数据中每个数据表进行数据量级别分类，确定所述待同步数据中每个数据表对应的数据量级别；The classification unit is used to classify each data table in the data to be synchronized according to the size of the data volume in the data to be synchronized in advance, and determine the data to be synchronized The data volume level corresponding to each data table in the data;
所述获取单元，具体用于根据所述待同步数据中每个数据表对应的数据量级别，获取业务方配置的待同步数据的初始调度参数。The obtaining unit is specifically configured to obtain the initial scheduling parameters of the data to be synchronized configured by the business party according to the data volume level corresponding to each data table in the data to be synchronized.
进一步地，所述初始调度参数包括同步时间参数、同步调度时间级别参数以及同步节点权重参数，所述获取单元包括：Further, the initial scheduling parameters include a synchronization time parameter, a synchronization scheduling time level parameter, and a synchronization node weight parameter, and the obtaining unit includes:
第一获取模块，用于根据所述待同步数据中每个数据表对应的数据量级别预估待同步数据完成同步任务所需要的时间，获取业务方配置的待同步数据的同步时间参数；The first acquisition module is used to estimate the time required for the data to be synchronized to complete the synchronization task according to the data volume level corresponding to each data table in the data to be synchronized, and obtain the synchronization time parameters of the data to be synchronized configured by the business party;
第二获取模块，用于根据所述待同步数据中每个数据表对应的数据量级别预估待同步数据在同步任务中调度的单位时间，获取业务方配置的待同步数据的同步调度时间级别参数；The second acquisition module is used to estimate the unit time for scheduling the data to be synchronized in the synchronization task according to the data volume level corresponding to each data table in the data to be synchronized, and obtain the synchronization scheduling time level of the data to be synchronized configured by the business party parameter;
第三获取模块，用于根据所述待同步数据中每个数据表对应的数据量级别预估待同步数据在同步任务中需要开启同步节点数量，获取业务方配置的待同步数据的同步节点权重参数。The third acquisition module is used to estimate the number of synchronization nodes that need to be enabled in the synchronization task of the data to be synchronized according to the data volume level corresponding to each data table in the data to be synchronized, and obtain the synchronization node weight of the data to be synchronized configured by the business party parameter.
进一步地，所述调整单元包括：Further, the adjustment unit includes:
第一比对模块，用于将所述各个节点上待同步数据在同步任务中调度的单位时间内完成同步任务的百分比与第一预设限值进行比对，得到第一比对结果；The first comparison module is used to compare the percentage of the data to be synchronized on each node that completes the synchronization task within the unit time scheduled in the synchronization task with a first preset limit value to obtain a first comparison result;
第一调整模块，用于根据所述第一比对结果，调整所述待同步数据的同步调度时间级别参数。A first adjustment module, configured to adjust the synchronization scheduling time level parameter of the data to be synchronized according to the first comparison result.
进一步地，所述第一预设限值包括第一预设上限值以及第一预设下限值，Further, the first preset limit value includes a first preset upper limit value and a first preset lower limit value,
所述第一调整模块，具体用于如果所述各个节点上待同步数据在同步任务中调度的单位时间内完成同步任务的百分比小于第一预设下限值，则调高所述待同步数据的同步调度时间级别参数；The first adjustment module is specifically configured to increase the data to be synchronized if the percentage of the data to be synchronized on each node that completes the synchronization task within the unit time scheduled in the synchronization task is less than a first preset lower limit value. Synchronous scheduling time level parameter of ;
所述第一调整模块，具体还用于如果所述各个节点上待同步数据在同步任务中调度的单位时间内完成同步任务的百分比大于第一预设上限值，则调低所述待同步数据的同步调度时间级别参数。The first adjustment module is specifically further configured to reduce the data to be synchronized if the percentage of completion of the synchronization task per unit time scheduled in the synchronization task on each node is greater than a first preset upper limit value. Data synchronization scheduling time level parameter.
进一步地，所述调整单元还包括：Further, the adjustment unit also includes:
第二比对模块，用于将所述各个节点上待同步数据在同步任务中处理器使用率百分比与第二预设限值进行比对，得到第二比对结果；The second comparison module is used to compare the percentage of processor usage in the synchronization task of the data to be synchronized on each node with a second preset limit value to obtain a second comparison result;
第二调整模块，用于根据所述第二比对结果，调整所述待同步数据的同步节点权重参数。The second adjustment module is configured to adjust the synchronization node weight parameter of the data to be synchronized according to the second comparison result.
进一步地，所述第二预设限值包括第二预设上限值以及第二预设下限值，所述根据所述第二比对结果，Further, the second preset limit value includes a second preset upper limit value and a second preset lower limit value, and according to the second comparison result,
所述第二调整模块，具体用于如果所述各个节点上待同步数据在同步任务中处理器使用率百分比小于第二预设下限值，则调低所述待同步数据的同步节点权重参数；The second adjustment module is specifically configured to lower the synchronization node weight parameter of the data to be synchronized if the percentage of processor usage in the synchronization task of the data to be synchronized on each node is less than a second preset lower limit value ;
所述第二调整模块，具体还用于如果所述各个节点上待同步数据在同步任务中处理器使用率百分比大于第二预设上限值，则调高所述待同步数据的同步节点权重参数。The second adjustment module is further configured to increase the weight of the synchronization node of the data to be synchronized if the percentage of processor usage in the synchronization task of the data to be synchronized on each node is greater than a second preset upper limit parameter.
进一步地，所述分配单元包括：Further, the allocation unit includes:
划分模块，用于根据所述待同步数据的初始调度参数将同步节点划分为至少一个部署单元中，每个部署单元中包含至少一个同步节点，并且每个部署单元中的同步节点用于处理相同初始调度参数的待同步数据；A dividing module, configured to divide the synchronization nodes into at least one deployment unit according to the initial scheduling parameters of the data to be synchronized, each deployment unit contains at least one synchronization node, and the synchronization nodes in each deployment unit are used to process the same The data to be synchronized for the initial scheduling parameters;
分配模块，用于将相同初始调度参数的待同步数据分配到一个部署单元中的同步节点上进行数据同步。An allocation module, configured to allocate data to be synchronized with the same initial scheduling parameters to synchronization nodes in a deployment unit for data synchronization.
依据本发明又一个方面，提供了一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现数据同步的方法的步骤。According to yet another aspect of the present invention, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the data synchronization method when executing the computer program.
依据本发明再一个方面，提供了一种计算机存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现数据同步的方法的步骤。According to still another aspect of the present invention, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data synchronization method are realized.
借由上述技术方案，本发明提供一种数据同步的方法及装置，通过获取业务方配置的待同步数据的初始调度参数，使得待同步数据根据初始调度参数执行同步调度任务，无需人为分配同步调度任务，在根据初始调度参数进行数据同步过程中，监控各个同步节点上待同步数据对应的同步状态参数，并根据同步状态参数实时调整待同步数据的初始调度参数，以避免出现数据同步过程中调度时间片不平衡的情况，提高数据同步效率。与现有技术中通过认为设置数据同步任务的方式相比，本发明实施例通过设置数据同步过程中的调度参数，将待同步数据以不同的调度时间片分配到各个同步节点中，使得同步调度的并发多线程得到合理利用，并且在数据同步过程中监控数据的同步状态参数，对调度时间不合适的待同步数据的初始调度参数进行调整，以保证待同步数据以适当的调度时间片完成数据同步任务，从而保证同步业务线独占时间片内的待同步数据可以快速执行。By means of the above technical solution, the present invention provides a method and device for data synchronization. By obtaining the initial scheduling parameters of the data to be synchronized configured by the business party, the data to be synchronized can perform synchronization scheduling tasks according to the initial scheduling parameters, without artificially assigning synchronization scheduling. Task, in the process of data synchronization according to the initial scheduling parameters, monitor the synchronization status parameters corresponding to the data to be synchronized on each synchronization node, and adjust the initial scheduling parameters of the data to be synchronized in real time according to the synchronization status parameters, so as to avoid scheduling during data synchronization When the time slice is unbalanced, the efficiency of data synchronization is improved. Compared with the way of setting data synchronization tasks in the prior art, the embodiment of the present invention allocates the data to be synchronized to each synchronization node with different scheduling time slices by setting the scheduling parameters in the data synchronization process, so that the synchronization scheduling The concurrent multi-threading is reasonably utilized, and the synchronization status parameters of the data are monitored during the data synchronization process, and the initial scheduling parameters of the data to be synchronized with inappropriate scheduling time are adjusted to ensure that the data to be synchronized completes the data in an appropriate scheduling time slice Synchronization tasks, so as to ensure that the data to be synchronized within the exclusive time slice of the synchronization business line can be quickly executed.
附图说明Description of drawings
通过阅读下文优选实施方式的详细描述，各种其他的优点和益处对于本领域普通技术人员将变得清楚明了。附图仅用于示出优选实施方式的目的，而并不认为是对本发明的限制。而且在整个附图中，用相同的参考符号表示相同的部件。在附图中：Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same components. In the attached picture:
图1示出了本发明实施例提供的一种数据同步的方法流程示意图；FIG. 1 shows a schematic flowchart of a data synchronization method provided by an embodiment of the present invention;
图2示出了本发明实施例提供的另一种数据同步的方法流程示意图；FIG. 2 shows a schematic flowchart of another data synchronization method provided by an embodiment of the present invention;
图3示出了本发明实施例提供的一种数据同步的装置的结构示意图；FIG. 3 shows a schematic structural diagram of a data synchronization device provided by an embodiment of the present invention;
图4示出了本发明实施例提供的另一种数据同步的装置的结构示意图。FIG. 4 shows a schematic structural diagram of another data synchronization device provided by an embodiment of the present invention.
具体实施方式Detailed ways
下面将参照附图更详细地描述本公开的示例性实施例。虽然附图中显示了本公开的示例性实施例，然而应当理解，可以以各种形式实现本公开而不应被这里阐述的实施例所限制。相反，提供这些实施例是为了能够更透彻地理解本公开，并且能够将本公开的范围完整的传达给本领域的技术人员。Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
本发明实施例提供了一种数据同步的方法，可以实现提高数据同步效率的目的，如图1所示，该方法包括：The embodiment of the present invention provides a method for data synchronization, which can achieve the purpose of improving the efficiency of data synchronization. As shown in FIG. 1, the method includes:
101、获取业务方配置的待同步数据的初始调度参数。101. Obtain the initial scheduling parameters of the data to be synchronized configured by the business party.
其中，业务方配置的待同步数据可以为数据库中存储运营过程产生的数据，例如，服务器生成的产数据，客户端输入的用户信息数据等，还可以为平台中记录用户行为的日志数据，例如，用户浏览网页的数据，用户购买商品的数据等，这里不进行限定。Among them, the data to be synchronized configured by the business side can be the data generated during the operation process stored in the database, for example, the production data generated by the server, the user information data input by the client, etc., or the log data that records user behavior in the platform, such as , the data of the user browsing the webpage, the data of the product purchased by the user, etc., are not limited here.
对于本发明实施例，待同步数据的初始调度参数可以包括同步时间参数、同步调度时间级别参数以及同步节点权重参数，同步时间参数为预估待同步数据完成同步所需要的时间，例如，配置1小时内完成待同步数据的同步，同步调度时间级别参数为待同步数据在同步过程中调度的单位时间，例如，5分钟级别、10分钟级别、30分钟级别、1小时级别、3小时级别以及1天级别，同步节点权重参数为同步过程中需要同时开启的节点数量，例如，同时开启5台服务器或者10台服务器进行数据同步。For this embodiment of the present invention, the initial scheduling parameters of the data to be synchronized may include a synchronization time parameter, a synchronization scheduling time level parameter, and a synchronization node weight parameter. The synchronization time parameter is an estimate of the time required for the data to be synchronized to complete synchronization, for example, configuration 1 The synchronization of the data to be synchronized is completed within an hour. The parameter of the synchronization scheduling time level is the unit time for scheduling the data to be synchronized during the synchronization process, for example, 5-minute level, 10-minute level, 30-minute level, 1-hour level, 3-hour level, and 1 At the day level, the synchronization node weight parameter is the number of nodes that need to be opened at the same time during the synchronization process, for example, 5 servers or 10 servers are opened at the same time for data synchronization.
对于本发明实施例，由于业务方在向业务中心发送数据同步任务的时候，业务方会预先统计待同步数据表的同步信息，如统计哪些数据表需要进行同步，同步的截止时间限制等等，进一步根据待同步数据表统计得到的同步信息设置初始调度参数，即完成数据同步所需要的时间、同步调度时间级别以及同步节点权重参数业务方，例如，业务方要求在3日内完成15个数据表的数据同步任务，这里设置初始调度参数为同步时间参数为3天，同步调度时间级别参数为1天级别，同步节点权重参数为5台服务器，每台服务器每日分配至少1个数据表完成数据同步任务。For the embodiment of the present invention, when the business party sends a data synchronization task to the business center, the business party will pre-statize the synchronization information of the data tables to be synchronized, such as counting which data tables need to be synchronized, the deadline for synchronization, etc. Further set the initial scheduling parameters according to the synchronization information obtained from the statistics of the data tables to be synchronized, that is, the time required to complete data synchronization, the synchronization scheduling time level, and the synchronization node weight parameters. For example, the business party requires 15 data tables to be completed within 3 days The data synchronization task, where the initial scheduling parameter is set as the synchronization time parameter is 3 days, the synchronization scheduling time level parameter is 1 day level, the synchronization node weight parameter is 5 servers, and each server is assigned at least 1 data table to complete the data every day Synchronize tasks.
102、根据所述待同步数据的初始调度参数将所述待同步数据分配到各个同步节点上进行数据同步。102. Distribute the data to be synchronized to each synchronization node according to the initial scheduling parameter of the data to be synchronized to perform data synchronization.
对于本发明实施例，业务方在将待同步数据发送到数据中心后已经预先为待同步数据设置好初始调度参数，使得各个待同步的数据表对应相同或者不同的初始调度参数，为了便于数据同步任务的管理，可以将具有相同初始参数的待同步数据表分配到共同的同步节点进行数据同步，使得具有相同初始参数的待同步数据表能够按时完成数据同步任务。For the embodiment of the present invention, after sending the data to be synchronized to the data center, the business party has set the initial scheduling parameters for the data to be synchronized in advance, so that each data table to be synchronized corresponds to the same or different initial scheduling parameters, in order to facilitate data synchronization For task management, the data tables to be synchronized with the same initial parameters can be assigned to a common synchronization node for data synchronization, so that the data tables to be synchronized with the same initial parameters can complete the data synchronization task on time.
现有的数据同步方案中往往需要手动设置将某些待同步数据表放置在一起进行同步调度，以解决业务场景维度、机器性能限制等问题，无法合理安排同步调度的资源分配，本发明实施例通过分时间级别以及调度级别将待同步数据分配到不同同步节点上进行数据同步，以实现更好的数据同步效果。In the existing data synchronization solutions, it is often necessary to manually set and place some data tables to be synchronized together for synchronization scheduling to solve problems such as business scenario dimensions and machine performance limitations, and it is impossible to reasonably arrange resource allocation for synchronization scheduling. The embodiment of the present invention The data to be synchronized is allocated to different synchronization nodes for data synchronization by time level and scheduling level, so as to achieve better data synchronization effect.
103、在根据所述初始调度参数进行数据同步的过程中，监控所述各个同步节点上待同步数据对应的同步状态参数。103. During data synchronization according to the initial scheduling parameters, monitor synchronization status parameters corresponding to data to be synchronized on each synchronization node.
由于待同步数据的初始调度参数为业务方在数据同步之前初步设置的调度参数，该初步设置的调度参数并非适用于待同步数据，所以在按照初始调度参数进行数据同步的过程中，通过实时监控各个同步节点上待同步数据的同步状态参数，可以更好的了解待同步数据的同步状态，从而对待同步数据是否处于合适的同步参数进行判定，如果是，则可以无需对待同步数据对应的初始同步参数进行调整，否则，需要对待同步数据对应的初始同步参数进行调整。Since the initial scheduling parameters of the data to be synchronized are the scheduling parameters initially set by the business party before data synchronization, the initially set scheduling parameters are not applicable to the data to be synchronized, so in the process of data synchronization according to the initial scheduling parameters, through real-time monitoring The synchronization status parameters of the data to be synchronized on each synchronization node can better understand the synchronization status of the data to be synchronized, so as to determine whether the data to be synchronized is in the appropriate synchronization parameters. If so, the initial synchronization corresponding to the data to be synchronized can be unnecessary If not, the initial synchronization parameters corresponding to the data to be synchronized need to be adjusted.
其中，各个同步节点上待同步数据对应的同步状态参数可以包括各个节点上待同步数据单位时间级别内完成数据同步的百分比以及各个节点上待同步数据在同步时处理器使用率等同步状态参数，这里不进行限定。。Wherein, the synchronization status parameters corresponding to the data to be synchronized on each synchronization node may include the percentage of data synchronization completed within the unit time level of the data to be synchronized on each node and the synchronization status parameters such as the processor usage rate of the data to be synchronized on each node during synchronization, There is no limitation here. .
本发明实施例中，通过实时监控各个同步节点上待同步数据的同步状态参数，可以了解待同步数据完成同步的情况以及各个同步节点在同步过程中是否处于合适的工作状态。In the embodiment of the present invention, by monitoring the synchronization state parameters of the data to be synchronized on each synchronization node in real time, it is possible to understand the completion of synchronization of the data to be synchronized and whether each synchronization node is in a proper working state during the synchronization process.
104、根据所述各个同步节点上待同步数据对应的同步状态参数，实时调整所述待同步数据的初始调度参数，并根据调整后的调度参数进行数据同步。104. Adjust the initial scheduling parameters of the data to be synchronized in real time according to the synchronization status parameters corresponding to the data to be synchronized on each synchronization node, and perform data synchronization according to the adjusted scheduling parameters.
需要说明的是，上述待同步数据对应的同步状态参数在数据同步的过程中是实时变化的，一旦待同步数据对应的同步状态参数处于预设范围内，该预设范围通常为同步节点无法负担正常工作的范围或者待同步数据同步速度过慢的范围，都是说明当前待同步数据并非处于合适的同步状态，需要通过调整待同步数据的初始调度参数，从而将待同步数据对应的同步状态参数调整至合适的同步状态，进而以更快速度完成数据同步任务。It should be noted that the synchronization status parameters corresponding to the above-mentioned data to be synchronized change in real time during the data synchronization process. Once the synchronization status parameters corresponding to the data to be synchronized are within the preset range, the preset range is usually unaffordable by the synchronization node. The range of normal work or the range where the synchronization speed of the data to be synchronized is too slow indicates that the current data to be synchronized is not in an appropriate synchronization state. It is necessary to adjust the initial scheduling parameters of the data to be synchronized, so that the synchronization state parameters corresponding to the data to be synchronized Adjust to the appropriate synchronization state, and then complete the data synchronization task faster.
一方面，如果各个同步节点上待同步数据在单位时间级别内完成数据同步任务的百分比小于下限值，则说明待同步的数据量可能较大，以目前单位时间级别的进行数据同步可能无法完成数据同步任务，需要调高调度的单位时间级别，同理，如果各个同步节点上待同步数据在单位时间级别内完成数据同同步任务的百分比大于上限值，则说明待同步的数据量较小，以目前单位时间级别的进行数据同步可能提前完成数据同步任务，可以将其他同步节点的待同步数据分配至该同步节点，或者降低调度的单位时间级别，使得同步调度时间级别大的待同步数据独占执行，同步调度时间级别小的待同步数据尽快执行；On the one hand, if the percentage of data to be synchronized on each synchronization node that completes the data synchronization task within a unit time level is less than the lower limit value, it means that the amount of data to be synchronized may be large, and data synchronization at the current unit time level may not be completed For data synchronization tasks, the unit time level of scheduling needs to be increased. Similarly, if the percentage of data synchronization tasks completed on each synchronization node within a unit time level is greater than the upper limit, it means that the amount of data to be synchronized is small , data synchronization at the current unit time level may complete the data synchronization task in advance, and the data to be synchronized from other synchronization nodes can be allocated to the synchronization node, or the unit time level of scheduling can be reduced, so that the data to be synchronized with a large time level can be synchronized Exclusive execution, synchronous scheduling of data to be synchronized with a small time level is executed as soon as possible;
另一方面，如果各个同步节点上待同步数据在数据同步时处理器使用率等状态参数大于预设阈值，则说明数据同步任务占用处理器资源的使用率超过一定限值，需要增加同步节点权重参数来分担数据同步任务才能够保证数据同步任务在时效内完成。On the other hand, if the state parameters such as the processor usage rate of the data to be synchronized on each synchronization node are greater than the preset threshold during data synchronization, it means that the usage rate of processor resources occupied by the data synchronization task exceeds a certain limit, and the weight of the synchronization node needs to be increased Parameters to share the data synchronization task can ensure that the data synchronization task is completed within the time limit.
本发明提供一种数据同步的方法，通过获取业务方配置的待同步数据的初始调度参数，使得待同步数据根据初始调度参数执行同步调度任务，无需人为分配同步调度任务，在根据初始调度参数进行数据同步过程中，监控各个同步节点上待同步数据对应的同步状态参数，并根据同步状态参数实时调整待同步数据的初始调度参数，以避免出现数据同步过程中调度时间片不平衡的情况，提高数据同步效率。与现有技术中通过认为设置数据同步任务的方式相比，本发明实施例通过设置数据同步过程中的调度参数，将待同步数据以不同的调度时间片分配到各个同步节点中，使得同步调度的并发多线程得到合理利用，并且在数据同步过程中监控数据的同步状态参数，对调度时间不合适的待同步数据的初始调度参数进行调整，以保证待同步数据以适当的调度时间片完成数据同步任务，从而保证同步业务线独占时间片内的待同步数据可以快速执行。The present invention provides a method for data synchronization. By obtaining the initial scheduling parameters of the data to be synchronized configured by the business party, the data to be synchronized can perform synchronization scheduling tasks according to the initial scheduling parameters, without artificially assigning synchronization scheduling tasks. During the data synchronization process, monitor the synchronization status parameters corresponding to the data to be synchronized on each synchronization node, and adjust the initial scheduling parameters of the data to be synchronized in real time according to the synchronization status parameters, so as to avoid the unbalanced scheduling time slice during the data synchronization process and improve Data synchronization efficiency. Compared with the way of setting data synchronization tasks in the prior art, the embodiment of the present invention allocates the data to be synchronized to each synchronization node with different scheduling time slices by setting the scheduling parameters in the data synchronization process, so that the synchronization scheduling The concurrent multi-threading is reasonably utilized, and the synchronization status parameters of the data are monitored during the data synchronization process, and the initial scheduling parameters of the data to be synchronized with inappropriate scheduling time are adjusted to ensure that the data to be synchronized completes the data in an appropriate scheduling time slice Synchronization tasks, so as to ensure that the data to be synchronized within the exclusive time slice of the synchronization business line can be quickly executed.
本发明实施例提供了另一种数据同步的方法，可以实现提高数据同步效率的目的，如图2所示，所述方法包括：The embodiment of the present invention provides another data synchronization method, which can achieve the purpose of improving data synchronization efficiency. As shown in FIG. 2, the method includes:
201、根据待同步数据中数据量大小预先对待同步数据中每个数据表进行数据量级别分类，确定所述待同步数据中每个数据表对应的数据量级别。201. Classify the data volume level of each data table in the data to be synchronized in advance according to the data volume in the data to be synchronized, and determine the data volume level corresponding to each data table in the data to be synchronized.
对于本发明实施例，由于业务方向业务中心发送同步任务的时候，为了便于业务方对待同步数据的管理，业务方会预先对待同步数据中每个数据表的数据量进行统计，并根据统计得到结果对待同步数据中每个数据表进行数据量级别分类，例如，可以将统计数据量大于1000条的数据表划分为大表类，统计数据量为500-1000条的数据表划分为中表类，统计数据量小于5000条的数据表划分为小表类，从而确定待同步数据表中每个数据表对应的数据量级别，并且为不同数据量级别的待同步数据表配置不同的调度参数。For the embodiment of the present invention, when the business side sends a synchronization task to the business center, in order to facilitate the management of the data to be synchronized by the business side, the business side will perform statistics on the data volume of each data table in the data to be synchronized in advance, and obtain the result according to the statistics Classify each data table in the data to be synchronized by data volume level. For example, a data table with a statistical data volume greater than 1,000 items can be classified as a large table category, and a data table with a statistical data volume of 500-1,000 items can be classified as a medium table category. Data tables with less than 5,000 statistical data are divided into small table categories, so as to determine the data volume level corresponding to each data table in the data table to be synchronized, and configure different scheduling parameters for data tables to be synchronized with different data volume levels.
例如，如果待同步数据表的数据量较大，则说明待同步数据表的数据量级别较高，通常需要较长的同步时间，如果待同步数据表的数据量较小，则说明待同步数据表的数据量级别较低，通常需要较少的同步时间。For example, if the data volume of the data table to be synchronized is large, it means that the data volume level of the data table to be synchronized is relatively high, and usually requires a longer synchronization time; if the data volume of the data table to be synchronized is small, it means Tables have a lower data volume level and generally require less synchronization time.
202、根据所述待同步数据中每个数据表对应的数据量级别，获取业务方配置的待同步数据的初始调度参数。202. According to the data volume level corresponding to each data table in the data to be synchronized, acquire the initial scheduling parameters of the data to be synchronized configured by the business party.
对于本发明实施例，具体可以通过统计待同步数据中每个数据表对应的数据量，确定待同步数据表对应的数据量级别，然后根据待同步数据中每个数据表对应的数据量级别，获取业务方配置的待同步数据的初始调度参数。For the embodiment of the present invention, the data volume level corresponding to each data table in the data to be synchronized can be determined by counting the data volume corresponding to each data table in the data to be synchronized, and then according to the data volume level corresponding to each data table in the data to be synchronized, Obtain the initial scheduling parameters of the data to be synchronized configured by the business side.
具体在获取业务方配置的待同步数据的初始调度参数的过程中，可以为待同步数据表设置数据条数上限阈值A和条数下限阈值B，如果待同步数据表中数据条数大于A，可以认为待同步数据表为大表，由于大表的数据量较大，需要耗费长时间来完成数据同步，所以可以将大表的同步调度时间级别可以设置1小时级别或者3小时级别等，如果待同步数据表中的数据条数处于A与B之间，可以认为待同步数据表为中表，由于中表的数据量居中，需要耗费中等级别的时间来完成数据同步，所以可以将中表的同步调度时间级别可以设置1小时级别或者30分钟级别等，对于如果待同步数据表中的数据条数小于B，可以认为待同步数据表为小表，由于小表中的数据条数较少，短时间即可完成数据同步，所以可以将小表的同步调度时间级别设置为5分钟级别或10分钟级别等。Specifically, in the process of obtaining the initial scheduling parameters of the data to be synchronized configured by the business party, the upper limit threshold A of the number of data items and the lower limit threshold B of the number of data items can be set for the data table to be synchronized. If the number of data items in the data table to be synchronized is greater than A, It can be considered that the data table to be synchronized is a large table. Since the data volume of the large table is large, it takes a long time to complete the data synchronization. Therefore, the synchronization scheduling time level of the large table can be set to 1 hour or 3 hours. If The number of data entries in the data table to be synchronized is between A and B, and the data table to be synchronized can be considered as the middle table. Since the data volume of the middle table is in the middle, it takes a medium level of time to complete the data synchronization, so the middle table can be The synchronization scheduling time level can be set to 1 hour level or 30 minutes level, etc. If the number of data items in the data table to be synchronized is less than B, the data table to be synchronized can be considered as a small table, because the number of data items in the small table is small , data synchronization can be completed in a short time, so you can set the synchronization scheduling time level of the small table to 5 minutes or 10 minutes.
对于本发明实施例，这里初始调度参数包括同步时间参数、同步调度时间级别参数以及同步节点权重参数，具体获取业务方配置的待同步数据的各个初始调度参数包括如下步骤：根据待同步数据对应的数据量级别预估待同步数据完成同步任务所需要的时间，获取业务方配置的待同步数据的同步时间参数；根据待同步数据对应的数据量级别预估待同步数据在同步任务中调度的单位时间，获取业务方配置的待同步数据的同步调度时间级别参数；根据待同步数据对应的数据量级别预估待同步数据在同步任务中需要开启同步节点数量，获取业务方配置的待同步数据的同步节点权重参数。For the embodiment of the present invention, the initial scheduling parameters here include synchronization time parameters, synchronization scheduling time level parameters, and synchronization node weight parameters. Specifically, obtaining each initial scheduling parameter of the data to be synchronized configured by the business party includes the following steps: according to the corresponding data to be synchronized The data volume level estimates the time required for the data to be synchronized to complete the synchronization task, and obtains the synchronization time parameters of the data to be synchronized configured by the business party; estimates the unit for scheduling the data to be synchronized in the synchronization task according to the data volume level corresponding to the data to be synchronized Time, to obtain the synchronization scheduling time level parameters of the data to be synchronized configured by the business party; to estimate the number of synchronization nodes that need to be enabled in the synchronization task of the data to be synchronized according to the data volume level corresponding to the data to be synchronized, and to obtain the data to be synchronized configured by the business party Synchronize node weight parameters.
需要说明的是，这里在根据待同步数据对应的数据量级别的基础上，还可以结合待同步数据对应的数据类型，获取业务方配置的待同步数据的出师调度参数。It should be noted that, on the basis of the data volume level corresponding to the data to be synchronized, the data type corresponding to the data to be synchronized can also be combined to obtain the dispatch scheduling parameters of the data to be synchronized configured by the business party.
203、根据所述待同步数据的初始调度参数将同步节点划分为至少一个部署单元中。203. Divide synchronization nodes into at least one deployment unit according to the initial scheduling parameters of the data to be synchronized.
对于本发明实施例，每个部署单元中包含至少一个同步节点，并且每个部署单元中的同步节点用于处理相同初始调度参数的待同步数据，例如，一共有10个同步节点，将同步节点划分至A、B、C部署单元中，部署单元A中包含2个同步节点，部署单元B中包含3个同步节点，部署单元C中包含4个同步节点。For the embodiment of the present invention, each deployment unit contains at least one synchronization node, and the synchronization nodes in each deployment unit are used to process the data to be synchronized with the same initial scheduling parameters. For example, there are 10 synchronization nodes in total, and the synchronization nodes It is divided into deployment units A, B, and C. Deployment unit A contains 2 synchronization nodes, deployment unit B contains 3 synchronization nodes, and deployment unit C contains 4 synchronization nodes.
204、将相同初始调度参数的待同步数据分配到一个部署单元中的同步节点上进行数据同步。204. Allocate the data to be synchronized with the same initial scheduling parameters to the synchronization nodes in one deployment unit to perform data synchronization.
由于相同初始调度参数的待同步数据在进行同步的过程中处于相同的同步节奏，适用于相同参数配置的同步节点，所以通过将相同初始调度参数的待同步数据分配到一个部署单元中的同步节点上进行同步，能够保证业务线独占某时间片的同步节点，提高数据同步的效率。Since the data to be synchronized with the same initial scheduling parameters is in the same synchronization rhythm during the synchronization process, it is suitable for the synchronization nodes with the same parameter configuration, so by assigning the data to be synchronized with the same initial scheduling parameters to the synchronization nodes in a deployment unit Synchronization on the Internet can ensure that the business line monopolizes the synchronization node of a certain time slice and improve the efficiency of data synchronization.
需要说明的是，如果所有待同步数据被设置为相同的同步调度时间级别，则将所有节点分配到一个部署单元，如果所有待同步数据设置有N个同步调度时间级别，则根据各个同步调度时间级别的待同步数据在所有待同步数据中的占比，将不同数量的同步节点分配到多个部署单元中，例如，同步调度时间级别为1小时的待同步数据在所有待同步数据中的占比达到50，则为将一半数量的同步节点分配到一个部署单元中，该部署单元用于处理1小时级别的待同步数据，同理，根据其他同步调度时间级别将相应数量的同步节点分配到用于处理该同步时间调度级的待同步数据。It should be noted that if all the data to be synchronized are set to the same synchronization scheduling time level, then all nodes will be assigned to one deployment unit; The proportion of the level of data to be synchronized in all the data to be synchronized, and different numbers of synchronization nodes are allocated to multiple deployment units. For example, the proportion of the data to be synchronized with the synchronization scheduling time level of 1 hour When the ratio reaches 50, half of the synchronization nodes are assigned to a deployment unit, which is used to process the data to be synchronized at the level of 1 hour. Similarly, according to other synchronization scheduling time levels, the corresponding number of synchronization nodes are assigned to It is used to process the data to be synchronized at the synchronization time scheduling level.
205、在根据所述初始调度参数进行数据同步的过程中，监控所述各个同步节点上待同步数据对应的同步状态参数。205. During the process of data synchronization according to the initial scheduling parameters, monitor the synchronization status parameters corresponding to the data to be synchronized on each synchronization node.
对于本发明实施例，完成数据同步的百分比能够表明数据同步任务处理的速度，具体可以通过监控各个同步节点上在单位同步时间级别内完成数据同步的百分比来了解各个同步节点上待同步数据对应的同步状态参数；同步节点的处理器使用率能够表明同步节点是否能负担当前数据同步任务，具体还可以通过监控各个同步节点在同步过程中处理器使用率来了解各个同步节点上待同步数据对应的同步状态参数，这里不进行限定。For the embodiment of the present invention, the percentage of completed data synchronization can indicate the processing speed of the data synchronization task. Specifically, the data corresponding to the data to be synchronized on each synchronization node can be known by monitoring the percentage of data synchronization completed on each synchronization node within the unit synchronization time level. Synchronization status parameter; the processor usage rate of the synchronization node can indicate whether the synchronization node can undertake the current data synchronization task. Specifically, the processor usage rate of each synchronization node during the synchronization process can be monitored to understand the corresponding data to be synchronized on each synchronization node. Synchronization status parameters, which are not limited here.
206、根据所述各个同步节点上待同步数据对应的同步状态参数，实时调整所述待同步数据的初始调度参数，并根据调整后的调度参数进行数据同步。206. According to the synchronization state parameters corresponding to the data to be synchronized on each synchronization node, adjust the initial scheduling parameters of the data to be synchronized in real time, and perform data synchronization according to the adjusted scheduling parameters.
对于本发明实施例，当各个节点上待同步数据对应的同步状态参数为各个节点上待同步数据单位时间级别内完成数据同步的百分比时，具体地，如果各个同步节点上待同步数据在单位时间级别内完成同步的百分比小于第一预设下限值，则说明待同步数据的数据量可能较大，以目前单位时间级别的进行数据同步可能无法完成同步任务，需要调高调度的单位时间，同理，如果各个节点上待同步数据在单位时间级别内完成同步的百分比大于第一预设上限值，则说明待同步数据的数据量可能较小，以目前单位时间级别的进行数据同步可能提前完成同步任务，可以将其他节点的待同步数据分配至该节点，或者降低调度的单位时间，使得同步调度时间级别大的待同步数据独占执行，同步调度时间级别小的待同步数据尽快执行。For the embodiment of the present invention, when the synchronization status parameter corresponding to the data to be synchronized on each node is the percentage of data synchronization completed within the unit time level of the data to be synchronized on each node, specifically, if the data to be synchronized on each node is within a unit time If the percentage of completed synchronization within the level is less than the first preset lower limit, it means that the amount of data to be synchronized may be large, and data synchronization at the current unit time level may not be able to complete the synchronization task, and the unit time of scheduling needs to be increased. Similarly, if the percentage of data to be synchronized on each node that is synchronized within a unit time level is greater than the first preset upper limit, it means that the data volume of the data to be synchronized may be small, and data synchronization at the current unit time level may To complete the synchronization task in advance, you can allocate the data to be synchronized from other nodes to this node, or reduce the unit time of scheduling, so that the data to be synchronized with a large synchronization scheduling time level is exclusively executed, and the data to be synchronized with a small synchronization scheduling time level is executed as soon as possible.
对于本发明实施例，当各个节点上待同步数据对应的同步状态参数为各个节点上待同步数据在同步时处理器使用率等同步状态参数时，具体地，如果各个节点上待同步数据在同步时处理器使用率等状态参数大于第二预设上限值，则说明数据同步任务占用处理器资源的使用率超过一定限值，需要增加同步节点权重参数来分担数据同步任务，同理，同理，如果各个节点上待同步数据在同步时处理器使用率等状态参数小于第二预设上限值，则说明数据同步任务占用较少的处理器资源，可以减少同步节点权重参数，如关闭该同步节点，或者将占用资源较多的同步节点上的同步任务分配至该节点。For the embodiment of the present invention, when the synchronization state parameters corresponding to the data to be synchronized on each node are synchronization state parameters such as the processor usage rate when the data to be synchronized on each node is synchronizing, specifically, if the data to be synchronized on each node is synchronizing When the status parameters such as the processor usage rate are greater than the second preset upper limit, it means that the usage rate of the processor resources occupied by the data synchronization task exceeds a certain limit, and it is necessary to increase the synchronization node weight parameter to share the data synchronization task. If the state parameters such as the processor usage rate of the data to be synchronized on each node are less than the second preset upper limit value during synchronization, it means that the data synchronization task occupies less processor resources, and the weight parameters of the synchronization nodes can be reduced, such as turning off The synchronization node, or assign the synchronization task on the synchronization node that occupies more resources to the node.
需要说明的是，这里对上述的第一预设上限值、第一预设下限值、第二预设上限值以及第二预设下限值不进行限定，并且第一上限值大于第一下限值，第二上限值大于第二下限值，具体数值可以根据处理数据同步任务的时间情况进行确定。It should be noted that the above-mentioned first preset upper limit, first preset lower limit, second preset upper limit, and second preset lower limit are not limited here, and the first upper limit greater than the first lower limit, and the second upper limit is greater than the second lower limit, and the specific value can be determined according to the time of processing the data synchronization task.
进一步地，作为图1所述方法的具体实现，本发明实施例提供了一种数据同步的装置，如图3所示，所述装置包括：获取单元31、分配单元32、监控单元33、调整单元34。Furthermore, as a specific implementation of the method described in FIG. 1 , an embodiment of the present invention provides a device for data synchronization. As shown in FIG. 3 , the device includes: an acquisition unit 31, an allocation unit 32, a monitoring unit 33, an adjustment Unit 34.
获取单元31，可以用于获取业务方配置的待同步数据的初始调度参数；The obtaining unit 31 can be used to obtain the initial scheduling parameters of the data to be synchronized configured by the business party;
分配单元32，可以用于根据所述待同步数据的初始调度参数将所述待同步数据分配到各个同步节点上进行数据同步；The allocation unit 32 may be configured to allocate the data to be synchronized to each synchronization node for data synchronization according to the initial scheduling parameters of the data to be synchronized;
监控单元33，可以用于在根据所述初始调度参数进行数据同步的过程中，监控所述各个同步节点上待同步数据对应的同步状态参数；The monitoring unit 33 may be configured to monitor the synchronization status parameters corresponding to the data to be synchronized on each synchronization node during the data synchronization process according to the initial scheduling parameters;
调整单元34，可以用于根据所述各个同步节点上待同步数据对应的同步状态参数，实时调整所述待同步数据的初始调度参数，并根据调整后的调度参数进行数据同步。The adjustment unit 34 may be configured to adjust the initial scheduling parameters of the data to be synchronized in real time according to the synchronization status parameters corresponding to the data to be synchronized on each synchronization node, and perform data synchronization according to the adjusted scheduling parameters.
本发明提供一种数据同步的装置，通过获取业务方配置的待同步数据的初始调度参数，使得待同步数据根据初始调度参数执行同步调度任务，无需人为分配同步调度任务，在根据初始调度参数进行数据同步过程中，监控各个同步节点上待同步数据对应的同步状态参数，并根据同步状态参数实时调整待同步数据的初始调度参数，以避免出现数据同步过程中调度时间片不平衡的情况，提高数据同步效率。与现有技术中通过认为设置数据同步任务的方式相比，本发明实施例通过设置数据同步过程中的调度参数，将待同步数据以不同的调度时间片分配到各个同步节点中，使得同步调度的并发多线程得到合理利用，并且在数据同步过程中监控数据的同步状态参数，对调度时间不合适的待同步数据的初始调度参数进行调整，以保证待同步数据以适当的调度时间片完成数据同步任务，从而保证同步业务线独占时间片内的待同步数据可以快速执行。The present invention provides a device for data synchronization. By obtaining the initial scheduling parameters of the data to be synchronized configured by the business party, the data to be synchronized can perform synchronization scheduling tasks according to the initial scheduling parameters, without artificially assigning synchronization scheduling tasks. During the data synchronization process, monitor the synchronization status parameters corresponding to the data to be synchronized on each synchronization node, and adjust the initial scheduling parameters of the data to be synchronized in real time according to the synchronization status parameters, so as to avoid the unbalanced scheduling time slice during the data synchronization process and improve Data synchronization efficiency. Compared with the way of setting data synchronization tasks in the prior art, the embodiment of the present invention allocates the data to be synchronized to each synchronization node with different scheduling time slices by setting the scheduling parameters in the data synchronization process, so that the synchronization scheduling The concurrent multi-threading is reasonably utilized, and the synchronization status parameters of the data are monitored during the data synchronization process, and the initial scheduling parameters of the data to be synchronized with inappropriate scheduling time are adjusted to ensure that the data to be synchronized completes the data in an appropriate scheduling time slice Synchronization tasks, so as to ensure that the data to be synchronized within the exclusive time slice of the synchronization business line can be quickly executed.
作为图3中所示数据同步的装置的进一步说明，图4是根据本发明实施例另一种数据同步的装置的结构示意图，如图4所示，所述装置还包括：As a further description of the data synchronization device shown in FIG. 3, FIG. 4 is a schematic structural diagram of another data synchronization device according to an embodiment of the present invention. As shown in FIG. 4, the device further includes:
分类单元35，可以用于在所述获取业务方配置的待同步数据的初始调度参数之前，根据待同步数据中数据量大小预先对待同步数据中每个数据表进行数据量级别分类，确定所述待同步数据中每个数据表对应的数据量级别；The classification unit 35 may be configured to classify each data table in the data to be synchronized in advance according to the data volume level in the data to be synchronized according to the size of the data volume in the data to be synchronized before the acquisition of the initial scheduling parameters of the data to be synchronized configured by the business party, and determine the The data volume level corresponding to each data table in the data to be synchronized;
所述获取单元，具体用于根据所述待同步数据中每个数据表对应的数据量级别，获取业务方配置的待同步数据的初始调度参数。The obtaining unit is specifically configured to obtain the initial scheduling parameters of the data to be synchronized configured by the business party according to the data volume level corresponding to each data table in the data to be synchronized.
进一步地，所述初始调度参数包括同步时间参数、同步调度时间级别参数以及同步节点权重参数，所述获取单元31包括：Further, the initial scheduling parameters include a synchronization time parameter, a synchronization scheduling time level parameter, and a synchronization node weight parameter, and the acquisition unit 31 includes:
第一获取模块311，可以用于根据所述待同步数据中每个数据表对应的数据量级别预估待同步数据完成同步任务所需要的时间，获取业务方配置的待同步数据的同步时间参数；The first acquisition module 311 can be used to estimate the time required for the data to be synchronized to complete the synchronization task according to the data volume level corresponding to each data table in the data to be synchronized, and obtain the synchronization time parameters of the data to be synchronized configured by the business party ;
第二获取模块312，可以用于根据所述待同步数据中每个数据表对应的数据量级别预估待同步数据在同步任务中调度的单位时间，获取业务方配置的待同步数据的同步调度时间级别参数；The second acquisition module 312 can be used to estimate the unit time for scheduling the data to be synchronized in the synchronization task according to the data volume level corresponding to each data table in the data to be synchronized, and obtain the synchronization schedule of the data to be synchronized configured by the business party time level parameters;
第三获取模块313，可以用于根据所述待同步数据中每个数据表对应的数据量级别预估待同步数据在同步任务中需要开启同步节点数量，获取业务方配置的待同步数据的同步节点权重参数。The third acquisition module 313 can be used to estimate the number of synchronization nodes that need to be enabled in the synchronization task of the data to be synchronized according to the data volume level corresponding to each data table in the data to be synchronized, and obtain the synchronization of the data to be synchronized configured by the business party Node weight parameter.
进一步地，所述调整单元34包括：Further, the adjustment unit 34 includes:
第一比对模块341，可以用于将所述各个节点上待同步数据在同步任务中调度的单位时间内完成同步任务的百分比与第一预设限值进行比对，得到第一比对结果；The first comparison module 341 can be used to compare the percentage of the data to be synchronized on each node that completes the synchronization task within the unit time scheduled in the synchronization task with the first preset limit value to obtain the first comparison result ;
第一调整模块342，可以用于根据所述第一比对结果，调整所述待同步数据的同步调度时间级别参数。The first adjustment module 342 may be configured to adjust the synchronization scheduling time level parameter of the data to be synchronized according to the first comparison result.
进一步地，所述第一预设限值包括第一预设上限值以及第一预设下限值，Further, the first preset limit value includes a first preset upper limit value and a first preset lower limit value,
所述第一调整模块342，具体可以用于如果所述各个节点上待同步数据在同步任务中调度的单位时间内完成同步任务的百分比小于第一预设下限值，则调高所述待同步数据的同步调度时间级别参数；The first adjustment module 342 can be specifically configured to increase the waiting time if the percentage of data to be synchronized on each node that completes the synchronization task within the unit time scheduled in the synchronization task is less than a first preset lower limit value. Synchronization scheduling time level parameter of synchronous data;
所述第一调整模块342，具体还可以用于如果所述各个节点上待同步数据在同步任务中调度的单位时间内完成同步任务的百分比大于第一预设上限值，则调低所述待同步数据的同步调度时间级别参数。The first adjustment module 342 can also be specifically configured to lower the synchronization task if the percentage of data to be synchronized on each node that completes the synchronization task within the unit time scheduled in the synchronization task is greater than a first preset upper limit value. Synchronization scheduling time level parameter of the data to be synchronized.
进一步地，所述调整单元34还包括：Further, the adjustment unit 34 also includes:
第二比对模块343，可以用于将所述各个节点上待同步数据在同步任务中处理器使用率百分比与第二预设限值进行比对，得到第二比对结果；The second comparison module 343 may be configured to compare the percentage of processor usage in the synchronization task of the data to be synchronized on each node with a second preset limit value to obtain a second comparison result;
第二调整模块344，可以用于根据所述第二比对结果，调整所述待同步数据的同步节点权重参数。The second adjustment module 344 may be configured to adjust the synchronization node weight parameter of the data to be synchronized according to the second comparison result.
进一步地，所述第二预设限值包括第二预设上限值以及第二预设下限值，所述根据所述第二比对结果，Further, the second preset limit value includes a second preset upper limit value and a second preset lower limit value, and according to the second comparison result,
所述第二调整模块344，具体可以用于如果所述各个节点上待同步数据在同步任务中处理器使用率百分比小于第二预设下限值，则调低所述待同步数据的同步节点权重参数；The second adjustment module 344 can be specifically configured to reduce the synchronization node of the data to be synchronized if the processor usage percentage of the data to be synchronized on each node in the synchronization task is less than a second preset lower limit value weight parameter;
所述第二调整模块344，具体还可以用于如果所述各个节点上待同步数据在同步任务中处理器使用率百分比大于第二预设上限值，则调高所述待同步数据的同步节点权重参数。The second adjustment module 344 can also be specifically configured to increase the synchronization rate of the data to be synchronized if the processor usage percentage in the synchronization task of the data to be synchronized on each node is greater than the second preset upper limit. Node weight parameter.
进一步地，所述分配单元32包括：Further, the distribution unit 32 includes:
划分模块321，可以用于根据所述待同步数据的初始调度参数将同步节点划分为至少一个部署单元中，每个部署单元中包含至少一个同步节点，并且每个部署单元中的同步节点用于处理相同初始调度参数的待同步数据；The division module 321 may be configured to divide the synchronization nodes into at least one deployment unit according to the initial scheduling parameters of the data to be synchronized, each deployment unit contains at least one synchronization node, and the synchronization nodes in each deployment unit are used for Process the data to be synchronized with the same initial scheduling parameters;
分配模块322，可以用于将相同初始调度参数的待同步数据分配到一个部署单元中的同步节点上进行数据同步。The allocation module 322 can be configured to allocate the data to be synchronized with the same initial scheduling parameters to the synchronization nodes in a deployment unit for data synchronization.
需要说明的是，本实施例提供的一种数据同步的装置所涉及各功能单元的其他相应描述，可以参考图1和图2中的对应描述，在此不再赘述。It should be noted that for other corresponding descriptions of the functional units involved in a data synchronization device provided in this embodiment, reference may be made to the corresponding descriptions in FIG. 1 and FIG. 2 , and details are not repeated here.
基于上述如图1和图2所示方法，相应的，本实施例还提供了一种存储介质，其上存储有计算机程序，该程序被处理器执行时实现上述如图1和图2所示的数据同步的方法。Based on the method shown in Figure 1 and Figure 2 above, correspondingly, this embodiment also provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned process as shown in Figure 1 and Figure 2 is realized. method of data synchronization.
基于这样的理解，本申请的技术方案可以以软件产品的形式体现出来，该软件产品可以存储在一个非易失性存储介质(可以是CD-ROM，U盘，移动硬盘等)中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施场景所述的方法。Based on this understanding, the technical solution of the present application can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be CD-ROM, U disk, mobile hard disk, etc.), including several The instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods described in various implementation scenarios of the present application.
基于上述如图1、图2所示的方法，以及图3、图4所示的虚拟装置实施例，为了实现上述目的，本申请实施例还提供了一种计算机设备，具体可以为个人计算机、服务器、网络设备等，该实体设备包括存储介质和处理器；存储介质，用于存储计算机程序；处理器，用于执行计算机程序以实现上述如图1和图2所示的数据同步的方法。Based on the method shown in Figure 1 and Figure 2 above, and the virtual device embodiment shown in Figure 3 and Figure 4, in order to achieve the above purpose, the embodiment of this application also provides a computer device, which can be a personal computer, A server, a network device, etc., the physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the above data synchronization method as shown in Figure 1 and Figure 2 .
可选地，该计算机设备还可以包括用户接口、网络接口、摄像头、射频(RadioFrequency，RF)电路，传感器、音频电路、WI-FI模块等等。用户接口可以包括显示屏(Display)、输入单元比如键盘(Keyboard)等，可选用户接口还可以包括USB接口、读卡器接口等。网络接口可选的可以包括标准的有线接口、无线接口(如蓝牙接口、WI-FI接口)等。Optionally, the computer device may further include a user interface, a network interface, a camera, a radio frequency (Radio Frequency, RF) circuit, a sensor, an audio circuit, a WI-FI module, and the like. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the like, and optional user interfaces may also include a USB interface, a card reader interface, and the like. Optionally, the network interface may include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface) and the like.
本领域技术人员可以理解，本实施例提供的数据同步的的实体设备结构并不构成对该实体设备的限定，可以包括更多或更少的部件，或者组合某些部件，或者不同的部件布置。Those skilled in the art can understand that the data synchronization entity device structure provided in this embodiment does not constitute a limitation on the entity device, and may include more or less components, or combine some components, or arrange different components .
存储介质中还可以包括操作系统、网络通信模块。操作系统是管理上述计算机设备硬件和软件资源的程序，支持信息处理程序以及其它软件和/或程序的运行。网络通信模块用于实现存储介质内部各组件之间的通信，以及与该实体设备中其它硬件和软件之间通信。The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned computer equipment, and supports the operation of information processing programs and other software and/or programs. The network communication module is used to realize the communication between various components inside the storage medium, and communicate with other hardware and software in the physical device.
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到本申请可以借助软件加必要的通用硬件平台的方式来实现，也可以通过硬件实现。通过应用本申请的技术方案，与目前现有技术相比，本发明实施例通过设置数据同步过程中的调度参数，将待同步数据以不同的调度时间片分配到各个同步节点中，使得同步调度的并发多线程得到合理利用，并且在数据同步过程中监控数据的同步状态参数，对调度时间不合适的待同步数据的初始调度参数进行调整，以保证待同步数据以适当的调度时间片完成数据同步任务，从而保证同步业务线独占时间片内的待同步数据可以快速执行。Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software plus a necessary general-purpose hardware platform, or by hardware. By applying the technical solution of this application, compared with the current prior art, the embodiment of the present invention allocates the data to be synchronized to each synchronization node with different scheduling time slices by setting the scheduling parameters in the data synchronization process, so that the synchronization scheduling The concurrent multi-threading is reasonably utilized, and the synchronization status parameters of the data are monitored during the data synchronization process, and the initial scheduling parameters of the data to be synchronized with inappropriate scheduling time are adjusted to ensure that the data to be synchronized completes the data in an appropriate scheduling time slice Synchronization tasks, so as to ensure that the data to be synchronized within the exclusive time slice of the synchronization business line can be quickly executed.
本领域技术人员可以理解附图只是一个优选实施场景的示意图，附图中的模块或流程并不一定是实施本申请所必须的。本领域技术人员可以理解实施场景中的装置中的模块可以按照实施场景描述进行分布于实施场景的装置中，也可以进行相应变化位于不同于本实施场景的一个或多个装置中。上述实施场景的模块可以合并为一个模块，也可以进一步拆分成多个子模块。Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art can understand that the modules in the devices in the implementation scenario can be distributed among the devices in the implementation scenario according to the description of the implementation scenario, or can be located in one or more devices different from the implementation scenario according to corresponding changes. The modules of the above implementation scenarios can be combined into one module, or can be further split into multiple sub-modules.
上述本申请序号仅仅为了描述，不代表实施场景的优劣。以上公开的仅为本申请的几个具体实施场景，但是，本申请并非局限于此，任何本领域的技术人员能思之的变化都应落入本申请的保护范围。The serial numbers of the above application are for description only, and do not represent the pros and cons of the implementation scenarios. The above disclosures are only a few specific implementation scenarios of the present application, but the present application is not limited thereto, and any changes conceivable by those skilled in the art shall fall within the protection scope of the present application.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910218795.6A CN110096548A (en) | 2019-03-21 | 2019-03-21 | Data synchronous method, apparatus, computer equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910218795.6A CN110096548A (en) | 2019-03-21 | 2019-03-21 | Data synchronous method, apparatus, computer equipment and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110096548A true CN110096548A (en) | 2019-08-06 |
Family
ID=67443348
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910218795.6A Pending CN110096548A (en) | 2019-03-21 | 2019-03-21 | Data synchronous method, apparatus, computer equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110096548A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111708841A (en) * | 2020-06-08 | 2020-09-25 | 中国工商银行股份有限公司 | Database data synchronization method and device |
| CN113918656A (en) * | 2021-12-13 | 2022-01-11 | 广东电网有限责任公司 | Distributed database data synchronization method |
| CN114358079A (en) * | 2021-12-29 | 2022-04-15 | 硕橙（厦门）科技有限公司 | A kind of multi-source heterogeneous data acquisition adaptive identification method, system and storage medium |
| CN118897757A (en) * | 2024-08-08 | 2024-11-05 | 浪潮通用软件有限公司 | A method, device and medium for automatic off-site backup of financial image files |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102096599A (en) * | 2009-12-14 | 2011-06-15 | 中国移动通信集团公司 | Multi-queue task scheduling method and related system and equipment |
| US20130332505A1 (en) * | 2012-06-08 | 2013-12-12 | Commvault Systems, Inc. | Intelligent scheduling for remote computers |
| CN108449394A (en) * | 2018-03-05 | 2018-08-24 | 北京华夏电通科技有限公司 | A kind of dispatching method of data file, dispatch server and storage medium |
| CN109359160A (en) * | 2018-10-12 | 2019-02-19 | 平安科技（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN109471903A (en) * | 2018-10-25 | 2019-03-15 | 深圳壹账通智能科技有限公司 | Method, device and computer equipment for sharing blockchain information |
| CN109492018A (en) * | 2018-09-12 | 2019-03-19 | 武汉达梦数据库有限公司 | A kind of adaptive dynamic adjusting method of data synchronous system and device |
-
2019
- 2019-03-21 CN CN201910218795.6A patent/CN110096548A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102096599A (en) * | 2009-12-14 | 2011-06-15 | 中国移动通信集团公司 | Multi-queue task scheduling method and related system and equipment |
| US20130332505A1 (en) * | 2012-06-08 | 2013-12-12 | Commvault Systems, Inc. | Intelligent scheduling for remote computers |
| CN108449394A (en) * | 2018-03-05 | 2018-08-24 | 北京华夏电通科技有限公司 | A kind of dispatching method of data file, dispatch server and storage medium |
| CN109492018A (en) * | 2018-09-12 | 2019-03-19 | 武汉达梦数据库有限公司 | A kind of adaptive dynamic adjusting method of data synchronous system and device |
| CN109359160A (en) * | 2018-10-12 | 2019-02-19 | 平安科技（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN109471903A (en) * | 2018-10-25 | 2019-03-15 | 深圳壹账通智能科技有限公司 | Method, device and computer equipment for sharing blockchain information |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111708841A (en) * | 2020-06-08 | 2020-09-25 | 中国工商银行股份有限公司 | Database data synchronization method and device |
| CN111708841B (en) * | 2020-06-08 | 2024-01-23 | 中国工商银行股份有限公司 | Database data synchronization method and device |
| CN113918656A (en) * | 2021-12-13 | 2022-01-11 | 广东电网有限责任公司 | Distributed database data synchronization method |
| CN113918656B (en) * | 2021-12-13 | 2022-02-18 | 广东电网有限责任公司 | Distributed database data synchronization method |
| CN114358079A (en) * | 2021-12-29 | 2022-04-15 | 硕橙（厦门）科技有限公司 | A kind of multi-source heterogeneous data acquisition adaptive identification method, system and storage medium |
| CN118897757A (en) * | 2024-08-08 | 2024-11-05 | 浪潮通用软件有限公司 | A method, device and medium for automatic off-site backup of financial image files |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110096548A (en) | Data synchronous method, apparatus, computer equipment and computer storage medium | |
| US8412810B1 (en) | Provisioning and managing a cluster deployed on a cloud | |
| Wang et al. | Dominant resource fairness in cloud computing systems with heterogeneous servers | |
| CN108683720B (en) | Container cluster service configuration method and device | |
| US8832249B2 (en) | Methods and apparatus to adjust resource allocation in a distributive computing network | |
| US11323336B2 (en) | Network slice management method and device | |
| CN110866167B (en) | Task allocation method, device, server and storage medium | |
| WO2020024442A1 (en) | Resource allocation method and apparatus, computer device and computer-readable storage medium | |
| CN107679718B (en) | List distribution method, device, and computer-readable storage medium | |
| US20140040573A1 (en) | Determining a number of storage devices to backup objects in view of quality of service considerations | |
| US8606905B1 (en) | Automated determination of system scalability and scalability constraint factors | |
| US10628766B2 (en) | Method and system for enabling dynamic capacity planning | |
| CN103713935B (en) | Method and device for managing Hadoop cluster resources in online manner | |
| US20150254090A1 (en) | System and method for modifying allocated resources | |
| CN104503831A (en) | Equipment optimization method and device | |
| US11489942B2 (en) | Time series data analysis | |
| CN110096547A (en) | Supply synchronous method, device, computer equipment and the computer storage medium of data | |
| Zhao et al. | A Heterogeneous Virtual Machines Resource Allocation Scheme in Slices Architecture of 5G Edge Datacenter. | |
| CN106384399A (en) | IOT (Internet of Things)-office-oriented network attendance method | |
| CN111831452A (en) | Task execution method, device, storage medium and electronic device | |
| WO2018157768A1 (en) | Method and device for scheduling running device, and running device | |
| Im et al. | Scheduling jobs with non-uniform demands on multiple servers without interruption | |
| CN108833592A (en) | Cloud host schedules device optimization method, device, equipment and storage medium | |
| CN109379235B (en) | Method based on high concurrent request scene creation cloud host | |
| CN108228355A (en) | Task processing method and device, method for scheduling task and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20190806 |
|
| WD01 | Invention patent application deemed withdrawn after publication |