CN113590713A - Distributed system parameter synchronization method and device, electronic equipment and computer readable storage medium - Google Patents
Distributed system parameter synchronization method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN113590713A CN113590713A CN202110861931.0A CN202110861931A CN113590713A CN 113590713 A CN113590713 A CN 113590713A CN 202110861931 A CN202110861931 A CN 202110861931A CN 113590713 A CN113590713 A CN 113590713A
- Authority
- CN
- China
- Prior art keywords
- version number
- parameter
- cache
- batch number
- batch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明提供了一种分布式系统参数同步方法、装置、电子设备及计算机可读存储介质，涉及分布式数据维护领域，可应用于金融领域和其他领域，所述方法包含：获取参数缓存对象类型，当参数缓存对象类型为单数据表缓存时，于结构化服务器中设置zset集合和list集合；通过zset集合存储参数数据表变更的批次号和批次号对应的版本号，通过所述list集合存储每个批次号更新的表主键值及对应批次号下主键的过期时间；根据接收到的参数数据表变更通知，比较本地缓存中参数数据表对应批次号的版本号与所述zset集合中存储的对应批次号的版本号，获得第一比较结果；根据所述第一比较结果清除更新所述zset集合或本地缓存中同步数据。
The present invention provides a distributed system parameter synchronization method, device, electronic device and computer-readable storage medium, which relate to the field of distributed data maintenance and can be applied to the financial field and other fields. The method includes: obtaining a parameter cache object type , when the parameter cache object type is single data table cache, set the zset collection and the list collection in the structured server; store the batch number and the version number corresponding to the batch number of the parameter data table change through the zset collection, through the list Collectively store the table primary key value updated by each batch number and the expiration time of the primary key under the corresponding batch number; according to the received parameter data table change notification, compare the version number of the parameter data table in the local cache corresponding to the batch number with the corresponding batch number. The version number of the corresponding batch number stored in the zset collection is obtained, and the first comparison result is obtained; according to the first comparison result, the synchronization data in the zset collection or the local cache is cleared and updated.
Description
Technical Field
The invention relates to the field of distributed data maintenance, which can be applied to the financial field and other fields, in particular to a distributed system parameter synchronization method, a distributed system parameter synchronization device, electronic equipment and a computer-readable storage medium.
Background
There is a class of parameters in the banking scenario that change less frequently but are read more frequently by the various application components, such as currency type, exchange rate, institution address, etc. Aiming at the characteristics of the service scene that the parameter is read more and written less, the distributed bank core system of the unit adopts a multi-level cache architecture, and the access times of an application component to a second-level cache and a database are reduced by configuring a first-level cache and a second-level cache, so that the phenomenon that the parameter data reading time is too long due to input and output and network bandwidth bottleneck is avoided, the parameter reading efficiency is improved, and the pressure of the second-level cache and the database is reduced. However, the adoption of a multi-level cache architecture still requires ensuring that the parameters accessed by all application components are consistent and up-to-date.
In the two-level cache architecture, the parameter read by the application component accesses the local cache, i.e., the first-level cache, if the data cannot be hit, the second-level cache needs to be read upwards, and if the data cannot be hit, the database is accessed, i.e., the cache penetration phenomenon occurs. At present, two main types of mainstream multi-level cache updating schemes are provided, one is a lazy loading type, namely, data in a timed out-of-date cache is read into an upper-level cache when the application is applied to a current cache and the data is loaded into the current-level cache to realize updating. The second is complementary, that is, when data is written into the cache, information such as expiration time is written into an asynchronous queue, the queue is periodically scanned by the background thread pool, and when the data is about to expire, the data is actively read into the cache again to realize updating. The lazy loading type cache updating scheme is simple to implement, has no automatic loading and asynchronous refreshing mechanism, can frequently generate cache breakdown under the conditions of high concurrency and no searched data in the cache, and ensures that the database bears great instantaneous flow pressure and the system runs at risk. The complementary cache update scheme changes the cache refresh into an asynchronous task, and once the asynchronous queue is backlogged by messages, synchronization delay is caused, and the application reads in possibly dirty data from the cache.
Disclosure of Invention
The invention aims to provide a distributed system parameter synchronization method, a distributed system parameter synchronization device, electronic equipment and a computer readable storage medium, which are used for ensuring that when parameters are updated, data copies in each level of cache in a cache architecture are updated synchronously.
To achieve the above object, the method for synchronizing parameters of a distributed system provided by the present invention specifically comprises: acquiring a parameter cache object type, and setting a zset set and a list set in a structured server when the parameter cache object type is single data table cache; storing the batch number changed by the parameter data table and the version number corresponding to the batch number through the zset set, and storing the table primary key value updated by each batch number and the expiration time of the primary key under the corresponding batch number through the list set; according to the received parameter data table change notice, comparing the version number of the corresponding batch number of the parameter data table in the local cache with the version number of the corresponding batch number stored in the zset set to obtain a first comparison result; and clearing and updating the synchronous data in the zset or the local cache according to the first comparison result.
In the above distributed system parameter synchronization method, preferably, the clearing and updating synchronization data in the structured server or the local cache according to the first comparison result includes: when the version number of the batch number corresponding to the parameter data table in the local cache is lower than the version number of the corresponding batch number stored in the structured server, obtaining a table primary key value corresponding to a high version number through the list set; and updating the corresponding data key value in the local cache through the acquired table primary key value, and updating the version number of the batch number corresponding to the parameter data table into the version number of the corresponding batch number stored in the structured server.
In the above distributed system parameter synchronization method, preferably, the clearing and updating synchronization data in the structured server or the local cache according to the first comparison result includes: and when the version number of the batch number corresponding to the parameter data table in the local cache is higher than the version number of the corresponding batch number stored in the structured server and the list set does not have the batch number corresponding to the version number, clearing the corresponding batch number and the version number in the zset set.
In the above distributed system parameter synchronization method, preferably, the zset set is an ordered set, and each element in the ordered set is not repeated and each element is associated with a fraction of a double-precision floating point type; wherein the batch number is each element in the ordered set, and the version number is a fraction of the double-precision floating-point type.
In the above distributed system parameter synchronization method, preferably, the list set includes a character string list sorted according to an insertion order, the updated table primary key value of each batch number is stored through the character string list, and an expiration time is set for the primary key of each batch number according to a preset rule.
In the above distributed system parameter synchronization method, preferably, the method further includes: when the parameter cache object type is a custom database access component cache, constructing an association configuration table and associating one or more parameter data tables at preset positions; recording the modified version number of the parameter data table by incremental accumulation by utilizing the associated configuration table, and informing the structured server to store the latest version number in the associated configuration table; when each application monitors that the updating information exists in the self-defined database access component, clearing data of a source data table of the self-defined database access component in a local cache, and comparing the version number stored in the structured server with the locally stored version number to obtain a second comparison result; and feeding back the target data obtained by the user-defined database access component query database to the query application or feeding back the query application after storing the target data in a local cache according to the second comparison result.
The invention also provides a distributed system parameter synchronization device, which comprises a construction module, a recording module, a comparison module and a synchronization module; the construction module is used for acquiring a parameter cache object type, and when the parameter cache object type is single data table cache, a zset set and a list set are set in the structured server; the recording module is used for storing batch numbers changed by the parameter data table and version numbers corresponding to the batch numbers through the zset set, and storing table primary key values updated by each batch number and expiration times of primary keys under corresponding batch numbers through the list set; the comparison module is used for comparing the version number of the batch number corresponding to the parameter data table in the local cache with the version number of the corresponding batch number stored in the zset set according to the received parameter data table change notice to obtain a first comparison result; and the synchronization module is used for clearing and updating the synchronization data in the zset set or the local cache according to the first comparison result.
In the above distributed system parameter synchronization apparatus, preferably, the synchronization module includes an updating unit, and the updating unit is configured to obtain, through the list set, a table primary key value corresponding to a high version number when a version number of a lot number corresponding to the parameter data table in the local cache is lower than a version number of a corresponding lot number stored in the structured server; and updating the corresponding data key value in the local cache through the acquired table primary key value, and updating the version number of the batch number corresponding to the parameter data table into the version number of the corresponding batch number stored in the structured server.
In the above distributed system parameter synchronization device, preferably, the update unit further includes: and when the version number of the batch number corresponding to the parameter data table in the local cache is higher than the version number of the corresponding batch number stored in the structured server and the list set does not have the batch number corresponding to the version number, clearing the corresponding batch number and the version number in the zset set.
In the above distributed system parameter synchronization apparatus, preferably, the apparatus further includes: the construction module is also used for constructing an associated configuration table and associating one or more parameter data tables at a preset position when the parameter cache object type is a custom database access component cache; the recording module is further configured to record a modified version number of the parameter data table by incremental accumulation using the associated configuration table, and notify the structured server to store a latest version number in the associated configuration table; the comparison module is further used for clearing data of a source data table of the customized database access component in a local cache and comparing the version number stored in the structured server with the locally stored version number to obtain a second comparison result when each application monitors that the customized database access component has the update message; and the synchronization module is also used for feeding back the target data obtained by the user-defined database access component query database to the query application or feeding back the target data to the query application after storing the target data in a local cache according to the second comparison result.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method when executing the computer program.
The present invention also provides a computer-readable storage medium storing a computer program for executing the above method.
The invention has the beneficial technical effects that: the timely synchronization of the designated key value data is realized by storing the key value batch version number of the updated data table, and the frequent cache penetration caused by all cache data which are overdue regularly is avoided. Meanwhile, the overdue key value data can be still ensured to be cleared, and the problems of message backlog and synchronous delay in asynchronous message processing are also avoided. For the self-defined database access component, whether the result obtained by reading the database is written into the local cache can be determined by comparing the version number in Redis and the local library, so that the data read from the local cache by the application is ensured to be synchronous with the database.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principles of the invention. In the drawings:
fig. 1 is a schematic flow chart illustrating a distributed system parameter synchronization method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating a parameter synchronization process of a custom database access component according to an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating an application flow of single-table parameter cache synchronization according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating data structures of zset and list according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an application flow of parameter cache synchronization of a custom database access component according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a distributed system parameter synchronization apparatus according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following detailed description of the embodiments of the present invention will be provided with reference to the drawings and examples, so that how to apply the technical means to solve the technical problems and achieve the technical effects can be fully understood and implemented. It should be noted that, unless otherwise specified, the embodiments and features of the embodiments of the present invention may be combined with each other, and the technical solutions formed are within the scope of the present invention.
Additionally, the steps illustrated in the flow charts of the figures may be performed in a computer system such as a set of computer-executable instructions and, although a logical order is illustrated in the flow charts, in some cases, the steps illustrated or described may be performed in an order different than here.
Referring to fig. 1, the distributed system parameter synchronization method provided by the present invention specifically includes:
s101, acquiring a parameter cache object type, and setting a zset set and a list set in a structured server when the parameter cache object type is single data table cache;
s102, storing batch numbers changed by the parameter data table and version numbers corresponding to the batch numbers through the zset set, and storing table primary key values updated by each batch number and expiration times of primary keys under corresponding batch numbers through the list set;
s103, comparing the version number of the batch number corresponding to the parameter data table in the local cache with the version number of the corresponding batch number stored in the zset set according to the received parameter data table change notice to obtain a first comparison result;
s104, clearing and updating the synchronous data in the zset or the local cache according to the first comparison result.
Specifically, the distributed system parameter synchronization method provided by the invention is mainly characterized in that a zset set is arranged in a structured server Redis to store the batch number of parameter data table change and the version number corresponding to the batch, and a list is also arranged to store the table primary key value updated by each batch number and the expiration time of the primary key under the batch number. When the parameter data table is updated, all applications receive the change notification, the version number of the batch corresponding to the parameter table in the local cache is compared with the version number of the batch stored by Redis, if the version number in the local cache is not the latest, the data of the batch corresponding to the key value in the local cache is cleared, the latest version number of the local batch is updated, and when the applications cannot find the cleared data, the applications read new data from the upper-level cache and write the new data into the local cache, so that the parameter cache synchronization is realized. The design avoids the problem that the asynchronous message backlog can not synchronize data in time in a supplementary cache updating scheme. In addition, updating the cache according to the version number of the batch also avoids that data which does not need to be updated is removed at regular time due to expiration in a lazy loading scheme, and the possibility of cache penetration is reduced.
In actual work, except for the cache synchronization of a common data table, the cache synchronization of a user-defined database access component is also realized; the custom database access component is a database access component developed by a programmer in a custom manner, and is generally a relatively complex SQL statement, such as multi-table connection query, aggregation query and the like. Such operations are capabilities specific to relational databases and cannot be supported by Redis caches. Meanwhile, the program is a relatively complex database operation, is often poor in performance, and is the most valuable component for caching. Thus, referring to fig. 2, in an embodiment of the present invention:
s201, when the parameter cache object type is a custom database access component cache, constructing an association configuration table and associating one or more parameter data tables at preset positions;
s202, recording the modified version number of the parameter data table by incremental accumulation by using the associated configuration table, and informing the structured server to store the latest version number in the associated configuration table;
s203, when each application monitors that the updating information exists in the self-defined database access component, clearing the data of the source data table of the self-defined database access component in the local cache, and comparing the version number stored in the structured server with the locally stored version number to obtain a second comparison result;
s204, according to the second comparison result, the target data obtained by the user-defined database access component through querying the database is fed back to the query application or stored in a local cache and then fed back to the query application.
Specifically, the customized database access component is a database access component developed by a programmer in a customized manner, and is generally a relatively complex SQL statement, such as multi-table join query, aggregation query, and the like. Such operations are capabilities specific to relational databases and cannot be supported by Redis caches. Meanwhile, the program is a relatively complex database operation, is often poor in performance, and is the most valuable component for caching. We have therefore devised a caching and synchronization scheme for such programs. A cache synchronization version table is designed to record the current version number of each table, when the DBBU cache is changed, the cache of the table in the local cache is deleted, and the comparison result of the version numbers in the local library and the Redis is stored. And when the application does not search the data in the local cache and the comparison result shows that the version number in the local library is latest, writing the result of reading the database into the local cache to realize parameter cache synchronization.
In an embodiment of the present invention, the clearing and updating the synchronization data in the structured server or the local cache according to the first comparison result includes: when the version number of the batch number corresponding to the parameter data table in the local cache is lower than the version number of the corresponding batch number stored in the structured server, obtaining a table primary key value corresponding to a high version number through the list set; and updating the corresponding data key value in the local cache through the acquired table primary key value, and updating the version number of the batch number corresponding to the parameter data table into the version number of the corresponding batch number stored in the structured server. Further, when the version number of the batch number corresponding to the parameter data table in the local cache is higher than the version number of the corresponding batch number stored in the structured server and the list set does not have the batch number corresponding to the version number, the corresponding batch number and the version number in the zset are removed.
Wherein the zset set is an ordered set, each element in the ordered set is not repeated, and each element is associated with a fraction of a double-precision floating point type; wherein the batch number is each element in the ordered set, and the version number is a fraction of the double-precision floating-point type; the list set comprises a character string list which is sorted according to an insertion sequence, the table primary key value updated by each batch number is stored through the character string list, and the expiration time of the primary key of each batch number is set according to a preset rule.
Specifically, referring to fig. 3, in actual work, the core flow of parameter cache synchronization of a common single data table includes the following steps:
firstly, monitoring the cache change condition of a data table by an AP (application unit), comparing the size between a batch number corresponding to the version number of the data table and a batch number corresponding to a local version number inquired in zset of Redis after receiving a data table cache change notice, and determining whether to perform synchronization or outdated data cleaning;
specifically, when a batch number of which the version number of the data table is greater than the local version number is queried from zset of Redis, an updated data key value is read from the Redis list according to the version number, and data of the key value in a first-level cache is cleared, so that the latest version number of the batch is locally recorded; therefore, the timely synchronization of the designated key value data is realized, and the frequent cache penetration caused by all cache data with the timing expiration is avoided. When a batch number of which the version number of the data table is smaller than the local version number is inquired from zset of Redis, whether a list corresponding to the batch number exists in the Redis is inquired; if not, the expired batch number and the version number thereof in the zset are cleared, so that the expired key-value data is cleared.
Redis Redis
Both zset and list data structures of Redis are shown in FIG. 4, where zset is an ordered set, each element in the set is not repeatable, and each element (value in zset) is associated with a score (score in zset) of double type, and Redis orders the elements in the set according to the scores. The batch number can be stored in zset as value and the version number as score, and the batch number of the latest version can be obtained by sorting the version numbers by Redis. List in Redis is a list of strings ordered in insertion order, and the expiration time of the list may be set. All keys for changing data under a certain batch number can be stored in the list, and the expiration time of the batch number is set, so that invalid keys can be deleted in time, and the memory space is saved. Therefore, the invention realizes the timely synchronization of the appointed key value data by storing the key value batch version number of the updated data table, and avoids the frequent cache penetration brought by all cache data with overdue timing. Meanwhile, the scheme still ensures that the expired key value data can be cleared. The scheme also avoids the problems of message backlog and synchronous delay in asynchronous message processing.
The parameter cache synchronization of the customized database access component can be referred to as shown in fig. 5, and the specific steps are as follows:
firstly, a configuration file is constructed for configuring tables associated with the DBBU, so that when the tables are changed, cache change and a cache synchronization version table are triggered to record the version number of each data table, when a data source table inquired by a user-defined database access component is updated, the version numbers of the table in the cache synchronization version table are increased and accumulated, and the latest version number of the table is stored in Redis.
Then, as shown in fig. 5, when the AP, that is, each application, monitors update messages of the DBBU, such as a DBBU cache change notification, the AP clears data in the local cache of the DBBU source data table, and then compares the version number in the Redis with the version number in the local library;
if the version numbers are inconsistent, the synchronization delay of the version numbers is indicated, the size relationship between the version number in the Redis and the version number in the local library needs to be judged at the moment, and if the version number in the Redis is larger than the version number of the local library, comparison is carried out again after the exponential sleep (the comparison times are limited in the process); if the version number in Redis is less than or equal to the version number of the local library, caching the comparison result to the local; therefore, whether the result obtained by reading the database is written into the local cache is determined by comparing the version number in Redis and the local library, so that the data read from the local cache by the application is ensured to be synchronous with the database.
In the actual operation process, when each application makes DBBU query, the local cache is read first, and if the application hits the local cache, the data in the local cache is returned;
if not, inquiring the database; reading the version number comparison result cached locally and determining whether the databases are synchronous or not; if the version numbers are consistent, storing the result of the database query into a local cache and then returning the data, otherwise, directly returning the query result of the database.
Therefore, for the user-defined database access component, whether the result obtained by reading the database is written into the local cache is determined by comparing the version number in the Redis and the version number in the local database, so that the data read from the local cache by the application is ensured to be synchronous with the database.
Referring to fig. 6, the present invention further provides a distributed system parameter synchronization apparatus, which includes a building module 601, a recording module 602, a comparing module 603, and a synchronization module 604; the building module 601 is configured to obtain a parameter cache object type, and set a zset set and a list set in a structured server when the parameter cache object type is a single data table cache; the recording module 602 is configured to store, through the zset set, a batch number changed by the parameter data table and a version number corresponding to the batch number, and store, through the list set, an updated table primary key value of each batch number and an expiration time of a primary key under a corresponding batch number; the comparing module 603 is configured to compare, according to the received parameter data table change notification, the version number of the batch number corresponding to the parameter data table in the local cache with the version number of the corresponding batch number stored in the zset set, and obtain a first comparison result; the synchronization module 604 is configured to clear and update the synchronized data in the zset set or the local cache according to the first comparison result. Therefore, for a common single data table, by using the data structure characteristics of the data structures zset and list of the Redis, associating the table and the key values by using the batch number and the version number, the specified clearing of the key value data of a certain table in the cache can be realized; meanwhile, invalid data is cleared in time by using the expiration time characteristic of list in Redis, and the storage space is saved.
In the above embodiment, the synchronization module includes an updating unit, and the updating unit is configured to obtain, through the list set, a table primary key value corresponding to a high version number when a version number of a batch number corresponding to a parameter data table in a local cache is lower than a version number of a corresponding batch number stored in the structured server; and updating the corresponding data key value in the local cache through the acquired table primary key value, and updating the version number of the batch number corresponding to the parameter data table into the version number of the corresponding batch number stored in the structured server. Further, the update unit further includes: when the version number of the batch number corresponding to the parameter data table in the local cache is higher than the version number of the corresponding batch number stored in the structured server and the list set does not have the batch number corresponding to the version number, clearing the corresponding batch number and the version number in the zset set; the detailed description of the embodiments is given in the foregoing embodiments, and the detailed description is omitted here.
In another embodiment of the present invention, the apparatus further comprises: the construction module is also used for constructing an associated configuration table and associating one or more parameter data tables at a preset position when the parameter cache object type is a custom database access component cache; the recording module is further configured to record a modified version number of the parameter data table by incremental accumulation using the associated configuration table, and notify the structured server to store a latest version number in the associated configuration table; the comparison module is further used for clearing data of a source data table of the customized database access component in a local cache and comparing the version number stored in the structured server with the locally stored version number to obtain a second comparison result when each application monitors that the customized database access component has the update message; and the synchronization module is also used for feeding back the target data obtained by the user-defined database access component query database to the query application or feeding back the target data to the query application after storing the target data in a local cache according to the second comparison result. Therefore, the version number of the cache synchronization version table record is designed for the user-defined database access component, whether the result of reading the database is written into the local cache is determined according to the comparison result of the version number between Redis and the local library when the data is not hit in the local cache, and the synchronized data in the local cache is ensured to be up-to-date and consistent with the database.
The invention has the beneficial technical effects that: the timely synchronization of the designated key value data is realized by storing the key value batch version number of the updated data table, and the frequent cache penetration caused by all cache data which are overdue regularly is avoided. Meanwhile, the overdue key value data can be still ensured to be cleared, and the problems of message backlog and synchronous delay in asynchronous message processing are also avoided. For the self-defined database access component, whether the result obtained by reading the database is written into the local cache can be determined by comparing the version number in Redis and the local library, so that the data read from the local cache by the application is ensured to be synchronous with the database.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method when executing the computer program.
The present invention also provides a computer-readable storage medium storing a computer program for executing the above method.
As shown in fig. 7, the electronic device 700 may further include: communication module 110, input unit 120, audio processing unit 130, display 160, power supply 170. It is noted that the electronic device 700 does not necessarily include all of the components shown in fig. 7; furthermore, the electronic device 700 may also comprise components not shown in fig. 7, reference being made to the prior art.
As shown in fig. 7, the central processor 100, sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, the central processor 100 receiving input and controlling the operation of the various components of the electronic device 700.
The memory 140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 100 may execute the program stored in the memory 140 to realize information storage or processing, etc.
The input unit 120 provides input to the cpu 100. The input unit 120 is, for example, a key or a touch input device. The power supply 170 is used to provide power to the electronic device 700. The display 160 is used to display an object to be displayed, such as an image or a character. The display may be, for example, an LCD display, but is not limited thereto.
The memory 140 may be a solid state memory such as Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 140 may also be some other type of device. Memory 140 includes buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application/function storage section 142, and the application/function storage section 142 is used to store application programs and function programs or a flow for executing the operation of the electronic device 700 by the central processing unit 100.
The memory 140 may also include a data store 143, the data store 143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by the electronic device. The driver storage portion 144 of the memory 140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging application, address book application, etc.).
The communication module 110 is a transmitter/receiver 110 that transmits and receives signals via an antenna 111. The communication module (transmitter/receiver) 110 is coupled to the central processor 100 to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method for synchronizing parameters of a distributed system, the method comprising:
acquiring a parameter cache object type, and setting a zset set and a list set in a structured server when the parameter cache object type is single data table cache;
storing the batch number changed by the parameter data table and the version number corresponding to the batch number through the zset set, and storing the table primary key value updated by each batch number and the expiration time of the primary key under the corresponding batch number through the list set;
according to the received parameter data table change notice, comparing the version number of the corresponding batch number of the parameter data table in the local cache with the version number of the corresponding batch number stored in the zset set to obtain a first comparison result;
and clearing and updating the synchronous data in the zset or the local cache according to the first comparison result.
2. The distributed system parameter synchronization method of claim 1, wherein the clearing and updating of the synchronization data in the structured server or the local cache according to the first comparison result comprises:
when the version number of the batch number corresponding to the parameter data table in the local cache is lower than the version number of the corresponding batch number stored in the structured server, obtaining a table primary key value corresponding to a high version number through the list set;
and updating the corresponding data key value in the local cache through the acquired table primary key value, and updating the version number of the batch number corresponding to the parameter data table into the version number of the corresponding batch number stored in the structured server.
3. The distributed system parameter synchronization method of claim 1, wherein the clearing and updating of the synchronization data in the structured server or the local cache according to the first comparison result comprises:
and when the version number of the batch number corresponding to the parameter data table in the local cache is higher than the version number of the corresponding batch number stored in the structured server and the list set does not have the batch number corresponding to the version number, clearing the corresponding batch number and the version number in the zset set.
4. The distributed system parameter synchronization method of claim 1, wherein the zset set is an ordered set, and each element in the ordered set is not repeated and each element is associated with a double-precision floating-point type score; wherein the batch number is each element in the ordered set, and the version number is a fraction of the double-precision floating-point type.
5. The distributed system parameter synchronization method according to claim 4, wherein the list set includes a list of strings sorted according to insertion order, the updated table primary key value of each batch number is stored through the list of strings, and an expiration time is set for the primary key of the batch number according to a preset rule.
6. The distributed system parameter synchronization method of claim 1, further comprising:
when the parameter cache object type is a custom database access component cache, constructing an association configuration table and associating one or more parameter data tables at preset positions;
recording the modified version number of the parameter data table by incremental accumulation by utilizing the associated configuration table, and informing the structured server to store the latest version number in the associated configuration table;
when each application monitors that the updating information exists in the self-defined database access component, clearing data of a source data table of the self-defined database access component in a local cache, and comparing the version number stored in the structured server with the locally stored version number to obtain a second comparison result;
and feeding back the target data obtained by the user-defined database access component query database to the query application or feeding back the query application after storing the target data in a local cache according to the second comparison result.
7. A distributed system parameter synchronization device is characterized by comprising a construction module, a recording module, a comparison module and a synchronization module;
the construction module is used for acquiring a parameter cache object type, and when the parameter cache object type is single data table cache, a zset set and a list set are set in the structured server;
the recording module is used for storing batch numbers changed by the parameter data table and version numbers corresponding to the batch numbers through the zset set, and storing table primary key values updated by each batch number and expiration times of primary keys under corresponding batch numbers through the list set;
the comparison module is used for comparing the version number of the batch number corresponding to the parameter data table in the local cache with the version number of the corresponding batch number stored in the zset set according to the received parameter data table change notice to obtain a first comparison result;
and the synchronization module is used for clearing and updating the synchronization data in the zset set or the local cache according to the first comparison result.
8. The distributed system parameter synchronization apparatus according to claim 7, wherein the synchronization module includes an updating unit, and the updating unit is configured to obtain, through the list set, a table primary key value corresponding to a high version number when a version number of a lot number corresponding to the parameter data table in the local cache is lower than a version number of a corresponding lot number stored in the structured server; and updating the corresponding data key value in the local cache through the acquired table primary key value, and updating the version number of the batch number corresponding to the parameter data table into the version number of the corresponding batch number stored in the structured server.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the method of any one of claims 1 to 6 by a computer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110861931.0A CN113590713A (en) | 2021-07-29 | 2021-07-29 | Distributed system parameter synchronization method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110861931.0A CN113590713A (en) | 2021-07-29 | 2021-07-29 | Distributed system parameter synchronization method and device, electronic equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113590713A true CN113590713A (en) | 2021-11-02 |
Family
ID=78251588
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110861931.0A Pending CN113590713A (en) | 2021-07-29 | 2021-07-29 | Distributed system parameter synchronization method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113590713A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114237806A (en) * | 2021-12-29 | 2022-03-25 | 瀚云科技有限公司 | Page information display method and device, electronic equipment and storage medium |
| CN114328629A (en) * | 2021-12-31 | 2022-04-12 | 中国联合网络通信集团有限公司 | Data caching method, device, electronic device and storage medium |
| CN114741373A (en) * | 2022-04-14 | 2022-07-12 | 中国银行股份有限公司 | Data access method and cache assembly |
| CN116108028A (en) * | 2022-02-21 | 2023-05-12 | 漳州立达信光电子科技有限公司 | Data processing method, device, server and computer-readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150334182A1 (en) * | 2012-12-17 | 2015-11-19 | Beijing Qihoo Technology Limited | System, Method and Browser Client for Enabling Browser Data Synchronization |
| CN109413127A (en) * | 2017-08-18 | 2019-03-01 | 北京京东尚科信息技术有限公司 | A kind of method of data synchronization and device |
| CN111723110A (en) * | 2019-03-21 | 2020-09-29 | 阿里巴巴集团控股有限公司 | Distributed cache system and associated query and update method, device and storage medium |
| CN111858556A (en) * | 2020-07-22 | 2020-10-30 | 浪潮云信息技术股份公司 | Distributed cache management method based on version control and manager |
| CN112749198A (en) * | 2021-01-21 | 2021-05-04 | 中信银行股份有限公司 | Multi-level data caching method and device based on version number |
-
2021
- 2021-07-29 CN CN202110861931.0A patent/CN113590713A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150334182A1 (en) * | 2012-12-17 | 2015-11-19 | Beijing Qihoo Technology Limited | System, Method and Browser Client for Enabling Browser Data Synchronization |
| CN109413127A (en) * | 2017-08-18 | 2019-03-01 | 北京京东尚科信息技术有限公司 | A kind of method of data synchronization and device |
| CN111723110A (en) * | 2019-03-21 | 2020-09-29 | 阿里巴巴集团控股有限公司 | Distributed cache system and associated query and update method, device and storage medium |
| CN111858556A (en) * | 2020-07-22 | 2020-10-30 | 浪潮云信息技术股份公司 | Distributed cache management method based on version control and manager |
| CN112749198A (en) * | 2021-01-21 | 2021-05-04 | 中信银行股份有限公司 | Multi-level data caching method and device based on version number |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114237806A (en) * | 2021-12-29 | 2022-03-25 | 瀚云科技有限公司 | Page information display method and device, electronic equipment and storage medium |
| CN114237806B (en) * | 2021-12-29 | 2024-03-12 | 瀚云科技有限公司 | Page information display method and device, electronic equipment and storage medium |
| CN114328629A (en) * | 2021-12-31 | 2022-04-12 | 中国联合网络通信集团有限公司 | Data caching method, device, electronic device and storage medium |
| CN116108028A (en) * | 2022-02-21 | 2023-05-12 | 漳州立达信光电子科技有限公司 | Data processing method, device, server and computer-readable storage medium |
| CN114741373A (en) * | 2022-04-14 | 2022-07-12 | 中国银行股份有限公司 | Data access method and cache assembly |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113590713A (en) | Distributed system parameter synchronization method and device, electronic equipment and computer readable storage medium | |
| US11119997B2 (en) | Lock-free hash indexing | |
| US9639468B2 (en) | Temporal clones to identify valid items from a set of items | |
| US11636083B2 (en) | Data processing method and apparatus, storage medium and electronic device | |
| US10725995B2 (en) | Automatically revising synopsis table structure | |
| JP5577350B2 (en) | Method and system for efficient data synchronization | |
| CN102111448B (en) | Data prefetching method of DHT memory system and node and system | |
| US11275759B2 (en) | Data storage method and apparatus, server, and storage medium | |
| CN106557578B (en) | Historical data query method and system | |
| CN102541757B (en) | Write cache method, cache synchronization method and device | |
| EP3465473B1 (en) | Versioning and non-disruptive servicing of in-memory units in a database | |
| CN111046034A (en) | Method and system for managing memory data and maintaining data in memory | |
| CN106462592A (en) | Systems and methods to optimize multi-version support in indexes | |
| US10055442B2 (en) | Efficient updates in non-clustered column stores | |
| US20130254240A1 (en) | Method of processing database, database processing apparatus, computer program product | |
| JP6820918B2 (en) | Methods and equipment for data processing | |
| US20170329836A1 (en) | Database transfer of changes | |
| CN105975638A (en) | NoSQL-based massive small file storage structure for aviation logistics and storage method of NoSQL-based massive small file storage structure | |
| US7403958B2 (en) | Synchronization-replication concurrency using non-shared snapshot query on a history table at read-uncommitted isolation level | |
| CN116340328A (en) | Data update method, data query method, device and storage medium | |
| CN115934583B (en) | Hierarchical caching method, device and system | |
| KR101806394B1 (en) | A data processing method having a structure of the cache index specified to the transaction in a mobile environment dbms | |
| CN113656380B (en) | Data lifecycle management method and device | |
| US11068451B2 (en) | Database column refresh via replacement | |
| CN115718770A (en) | Barrage processing method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |