CN116226080A - Data migration method, device, electronic equipment and storage medium - Google Patents

Abstract

The present application discloses a data migration method, device, electronic device, and storage medium. The method includes: when monitoring a user request to be sent to the first application, sending the user request to the first application and copying the user request, wherein the user request carries identification information, and the identification information is used to indicate to read the user request or migrate the user request; according to the identification information, control the second application Reading the copied user request or sending the copied user request to a third application and writing it into a target database corresponding to the third application, where the target database includes the source database corresponding to the first application full amount of data. This embodiment ensures service continuity during data migration.

Description

Data migration method, device, electronic device and storage medium

technical field

The present application belongs to the technical field of data migration, and in particular relates to a data migration method, device, electronic equipment and storage medium.

Background technique

Data migration is a difficult point in the process of application system reconstruction and technical transformation. At present, in the database migration scenario, the data in the source database is usually migrated to the target database at one time, but a business downtime window is required, and the size of this window depends on the time for full data migration.

In the above solution, when the amount of data is large, the entire migration process may take several days, which results in long business downtime and cannot guarantee business continuity.

Contents of the invention

Embodiments of the present application provide a data migration method, device, electronic equipment, and storage medium, so as to solve the problem in the related art that the data migration process cannot guarantee business continuity.

In the first aspect, the embodiment of the present application provides a data migration method, including:

When a user request to be sent to the first application is detected, the user request is sent to the first application and the user request is copied, wherein the user request carries identification information, and the identification information uses Instructing to read the user request or migrate the user request;

According to the identification information, control the second application to read the copied user request or send the copied user request to a third application and write it into a target database corresponding to the third application, the target database includes The full data of the source database corresponding to the first application.

In the second aspect, the embodiment of the present application further provides a data migration device, including:

a sending module, configured to send the user request to the first application and copy the user request when monitoring a user request to be sent to the first application, wherein the user request carries identification information, The identification information is used to indicate to read the user request or to migrate the user request;

A processing module, configured to control the second application to read the copied user request or send the copied user request to a third application and write it into a target database corresponding to the third application according to the identification information, The target database includes all data of the source database corresponding to the first application.

In a third aspect, an embodiment of the present application provides an electronic device, the electronic device includes a processor, a memory, and a program or instruction stored in the memory and operable on the processor, and the program or instruction is The processor implements the steps of the method described in the first aspect when executed.

In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or an instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented .

In the solution provided by the embodiment of this application, when a user request to be sent to the first application is detected, the user request is sent to the first application and the user request is copied, wherein the user request carries Identification information, where the identification information is used to indicate to read the user request or to migrate the user request; according to the identification information, control the second application to read the copied user request, or to The copied user request is sent to the third application and written into the target database corresponding to the third application, and the target database includes the full amount of data of the source database corresponding to the first application, which realizes uninterrupted processing of user requests While sending to the first application for processing, the user request is copied, and the copied user request is read or written into the target database according to the requirements, thereby realizing the migration of incremental data to the target database while ensuring It ensures the continuity of business processing and solves the problem that the data migration process cannot guarantee business continuity.

Description of drawings

Fig. 1 is a schematic flow chart of the data migration method in the embodiment of the present application;

Fig. 2 is a schematic diagram of the process of the data migration method in the embodiment of the present application;

FIG. 3 is a schematic structural diagram of a data migration device in an embodiment of the present application;

FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

The terms "first", "second" and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific sequence or sequence. It should be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application can be practiced in sequences other than those illustrated or described herein, and that references to "first," "second," etc. distinguish Objects are generally of one type, and the number of objects is not limited. For example, there may be one or more first objects. In addition, "and/or" in the specification and claims means at least one of the connected objects, and the character "/" generally means that the related objects are an "or" relationship.

The data migration method provided by the embodiment of the present application will be described in detail below through specific embodiments and application scenarios with reference to the accompanying drawings.

Fig. 1 shows a data migration method provided by an embodiment of the present invention, the method may be executed by an electronic device, and the electronic device may include: a server and/or a terminal device. In other words, the method can be executed by software or hardware installed in the electronic device, and the method includes the following steps:

Step 101: In a case where a user request to be sent to a first application is detected, send the user request to the first application and copy the user request.

Specifically, the user request may be a hypertext transfer protocol (Hyper Text Transfer Protocol, HTTP) request.

When a user request to be sent to the first application is detected, the user request may be sent to the first application, so that the user request can be processed by the first application. Wherein, the first application corresponds to an active database, and the source database stores all data up to the current time.

In addition, when the user request is sent to the first application, the user request needs to be copied, and the user request carries identification information, and the identification information is used to indicate that the user request is read or the user request is read. A user requested a migration. In this way, the copied user request can be read or migrated through the identification information carried by the user request.

Step 102: According to the identification information, control the second application to read the copied user request, or send the copied user request to a third application and write it into a target database corresponding to the third application.

Specifically, if the identification information is used to indicate to read the user request, the second application can be controlled to read the copied user request; if the identification information is used to indicate to migrate the user request, the copied The user request is sent to the third application and written into the target database corresponding to the third application, so that the newly added user request can be smoothly migrated to the target database without business awareness. Moreover, the process of reading or writing data is realized through different information indicated by the identification information, and different service requirements are met.

In addition, the second application may be a test application. Of course, the test application corresponds to a test database. In this case, the stress test or function verification of the system may be implemented through the second application.

The business attributes of the third application and the first application may be the same, so that the request result fed back by the third application can be the same as the request result fed back by the first application, thereby realizing the reconstruction of the application system.

It should be noted that the target database includes the full amount of data of the source database corresponding to the first application, which makes it possible to complete the data integrity of the target database after the newly added user request is migrated to the target database.

In this way, when a user request to be sent to the first application is detected in this embodiment, the user request is sent to the first application and the user request is copied, and the second application is controlled to copy the copied user request according to the identification information carried by the user request. user request to read, or send the copied user request to a third application and write it into the target database corresponding to the third application, the target database includes the full amount of data in the source database corresponding to the first application, It realizes copying the user request while continuously sending the user request to the first application for processing, and reads the copied user request according to the requirement

Or write it into the target database, so as to realize the migration of newly added data to the target database, 5 to ensure the continuity of business processing, and solve the problem that the data migration process cannot guarantee business continuity.

In addition, in this embodiment, the duplication process requested by the user can be realized by means of openrestry+lua+kafka at the application layer. Specifically, in one embodiment, sending the user request to the first application and copying the user request includes the following steps:

The user request is sent to the first application through OpenResty, and the user request is sent to the first Topic or the second Topic in Kafka according to the identification information, wherein the first Topic is used to store the Read user requests, and the second Topic is used to store user requests to be migrated.

Specifically, OpenResty is a high-performance global wide area network (World WideWeb, Web) platform based on Nginx and Lua. Handle dynamic web applications, web services, and dynamic gateways with ultra-high concurrency and high scalability.

In this embodiment, in the nginx system, ngx.location.capture_multi of openrestry can be configured to implement concurrent replication of the same user request. Among them, this embodiment can be in the configuration file of openrestry

Set relevant parameters in the file, such as connection timeout, send timeout, read timeout, timeout for maintaining 0 high availability, connection pool size, cache size, etc., and obtain the header and body of the user request. The capture_multi API is used to copy the request message body, and forward the user request to the first application.

In addition, this embodiment can also create the Kafka producer of openrestry, and formulate the producer's message

information format, the message broker list (broker list) of the Kafka cluster, and create a first Topic for storing user requests to be read 5 and a second Topic for storing user requests to be migrated. This enables this embodiment to send the user request to the first application through OpenResty, and send the user request indicating the identification information to be read to the first topic in Kafka through OpenResty, and send the user request indicating the identification information to be migrated to In the second Topic, the data replication process is realized through OpenResty+lua+kafka.

In addition, in an embodiment, according to the identification information, the second application is controlled to read the copied user request or send the copied user request to a third application and write it into the corresponding Target database, including:

Controlling the second application to read the user request from the first Topic, or sending the user request in the second Topic to a third application and writing it into a target database corresponding to the third application.

Specifically, since the first Topic is used to store user requests to be read, and the second Topic is used to store user requests to be migrated, the second application can be controlled to read user requests from the first Topic, or the second Topic The user request in the second topic is sent to the third application and written into the target database corresponding to the third application, so that user requests of different indication types can be obtained from different topics, and user requests to be read are avoided. Write to the target database or read user requests to be migrated to the second application.

In addition, in order to ensure the processing order of requests for the same business master key, in one embodiment, both the first topic and the second topic are pre-partitioned according to the business master key information, and the user request also carries the user's business primary key information;

At this time, the sending the user request to the first Topic or the second Topic in Kafka according to the identification information includes: sending the user request to the first Topic or the second Topic according to the business master key information In the partition corresponding to the business primary key information in the second Topic.

Specifically, the service primary key information may include a user identifier, a user service order number, etc. Of course, the service primary key information corresponding to each user is unique.

By sending the user request to the partition corresponding to the business master key information in the first Topic or the second Topic according to the business master key information, only one business request corresponding to the business master key is stored in each partition, so that the messages in the partition can be passed first Out of features, to ensure the processing order of the request of the same business primary key.

The present embodiment will be specifically described below with reference to FIG. 2 .

As shown in Figure 2, assume that the application system is the nginx system. When the nginx system receives a user request, it can send the user request to the first application through OpenResty. Of course, the first application can query the user request from the source database at this time. result;

This embodiment can create the Kafka producer of openrestry, formulate the message format of the producer, the broker list of the Kafka cluster, and the Kafka consumer, and create the first Topic for storing the user request to be read and the first topic for storing the user request to be read The second topic requested by the migrated user. The user request carries identification information, and the identification information is used to indicate to read the user request or migrate the user request. At this time, the user request can be copied to the first or second topic in Kafka through OpenResty, and according to Identification information, control the second application to read the user request from the first Topic, or call the application program interface (Application Program Interface, API) related to the third application through the Kafka messenger, and send the user request in the second Topic to the second Topic The third application is written into the target database corresponding to the third application. Specifically, the second application may be a test application, and corresponds to a test database, so as to implement stress or function testing of the reconfigured application system.

It should be noted that the target database includes all data of the source database, thereby ensuring the integrity of the target database.

In this way, in this embodiment, the user request is copied while the user request is continuously sent to the first application for processing, and the copied user request is read or written into the target database according to the requirement, thereby realizing While migrating incremental data to the target database, it ensures the continuity of business processing, and can automatically adapt to data model changes and application logic optimization in the process of application system refactoring and data migration, thereby solving the problem of data chain in the process of data migration. The problem of road redundancy and data processing logic duplication.

Fig. 3 shows a schematic structural diagram of a data migration device provided by an embodiment of the present invention. As shown in Figure 3, the data migration device includes:

A sending module 301, configured to send the user request to the first application and copy the user request when monitoring a user request to be sent to the first application, wherein the user request carries identification information , the identification information is used to indicate to read the user request or to migrate the user request;

The processing module 302 is configured to control the second application to read the copied user request according to the identification information, or send the copied user request to a third application and write it into the corresponding A target database, where the target database includes all data of the source database corresponding to the first application.

In one embodiment, the sending module 301 is configured to send the user request to the first application through OpenResty, and send the user request to Kafka according to the identification information

The first Topic or the second Topic, wherein the first Topic is used to store user requests to be read, and the second Topic is used to store user requests to be migrated.

In one embodiment, both the first Topic and the second Topic are pre-partitioned according to the business master key information, and the user request also carries the user's business master key information;

The sending module is configured to send the user request to the partition corresponding to the service master key information in the first Topic or the second Topic according to the service master key information.

5 In an embodiment, the processing module is configured to control the second application to read the user request from the first Topic, or send the user request in the second Topic to a third application And write into the target database corresponding to the third application.

The data migration device provided in the embodiment of the present application can realize each process realized in the method embodiment in Fig. 1-2, and to avoid repetition, details are not repeated here.

0 It should be noted that the embodiment of the data migration device in this specification is based on the same inventive concept as the embodiment of the data migration method in this specification, so for the specific implementation of the embodiment of the data migration device, please refer to the above corresponding data migration The implementation of the method embodiment will not be described repeatedly.

The data migration device in the embodiment of the present application may be a device, or a component, an integrated circuit, or a chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, tablet computer, notebook computer, palmtop computer, vehicle electronic device, wearable device, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook or personal digital assistant (personal digital assistant) , PDA), etc., the non-mobile electronic device can be a server, a network attached storage (NetworkAttached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., the embodiment of the present application Not specifically limited.

The data migration device in the embodiment of the present application may be a device with an operating system. The operating system may be an Android (Android) operating system, an ios operating system, or other possible operating systems, which are not specifically limited in this embodiment of the present application.

Based on the same technical concept, an embodiment of the present application also provides an electronic device, which is used to execute the above-mentioned data migration method. FIG. 4 is a schematic structural diagram of an electronic device implementing various embodiments of the present application. Electronic equipment may have relatively large differences due to different configurations or performances, and may include a processor (processor) 410, a communication interface (Communications Interface) 420, a memory (memory) 430, and a communication bus 440, wherein the processor 410, the communication interface 420 , the memories 430 communicate with each other through the communication bus 440 . The processor 410 can invoke a computer program stored on the memory 430 and executable on the processor 410 to perform the following steps:

When a user request to be sent to the first application is detected, the user request is sent to the first application and the user request is copied, wherein the user request carries identification information, and the identification information uses Instructing to read the user request or migrate the user request;

According to the identification information, control the second application to read the copied user request, or send the copied user request to a third application and write it into a target database corresponding to the third application, and the target database It includes all data of the source database corresponding to the first application.

In one embodiment, the sending the user request to the first application and copying the user request includes: sending the user request to the first application through OpenResty, and copying the user request according to the identification information The user request is sent to the first Topic or the second Topic in Kafka, wherein the first Topic is used to store the user request to be read, and the second Topic is used to store the user request to be migrated.

In one embodiment, both the first Topic and the second Topic are pre-partitioned according to the business master key information, and the user request also carries the user's business master key information;

The sending the user request to the first Topic or the second Topic in Kafka according to the identification information includes: sending the user request to the first Topic or the second Topic according to the business master key information In the partition corresponding to the business primary key information described in Topic 2.

In one embodiment, according to the identification information, the second application is controlled to read the copied user request, or send the copied user request to a third application and write it into the corresponding The target database includes: controlling the second application to read the user request from the first Topic according to the identification information, or sending the user request in the second Topic to a third application and writing Enter the target database corresponding to the third application.

For specific execution steps, please refer to the steps in the above embodiments of the data migration method, and the same technical effect can be achieved. To avoid repetition, details are not repeated here.

It should be noted that the electronic device in the embodiment of the present application includes: a server, a terminal or other devices except the terminal.

The above electronic device structure does not constitute a limitation to the electronic device, and the electronic device may include more or fewer components than shown in the illustration, or combine certain components, or arrange different components, for example, an input unit may include a graphics processor (Graphics Processing Unit, GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit includes at least one of a touch panel and other input devices. A touch panel is also called a touch screen. Other input devices may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, switch buttons, etc.), trackballs, mice, and joysticks, which will not be repeated here.

The memory can be used to store software programs as well as various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, application programs or instructions required by at least one function (such as sound playback function, image playback function, etc.) etc. Also, memory may include volatile memory or nonvolatile memory, or memory may include both volatile and nonvolatile memory. Wherein, the non-volatile memory may be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electronically programmable Erase Programmable Read-Only Memory (Electrically EPROM, EEPROM) or Flash. Volatile memory can be random access memory (Random Access Memory, RAM), static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDRSDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (Synchlink DRAM, SLDRAM) and Direct Memory Bus Random Access Memory (Direct Rambus RAM, DRRAM).

The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and application programs, and the modem The tone processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the foregoing modem processor may not be integrated into the processor.

The embodiment of the present application also provides a readable storage medium, the readable storage medium stores a program or an instruction, and when the program or instruction is executed by a processor, the various processes of the above-mentioned data migration method embodiment are realized, and the same To avoid repetition, the technical effects will not be repeated here.

Wherein, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

The embodiment of the present application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above method embodiments , and can achieve the same technical effect, in order to avoid repetition, it will not be repeated here.

It should be understood that the chips mentioned in the embodiments of the present application may also be called system-on-chip, system-on-chip, system-on-a-chip, or system-on-a-chip.

It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. Functions are performed, for example, the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product in essence or the part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to enable a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in various embodiments of the present application.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Under the inspiration of this application, without departing from the purpose of this application and the scope of protection of the claims, many forms can also be made, all of which belong to the protection of this application.

Claims (10)

1. A method of data migration, comprising:

under the condition that a user request to be sent to a first application is monitored, sending the user request to the first application and copying the user request, wherein the user request carries identification information, and the identification information is used for indicating reading of the user request or migration of the user request;

and according to the identification information, controlling the second application to read the copied user request, or sending the copied user request to a third application and writing the copied user request into a target database corresponding to the third application, wherein the target database comprises the full data of the source database corresponding to the first application.

2. The data migration method of claim 1, wherein said sending the user request to the first application and replicating the user request comprises:

and sending the user request to the first application through OpenResity, and sending the user request to a first Topic Topic or a second Topic in Kafka according to the identification information, wherein the first Topic is used for storing the user request to be read, and the second Topic is used for storing the user request to be migrated.

3. The data migration method according to claim 2, wherein the first Topic and the second Topic are each partitioned in advance according to service primary key information, and the user request further carries service primary key information of a user;

the sending the user request to the first Topic or the second Topic in Kafka according to the identification information comprises the following steps:

and sending the user request to the partition corresponding to the service primary key information in the first Topic or the second Topic according to the service primary key information.

4. The data migration method according to claim 2, wherein the controlling the second application to read the copied user request according to the identification information, or sending the copied user request to a third application and writing the copied user request into a target database corresponding to the third application includes:

and according to the identification information, controlling the second application to read the user request from the first Topic, or sending the user request in the second Topic to a third application and writing the user request in the second Topic into a target database corresponding to the third application.

5. A data migration apparatus, comprising:

the sending module is used for sending the user request to the first application and copying the user request under the condition that the user request to be sent to the first application is monitored, wherein the user request carries identification information, and the identification information is used for indicating reading of the user request or migration of the user request;

and the processing module is used for controlling the second application to read the copied user request according to the identification information, or sending the copied user request to a third application and writing the copied user request into a target database corresponding to the third application, wherein the target database comprises the full data of the source database corresponding to the first application.

6. The data migration apparatus according to claim 5, wherein the sending module is configured to send the user request to the first application through openredundancy, and send the user request to a first Topic or a second Topic in Kafka according to the identification information, where the first Topic is used for storing a user request to be read, and the second Topic is used for storing a user request to be migrated.

7. The data migration apparatus of claim 6, wherein the first Topic and the second Topic are each partitioned in advance according to service primary key information, and the user request further carries service primary key information of a user;

the sending module is configured to send the user request to a partition corresponding to the service primary key information in the first Topic or the second Topic according to the service primary key information.

8. The data migration apparatus of claim 6, wherein the processing module is configured to control the second application to read the user request from the first Topic, or send the user request in the second Topic to a third application and write the user request in the third application into a target database corresponding to the third application.

9. An electronic device comprising a processor, a memory and a program or instruction stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the data migration method of any one of claims 1-4.

10. A readable storage medium, characterized in that the readable storage medium has stored thereon a program or instructions which, when executed by a processor, implement the steps of the data migration method according to any one of claims 1-4.

Images