CN112256481B

CN112256481B - Data recovery method and system in agile development

Info

Publication number: CN112256481B
Application number: CN202011155979.1A
Authority: CN
Inventors: 张文强; 孙伟; 管文琦; 王新明
Original assignee: Huatai Securities Co ltd
Current assignee: Huatai Securities Co ltd
Priority date: 2020-10-26
Filing date: 2020-10-26
Publication date: 2024-09-20
Anticipated expiration: 2040-10-26
Also published as: CN112256481A

Abstract

The invention aims to provide a data recovery method in agile development, which solves the problem of agile development data use and recovers data to any time point, and comprises the following steps: generating a data copy by taking a snapshot at a key point in time of a data copy lifecycle; establishing a historical database according to the generated data copy; receiving a data recovery request; and according to the database recovery request, a corresponding data copy is called from the historical database to recover the data of the target database. The invention solves the problem that the data is difficult to multiplex in the testing process, generates the data copy by taking the snapshot at the key time point of the life cycle of the data copy and establishes the historical database according to the generated data copy, so that the subsequent modification does not influence the basic data, can fall back to any historical data version at any time, and indirectly saves the time of exporting the data from the production environment again.

Description

Data recovery method and system in agile development

Technical Field

The invention relates to the field of data recovery in agile development, in particular to a data recovery method and a system in agile development.

Background

The agile development takes the evolution of the user's demands as the core, and adopts an iterative and progressive method to develop software. In agile development, a software project is split into a plurality of sub-projects at the initial stage of construction, and the results of all the sub-projects are tested, so that the method has the characteristics of visibility, integration and operation. In other words, a large item is divided into a plurality of small items which are connected with each other but can be independently operated, and the small items are respectively completed, and the software is always in a usable state in the process.

Along with the rapid development of company business and the promotion of agile development, the iteration speed of application versions is obviously accelerated, and the frequency of development test using different versions of data for testing is also higher and higher. In agile development, code management, test case execution, version release and the like are basically automated and tend to be mature. And for changing test data of different versions or simultaneously carrying out a plurality of scene tests on one data, a manual import and export mode is still adopted, so that the test efficiency is greatly reduced.

According to the test personnel, the problems of difficult multiplexing of data, high data preparation cost and the like exist in the process of using a plurality of data versions for testing, so that the data recovery period is long, if the data derived from production is used as basic data, the data cannot be restored to the basic data state after a plurality of tests are carried out, the testing efficiency is influenced, and no better solution exists in the industry. Meanwhile, in the test development process, when multiple branches of different projects are developed and tested in parallel, the problems of inconsistent library structures and data conflict can be caused.

Disclosure of Invention

The invention aims to solve the problem of using agile development data, and provides a data recovery method and a system in agile development, which can save time and efficiently recover the data to any time point.

In order to achieve the above purpose, the invention is realized by adopting the following technical scheme:

In a first aspect, the present invention provides a method for recovering data in agile development, including the following steps: generating a data copy by taking a snapshot at a key point in time of a data copy lifecycle; establishing a historical database according to the generated data copy; receiving a data recovery request; and according to the database recovery request, a corresponding data copy is called from the historical database to recover the data of the target database.

Further, the method for retrieving the corresponding data copy from the history database to recover the data comprises the following steps: the historical data copy is mounted when the data copy is rolled back and/or the data copy is mounted to multiple servers when tested in parallel.

Further, the snapshot at the key time point of the life cycle of the data copy includes the following steps: step A: positioning a host and a mounting point according to the IP; and (B) step (B): c, confirming whether the host computer has the mounted snapshot, if so, jumping to the step C, otherwise, ending the flow; step C: c, confirming whether the data copy is protected, if so, jumping to the step D, otherwise, adding protection to the data copy; if the added protection is successful, jumping to the step D, otherwise ending the flow; step D: stopping the target database, if the target database is successful, jumping to the step E, otherwise ending the flow; step E: taking a snapshot, if the snapshot is successful, jumping to the step F, otherwise ending the flow; step F: and starting the target database, and ending the flow after the completion.

Further, the step of mounting the historical data copy when the data copy is rolled back includes the following steps: stopping the software environment before unloading the data snapshot; unloading the current data snapshot and mounting the target data snapshot; and starting the software environment after the data snapshot is mounted.

Further, the unloading the current data snapshot includes the following steps: step A2: positioning the host according to the IP; step B2: c2, confirming whether the host computer has a mounting snapshot, if so, jumping to the step C2, otherwise, ending the flow; step C2: obtaining a current snapshot, if the current snapshot is successful, jumping to the step D2, otherwise ending the flow; step D2: stopping the target database, if the target database is successful, jumping to the step E2, otherwise ending the flow; step E2: and unloading the snapshot, and ending the flow after the snapshot is completed.

Further, the unloading the current data snapshot includes the following steps: the mounting target data snapshot comprises the following steps: step A3: positioning the host according to the IP; step B3: c3, confirming whether the host computer has the mounted snapshot, if so, jumping to the step C3, otherwise, ending the flow; step C3: obtaining a mounting point, if yes, jumping to the step D3, otherwise ending the flow; step D3: e3, confirming whether the data copy is labeled, if so, jumping to the step E3, otherwise, acquiring an original label; if the original label is successfully acquired, the step F3 is skipped, otherwise, the flow is ended; step E3: marking, if the marking is successful, jumping to the step F3, otherwise ending the flow; step F3: mounting the appointed snapshot, if successful, jumping to the step G3, otherwise ending the flow; step G3: and starting the target database, and ending the flow after the completion.

Further, the mounting the data copy to the plurality of servers during the parallel test includes the following steps: mounting the data copy to a server; and starting the software environment after the data copy is mounted.

Further, the method further comprises: preparing server resources capable of storing data copies according to the number of parallel test scenes; deploying a software environment using the data copy according to the server resource capable of storing the data copy; the server resources that can store the data copies include servers that have deployed CDM (Copy DATA MANAGEMENT, data Copy management) product drivers and turned on ISCSI (INTERNET SMALL Computer SYSTEM INTERFACE ) services.

In a second aspect, the present invention provides a data recovery system in agile development, the system comprising: and a copy generation module: for generating a data copy by taking a snapshot at a critical point in the data copy lifecycle; a database module: the target database is established according to the generated data copy; a request receiving module: for receiving a data recovery request; and a data recovery module: and the data recovery module is used for calling corresponding data copies from the target database according to the database recovery request to carry out data recovery.

Further, the request receiving module comprises a graphical page, wherein the graphical page is used for realizing the storage and mounting use of the data version, controlling the starting and stopping of the database and displaying the topological relation among a plurality of version data.

Compared with the prior art, the invention has the beneficial effects that:

1. The method solves the problem that the data is difficult to multiplex in the testing process, generates the data copy by taking the snapshot at the key time point of the life cycle of the data copy and establishes the historical database according to the generated data copy, so that the subsequent modification does not influence the basic data, can fall back to any historical data version at any time, and indirectly saves the time of exporting the data from the production environment again.

2. The invention solves the problems of inconsistent library structure and data conflict caused by multi-branch parallel development test of different projects in the test development process. During parallel testing, the data copy is mounted to a plurality of prepared servers, the same data can be mounted into a plurality of data for different project branches, the data are isolated from each other, manual input is not needed, and the testing efficiency is improved.

Drawings

FIG. 1 is a flow chart of a data recovery method in agile development of the invention;

FIG. 2 is a snapshot flow;

FIG. 3 is a data version rollback scenario;

FIG. 4 is an unload snapshot flow;

FIG. 5 is a mount snapshot flow;

fig. 6 is a parallel test scenario.

Detailed Description

The invention is further described below with reference to the accompanying drawings. The following examples are only for more clearly illustrating the technical aspects of the present invention, and are not intended to limit the scope of the present invention.

The invention has the following overall conception: the product with CDM characteristics (the product has the functions of taking a snapshot of one part of data, mounting the snapshot to a server, unloading the snapshot and the like) is subjected to secondary development, the operation between the product and the database is completed, a user can backup the current database (taking the snapshot) and recover the historical database (mounting the snapshot and unloading the snapshot) by one key, and the capacity is required when the data is recovered in agile development, so that a secondarily developed system is applied to agile development, rapid recovery of assistance data, parallel test and the like; modifications and variations may be made without departing from the technical principles of the present invention, and such modifications and variations are also considered to be within the scope of the present invention.

Embodiment one:

The embodiment provides a data recovery method and a system in agile development, the method is based on a product with CDM (data copy management) technical characteristics, server resources capable of storing data copies are prepared according to the number of parallel test scenes, a required environment is deployed, a snapshot is made at a key time point of a life cycle of the data copies, only historical data copies need to be mounted when the data copies are retracted, and the data copies need to be mounted to a plurality of prepared servers during parallel test. The method uses the CDM product in a mode of calling the CDM product API, three core technologies of generating the snapshot, unloading the snapshot and mounting the snapshot are provided for the CDM product, and data is processed (producing the snapshot, unloading the snapshot and mounting the snapshot) through the CDM product.

Referring to fig. 1, the method and system for recovering data in agile development of the invention comprise the following steps:

step 1: preparing server resources capable of storing data copies according to the number of parallel test scenes;

step 2: deploying a software environment using the data copy according to the resources prepared in the step 1;

Step 3: taking a snapshot at a key time point of a data copy life cycle;

Step 4: and selecting different operations according to the required functions, namely, only mounting historical data copies when the data copies fall back, and mounting the data copies to a plurality of prepared servers when in parallel testing.

Specifically, the server resource capable of storing a data Copy in step 1 refers to a server that has deployed CDM (Copy DATA MANAGEMENT, data Copy management) product driver and turned on ISCSI (INTERNET SMALL Computer SYSTEM INTERFACE ) service.

Specifically, the software environment for deploying the data copy in step 2 refers to installing database software (MySQL, oracle, etc.) or other software for managing data.

Specifically, the step 3 of taking the snapshot at the key time point of the life cycle of the data copy refers to taking the snapshot specific flow as shown in fig. 2 by using the snapshot technology to backup the data when the current data state needs to be maintained.

Specifically, the different operations described in step 4 are selected according to the required functions, and the specific operations are as follows:

And if the data version is returned, unloading the current data snapshot and mounting the target data snapshot, automatically stopping the software deployed in the step 2 before unloading, and automatically starting the software deployed in the step 2 after unloading. With reference to fig. 3, it is assumed that in one iteration of 4 months, the data accumulation has changed 5 times, where the two changes are large, and after testing of version 3.0 data on 12 days has ended, it is desirable to continue testing the data version for 1 day. By using the method, the current 3.0 data version is simply unloaded, and the 1.0 data version is mounted, wherein the specific flow of unloading the snapshot is shown in fig. 4, and the specific flow of mounting the snapshot is shown in fig. 5;

and if the parallel test is performed, the data copy is mounted to the servers prepared in the step 1, and the software deployed in the step 2 is automatically started after the data copy is mounted. With reference to fig. 6, it is assumed that at 8 days, the tester is temporarily informed that a test of a different scenario is required for the data of 6 days, and the data is restored after the test is finished, and continues for the test of 10 days. By using the method, the system and the device, the version 2.0 data can be simultaneously mounted to a plurality of servers for parallel test, and the version 2.1/2.2 can be deleted after the end, so that the version 2.0 is not influenced, and the specific flow of mounting the snapshot is shown in figure 5.

The method solves the problem that data is difficult to multiplex in the testing process, generates the data copy by taking the snapshot at the key time point of the life cycle of the data copy and establishes the historical database according to the generated data copy, so that the subsequent modification does not influence the basic data, can fall back to any historical data version at any time, and indirectly saves the time of exporting the data from the production environment again. Meanwhile, the problems of inconsistent library structure and data conflict caused by multi-branch parallel development testing of different projects in the test development process are solved. During parallel testing, the data copy is mounted to a plurality of prepared servers, the same data can be mounted into a plurality of data for different project branches, the data are isolated from each other, manual input is not needed, and the testing efficiency is improved.

Embodiment two:

The embodiment provides a data recovery method in agile development, which comprises the following steps:

generating a data copy by taking a snapshot at a key point in time of a data copy lifecycle;

Establishing a historical database according to the generated data copy;

receiving a data recovery request; and according to the database recovery request, a corresponding data copy is called from the historical database to recover the data of the target database.

Specifically, the method for retrieving the corresponding data copy from the historical database to recover the data comprises the following steps: the historical data copy is mounted when the data copy is rolled back and/or the data copy is mounted to multiple servers when tested in parallel.

Specifically, as shown in fig. 2, the snapshot at the key time point of the life cycle of the data copy includes the following steps:

Step A: positioning a host and a mounting point according to the IP;

And (B) step (B): c, confirming whether the host computer has the mounted snapshot, if so, jumping to the step C, otherwise, ending the flow;

step C: c, confirming whether the data copy is protected, if so, jumping to the step D, otherwise, adding protection to the data copy;

If the added protection is successful, jumping to the step D, otherwise ending the flow;

step D: stopping the target database, if the target database is successful, jumping to the step E, otherwise ending the flow;

Step E: taking a snapshot, if the snapshot is successful, jumping to the step F, otherwise ending the flow;

step F: and starting the target database, and ending the flow after the completion.

Specifically, as shown in fig. 3, the step of mounting the historical data copy when the data copy is rolled back includes the following steps:

Stopping the software environment before unloading the data snapshot;

Unloading the current data snapshot and mounting the target data snapshot;

And starting the software environment after the data snapshot is mounted.

Specifically, the unloading the current data snapshot includes the following steps:

Step one: positioning the host according to the IP;

step two: c, confirming whether the host computer has the mounted snapshot, if so, jumping to the step C, otherwise, ending the flow;

step three: the current snapshot is obtained, if the current snapshot is successful, the step D is skipped, and if the current snapshot is not successful, the process is ended;

step four: stopping the target database, if the target database is successful, jumping to the step E, otherwise ending the flow;

step five: and unloading the snapshot, and ending the flow after the snapshot is completed.

Specifically, as shown in fig. 5, the mounting target data snapshot includes the following steps:

Step one: positioning the host according to the IP;

Step three: obtaining a mounting point, if yes, jumping to the step D, otherwise ending the flow;

Step four: e, confirming whether the data copy is labeled, if so, jumping to the step E, otherwise, acquiring an original label; if the original label is successfully obtained, the step F is skipped, otherwise, the flow is ended;

Step five: marking, if the marking is successful, jumping to the step F, otherwise ending the flow;

step six: mounting the appointed snapshot, if successful, jumping to the step G, otherwise ending the flow;

Step seven: and starting the target database, and ending the flow after the completion.

Specifically, the mounting the data copy to the plurality of servers during the parallel test includes the following steps: mounting the data copy to a server; and starting the software environment after the data copy is mounted.

Specifically, the method further comprises the following steps: preparing server resources capable of storing data copies according to the number of parallel test scenes; deploying a software environment using the data copy according to the server resource capable of storing the data copy; the server resources that can store the data copies include servers that have deployed CDM (Copy DATA MANAGEMENT, data Copy management) product drivers and turned on ISCSI (INTERNET SMALL Computer SYSTEM INTERFACE ) services. Deploying a software environment using copies of data refers to installing database software (MySQL, oracle, etc.) or other software that manages data.

And if the data version is returned, unloading the current data snapshot and mounting the target data snapshot, automatically stopping deploying software before unloading, and automatically starting the deployed software after unloading. With reference to fig. 3, it is assumed that in one iteration of 4 months, the data accumulation has changed 5 times, where the two changes are large, and after testing of version 3.0 data on 12 days has ended, it is desirable to continue testing the data version for 1 day. According to the embodiment, only the current 3.0 data version is needed to be simply unloaded, and the 1.0 data version is installed, the specific flow of unloading the snapshot is shown in fig. 4, and the specific flow of installing the snapshot is shown in fig. 5;

If the parallel test is performed, the data copy is mounted to a plurality of prepared servers, and deployed software is automatically started after the data copy is mounted. With reference to fig. 6, it is assumed that at 8 days, the tester is temporarily informed that a test of a different scenario is required for the data of 6 days, and the data is restored after the test is finished, and continues for the test of 10 days. By using the method, the system and the device, the version 2.0 data can be simultaneously mounted to a plurality of servers for parallel test, and the version 2.1/2.2 can be deleted after the end, so that the version 2.0 is not influenced, and the specific flow of mounting the snapshot is shown in figure 5.

Embodiment III:

the invention provides a data recovery system in agile development, which comprises:

And a copy generation module: for generating a data copy by taking a snapshot at a critical point in the data copy lifecycle;

a database module: the method comprises the steps of establishing a historical database according to a generated data copy;

a request receiving module: for receiving a data recovery request;

And a data recovery module: and the database recovery module is used for calling corresponding data copies from the target database according to the database recovery request to recover the data of the target database.

The request receiving module comprises a graphical page, wherein the graphical page is used for realizing the storage and mounting use of the data version, controlling the starting and stopping of the database and displaying the topological relation among a plurality of version data.

The hardware of the system provided by the embodiment comprises a general server, the generated backup data is stored on a software server for installing management data, and the software for managing the data is installed on an independent server; the software is transmitted through a network, and the network communication protocol is ISCSI.

CDM products on the market were designed primarily for two purposes:

1. Data flow from the production environment to the test environment. Copying the production data to a research and development or test environment for development or test, and realizing the instant multiplexing of the data;

2. And (5) ultra-large database preparation. The backup mode of 'initial total and permanent increment' is adopted, so that the backup time window of the oversized database is greatly shortened.

The "snapshot+virtualization" function of CDM products has the ability to quickly recover copies of data. Therefore, in view of the problems existing in the data use in the agile development of the company, the present embodiment is used for the fast recovery scenario of the data in the CI/CD in combination with the technical advantages of CDM.

The main technical key points of the data recovery system in agile development provided by the embodiment are as follows:

1. the data version can be saved by one key and used by one key in a mounting way through the graphical page, and the database is automatically started and stopped, and the topological relation among a plurality of versions of data is displayed. The embodiment provides a web page for the user to realize the operation.

2. The reconfiguration package REST API encapsulates complex business logic and database management operations into 5 extremely simple friendly interfaces: logging in, taking a snapshot, mounting, unloading and viewing the snapshot version. The developer does not need to know the underlying technical principle, and can realize quick backup and recovery of data in the CI/CD only by simple calling, thereby greatly reducing the use threshold.

Embodiment four:

the embodiment of the invention also provides a data recovery device in agile development, which comprises a processor and a storage medium;

The storage medium is used for storing instructions;

the processor is configured to operate according to the instructions to perform the steps of the method of:

Establishing a historical database according to the generated data copy;

Fifth embodiment:

the embodiment of the invention also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

Establishing a historical database according to the generated data copy;

It should be noted that, the implementation environment of the present invention does not have a standard implementation environment, and the present invention can be implemented only by installing corresponding software on a server, and the application scenario is not limited to the following two types: data version rollback scenarios and parallel test scenarios, the scope of the present invention cannot be limited with the software environment and applicable scenarios in an embodiment.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The foregoing is merely a preferred embodiment of the present invention, and it should be noted that modifications and variations could be made by those skilled in the art without departing from the technical principles of the present invention, and such modifications and variations should also be regarded as being within the scope of the invention.

Claims

1. The data recovery method in agile development is characterized by comprising the following steps:

Generating a data copy by taking a snapshot of the data at the current moment at a key time point of the life cycle of the data copy;

Establishing a historical database according to the generated data copy;

receiving a data recovery request;

According to the database recovery request, corresponding data copies are called from the historical database to recover the data of the target database;

The method for retrieving the corresponding data copy from the historical database to recover the data comprises the following steps:

Mounting historical data copies when the data copies fall back, and/or mounting the data copies to a plurality of servers when parallel testing is performed;

the snapshot at the key time point of the life cycle of the data copy comprises the following steps:

Step A: positioning a host and a mounting point according to the IP;

step C: c, confirming whether the data copy is protected, if so, jumping to the step D, otherwise, adding protection to the data copy; if the added protection is successful, jumping to the step D, otherwise ending the flow;

Step E: taking a snapshot of the data in the target database, if the snapshot is successful, jumping to the step F, otherwise ending the flow;

2. The method for recovering data in agile development according to claim 1, wherein the step of mounting the historical data copy when the data copy is rolled back comprises the steps of:

Stopping the software environment before unloading the data snapshot;

Unloading the current data snapshot and mounting the target data snapshot;

And starting the software environment after the data snapshot is mounted.

3. A method of recovering data in agile development according to claim 2, wherein the offloading of the current data snapshot comprises the steps of:

Step A2: positioning the host according to the IP;

step B2: c2, confirming whether the host computer has a mounting snapshot, if so, jumping to the step C2, otherwise, ending the flow;

Step C2: obtaining a current snapshot, if the current snapshot is successful, jumping to the step D2, otherwise ending the flow;

step D2: stopping the target database, if the target database is successful, jumping to the step E2, otherwise ending the flow;

step E2: and unloading the snapshot, and ending the flow after the snapshot is completed.

4. The method for recovering data in agile development according to claim 2, wherein the mounting target data snapshot comprises the steps of:

Step A3: positioning the host according to the IP;

Step B3: c3, confirming whether the host computer has the mounted snapshot, if so, jumping to the step C3, otherwise, ending the flow;

Step C3: obtaining a mounting point, if yes, jumping to the step D3, otherwise ending the flow;

step D3: e3, confirming whether the data copy is labeled, if so, jumping to the step E3, otherwise, acquiring an original label; if the original label is successfully acquired, the step F3 is skipped, otherwise, the flow is ended;

Step E3: marking, if the marking is successful, jumping to the step F3, otherwise ending the flow;

step F3: mounting the appointed snapshot, if successful, jumping to the step G3, otherwise ending the flow;

step G3: and starting the target database, and ending the flow after the completion.

5. A method of recovering data in agile development according to claim 1, wherein mounting copies of data to a plurality of servers when tested in parallel comprises the steps of:

mounting the data copy to a server;

and starting the software environment after the data copy is mounted.

6. The method for recovering data in agile development of claim 1, further comprising: preparing server resources capable of storing data copies according to the number of parallel test scenes;

deploying a software environment using the data copy according to the server resource capable of storing the data copy;

The server resources that can store the data copies include servers that have deployed data copy management product drivers and open internet small computer system interface services.

7. A agile in-development data retrieval system, the system comprising:

a request receiving module: for receiving a data recovery request;

And a data recovery module: the method is used for retrieving corresponding data copies from the historical database according to the database recovery request to recover the data of the target database;

Step A: positioning a host and a mounting point according to the IP;

8. The system of claim 7, wherein the request receiving module comprises a graphical page for storing and mounting the data version, and controlling the database to be started and stopped and showing the topological relation among the data versions.