CN117056313A - Memory database transaction management method and device, electronic equipment and medium - Google Patents

Abstract

The present application relates to the field of in-memory database management, and in particular to an in-memory database transaction management method, device, electronic device and medium. The method includes obtaining the transaction identifier of the read-only transaction when a read-only transaction request is detected to access a data row. Several versions of data information corresponding to data rows. Each version of data information includes version data rows, version transaction identifiers, read times, transaction access intervals, and log persistence parameters; transaction identifiers and several versions based on read-only transactions The corresponding transaction access interval of the data information determines the target version data information from several versions of data information. The target version data information is the version data information accessed by the read-only transaction. Based on the log persistence parameters of the target version data information, the visible version data information is determined. Version data row. This application can ensure the correctness of transactions in the DolphinDB in-memory database.

Description

In-memory database transaction management method, device, electronic equipment and medium

Technical field

The present application relates to the field of memory database management, and in particular to a memory database transaction management method, device, electronic equipment and media.

Background technique

DolphinDB is a high-performance, distributed in-memory database designed to handle large-scale data analysis and real-time data processing tasks. In order to ensure the atomicity and durability of transactions in the in-memory database, the DolphinDB in-memory database uses a two-phase locking protocol and multi-version Concurrency control mechanism. Among them, the two-stage lock protocol requires that the write lock of the read-write transaction cannot be released before the log of the read-write transaction is written to disk. As a result, concurrent read-only transactions that have data intersection with the read-write transaction cannot obtain the read lock in time.

In related technologies, the write lock of read-write transactions is released before the log is written to disk, so that concurrent read-only transactions can obtain read locks, reducing the lock time of data rows and improving the concurrency performance of the in-memory database. Among them, the read-write transaction releases the lock immediately after completing the data modification. If there is a concurrent read-only transaction of the read-write transaction at this time and needs to read the modified data of the read-write transaction and the concurrent read-only transaction has completed the submission, but the read-write transaction If the data modification operation of the write transaction is lost and has not been saved to the persistent storage medium (hard disk), the read-write transaction is a transaction that needs to be rolled back, which in turn causes the read-only transaction to read a modification that needs to be rolled back. data, which violates the isolation of transactions, which in turn affects the correctness of transactions. Therefore, how to ensure the correctness of transactions in DolphinDB in-memory database is an urgent problem to be solved.

Contents of the invention

In order to ensure the correctness of transactions in the DolphinDB in-memory database, this application provides an in-memory database transaction management method, device, electronic equipment and media.

In the first aspect, this application provides an in-memory database transaction management method, adopting the following technical solution:

An in-memory database transaction management method, including:

When it is detected that a read-only transaction requests access to a data row, the transaction identifier of the read-only transaction and several version data information corresponding to the data row are obtained. Each of the version data information includes a version data row and a version transaction identifier. , the number of reads, the transaction access interval and the log persistence parameters. The transaction access interval is the transaction identifier interval of the read-only transaction that can read the version data row. The log persistence parameters are used to characterize the creation of the Whether the modification operation log of the read and write transactions of the version data row has been written to the persistent storage medium;

Based on the transaction identifier of the read-only transaction and the transaction access interval corresponding to each of the several versions of data information, the target version data information is determined from the several versions of data information, and the target version data information is the only The read transaction specifies the version data information accessed;

Visible version data rows are determined based on log persistence parameters of the target version data information.

By adopting the above technical solution, based on the transaction identifier of the read-only transaction, the target version data information that the read-only transaction needs to read is determined from several versions of data information; and then based on the log persistence parameters of the target version data information, the read Whether the modification operation log of the write transaction has been written to the persistent storage medium. Only when the modification operation log of the read and write transaction has been written to the persistent storage medium, the corresponding version data row will be considered visible, that is, the version data will be determined. OK. This ensures the correctness of the transaction and avoids reading deleted or uncommitted data.

In a possible implementation manner, determining the visible version data row based on the log persistence parameters of the target version data row includes:

After the modification operation log of the read-write transaction corresponding to the target version data information is written to the persistent storage medium, modify the log persistence parameters in the version data information to a preset visible value;

When the log persistence parameter of the target version data information is the preset visible value, the version data row in the target version data information is determined as a visible version data row, and the version data row in the target version data information is The number of reads is modified;

When the log persistence parameter of the target version data information is not the preset visible value, the read-write transaction corresponding to the target version data information is added to the dependent transaction set of the read-only transaction, based on the Depend on the transaction set, determine the visible version data rows, and modify the number of reads in the target version data information.

By adopting the above technical solution, if the modification operation log of the read-write transaction corresponding to the target version data information has been written to the persistent storage medium, the corresponding log persistence parameters are modified to the preset visible values; when the log persistence parameters are the preset visible values When setting the visible value, determine the modified version data row in the target version data information as the visible version data row to ensure that the data read by the read-only transaction is correct. When the log persistence parameter of the target version data information is not a preset visible value, add the read-write transaction corresponding to the target version data information to the dependent transaction set of the read-only transaction. This can establish a dependency relationship and ensure the read-only transaction. Ability to read updated data that relies on transactions committed. By recording the number of reads, the access status of each version of the data row can be tracked for subsequent optimization and statistical analysis.

In a possible implementation manner, adding the read-write transaction corresponding to the target version data information to the dependent transaction set of the read-only transaction includes:

Based on the transaction access interval of the target version data information, determine the transaction identifier of the read and write transaction corresponding to the target version data information;

Based on the transaction identifier of the read-write transaction corresponding to the target version data information, the read-write transaction corresponding to the target version data information is added to the dependent transaction set of the read-only transaction.

By adopting the above technical solution, when creating the target version data information, the lower limit of the transaction access interval is the transaction identifier of the read and write transaction that created the target version data information. Therefore, through the transaction access interval of the target version data information, it can be determined Create the transaction identifier of the read-write transaction of the target version data information, and then add the read-write transaction to the dependent transaction set of the read-only transaction based on the transaction identifier of the read-write transaction, which can more quickly determine the creation of the target version Read and write transactions of data information.

In a possible implementation manner, before obtaining several versions of data information corresponding to the data row, the method further includes:

When it is detected that a read-write transaction requests modification of the data row, based on the read-write transaction, version data information of the data row is created. The version data information includes a version data row, a version transaction identifier, the number of reads, and a transaction Access interval and log persistence parameters. The initial value of the version transaction identifier is the transaction identifier of the read-write transaction that created the version data information. The initial value of the log persistence parameter is a preset invisible value.

By adopting the above technical solution, when it is detected that a read-write transaction requests to modify a data row, the version data information of the data row is created based on the read-write data row. The version data information includes the version data row that needs to be modified, the version transaction identifier, the read-write transaction ID, and the version data row that needs to be modified. Fetch times, transaction access interval and log persistence parameters. By setting the version transaction identifier to the transaction identifier of the read-write transaction that created that version of data information, you can accurately track the source transaction of each version of data information. By setting the initial value of the log persistence parameter to the preset invisible value, you can ensure that the newly created version data is invisible to other transactions; only when the modification operation log of the read-write transaction has been written to the persistent storage medium, that is, Only when the log persistence parameters of this version of data information are modified to the preset visible value can other transactions read this version of data, ensuring transaction consistency. At the same time, when a read-only transaction needs to access the data of this version, the log persistence parameters can be used to determine that the data of this version can be read safely. There is no need to query the data in the global transaction collection, which can reduce the overhead of the database management system.

In a possible implementation manner, after creating the version data information of the data row, it also includes:

Determine that the read-write transaction is in a write lock-bound state;

After the read-write transaction completes the modification operation on the data row data, it is determined that the read-write transaction is in the write lock release state, and the version transaction identifier in the version data information corresponding to the read-write transaction is modified to the preset state. Assume to obtain the lock value;

Write the modification operation log of the read-write transaction to the persistent storage medium, where the modification operation log includes the modification operation of the data row by the read-write transaction.

By adopting the above technical solution, after the read-write transaction creates the version data information corresponding to the data row, the read-write transaction needs to obtain the write lock of the data row, that is, it is determined that the read-write transaction is in the write lock-bound state to ensure other needs The read-only transaction accessing the data row of this version cannot read the data row being modified, thereby ensuring the consistency of the data; after the read-write transaction completes the modification operation on the data row, the read-write transaction releases the write lock of the data row, that is, Determine that the read-write transaction is in the write lock release state, and at the same time modify the version transaction identifier corresponding to the read-write transaction to the default acquisition lock value. This modification can be used as a signal to inform other read-only transactions that they can start reading this version of data rows. Lock; then write the modification operation log of the read-write transaction to the persistent storage medium to ensure that the modification operation made by the read-write transaction is persisted. Ensure the consistency and concurrency of concurrent read-only transactions by determining the write lock binding and release status and modifying the version transaction identifier.

In one possible implementation manner, after determining the target version data information from the several versions of data information based on the transaction identifier of the read-only transaction and the transaction access intervals corresponding to the several versions of data information, ,Also includes:

Obtain the version transaction identifier of the target version data information;

When the version transaction identifier of the target version data information is a preset acquisition lock value, it is determined that the read-only transaction is in a read lock binding state.

By adopting the above technical solution and using the version transaction identifier of the target version data information, it is judged whether the version data row in the target version data information has been modified. If the modification has been completed (the version transaction identifier is the preset acquisition lock value), then access The read-only transaction of this version of data can obtain the corresponding read lock, that is, the read-only transaction is in the read lock binding state. The read lock binding state means that read-only transactions can concurrently read target version data information without conflicting with write transactions. This can improve concurrency performance, allow multiple read-only transactions to read target version data information at the same time, and increase system throughput.

In the second aspect, this application provides an in-memory database transaction management device that adopts the following technical solution:

An in-memory database transaction management device, including:

A transaction information and version data information acquisition module, configured to obtain the transaction identifier of the read-only transaction and several version data information corresponding to the data row when a read-only transaction request is detected to access a data row, each of which The version data information includes version data rows, version transaction identifiers, read times, transaction access intervals, and log persistence parameters. The transaction access intervals are the transaction identifier intervals of read-only transactions that can read the version data rows, so The log persistence parameter is used to indicate whether the modification operation log of the read-write transaction that created the version data row has been written to the persistent storage medium;

The target version data information determination module is configured to determine the target version data information from the several version data information based on the transaction identifier of the read-only transaction and the transaction access intervals corresponding to the several version data information, so The target version data information is the version data information specified to be accessed by the read-only transaction;

The visible version data row determination module is used to determine the visible version data row based on the log persistence parameters of the target version data information.

By adopting the above technical solution, based on the transaction identifier of the read-only transaction, the target version data information that the read-only transaction needs to read is determined from several versions of data information; and then based on the log persistence parameters of the target version data information, the read Whether the modification operation log of the write transaction has been written to the persistent storage medium. Only when the modification operation log of the read and write transaction has been written to the persistent storage medium, the corresponding version data row will be considered visible, that is, the version data will be determined. OK. This ensures the correctness of the transaction and avoids reading deleted or uncommitted data.

In a possible implementation manner, when the visible version data row determination module determines the visible version data row based on the log persistence parameters of the target version data row, it is specifically used to:

After the modification operation log of the read-write transaction corresponding to the target version data information is written to the persistent storage medium, modify the log persistence parameters in the version data information to a preset visible value;

When the log persistence parameter of the target version data information is the preset visible value, the version data row in the target version data information is determined as a visible version data row, and the version data row in the target version data information is The number of reads is modified;

When the log persistence parameter of the target version data information is not the preset visible value, the read-write transaction corresponding to the target version data information is added to the dependent transaction set of the read-only transaction, based on the Depend on the transaction set, determine the visible version data rows, and modify the number of reads in the target version data information.

In a possible implementation manner, when the visible version data row determination module adds the read-write transaction corresponding to the target version data information to the dependent transaction set of the read-only transaction, it is specifically used to:

Based on the transaction access interval of the target version data information, determine the transaction identifier of the read and write transaction corresponding to the target version data information;

Based on the transaction identifier of the read-write transaction corresponding to the target version data information, the read-write transaction corresponding to the target version data information is added to the dependent transaction set of the read-only transaction.

In a possible implementation manner, an in-memory database transaction management device further includes:

A version data information creation module, configured to create version data information of the data row based on the read-write transaction when a read-write transaction request is detected to modify the data row. The version data information includes a version data row, a version Transaction identifier, number of reads, transaction access interval and log persistence parameters. The initial value of the version transaction identifier is the transaction identifier of the read and write transaction that created the version data information. The initial value of the log persistence parameter is Default invisible value.

In a possible implementation manner, an in-memory database transaction management device further includes:

A write lock binding state determination module, used to determine that the read and write transaction is in a write lock binding state;

A write lock release state determination module, configured to determine that the read and write transaction is in the write lock release state after the read and write transaction completes the modification operation of the data row data, and store the version data corresponding to the read and write transaction. The version transaction identifier in the information is changed to the default acquisition lock value;

A log writing module is configured to write the modification operation log of the read-write transaction to the persistent storage medium, where the modification operation log includes the modification operation of the data row by the read-write transaction.

In a possible implementation manner, an in-memory database transaction management device further includes:

A version transaction identification acquisition module, used to obtain the version transaction identification of the target version data information;

The read lock binding state determination module is configured to determine that the read-only transaction is in the read lock binding state when the version transaction identifier of the target version data information is a preset acquisition lock value.

In the third aspect, this application provides an electronic device that adopts the following technical solution:

An electronic device including:

at least one processor;

memory;

At least one application program, wherein the at least one application program is stored in a memory and configured to be executed by at least one processor, the at least one application program is configured to: execute the above-mentioned in-memory database transaction management method.

In the fourth aspect, this application provides a computer-readable storage medium, adopting the following technical solution:

A computer-readable storage medium includes: storing a computer program that can be loaded by a processor and execute the above-mentioned memory database transaction management method.

To sum up, this application includes at least one of the following beneficial technical effects:

1. Based on the transaction identifier of the read-only transaction, determine the target version data information that the read-only transaction needs to read from several versions of data information; and then determine the modification of the read-write transaction based on the log persistence parameters of the target version data information. Whether the operation log has been written to the persistent storage medium. Only when the modification operation log of the read-write transaction has been written to the persistent storage medium, the corresponding version data row will be considered visible, that is, the version data row will be determined. This ensures the correctness of the transaction and avoids reading deleted or uncommitted data.

2. If the modification operation log of the read-write transaction corresponding to the target version data information has been written to the persistent storage medium, the corresponding log persistence parameters are modified to the preset visible values; when the log persistence parameters are the preset visible values , determine the modified version data rows in the target version data information as visible version data rows to ensure that the data read by the read-only transaction is correct. When the log persistence parameter of the target version data information is not a preset visible value, add the read-write transaction corresponding to the target version data information to the dependent transaction set of the read-only transaction. This can establish a dependency relationship and ensure the read-only transaction. Ability to read updated data that relies on transactions submitted. By recording the number of reads, the access status of each version of the data row can be tracked for subsequent optimization and statistical analysis.

3. When it is detected that a read-write transaction requests to modify a data row, the version data information of the data row is created based on the read-write data row. The version data information includes the version data row that needs to be modified, the version transaction identifier, the number of reads, and the transaction Access interval and log persistence parameters. By setting the version transaction identifier to the transaction identifier of the read-write transaction that created that version of data information, you can accurately track the source transaction of each version of data information. By setting the initial value of the log persistence parameter to the preset invisible value, you can ensure that the newly created version data is invisible to other transactions; only when the modification operation log of the read-write transaction has been written to the persistent storage medium, that is, Only when the log persistence parameters of this version of data information are modified to the preset visible value can other transactions read this version of data, ensuring transaction consistency. At the same time, when a read-only transaction needs to access the data of this version, the log persistence parameters can be used to determine that the data of this version can be read safely. There is no need to query the data in the global transaction collection, which can reduce the overhead of the database management system.

Description of the drawings

Figure 1 is a schematic flow chart of an in-memory database transaction management method in an embodiment of the present application;

Figure 2 is a timeline diagram of a read-only transaction acquiring a read lock in the embodiment of the present application;

Figure 3 is a schematic structural diagram of a memory database transaction management device in an embodiment of the present application;

Figure 4 is a schematic structural diagram of an electronic device in an embodiment of the present application.

Detailed ways

The present application will be further described in detail below in conjunction with Figures 1-4.

Those skilled in the art can make modifications to this embodiment without creative contribution as needed after reading this description, but as long as they are within the scope of this application, they are protected by patent law.

In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments These are part of the embodiments of this application, but not all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

In addition, the term "and/or" in this article is only an association relationship that describes related objects, indicating that there can be three relationships. For example, A and/or B can mean: A alone exists, and A and B exist simultaneously. There are three cases of B alone. In addition, the character "/" in this article, unless otherwise specified, generally indicates that the related objects are in an "or" relationship.

The embodiment of the present application provides a memory database transaction management method, which is executed by an electronic device. Referring to Figure 1, the method includes steps S101 to S103, wherein:

Step S101: When it is detected that a read-only transaction requests access to a data row, obtain the transaction identifier of the read-only transaction and several version data information corresponding to the data row. Each version data information includes a version data row, a version transaction identifier, a read times, transaction access interval and log persistence parameters. The transaction access interval is the transaction identifier interval of a read-only transaction that can read version data rows. The log persistence parameters are used to characterize the modification operation log of read-write transactions that create version data rows. Whether it has been written to persistent storage media.

For the embodiment of the present application, in the in-memory database system, each transaction can be automatically assigned a unique transaction identifier, so that the system can identify and track the execution and status of the transaction. At the same time, the in-memory database system will record the transaction identifier of the transaction and associate the transaction identifier with the transaction. The transaction identifier can be a number, a string, or other types of identifiers. The transaction identifier can be generated in different ways. , depending on the requirements and design of the system. Ways to generate transaction identifiers can include: Incrementing counter: The database system can use an incrementing counter to generate transaction identifiers. Every time a new transaction is created, the value of the counter is increased, and this value is used as the identifier of the new transaction. Timestamp: Use the current timestamp as a transaction identifier, usually by converting the current time into a string in a specific format or using a specific data type to store the timestamp. Globally unique identifier (GUID): Use a globally unique identifier as a transaction identifier. A GUID is a 128-bit unique identifier that is usually generated using algorithm and system-specific information. The generation method and expression form of the transaction identifier are not specifically limited in the embodiment of this application, as long as it can accurately represent the corresponding transaction.

Further, when it is detected that a read-only transaction requests access to a data row, the transaction identifier of the read-only transaction is obtained, and at the same time, according to the data row that the read-only transaction needs to access, several versions of data information corresponding to the data row are obtained, where provided here The version data information obtained is the version data information created by different read-write transactions for the data row before the read-only transaction request is detected to access the data row. Each version of the data information of the data row is associated with the transaction identifier of the read-write transaction that created the version of the data information. The initial value of the version transaction identifier is the transaction identifier of the read-write transaction that created the version of the data information. When reading and writing After the transaction completes the read and write operations, the corresponding version identifier will be changed to a preset identification value. The preset identification value is used to distinguish it from the transaction identifier of the read and write transactions.

Users can understand the number of read-only transactions currently reading data rows of this version based on the value corresponding to the number of reads. For example, the initial value of the number of reads is 0. When a read-only transaction reads a data row of this version, The number of reads is increased by 1; when the read-only transaction completes reading, the corresponding number of reads is decremented by 1.

The transaction access interval is used to represent the transaction identifier interval of the read-only transaction that can read the data rows of this version. The lower threshold of the transaction access interval can be the transaction identifier of the read-write transaction that created the data information of this version, that is, the version transaction. The initial value of the identifier. The upper threshold of the transaction access interval can be determined based on user needs and the usage of the next read-write transaction. Taking the transaction identifier as the timestamp of creating the corresponding transaction as an example: the lower limit of the transaction access interval is the timestamp of creating this version of the read-write transaction, and the upper limit of the transaction access interval can be the start timestamp of the next read-write transaction minus 1. Suppose there are two transactions T1 and T2. T1 starts when the timestamp is 1000, and T2 starts when the timestamp is 2000. When T1 modifies a certain data row, a new version will be created. The lower limit of the transaction access interval is 1000, but the upper limit will be set to 1999 (the start timestamp of T2 minus 1). In this way, the data modified by T1 is invisible to T2, and the transaction access intervals of each version of the data information do not overlap.

Log persistence parameters correspond to a set of numbers or characters, for example, 0 and 1, false and true, etc. After the operation log of the read-write transaction has been written to the persistent storage medium, the log persistence parameters will be modified from the initial value to another non-initial value in the corresponding array or character group. For example, the initial value of the log persistence parameter is false, true when the operation data of the read-write transaction is written to the persistent storage medium.

Step S102: Based on the transaction identifier of the read-only transaction and the corresponding transaction access intervals of several versions of data information, determine the target version data information from the several versions of data information. The target version data information is the version data designated to be accessed by the read-only transaction. information;

For the embodiment of this application, when a read-only transaction starts to execute, it will specify the version of the data row to be read through the transaction identifier. Therefore, it can be judged by judging the transaction identifier of the read-only transaction and the transaction access of each version of the data information. The relationship between intervals determines the target version data information from several versions of data information. The transaction identifier of the read-only transaction is within the transaction access interval of the target version data information, and the transaction access intervals of each version of data information are mutually exclusive. .

Step S103: Determine the visible version data rows based on the log persistence parameters of the target version data information.

For the embodiment of this application, the log persistence parameters of the current target version data information are determined. If the log persistence parameter will be a certain preset value, the preset value indicates that the data modification operation corresponding to the target version data information has been written. In the persistent storage medium, the modified version data rows in the target version data information are used to determine the visible version data rows that can be read by the read-only transaction.

According to the transaction identifier of the read-only transaction, the target version data information that the read-only transaction needs to read is determined from several version data information; and then based on the log persistence parameters of the target version data information, the modification operation log of the read-write transaction is determined. Whether it has been written to the persistent storage medium, only when the modification operation log of the read-write transaction has been written to the persistent storage medium, the corresponding version data row will be considered visible, that is, the version data row will be determined. This ensures the correctness of the transaction and avoids reading deleted or uncommitted data.

Further, based on the log persistence parameters of the target version data row, determining the visible version data row includes steps S1031 (not shown in the figure) - step S1033 (not shown in the figure), wherein:

Step S1031: After the modification operation log of the read-write transaction corresponding to the target version data information is written to the persistent storage medium, modify the log persistence parameters in the version data information to a preset visible value.

Specifically, after the operation log of the read-write transaction corresponding to the target version data information is written to the persistent storage medium (log disk), the log persistence parameters in the target version data information need to be modified from the initial value to the preset visible value. The preset visible value of the log persistence parameter is different from the initial value. You can learn whether the corresponding read and write transaction has completed the log placement by reading the log persistence parameter of the version data information. For example, when read-write transaction A creates version data information

Further, step S1031 can be executed before any step from step S101 to step S103. Step S1031 can also be executed after any step from step S101 to step S103. Step S1031 can also be executed simultaneously with any step from step S101 to step S103. As long as the modification operation log of the read-write transaction corresponding to the version data information has been written to the persistent storage medium, the log persistence parameters in the version data information can be modified to the preset visible value. Steps S1032 and S1033 need to be executed after step S102. If step S1031 has been executed before the target version data information is determined, continue to execute S1032; if step S1031 has not been executed before the target version data information is determined, continue to execute S1033.

Step S1032: When the log persistence parameter of the target version data information is a preset visible value, determine the version data row in the target version data information as a visible version data row, and modify the number of reads in the target version data information. ;

Step S1033: When the log persistence parameter of the target version data information is not a preset visible value, add the read-write transaction corresponding to the target version data information to the dependent transaction set of the read-only transaction, and determine the visible version based on the dependent transaction set. data row, and modify the number of reads in the target version data information.

Specifically, when the log persistence parameter of the target version data information is a preset visible value, the version data row corresponding to the target version data information is determined to be a visible version data row. When the log persistence parameter of the target version data information is not a preset visible value, in order to ensure that the read-only transaction can read the correct version data, the read-write transaction corresponding to the target version data information needs to be added to the read-only transaction itself. in the set of dependent transactions. Then, through the dependency tracking method, the transactions in the dependent transaction set are monitored and analyzed until the log persistence parameter of each dependent read-write transaction in the dependent transaction set becomes a preset visible value, and a visible version data row is obtained. After the visible version data row corresponding to the read-only transaction is determined, the number of reads in the target version data information to which the visible version data row belongs needs to be modified, for example, the current number of reads is increased by 1.

Further, the read-write transaction corresponding to the target version data information is added to the dependent transaction set of the read-only transaction, including step SA1 (not shown in the figure) - step SA2 (not shown in the figure), where:

Step SA1: Based on the transaction access interval of the target version data information, determine the transaction identifier of the read and write transaction corresponding to the target version data information;

Step SA2: Based on the transaction identifier of the read-write transaction corresponding to the target version data information, add the read-write transaction corresponding to the target version data information to the dependent transaction set of the read-only transaction.

Specifically, when creating a transaction access interval for target version data information, the lower limit of the transaction access interval is the transaction identifier of the read-write transaction that creates the target version data information. Therefore, the transaction identifier of the read-write transaction that requires dependency tracking can be determined through the transaction access interval. Then, the read-write transaction corresponding to the target version data information is added to the dependent transaction set of the read-only transaction to implement dependency tracking.

Further, before obtaining several version data information corresponding to the data row, step S01 is also included. When it is detected that a read-write transaction requests to modify the data row, based on the read-write transaction, create version data information of the data row. The version data information includes Version data row, version transaction identifier, number of reads, transaction access interval, and log persistence parameters. The initial value of the version transaction identifier is the transaction identifier of the read-write transaction that created the version data information. The initial value of the log persistence parameter is the preset value. Let the value be invisible.

For the embodiment of this application, the DolphinDB in-memory database adopts a concurrency control protocol to manage concurrent access of transactions in the database system. The concurrent execution and isolation of transactions are achieved by using a multi-version concurrency control mechanism and a two-stage locking protocol. When a read-write transaction needs to modify a data row, it will request to obtain a write lock for the corresponding data row. If it is detected that a read-write transaction requests to modify a data row, the read-write transaction will create version data information of the data row based on the read-write transaction before starting to modify the data row. The version data information contains the data row that needs to be modified by the read-write transaction. Data, that is, version data rows. The version data information also includes the version transaction identifier, the current number of read-only transactions that read the data rows of this version, the transaction identifier interval (transaction access interval) of the read-only transactions that can read the data rows of this version, and the transaction access interval used to characterize the read The log persistence parameter determines whether the write transaction operation has completed log writing. Among them, the initial value of the version transaction identifier is the transaction identifier of the read-write transaction that created the version data information; the initial value of the log persistence parameter is the preset invisible value. The preset invisible value is different from the preset visible value. Use Used to represent different log placement statuses.

Further, after creating the version data information of the data row, steps S02 (not shown in the figure) to step S04 (not shown in the figure) are also included, wherein:

Step S02: Determine that the read and write transaction is in the write lock binding state;

Step S03: After the read-write transaction completes the modification operation on the data row data, determine that the read-write transaction is in the write lock release state, and modify the version transaction identifier in the version data information corresponding to the read-write transaction to the default acquisition lock value.

For the embodiment of this application, before starting to modify the data row, the read-write transaction will need to obtain a write lock (Exclusive Lock) to protect the modified data row, that is, it is determined that the read-write transaction is in a write lock-bound state. Each transaction obtains a view (snapshot) at the beginning, which represents the consistent state of the database at the beginning of the transaction. When a read-write transaction wants to modify a data row, it will determine whether the data row can be modified based on the view information. If the view of the read-write transaction does not conflict with the versions of other concurrent read-write transactions, then the read-write transaction can Generate a new version and obtain the write lock of the data row for write operations. After the read-write transaction successfully obtains the write lock of the data row, it can perform modification operations on the data row, such as update, insert, and delete operations. Once the read-write transaction completes the modification operation on the data row, it needs to release the write lock it holds, that is, change the status of the read-write transaction to the write lock release state.

Further, after the read-write transaction corresponding to the version data information completes the modification operation and releases the write lock, that is, when the read-write transaction corresponding to the version data information is in the write lock release state, the version transaction identifier in the version data information needs to be modified to Default acquisition lock value. The version transaction identifier can be used to determine whether the read-write transaction corresponding to the version data information to which it belongs has completed the operation. The default acquisition lock value is different from the transaction identifier of the read-write transaction. When the version transaction identifier is the default acquisition lock value ( When the status of a read-write transaction is the write lock release state), other read-only transactions can acquire the read lock of the data row. For example, the transaction identifier of a transaction is the timestamp when it was created. The default acquisition lock value is 0. The transaction identifier of read-write transaction A is 100. Then the initial value of the version transaction identifier is 100. After the read-write transaction A is completed, After the modification operation releases the write lock, the initial value of the version transaction identifier is modified to 0.

Step S04: Write the modification operation log of the read-write transaction to the persistent storage medium. The modification operation log includes the modification operation of the data row by the read-write transaction.

For the embodiment of this application, in order to persist the operations of read and write transactions and ensure ACID properties (atomicity, consistency, isolation, durability), the operation records of read and write transactions are usually written to a persistent storage medium, such as a disk. Or log file, this can ensure that even if a system failure occurs, the modification operation of the data row can still be recovered. Writing the operation records of read and write transactions to the persistent storage medium is a process of waiting for the log to be written to the disk. The "waiting for the log to be written to the disk" stage involves disk I/O operations, and in order to ensure the absolute security of the data, this stage usually requires a " fsync" call. "fsync" is a synchronization operation in the file system, which is used to ensure that the data in the buffer or memory can be persisted to the persistent storage medium (disk). At the same time, the overhead of "fsync" is very large, and it usually costs money for mechanical hard disks. A few milliseconds, even SSDs take hundreds of microseconds. In order to improve transaction throughput, this application needs to release the write lock before the log of the read-write transaction is placed on the disk. As shown in Figure 2, while the previous read-write transaction is waiting for the log to be placed on the disk, the concurrent read-only transaction will Can carry out work normally.

Further, after determining the target version data information from the several version data information based on the transaction identifier of the read-only transaction and the corresponding transaction access intervals of the several version data information, step SB1 (not shown in the figure) is also included. - step SB2 (not shown in the figure), wherein:

Step SB1: Obtain the version transaction identifier of the target version data information;

Step SB2: When the version transaction identifier of the target version data information is the preset acquisition lock value, determine that the read-only transaction is in the read lock binding state.

For the embodiment of the present application, after determining the target version data information from several versions of data information, the version transaction identifier of the target version data information is obtained. When the version transaction identifier of the target version data information is the preset acquisition lock value, this means that other transactions can read the version data row, and the read-only transaction is determined to be in a read lock binding state.

Use the version transaction identifier of the target version data information to determine whether the version data row in the target version data information has been modified. If the modification has been completed (the version transaction identifier is the default acquisition lock value), the read-only access to the version data The transaction can obtain the corresponding read lock, that is, the read-only transaction is in the read lock binding state. The read lock binding state means that read-only transactions can concurrently read target version data information without conflicting with write transactions. This can improve concurrency performance, allow multiple read-only transactions to read target version data information at the same time, and increase system throughput.

The above embodiment introduces a method of in-memory database transaction management from the perspective of method flow. The following embodiment introduces a device for in-memory database transaction management from the perspective of virtual modules or virtual units. For details, see the following embodiments.

The embodiment of the present application provides a device for memory database transaction management. As shown in Figure 3, the device for memory database transaction management may specifically include a transaction information and version data information acquisition module 201, a target version data information determination module 202, and a visible version. Data row determination module 203, wherein:

The transaction information and version data information acquisition module 201 is used to obtain the transaction identifier of the read-only transaction and several version data information corresponding to the data row when a read-only transaction request is detected to access the data row. Each version data information includes a version Data rows, version transaction identifiers, read times, transaction access intervals, and log persistence parameters. The transaction access interval is the transaction identifier interval of a read-only transaction that can read version data rows. The log persistence parameters are used to characterize the creation of version data. Whether the modification operation log of the row's read-write transaction has been written to the persistent storage medium;

The target version data information determination module 202 is used to determine the target version data information from several version data information based on the transaction identifier of the read-only transaction and the corresponding transaction access intervals of several version data information. The target version data information is only The read transaction specifies the version data information accessed;

The visible version data row determination module 203 is used to determine the visible version data row based on the log persistence parameters of the target version data information.

By adopting the above technical solution, based on the transaction identifier of the read-only transaction, the target version data information that the read-only transaction needs to read is determined from several versions of data information; and then based on the log persistence parameters of the target version data information, the read Whether the modification operation log of the write transaction has been written to the persistent storage medium. Only when the modification operation log of the read and write transaction has been written to the persistent storage medium, the corresponding version data row will be considered visible, that is, the version data will be determined. OK. This ensures the correctness of the transaction and avoids reading deleted or uncommitted data.

In a possible implementation manner, when determining the visible version data row based on the log persistence parameters of the target version data row, the visible version data row determination module 203 is specifically used to:

After the modification operation log of the read-write transaction corresponding to the target version data information is written to the persistent storage medium, modify the log persistence parameters in the version data information to the preset visible value;

When the log persistence parameter of the target version data information is a preset visible value, determine the version data row in the target version data information as a visible version data row, and modify the number of reads in the target version data information;

When the log persistence parameter of the target version data information is not a preset visible value, add the read-write transaction corresponding to the target version data information to the dependent transaction set of the read-only transaction, and determine the visible version data row based on the dependent transaction set. And modify the number of reads in the target version data information.

In one possible implementation manner, when adding the read-write transaction corresponding to the target version data information to the dependent transaction set of the read-only transaction, the visible version data row determination module 203 is specifically used to:

Based on the transaction access interval of the target version data information, determine the transaction identifier of the read and write transaction corresponding to the target version data information;

Based on the transaction identifier of the read-write transaction corresponding to the target version data information, the read-write transaction corresponding to the target version data information is added to the dependent transaction set of the read-only transaction.

In a possible implementation manner, an in-memory database transaction management device further includes:

The version data information creation module is used to create version data information of the data row based on the read-write transaction when a read-write transaction request is detected to modify the data row. The version data information includes version data rows, version transaction identifiers, read times, and transactions. Access interval and log persistence parameters. The initial value of the version transaction identifier is the transaction identifier of the read-write transaction that created the version data information. The initial value of the log persistence parameters is the preset invisible value.

In a possible implementation manner, an in-memory database transaction management device further includes:

The write lock binding status determination module is used to determine that the read and write transactions are in the write lock binding status;

The write lock release state determination module is used to determine that the read and write transaction is in the write lock release state after the read and write transaction completes the modification operation of the data row data, and modifies the version transaction identifier in the version data information corresponding to the read and write transaction to Default acquisition lock value;

The log writing module is used to write the modification operation log of the read-write transaction to the persistent storage medium. The modification operation log includes the modification operation of the data row by the read-write transaction.

In a possible implementation manner, an in-memory database transaction management device further includes:

The version transaction identifier acquisition module is used to obtain the version transaction identifier of the target version data information;

The read lock binding state determination module is used to determine that the read-only transaction is in the read lock binding state when the version transaction identifier of the target version data information is a preset acquisition lock value.

An electronic device is provided in an embodiment of the present application, as shown in FIG. 4 . The electronic device 300 shown in FIG. 4 includes: a processor 301 and a memory 303 . Among them, the processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, electronic device 300 may also include a transceiver 304. It should be noted that in practical applications, the number of transceivers 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present application.

The processor 301 may be a CPU (Central Processing Unit, central processing unit), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), ASIC (Application Specific Integrated Circuit, application specific integrated circuit), FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various illustrative logical blocks, modules, and circuits described in connection with this disclosure. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

Bus 302 may include a path that carries information between the above-mentioned components. The bus 302 may be a PCI (Peripheral Component Interconnect, Peripheral Component Interconnect Standard) bus or an EISA (Extended Industry Standard Architecture, Extended Industry Standard Architecture) bus, etc. The bus 302 can be divided into an address bus, a data bus, a control bus, etc. For ease of presentation, only one thick line is used in Figure 4, but it does not mean that there is only one bus or one type of bus.

The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory, random access memory) or other types that can store information and instructions. Dynamic storage devices can also be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact DiscRead Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, Laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage device, or any other device capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer medium, but not limited to this.

The memory 303 is used to store application program code for executing the solution of the present application, and is controlled by the processor 301 for execution. The processor 301 is used to execute the application program code stored in the memory 303 to implement the contents shown in the foregoing method embodiments.

Among them, electronic devices include but are not limited to: mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PAD (tablet computers), PMP (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc. mobile terminals such as digital TVs, desktop computers, etc. Also for servers etc. The electronic device shown in FIG. 4 is only an example and should not impose any restrictions on the functions and usage scope of the embodiments of the present application.

Embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When run on a computer, the computer can execute the corresponding content in the foregoing method embodiments.

It should be understood that although various steps in the flowchart of the accompanying drawings are shown in sequence as indicated by arrows, these steps are not necessarily performed in the order indicated by arrows. Unless explicitly stated in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least some of the steps in the flow chart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and their execution order is also It does not necessarily need to be performed sequentially, but may be performed in turn or alternately with other steps or sub-steps of other steps or at least part of the stages.

The above are only some of the embodiments of the present application. It should be pointed out that those of ordinary skill in the technical field can also make several improvements and modifications without departing from the principles of the present application. These improvements and modifications can also be made. should be regarded as the scope of protection of this application.

Claims (9)

1. A memory database transaction management method, comprising:

when a read-only transaction request access data line is detected, acquiring a transaction identifier of the read-only transaction and a plurality of version data information corresponding to the data line, wherein each version data information comprises a version data line, a version transaction identifier, reading times, a transaction access interval and a log persistence parameter, the transaction access interval is a transaction identifier interval of the read-only transaction capable of reading the version data line, and the log persistence parameter is used for representing whether a modification operation log of a read-write transaction creating the version data line is written into a persistent storage medium;

determining target version data information from the plurality of version data information based on a transaction identifier of the read-only transaction and transaction access intervals corresponding to the plurality of version data information, wherein the target version data information designates accessed version data information for the read-only transaction;

And determining visible version data lines based on the log persistence parameters of the target version data information.

2. The method of claim 1, wherein determining visible version data lines based on log persistence parameters of the target version data lines comprises:

after a modification operation log of a read-write transaction corresponding to the target version data information is written into a persistent storage medium, modifying a log persistence parameter in the version data information into a preset visible value;

when the log persistence parameter of the target version data information is the preset visible value, determining the version data line in the target version data information as a visible version data line, and modifying the reading times in the target version data information;

when the log persistence parameter of the target version data information is not the preset visible value, adding the read-write transaction corresponding to the target version data information into a dependent transaction set of the read-only transaction, determining visible version data lines based on the dependent transaction set, and modifying the reading times in the target version data information.

3. The method for managing memory database transactions according to claim 2, wherein adding the read-write transaction corresponding to the target version data information to the dependent transaction set of the read-only transaction includes:

determining a transaction identifier of a read-write transaction corresponding to the target version data information based on the transaction access interval of the target version data information;

and adding the read-write transaction corresponding to the target version data information into a dependent transaction set of the read-only transaction based on the transaction identifier of the read-write transaction corresponding to the target version data information.

4. The method for managing memory database transactions according to claim 1, further comprising, before obtaining the version data information corresponding to the data line:

when the read-write transaction request is detected to modify the data row, based on the read-write transaction, version data information of the data row is created, wherein the version data information comprises a version data row, a version transaction identifier, reading times, a transaction access interval and a log persistence parameter, the initial value of the version transaction identifier is a transaction identifier of the read-write transaction for creating the version data information, and the initial value of the log persistence parameter is a preset invisible value.

5. The method of claim 4, further comprising, after creating version data information for the data line:

determining that the read-write transaction is in a write lock binding state;

after the read-write transaction completes the modification operation on the data of the data line, determining that the read-write transaction is in a write lock release state, and modifying a version transaction identifier in version data information corresponding to the read-write transaction into a preset acquisition lock value;

writing a modification operation log of the read-write transaction to a persistent storage medium, the modification operation log comprising modification operations of the read-write transaction to the data line.

6. The method of claim 5, further comprising, after determining the target version data information from the plurality of version data information based on the transaction identifier of the read-only transaction and the transaction access interval corresponding to each of the plurality of version data information:

a version transaction identifier of the target version data information is obtained;

and when the version transaction identifier of the target version data information is a preset acquisition lock value, determining that the read-only transaction is in a read lock binding state.

7. A memory database transaction management device, comprising:

the system comprises a transaction information and version data information acquisition module, a data storage module and a data storage module, wherein the transaction information and version data information acquisition module is used for acquiring a transaction identifier of a read-only transaction and a plurality of version data information corresponding to the data row when detecting that the read-only transaction requests to access the data row, each version data information comprises a version data row, a version transaction identifier, reading times, a transaction access interval and a log persistence parameter, the transaction access interval is a transaction identifier interval of the read-only transaction capable of reading the version data row, and the log persistence parameter is used for representing whether a modification operation log of a read-write transaction creating the version data row is written into a persistent storage medium;

the target version data information determining module is used for determining target version data information from the plurality of version data information based on the transaction identifier of the read-only transaction and the transaction access interval corresponding to the plurality of version data information, wherein the target version data information is version data information appointed for access by the read-only transaction;

and the visible version data row determining module is used for determining visible version data rows based on the log persistence parameters of the target version data information.

8. An electronic device, comprising:

at least one processor;

a memory;

at least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application configured to: performing the in-memory database transaction management method of any one of claims 1-6.

9. A computer-readable storage medium, comprising: a computer program stored with instructions capable of being loaded by a processor and executing the method of memory database transaction management as claimed in any one of claims 1 to 6.