CN115291838A - Multi-user electronic mailbox mail indexing method based on KV system - Google Patents

Abstract

The invention provides a multi-user electronic mailbox mail indexing method based on a KV system, which comprises the following steps: constructing a Server program architecture; constructing a Key-List mailbox structure based on a KV system according to the Server program architecture; constructing a Map structure thread lock barrel based on a memory; and storing a plurality of field information of the mail by adopting a ProtoBuf structure. The newly added mail information field is flexible and efficient, backward/forward compatibility of the mail information data field structure is effectively realized, and the summary information of the user mailbox folder and the index of the mail information are realized.

Description

A method of indexing multi-user email boxes based on KV system

technical field

The invention relates to the field of e-mail servers, in particular to a KV system-based multi-user e-mail indexing method.

Background technique

One of the important functions of the email server is to store the mailbox information and original emails of multiple users. Users access the email server through MUA (such as webmail, pop, imap) clients to read the summary information of individual mailboxes and the original email content. With the sharp increase in the number of users and their emails, the email server is getting slower and slower when displaying the summary information of personal mailboxes (folder information, number of emails) and reading the list of original emails , Even in the case of large concurrency and high load, the email server will be down or unable to serve.

Based on the above reasons, it is necessary to cache or index the user's mailbox summary information and mail text list information to speed up the user's mailbox summary information and mailing list information when receiving mail through the MUA client. At the same time, it is also necessary to improve The speed at which the user traverses the mailing list, and when the number of mails in the user's mailbox changes (such as newly received mail, deleted mail, moving mail from one mail folder to another, etc.) can quickly update the user's mailbox summary information and adjust Mailing list information.

The mailbox format of the usual email server is mbox and maildir format, which respectively organize the storage of the original text of the email in the form of a single file or a directory structure file, and there is no cache or index of the relevant mailbox summary information. When it is necessary to display the mailbox summary For information, the mbox format needs to collect statistics on the original text information and quantity of all emails in a single file, and the maildir format needs to count the number and size of files in the entire directory. It is also very complicated when changing or moving, so Efficiency is usually lower.

The mailbox format of most improved email servers still uses maildir format or cloud storage to store the original text of the mail, but at the same time, an index device will be added to store the summary information of the mailbox and the list information of the original text of the mail, so as to speed up the reading of relevant speed of information.

One of the indexing devices is to store the user's original email information in a binary structured index file, and use each fixed-sized structural block in the structured file to record the relevant information (sender, recipient, subject, etc.) of the original email. , size, file ID, etc.), when reading information, calculate the relevant summary information by calculating the number of structural blocks in the structured file and traversing the structural block information. When displaying mailing list information, you can directly read the structure The structural block information in the file is optimized to achieve the effect of increasing speed. Since the data structure of the structured block is fixed, if an email information field needs to be added, the entire structured file needs to be changed, which requires a very large amount of work whether online or offline.

Another kind of indexing device is to store the user's mail original text information in the MySQL database table, use the mail folder information table and mail information table of the database, and collect all users' mail folder information (mail folder name, mail number, etc.) and mail The relevant information of the original text (sender, recipient, subject, size, mail folder ID, file ID, etc.) is stored in the database records in the form of table fields. When reading information, query and traverse through SQL related statements , when performing data changes such as adding or deleting emails, due to the associated update relationship between the mail folder information table and the email information table, in order to ensure atomic updates, it is necessary to use database transaction operations, and the performance of database transaction operations is relatively low. At the same time, as the amount of user mail data increases, there will be a certain bottleneck in the query performance of the database. When adding email information fields, table locking operations are required. When the amount of data is very large, table locking is required for a long time to update new fields in the database.

To sum up, the working efficiency of the indexing device in the prior art is low.

Contents of the invention

In view of the above problems, the present invention is proposed to provide a KV system-based multi-user email indexing method that overcomes the above problems or at least partially solves the above problems.

According to one aspect of the present invention, there is provided a kind of multi-user electronic mailbox mail indexing method based on KV system comprising:

Construct the Server program architecture;

Build the Key-List mailbox structure based on the KV system according to the Server program framework;

Build a memory-based Map structure thread lock bucket;

Use the ProtoBuf structure to store multiple field information of the mail.

Optionally, said building the Server program architecture specifically includes:

The Server program is a multi-threaded service program, including a network IO thread pool, which is used to provide the Restful interface access service of the HTTP protocol; a worker thread pool, which is used to perform data query and update services of the KV storage system;

The network IO thread pool and the worker thread pool interact through a message queue.

Optionally, the construction of the Key-List mailbox structure based on the KV system according to the Server program architecture specifically includes:

The working thread pool thread uses multiple Key-Value data based on the KV storage system to construct the mail folder information and mail information list structure of the Key-List, one of the Key-Value data is the mail folder information data, and the multiple Key-Value data The Value data is mail information data, and each Key-Value data represents a piece of mail data, and stores relevant information of the mail;

Carry out content format planning to the Key in the Key-Value data, Key includes TAG identification field, shows the type of Key, mail folder ID and mail ID field; Prefix part includes identical user ID;

Value includes multiple fields. The Value of the mail folder information includes the time stamp of the mail folder update, the number of mails in the mail folder, the total size of the mail in the mail folder, the maximum mail ID and the minimum mail ID in the mail folder, and the Value of the mail information includes the time of the mail ProtoBuf serialization data of the stamp, the size of the mail, and the summary information of the mail.

Optionally, the construction of the memory-based Map structure thread lock bucket specifically includes:

When the thread of the worker thread pool updates the user's mail folder information and mail information, in order to ensure the atomic operation of the data, a user-level thread mutex is used;

A memory-based Map structure thread lock bucket is used;

When updating KV data, the KV system batch operation interface is adopted.

Optionally, the multiple field information of storing emails using the ProtoBuf structure specifically includes:

The ProtoBuf structure stores multiple fields of email information. When storing to the Key-Value of the KV system, the entire structure is serialized into a string. When reading, the string is deserialized. The whole process is transparent to the user ;

If you need to add an email information field, you only need to add a field of the ProtoBuf structure. The newly added field will automatically adopt the default value during the deserialization process of the old data. When updating, the newly added field will be automatically serialized to In the string, the whole process has no impact on the overall data, the data conversion is flexible and efficient, and the backward/forward compatibility of the data structure is realized.

A kind of multi-user electronic mailbox mail indexing method based on the KV system provided by the present invention includes: constructing the Server program architecture; constructing the Key-List mailbox structure based on the KV system according to the Server program architecture; constructing a memory-based Map structure thread lock bucket ; Use the ProtoBuf structure to store multiple field information of the mail. The new mail information field is flexible and efficient, effectively realizing the backward/forward compatibility of the data field structure of the mail information, and realizing the index of the user mailbox folder summary information and mail information.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.

Fig. 1 is a schematic diagram of the Server program architecture provided by the embodiment of the present invention;

Fig. 2 is the Key-List mailbox structural representation based on KV system that the embodiment of the present invention provides;

FIG. 3 is a schematic diagram of a memory-based MAP structure thread lock bucket provided by an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

The terms "comprising" and "having" and any variations thereof in the description, embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusion, for example, including a series of steps or units.

The technical solutions of the present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments.

As shown in Figure 1, build the server program architecture: the server program is a multi-threaded service program, mainly composed of two sets of thread pools, one set is the network IO thread pool, which provides the Restful interface access service of the HTTP protocol, and the other set is the work Thread pool for data query and update services of the KV storage system. The interaction between the network IO thread and the worker thread pool is through the message queue.

The KV storage engine system is integrated internally, and the index data such as mailbox summary information and mail information of multiple users are stored in the KV system. The KV system is a very fast NoSQL storage system that can store massive data and provide fast access to data.

As shown in Figure 2, the establishment of a Key-List mailbox structure based on the KV system includes:

The working thread pool thread uses multiple Key-Value data based on the KV storage system to construct the Key-List mail folder information and mail information list structure. One of the KV data is the mail folder information data, and the rest of the KV data are mail information data. , where each piece of KV data represents a piece of mail data, storing various relevant information of the mail. In order to establish the association relationship between mail folder information and mail information, it is necessary to plan the content format of the Key in these KV data. The Key consists of multiple fields, and the prefix part includes the same user ID to ensure that the data of the same user has the same At the same time, Key also includes a TAG identification field, indicating the type of Key (mail folder information, mail information), and mail folder ID and mail ID fields. Value is also composed of multiple fields. The Value of mail folder information is mainly composed of mail folder The update timestamp, the number of mails in the mail folder, the total size of the mail in the mail folder, the largest mail ID and the smallest mail ID in the mail folder, the value of the mail information is mainly composed of the timestamp of the mail, the size of the mail and the ProtoBuf sequence of the summary information of the mail composition of data.

When adding or deleting an email, you need to add or delete an email Key-Value data, and at the same time update the Count, Bytes, MaxID, MinID field values in the Value of the Mail Folder Key-Value data.

When reading mail folder information, you only need to read the data of the mail folder Key-Value, which is a very fast reading action in the NoSQL system, and you can directly obtain the information of the mail folder.

When traversing the mail in the mail folder, you only need to read the Key-Value data in the NoSQL system sequentially according to the UID prefix and MailID order, and use the KV traversal pointer of the NoSQL system to quickly read the Key-Value data even in massive data. Value data.

Construct the Key-List structure through multiple Key-Value data of the KV system, that is: Key is a record, which records the user's mail folder information, and List is multiple records, each of which records a user's mail information. In this way Build into the user's mailbox index structure. When reading mail folder information, you only need to read a Key-Value data, using the advantages of the NoSQL KV storage system, quickly locate and read a Key-Value mail folder information data in massive data, and the reading speed is very fast . When adding or deleting emails, only two records need to be updated at the same time, one is the mail folder information, and the other is the added or deleted email information, and there is no need to use transactions, but use user-level memory Mutex thread locks and KV storage The batch operation (BatchPut) interface of the system is completed at one time, and the speed and efficiency are very high.

As shown in Figure 3, the construction of a memory-based Map structure thread lock bucket includes: when the threads of the worker thread pool update the user's mail folder information and mail information, in order to ensure the atomic operation of data, use user-level (UID) threads Mutex locks, and in order to satisfy large concurrent users to update data at the same time, a memory-based Map structure thread lock bucket is used, and 10,000 buckets are pre-allocated. The element of the Map is the hash value of the UID, and the data corresponding to this element is The thread locks the Mutex object. According to the characteristics of the Map data structure, the time complexity for a UID to acquire a lock is O(1) or O(logn). When updating the KV data, the KV system batch operation interface (BatchPut) is used. The speed and atomicity of updating data are guaranteed, and at the same time, the automatic release function of the Mutex lock is realized in the working thread, which effectively avoids the situation of system deadlock.

In order to ensure the atomicity of updating the Key-List data, the thread lock bucket based on the memory Map structure is used in the server program, and the thread lock is obtained by hashing the Key and sharing it to different buckets. When the number of elements in the Map (that is, the number of buckets) is set to 10,000 (or even more), the requirements for concurrent access by more users can be obtained. The time complexity of Map element positioning can reach O(1) or O(logn), and specific locks can be quickly located to ensure fast concurrent update of data.

The multiple field information of email stored in ProtoBuf structure includes: multiple fields of email information stored in ProtoBuf structure, when stored in the Key-Value of the KV system, the entire structure is serialized into a string, and when read, the This string is deserialized, and the whole process is transparent to the user. When you need to add a new mail information field, you only need to add a field of the ProtoBuf structure. During the deserialization process of the old data, the newly added field will automatically adopt the default value without affecting the integrity of the data. When updating, it will automatically Serialize the newly added fields into strings. The whole process has no impact on the overall data. The data conversion is flexible and efficient, and the backward/forward compatibility of the data structure is effectively realized.

For a piece of Key-Value data of an email message, multiple fields of the email message are serialized using the ProtoBuf structure and stored in the Value, and then deserialized when reading. When a new field is added, all the mail information data stored in the current store does not need to be updated. The ProtoBuf structure will provide the default data value of the newly added field. Only when the Key-Value data needs to be updated, the mail information data stored in ProtoBuf will be updated, and it is Updating a single piece of data does not affect all stored data. This dynamic and flexible data storage method effectively improves the flexibility of storing and updating email information data fields.

Beneficial effects: Through the multiple Key-Value data in the KV system, a mailbox index structure in the form of Key-List for multiple users is constructed to realize the indexing of folder summary information and mail information of user mailboxes, and fast query/update can be realized Mail folder information, and quickly add/delete/update mail information.

Through the thread lock bucket of the memory-level Map structure, it solves the fast atomic update of mail folder information and mail information in the Key-List structure of multiple concurrent users.

Multiple fields of mail information are stored in the Value of Key-Value with ProtoBuf structure data, and the new mail information field is flexible and efficient, effectively realizing the backward/forward compatibility of the data field structure of mail information.

The above specific implementation manners have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation modes of the present invention, and are not used to limit the protection scope of the present invention. Within the spirit and principles of the present invention, any modifications, equivalent replacements, improvements, etc., shall be included in the protection scope of the present invention.

Claims (5)

1. A multi-user E-mail indexing method based on a KV system is characterized by comprising the following steps:

constructing a Server program architecture;

constructing a Key-List mailbox structure based on a KV system according to the Server program architecture;

constructing a Map structure thread lock barrel based on a memory;

and storing a plurality of field information of the mail by adopting a ProtoBuf structure.

2. The method according to claim 1, wherein the constructing of the Server program architecture specifically comprises:

the Server program is a multi-thread service program and comprises a network IO thread pool, a network IO thread pool and a Server module, wherein the network IO thread pool is used for providing Restful interface access service of an HTTP protocol; the working thread pool is used for carrying out data query and update service of the KV storage system;

and the network IO thread pool and the work thread pool are interacted through a message queue.

3. The method according to claim 2, wherein the constructing a Key-List mailbox structure based on the KV system according to the Server program architecture specifically comprises:

the method comprises the steps that a working thread pool thread utilizes a plurality of pieces of Key-Value data based on a KV storage system to construct a mail folder information and mail information List structure of a Key-List, one piece of Key-Value data is mail folder information data, a plurality of pieces of Key-Value data are mail information data, each piece of Key-Value data represents one piece of mail data, and relevant information of a mail is stored;

planning a content format of a Key in the Key-Value data, wherein the Key comprises a TAG identification field which indicates the type of the Key, a mail folder ID and a mail ID field; the prefix portion includes the same user ID;

the Value includes a plurality of fields, the Value of the folder information includes a folder update time stamp, the number of mails in the folder, a total Size of the mails in the folder, a maximum mail ID and a minimum mail ID in the folder, and the Value of the mail information includes a time stamp of the mail, a Size of the mail and summary information ProtoBuf serialization data of the mail.

4. The method for indexing the multi-user electronic mailbox mail based on the KV system as claimed in claim 1, wherein the constructing of the Map-structured thread lock bucket based on the memory specifically comprises:

when the thread of the working thread pool updates the mail folder information and the mail information of a user, a user-level thread mutual exclusion lock is used for ensuring the atomicity operation of data;

a Map structure thread lock barrel based on a memory is used;

when the KV data is updated, a KV system batch operation interface is adopted.

5. The method according to claim 1, wherein the storing of the information on the plurality of fields of the mail by using the ProtoBuf structure specifically comprises:

the ProtoBuf structure stores a plurality of fields of mail information, when the fields are stored in Key-Value of a KV system, the whole structure is serialized into a character string, and when the fields are read, the character string is deserialized, so that the whole process is transparent to a user;

if a new mail information field is needed, only one field of the ProtoBuf structure needs to be added, the newly added field can automatically adopt a default value in the de-serialization process of old data, and the newly added field can be automatically serialized into a character string during updating, so that the whole process has no influence on the whole data, the data conversion is flexible and efficient, and the backward/forward compatibility of the data structure is realized.

Images