CN111988241B

CN111988241B - Message queuing method, system, device and storage medium

Info

Publication number: CN111988241B
Application number: CN202010842359.9A
Authority: CN
Inventors: 樊荣; 何俞勇
Original assignee: Hundsun Technologies Inc
Current assignee: Hundsun Technologies Inc
Priority date: 2020-08-20
Filing date: 2020-08-20
Publication date: 2022-09-30
Anticipated expiration: 2040-08-20
Also published as: CN111988241A

Abstract

The application discloses a message queuing method, a system, equipment and a storage medium, wherein many-to-one RDMA datagram communication is realized between a client and a server, the server uses a message completion queue of an RDMA protocol to detect whether a queuing message receiving result exists, and if so, a source identifier and a source key value in the queuing message are obtained; the server side judges whether the source key value exists in a local registered client side information list or not, and the registered client side information list stores the key value distributed to the registered client side and corresponding source identification information; if so, judging whether the source identification is consistent with a locally stored source identification corresponding to the source key value, if so, confirming that the queuing message is valid, otherwise, confirming that the queuing message is invalid. The method and the device determine the arrival sequence of the queued messages by using the queue of the RDMA protocol, and in the RDMA communication process, the receiving process of data is the queuing process, and the messages are stored by using the inherent receiving queue of the RDMA protocol, so that the queuing of the messages is realized.

Description

Message queuing method, system, device and storage medium

Technical Field

The present application relates to the field of message processing technologies, and in particular, to a message queuing method, system, device, and storage medium.

Background

Message queuing means that a plurality of user clients on a network initiate message requests to the same server, the server needs to process the messages in sequence according to the arrival sequence of the messages, and if the message requests received by the server in a short time exceed the processing capacity, the messages need to be cached according to the arrival sequence, namely, message queuing.

In the prior art, a message receiving process and a message queuing process are independent of each other. Conventionally, a message receiving module is responsible for receiving queuing messages, the queuing module subscribes the queuing messages through an intermediate adapter, the queuing messages are converted into internal messages and called back to the queuing module, and the queuing module sequentially queues the data packets and stores the data packets in a shared memory through locking control.

The message queuing method needs additional message conversion and callback process because the message queuing process is independent of the message receiving process, thereby reducing the message queuing processing efficiency.

Disclosure of Invention

In view of the above problems, the present application provides a message queuing method, system, device and storage medium, so as to solve the problem of low message queuing processing efficiency in the prior art. The specific scheme is as follows:

a message queuing method is applied to a server, and the server and a client communicate through RDMA technology, and the method comprises the following steps:

when a queuing message receiving result is detected in a local message completion queue, acquiring a source identifier and a source key value in the queuing message, wherein the source identifier is QPN information of a queue pair QPN of a sending client of the queuing message;

judging whether the source key value exists in a local registered client information list or not, wherein the registered client information list stores key values distributed to registered clients and corresponding source identification information;

if so, judging whether the source identification is consistent with a locally stored source identification corresponding to the source key value; if the judgment result is consistent, the queuing message is confirmed to be valid, otherwise, the queuing message is confirmed to be invalid.

Preferably, when the queued message is confirmed to be valid, the method further comprises:

and adding sequence numbers to the queued messages according to the receiving sequence of the queued messages.

and recording the memory address of the queuing message into a queuing message address list.

Preferably, the adding sequence numbers to the queued messages according to the receiving order of the queued messages includes:

determining the sequence number of the queuing message according to the number of the effective queuing messages received before the queuing message;

adding the determined sequence number to a sequence number field reserved for the message content part of the queuing message;

or the like, or a combination thereof,

adding the determined sequence number to a reserved space before a message content part of the queued message.

Preferably, the recording the memory address of the queued message into a queued message address list includes:

if the sequence number of the queuing message exists in the sequence number field of the message content part of the queuing message, recording the memory address of the message content part in the queuing message into a queuing message address list;

if the sequence number of the queuing message exists in a reserved space before the message content part of the queuing message, recording the reserved space and the memory address where the message content part is located in the queuing message into a queuing message address list.

Preferably, before receiving the queued message, the method further comprises: and finishing a queuing registration process by interacting with the client, wherein the process comprises the following steps:

responding to a reliable connection establishment request for registration and cancellation initiated by a client, and establishing a reliable connection with the client;

receiving a queuing registration request sent by a client, wherein the queuing registration request comprises QPN information of the client;

allocating a unique key value for the client, and correspondingly recording the key value and QPN information in the queued registration request into a local registered client information list;

and returning the queuing registration response information containing the distributed key value to the client.

Preferably, before the establishing a reliable connection with the client, the queuing registration process further includes:

an RDMA protocol queue pair that receives the queued message is allocated.

Preferably, the method further comprises the following steps: and (3) interacting with the client to finish a queuing logout process, wherein the process comprises the following steps:

receiving a queuing cancellation request sent by a client, wherein the queuing cancellation request comprises QPN information of the client and a corresponding key value;

deleting QPN information in the queuing logout request from a local registered client information list, and releasing a key value corresponding to the QPN information;

returning queuing logout response information to the client;

and responding to a request for disconnecting the reliable connection initiated by the client, and disconnecting the reliable connection with the client.

Preferably, the queued logout procedure further comprises:

judging whether a client side with established connection still exists;

and if not, releasing the allocated RDMA protocol queue pair.

A message queuing system comprising: the client and the server communicate through RDMA technology;

the client responds to a message sending request of an upper layer service, QPN information and a key value of the client are added in the message to be sent to obtain a queuing message, and the key value is distributed to the client by a server in the queuing registration process;

the client sends the queuing message to the server based on the datagram communication of the RDMA protocol;

when detecting that a queuing message receiving result exists in a local message completion queue, the server side acquires a source identifier and a source key value in the queuing message, wherein the source identifier is QPN information of a sending client side of the queuing message; judging whether the source key value exists in a local registered client information list or not, wherein the registered client information list stores key values distributed to registered clients and corresponding source identification information; if so, judging whether the source identification is consistent with a locally stored source identification corresponding to the source key value; if the judgment result is consistent, the queuing message is confirmed to be valid, otherwise, the queuing message is confirmed to be invalid.

Preferably, the client comprises a queuing registration request module and a queuing message sending module; the server comprises a queuing registration processing module and a queuing message confirmation module;

the queuing registration request module is used for establishing reliable connection with the queuing registration processing module before sending the queuing message and sending a queuing registration request containing a client identifier to the queuing registration processing module through the established reliable connection;

the queuing registration processing module is used for allocating a unique key value for the registered client, correspondingly recording the key value and QPN information in the queuing registration request into a local registered client information list, and returning queuing registration response information containing the allocated key value to the queuing registration request module;

the queuing message sending module is used for responding to a message sending request of an upper layer service, adding the key value distributed by the queuing registration processing module in a message to be sent to obtain a queuing message, adding the registered QPN information into the queuing message by using a sending queue corresponding to the registered QPN information, and sending the queuing message to the queuing message confirmation module;

the queuing message confirming module is used for confirming the validity of the queuing message.

Preferably, the queued registration request module is further configured to interact with the queued registration processing module, log off the QPN information that is registered, release a key value corresponding to the QPN information by the queued registration processing module, and disconnect a reliable connection between the queued registration request module and the queued registration processing module after the log-off.

Preferably, the queued message confirmation module is further configured to:

and when the queuing message is confirmed to be valid, adding a sequence number to the queuing message according to the receiving sequence of the queuing message.

Preferably, the queued message confirmation module is further configured to:

and when the queuing message is confirmed to be valid, recording the memory address of the queuing message into a queuing message address list.

A message queuing apparatus comprising: a memory and a processor;

the memory is used for storing programs;

the processor is configured to execute the program to implement the steps of the message queuing method.

A storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the message queuing method as described above.

By means of the technical scheme, the message queuing method achieves many-to-one RDMA datagram communication between the server and the client, the server detects whether a queuing message receiving result exists by using a message completion queue of an RDMA protocol, and when the receiving result is detected, a source identifier and a source key value in the queuing message are obtained, wherein the source identifier is QPN information of a queue pair of a sending client of the queuing message; the server side judges whether the source key value exists in a local registered client side information list or not, and the registered client side information list stores the key value distributed to the registered client side and corresponding source identification information; if so, judging whether the source identification is consistent with a locally stored source identification corresponding to the source key value, if so, confirming that the queuing message is valid, otherwise, confirming that the queuing message is invalid. According to the method and the device, the arrival sequence of the queued messages is determined by utilizing the characteristic that the queues of the RDMA protocol are processed in a first-in first-out sequence, in the RDMA communication process, the receiving process of data is the queuing process, the messages are stored by utilizing the inherent receiving queues of the RDMA protocol, the queuing of the messages is realized, extra message conversion and callback processes are not needed, and the message queuing processing efficiency is greatly improved.

In addition, the source identification and the registered key value are added in the queuing message, so that the validity of the queuing message can be conveniently verified by the server, and the accuracy of the message is ensured.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

fig. 1 is a schematic flow chart of a message queuing method provided from a server side according to an embodiment of the present application;

fig. 2 is a signaling interaction diagram of a client and a server performing queuing registration according to an embodiment of the present application;

FIG. 3 illustrates a schematic diagram of establishing a connection between multiple clients and a server;

fig. 4 is a signaling interaction diagram of another queuing registration between a client and a server according to an embodiment of the present application;

fig. 5 is a signaling interaction diagram for queuing and deregistering between a client and a server according to an embodiment of the present application;

fig. 6 is a signaling interaction diagram of another queuing cancellation performed between a client and a server according to an embodiment of the present application;

fig. 7 is another schematic flow chart of a message queuing method provided from the perspective of a server according to an embodiment of the present application;

FIG. 8 is a schematic flow chart of a message queuing method provided from the perspective of a server according to an embodiment of the present application;

FIG. 9 illustrates a schematic diagram of the manner in which messages are queued for non-sequence number fields;

FIG. 10 is a diagram illustrating the processing of queued messages for a field of ordered numbers;

fig. 11 is a schematic structural diagram of a message queuing system according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of a message queuing apparatus according to an embodiment of the present application.

Detailed Description

Before introducing the present application, some concepts used in the present application will be introduced:

rdma (remote Direct Memory access): remote direct memory access, a technique that allows data to be transferred from the memory of one computer to the memory of another computer over a network without the intervention of an operating system.

RoCE (RDMA over Converged Ethernet): RDMA protocol over converged Ethernet.

Infiniband: a high-performance IO interconnection technology is an RDMA protocol realized on a special Infiniband network.

RC (replaceable connection): reliable connection is a transmission service type of RDMA protocol, and establishes exclusive queue association relationship between two communication parties to ensure reliable transmission of messages between two queues.

UD (Unreliable datagram): unreliable datagram, a type of transport service of RDMA protocol, does not establish queue-to-queue association, and a queue can communicate with any other queue.

QP (queue Pair) RDMA: the queue pair of the protocol comprises a sending queue and a receiving queue.

Qpn (queue Pair number): a queue pair number for identifying a queue pair in a network adapter.

Cq (completion queue): completion queues of the RDMA protocol return the results of message sending or receiving.

Rqe (receive Queue element): and the receiving queue unit comprises cache information stored in the received message.

CQE (completion Queue entry): the completion queue entry contains work request processing result information, and if the CQE generated by the reception processing contains the sender-side information.

GRH (Global Route header): the global routing header of the RoCE protocol and the Infiniband protocol.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The application provides a message queuing method which can be realized based on a message queuing system, wherein the message queuing system comprises a server side for receiving queued messages and a client side for sending the queued messages, the number of the client sides can be multiple, and the client side is a message source needing queuing.

Before the message queuing processing is realized, the server and the client need to realize communication based on RDMA technology, such as RDMA datagram communication, so that one-to-many communication between the server and the client can be realized.

Specifically, the client may include a sorting message sending module, and the server may include a queuing message confirming module. The RDMA communication mode between the server and the client may specifically be that a communication mode of RDMA datagram is adopted between the sequencing message sending module and the queuing message confirming module to implement sending and processing of the queuing message.

The RDMA protocol applied by the scheme of the application can be a RoCE protocol and can also be an Infiniband protocol. If the RoCE protocol is used, the TCP/IP protocol can be simultaneously operated because the RoCE protocol is based on the Ethernet; if the Infiniband protocol is used, all connections need to walk the RDMA software stack since it is a private network.

In order to realize message transmission of queues between a server and a plurality of clients, the communication mode of RDMA datagram, such as RoCE UD type, or Infiniband UD type, or other datagram types, is selected in the embodiment of the application.

Taking RDMA datagram communication mode as an example, if the RoCE protocol is used in this embodiment, the RoCE UD type communication can be performed between the client and the server. The RoCE UD communication mode does not bind queues at two ends, and can realize that a plurality of queues transmit messages to the same queue.

If the Infiniband protocol is used in this embodiment, Infiniband UD type communication can be performed between the client and the server. Similarly, the Infiniband UD communication mode does not bind queues at two ends, and a plurality of queues can send messages to the same queue.

Next, referring to fig. 1, a message queuing method according to the present application is described from a server side, where the method may include the following steps:

step S100, when detecting that a queue message receiving result exists in a local message completion queue, acquiring a source identifier and a source key value in the queue message.

Specifically, the server may detect a message completion queue CQ of the RDMA protocol according to a set detection policy, and determine whether there is a queued message reception result in the CQ. If the message receiving result is determined to be available, the new queuing message is received, so that the source identification and the source key value in the queuing message can be acquired.

Specifically, the server may extract the sending-end information, including the source identifier and the source key value, from the message reception result CQE of the CQ.

Wherein the source identifier is a queue pair QPN information, such as a QPN number, of the sending client of the queued message.

Step S110, judging whether the source key value exists in a local registered client information list, if so, executing step S120, and if not, executing step S140;

the server may locally pre-store a registered client information list, where a KEY value Q _ KEY and corresponding source identification information QPN allocated to a registered client are stored in the registered client information list.

Specifically, after the server allocates the key value to the QPN information of the registered client, the server records the allocated key value and the QPN information of the client in the registered client information list.

The process of registering the client with the server may be described with reference to the following description.

Step S120, determining whether the source identifier is consistent with a locally stored source identifier corresponding to the source key value, if yes, performing step S130, and if not, performing step S140.

Specifically, the server queries a source identifier corresponding to the source key value stored in the local registered client information list, and further determines whether the queried source identifier is consistent with the source identifier in the queued message, thereby implementing validation of the queued message. It is understood that if they are consistent, they indicate that the queued message is valid, and if not, they indicate that the queued message is invalid.

And step S130, confirming that the queuing message is valid.

And step S140, confirming that the queuing message is invalid.

The message queuing method of the embodiment of the application realizes many-to-one RDMA datagram communication between the server and the client, the server detects whether a queuing message receiving result exists by using a message completion queue of an RDMA protocol, and when the receiving result is detected, a source identifier and a source key value in the queuing message are obtained, wherein the source identifier is QPN information of a queue pair of a sending client of the queuing message; the server side judges whether the source key value exists in a local registered client side information list or not, and the registered client side information list stores the key value distributed to the registered client side and corresponding source identification information; if so, judging whether the source identification is consistent with a locally stored source identification corresponding to the source key value, if so, confirming that the queuing message is valid, otherwise, confirming that the queuing message is invalid. According to the method and the device, the arrival sequence of the queued messages is determined by utilizing the characteristic that the queues of the RDMA protocol are processed in a first-in first-out sequence, in the RDMA communication process, the receiving process of data is the queuing process, the messages are stored by utilizing the inherent receiving queues of the RDMA protocol, the queuing of the messages is realized, extra message conversion and callback processes are not needed, and the message queuing processing efficiency is greatly improved.

Further, under the condition that the network card supports the loop of the RDMA message, the client and the server may be deployed on the same server to implement local message queuing.

It can be understood that the above steps S110 and S120 may exchange execution sequences with each other, that is, it may be determined whether the source identifier is consistent with a locally stored source identifier corresponding to the source key value, if not, the queued message is determined to be invalid, if so, it is further determined whether the source key value exists in a local registered client information list, if so, the queued message is determined to be valid, otherwise, the queued message is determined to be invalid.

Optionally, before the server detects whether there is a queued message receiving result in the local message completion queue in step S100, the following steps may be further added in the solution of the present application:

the server side detects whether a queuing client side for establishing connection exists according to a set detection strategy, if not, the server side is ended and waits for the next detection according to the set detection strategy, and if so, the server side can execute the step S100.

The set detection strategy may be to detect once every set time length.

The process of detecting whether there is a client that establishes a connection by the server may be detecting whether there is locally stored registered client information, such as client QPN information, and determining that there is a client that establishes a connection if it is detected that there is locally stored registered client information, otherwise, determining that there is no client that establishes a connection.

In another embodiment of the present application, a process of completing a queuing registration by interacting with a client before a server receives a queuing message is described.

Optionally, the client may include a queuing registration request module, and the server may include a queuing registration processing module, so that the client and the server interact to complete the queuing registration process, or the queuing registration request module and the queuing registration processing module interact to complete the queuing registration process.

As can be seen from the signaling interaction diagram of the client and server registration process illustrated in fig. 2, the queuing registration process may include:

and step S10, the client establishes reliable connection with the server.

Specifically, the client initiates a reliable connection establishment request to the server, and the server responds to the request to establish a reliable connection between the client and the server.

Wherein a reliable connection is a connection for queued registrations and deregistrations.

The reliable connection established between the client and the server may specifically be established between the queued registration request module and the queued registration processing module.

The reliable connection type may be TCP or RoCE RC or Infiniband RC, and may be selected according to the network type of the actual networking.

Referring to fig. 3, a schematic diagram illustrating connections established between multiple clients and a single server is shown.

As can be seen from fig. 3, in order to implement reliable transmission of registration and deregistration data between the client and the server, a reliable connection needs to be established between the queued registration request module of the client and the queued registration processing module of the server, so as to implement the processes of queued registration and deregistration.

Meanwhile, in order to realize the sending and processing of the queuing messages after the queuing registration and ensure that a single server can simultaneously process the queuing messages of a plurality of clients, a datagram type communication mode is adopted between a queuing message sending module of the client and a queuing message confirming module of the server.

Of course, fig. 3 merely illustrates an example of three connections between clients and a server, where the number of clients may be other values.

Step S11, the client sends a queued registration request containing QPN information to the server.

Specifically, the client may send a queuing registration request to the server through a reliable connection established with the server. The queued registration request carries QPN information of the client.

Step S12, the server allocates a unique key value to the client, and records the key value and the QPN information in a local registered client information list.

Specifically, the server allocates a unique key value to the registered client, and simultaneously, records the key value and QPN information in the queued registration request into a local registered client information list, so as to verify the validity of the received queued message in the following.

And step S13, the server side returns the queuing registration response information containing the distributed key value to the client side.

For the client, it may store the key value for later addition when sending queued messages.

Optionally, referring to fig. 4, it illustrates another signaling interaction diagram of a client and server registration process, where:

steps S22-S25 correspond one-to-one to steps S10-S13 in FIG. 2.

Compared with the flow illustrated in fig. 2, the following steps are further added in fig. 4 before the client establishes a reliable connection with the server:

step S20, the server allocates QP for receiving the queuing message.

The server needs to complete the distribution process of the QP for locally receiving the queuing message before the client initiates the registration request.

Step S21, the client allocates a QP for sending the queuing message.

Before initiating the queuing registration, the client needs to allocate a QP for sending the queuing message.

It should be understood that there is no specific sequence between step S20 and step S21, and fig. 4 is only an example and should not be construed as a limitation to the implementation of the scheme.

The above embodiment describes the flow of the service end and the client end for interactively completing the queuing registration. Further, in this embodiment of the present application, a process of queuing and logout between the server and the client is introduced.

Optionally, the process of queue logout is completed by the client and the server in an interactive manner, which may be a process of queue logout completed by the interaction of the queue registration request module and the queue registration processing module.

As can be seen from the signaling interaction diagram of the client and server logout procedure illustrated in fig. 5, the queuing logout procedure may include:

and step S30, the client sends a queue logout request containing QPN information and the key value to the server.

Specifically, after the client finishes sending the queuing message to the server, the client may send a queuing deregistration request to the server to deregister the connection established with the server. The queue logout request comprises QPN information registered by the client and a corresponding key value.

Optionally, after receiving a queuing cancellation request sent by the client, if there are messages sent by the client that do not complete queuing confirmation locally, the server may discard the messages according to a policy selection or wait for completion of processing of the messages before cancellation; queued messages that do not reach the server are not guaranteed to be executed.

Step S31, the server deletes the QPN information in the queuing deregistration request from the registered client information list, and releases the key value corresponding to the QPN information.

Specifically, the server responds to the queuing logout request, deletes the QPN information of the client from the registry, and meanwhile, may release the key value corresponding to the QPN information, and the released key value may be used for allocation to other clients requesting registration.

And step S32, the server returns queuing logout response information to the client.

And step S33, the client disconnects the reliable connection with the server.

Specifically, after receiving the queuing and logout response information, the client can actively disconnect the reliable connection with the server. Alternatively, a request for disconnecting the reliable connection may be further sent to the server, and the server disconnects the reliable connection with the client.

Optionally, referring to fig. 6, it illustrates another signaling interaction diagram of a client and server logout procedure, where:

steps S40-S43 correspond one-to-one to steps S30-S33 in FIG. 5.

Compared with the flow illustrated in fig. 5, the following steps are further added in fig. 6:

step S44, the client releases the QP for sending the queued message.

Specifically, after the client disconnects the reliable connection with the server, the QP for sending the queuing message may be released.

Step S45, the server determines whether there is a client that has already established a connection, and if not, executes step S46.

Step S46, the server releases the allocated QP.

It should be understood that there is no specific sequence restriction between step S44 and step S45, and fig. 6 is only an example, and should not be construed as a limitation on the implementation of the scheme.

After the server breaks the reliable connection with the client requesting to logout, the server can further judge whether other clients establish connection with the server, if so, the server can continue working, and if not, the server indicates that no client currently establishes reliable connection with the server, namely no client needs to send a queuing message. Based on this, the server can choose to release the allocated QP or choose to keep the allocated QP according to the set policy.

The service end can not provide the message queuing function after releasing the allocated QP, and can only return failure if receiving a new registration request until the QP for receiving the queued message is reallocated.

After the flow of queuing, registering, and deregistering between the client and the server is described in the above embodiment, another embodiment of the present application continuously introduces the message queuing method of the present application from the perspective of the server.

Referring to fig. 7, another message queuing method is introduced, which may include:

step S200, when detecting that a queue message receiving result exists in the local message completion queue, acquiring a source identifier and a source key value in the queue message.

Step S210, determining whether the source key value exists in the local registered client information list, if yes, executing step S220, and if not, executing step S240.

Step S220, determining whether the source identifier is consistent with a locally stored source identifier corresponding to the source key value, if so, performing step S230, and if not, performing step S240.

Specifically, the steps S200 to S220 correspond to the steps S100 to S120 one to one, and the detailed description is omitted here for brevity.

And step S240, confirming that the queuing message is invalid.

And step S230, confirming that the queuing message is valid, and adding a sequence number to the queuing message according to the receiving sequence of the queuing message.

Compared with the embodiment corresponding to fig. 1, in this embodiment, when the queued message is confirmed to be valid, an operation of adding a sequence number to the queued message is further added, that is, a sequence number is added to the queued message according to the receiving sequence of the valid queued message, so as to ensure the accuracy of the sequence of the message in the subsequent forwarding process.

Optionally, when the server determines that the current queued message is valid, the server may refer to the number of valid queued messages received before the current queued message, and determine the sequence number of the current queued message, and if the number of valid queued messages received before the current queued message is m, the sequence number of the current queued message may be defined as m + 1. That is, the queued messages for which the acknowledgement is invalid can be discarded by the server without adding a sequence number thereto.

After determining the sequence number of the currently queued message, a sequence number may further be added to the currently queued message.

Two alternative implementations of adding sequence numbers to queued messages are provided in this embodiment. Respectively as follows:

a first kind,

For the case where the queued message itself does not have a sequence number field, the determined sequence number may be added using the reserved space before the message content portion of the queued message. The reserved space may be in different forms according to different connection modes, and for example, a RoCE UD communication mode is established between the client and the server, and the communication mode needs to reserve a 40-byte UD Addition space in advance of the message content to store GRH information, which is physically continuous with the message content space. Therefore, the sequence number of the queued message may be added to the UD Addition space.

A second kind,

For the case where the queued message itself reserves the sequence number field, the determined sequence number may be added to the sequence number field reserved for the message content portion of the queued message, without using the UD Addition space.

Still further, referring to fig. 8, yet another message queuing method is introduced, which may include:

step S300, when detecting that a queue message receiving result exists in the local message completion queue, acquiring a source identifier and a source key value in the queue message.

Step S310, determining whether the source key value exists in the local registered client information list, if yes, executing step S320, and if not, executing step S340.

Step S320, determining whether the source identifier is consistent with a locally stored source identifier corresponding to the source key value, if so, performing step S330, and if not, performing step S340.

Specifically, the steps S300 to S320 correspond to the steps S200 to S220 one to one, and the detailed description is omitted here for brevity.

And step S340, confirming that the queuing message is invalid.

And step S330, confirming that the queuing message is valid, and adding a sequence number to the queuing message according to the receiving sequence of the queuing message.

And step S350, recording the memory address of the queuing message into a queuing message address list.

Specifically, when the queued message is confirmed to be valid, the memory address where the queued message is located is further recorded in the queued message address list in this embodiment.

The message buffer pointed by the receiving queue unit RQE of the RDMA protocol is pre-allocated, so that the server can index the memory address stored in the queuing message through the RQE and further record the address in the queuing message address list.

In the embodiment, the memory storage address of the effective queuing message is recorded by adding the queuing message address list, the message is not required to be copied in the subsequent queuing message processing process, the message address only needs to be transmitted through the queuing message address list, and the processing efficiency is improved.

Optionally, the queued message address list may include a virtual address and a physical address of the queued message, where the virtual address is used for subsequent processing and calling by local software, and the physical address is used for forwarding by hardware.

It is understood that the process of recording the memory address of the queued message in the queued message address list may be added to the corresponding embodiments of fig. 1 or 7, and fig. 8 illustrates only one of the optional cases.

On the basis of adding the sequence number to the queued message described in the previous embodiment, according to two different adding manners of the sequence number of the queued message, in step S350 in this embodiment, there are also two different implementation manners for recording the memory address where the queued message is located in the address list of the queued message, which are respectively as follows:

a first kind,

A second kind,

And if the sequence number of the queuing message exists in the sequence number field of the message content part of the queuing message, recording the memory address of the message content part in the queuing message into a queuing message address list.

Next, referring to fig. 9 and fig. 10, the processing modes of the service end for two different types of queued messages will be described according to whether the queued message has a sequence number field or not.

Fig. 9 illustrates a processing manner of queued messages without sequence number field by the service end:

each RQE in a receiving queue of the RDMA protocol at the server end points to a section of memory for storing messages, when the receiving of the queued messages is completed, the messages are stored in the specified memory, and a corresponding completion queue entry CQE is generated at the same time. And (4) queuing messages indexed according to the sequence of the RQE in the queue, namely the message sequence which is queued.

The server side carries out message validity verification on the received queuing messages based on QPN and Q _ KEY in the queuing registry, if a certain queuing message is found to be invalid, the queuing sequence number skips the message, and the message cannot be recorded in a queuing message address list. As shown in fig. 9, the server receives n queued messages in total, where the CQE2 checks that the queued message pointed by the corresponding RQE2 is invalid, and no queued sequence number is added thereto, and at the same time, the memory address of the queued message is not recorded in the queued message address list, and finally, the valid queued message sequence number is recorded in n-1, and there are only n-1 records in the queued message address list.

It should be noted that, since the sequence number of the queued message is added to the UD Addition space, when recording the memory address of the queued message, it is necessary to record the memory address of the queued message from the UD Addition space start address of the queued message.

FIG. 10 illustrates how the server processes queued messages with sequence number fields:

when the server adds the sequence number to the queuing message, the sequence number can be directly added to the sequence number field, and the UD Addition space is not needed. When recording the memory address of the queued message, the memory address of the queued message may be recorded starting from the start address of the message content field of the queued message. The other processing is the same as that of fig. 9.

In another embodiment of the present application, there is further provided a message queuing system, as shown in fig. 11, the message queuing system may include:

the client 10 and the server 20, the client 10 and the server 20 realize communication based on RDMA technology.

The client 10 responds to the message sending request of the upper layer service, and adds QPN information and key values of the client to the message to be sent to obtain a queuing message. Wherein the key value is that the queue registration process is allocated by the server 20 for the client.

The client 10 sends the queued message to the server 20 based on datagram communication of the RDMA protocol.

When detecting that there is a queued message receiving result in the local message completion queue, the server 20 obtains a source identifier and a source key value in the queued message, where the source identifier is QPN information of a sending client of the queued message. Further, judging whether the source key value exists in a local registered client information list or not, wherein the registered client information list stores the key value distributed to the registered client and corresponding source identification information; if so, judging whether the source identification is consistent with a locally stored source identification corresponding to the source key value; if the judgment result is consistent, the queuing message is confirmed to be valid, otherwise, the queuing message is confirmed to be invalid.

Further, still referring to fig. 11, the client 10 may include a queued registration request module 100 and a queued message sending module 101. The server 20 may include a queued registration processing module 200 and a queued message confirmation module 201.

The queuing registration request module 100 is configured to establish a reliable connection with the queuing registration processing module 200 before sending the queuing message, and send a queuing registration request including the client identifier to the queuing registration processing module 200 through the established reliable connection.

The queued registration processing module 200 is configured to allocate a unique key value to the registered client, record the key value and QPN information in the queued registration request into a local registered client information list, and return queued registration response information including the allocated key value to the queued registration request module 100.

The queuing message sending module 101 is configured to respond to a message sending request of an upper layer service, add a key value to a message to be sent, obtain a queuing message, add the registered QPN information to the queuing message by using a sending queue corresponding to the registered QPN information, and send the queuing message to the queuing message confirmation module 201.

The queued message confirming module 201 is configured to confirm validity of the queued message.

Further, the queue registration request module 100 is further configured to interact with the queue registration processing module 200, log off the registered QPN information, release a key value corresponding to the QPN information by the queue registration processing module 200, and disconnect a reliable connection between the QPN information and the key value after logging off.

Still further, the queued message confirmation module 201 may be further configured to:

The specific implementation process of adding the sequence number in the queued message may refer to the foregoing description, and is not described herein again.

The specific implementation process of recording the memory address of the queuing message in the queuing message address list may refer to the foregoing description, and is not described here any more.

The message queuing device provided by the embodiment of the application can be applied to message queuing equipment, such as a terminal: mobile phones, computers, etc. Alternatively, fig. 12 is a block diagram showing a hardware configuration of the message queuing apparatus, and referring to fig. 12, the hardware configuration of the message queuing apparatus may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

in the embodiment of the application, the number of the processor 1, the communication interface 2, the memory 3 and the communication bus 4 is at least one, and the processor 1, the communication interface 2 and the memory 3 complete mutual communication through the communication bus 4;

the processor 1 may be a central processing unit CPU, or an application Specific Integrated circuit asic, or one or more Integrated circuits configured to implement embodiments of the present invention, etc.;

the memory 3 may include a high-speed RAM memory, and may further include a non-volatile memory (non-volatile memory) or the like, such as at least one disk memory;

wherein the memory stores a program and the processor can call the program stored in the memory, the program for:

when a queuing message receiving result is detected in a local message completion queue, acquiring a source identifier and a source key value in the queuing message, wherein the source identifier is QPN information of a queue pair of a sending client of the queuing message;

if so, judging whether the source identification is consistent with a locally stored source identification corresponding to the source key value;

if the judgment result is consistent, the queuing message is confirmed to be valid, otherwise, the queuing message is confirmed to be invalid.

Alternatively, the detailed function and the extended function of the program may be as described above.

Embodiments of the present application further provide a storage medium, where a program suitable for execution by a processor may be stored, where the program is configured to:

Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, the embodiments may be combined as needed, and the same and similar parts may be referred to each other.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A message queuing method applied to a server, wherein the server communicates with a client by using RDMA technology, the method comprising:

if yes, confirming that the queuing message is valid, and adding a sequence number to the queuing message according to the receiving sequence of the queuing message; determining the sequence number of the queuing message according to the number of the effective queuing messages received before the queuing message;

or the like, or, alternatively,

adding the determined sequence number to a reserved space before a message content part of the queued message; and if the judgment result is inconsistent, the queuing message is confirmed to be invalid.

2. The method of claim 1, wherein upon confirming that the queued message is valid, the method further comprises:

3. The method according to claim 2, wherein the recording the memory address of the queued message in a queued message address list comprises:

4. A method according to any of claims 1-3, wherein prior to receiving the queued message, the method further comprises: and finishing a queuing registration process by interacting with the client, wherein the process comprises the following steps:

5. The method of claim 4, wherein queuing the registration process further comprises, prior to said establishing the reliable connection with the client:

RDMA protocol queue pairs that receive queued messages are allocated.

6. The method according to any one of claims 1-3, further comprising: and finishing a queuing logout process by interacting with the client, wherein the process comprises the following steps:

receiving a queuing logout request sent by a client, wherein the queuing logout request comprises QPN information of the client and a corresponding key value;

returning queuing logout response information to the client;

7. The method of claim 6, wherein the queued logoff flow further comprises:

judging whether a client side which establishes connection still exists or not;

if not, releasing the allocated RDMA protocol queue pair.

8. A message queuing system, comprising: the client side and the server side comprise a queuing message confirmation module, and the client side and the server side communicate through an RDMA technology;

when detecting that a queuing message receiving result exists in a local message completion queue, the server side acquires a source identifier and a source key value in the queuing message, wherein the source identifier is QPN information of a sending client side of the queuing message; judging whether the source key value exists in a local registered client information list or not, wherein the registered client information list stores the key value distributed to the registered client and corresponding source identification information; if so, judging whether the source identification is consistent with a locally stored source identification corresponding to the source key value; if the judgment result is consistent, the queuing message is confirmed to be valid, otherwise, the queuing message is confirmed to be invalid;

the queuing message confirming module is used for confirming the validity of the queuing message; when the queuing message is confirmed to be valid, adding a sequence number to the queuing message according to the receiving sequence of the queuing message; determining the sequence number of the queuing message according to the number of the effective queuing messages received before the queuing message; adding the determined sequence number to a sequence number field reserved for the message content part of the queuing message; or, adding the determined sequence number to a reserved space before a message content part of the queued message.

9. The system of claim 8, wherein the client comprises a queued registration request module and a queued message sending module; the server also comprises a queuing and registering processing module;

the queuing registration processing module is used for distributing a unique key value for the registered client, correspondingly recording the key value and QPN information in the queuing registration request into a local registered client information list, and returning queuing registration response information containing the distributed key value to the queuing registration request module;

the queuing message sending module is used for responding to a message sending request of an upper layer service, adding a key value distributed by the queuing registration processing module in a message to be sent to obtain a queuing message, adding the registered QPN information into the queuing message by using a sending queue corresponding to the registered QPN information, and sending the queuing message to the queuing message confirmation module.

10. The system according to claim 9, wherein the queued registration request module is further configured to interact with the queued registration processing module, perform cancellation on the QPN information that is registered, release a key value corresponding to the QPN information by the queued registration processing module, and disconnect a reliable connection between the queued registration request module and the queued registration processing module after cancellation.

11. The system of claim 9, wherein the queued message confirmation module is further configured to:

and when the queued message is confirmed to be valid, recording the memory address of the queued message into a queued message address list.

12. A message queuing apparatus, comprising: a memory and a processor;

the memory is used for storing programs;

the processor is used for executing the program and realizing the steps of the message queuing method according to any one of claims 1-7.

13. A storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, performs the steps of the message queuing method as claimed in any one of claims 1 to 7.