CN118796355A - Data processing method, device, computer readable storage medium and electronic device - Google Patents

Abstract

The present invention discloses a data processing method, device, computer-readable storage medium and electronic device. It relates to the field of distributed computing, and the method comprises: obtaining a matching relationship set and container group information of the first container group through an application in a first container group, wherein the matching relationship set includes matching relationships between multiple first container groups and multiple storage directories, and different first container groups match different storage directories; determining the storage directory matched by the first container group according to the container group information and the matching relationship set through an application in the first container group to obtain a target storage directory; and storing the data processed by the application in the first container group to the target storage directory through an application in the first container group. The present invention solves the technical problem in the related art that the applications in each container group write the processed data into the same file, which easily leads to data loss or confusion, and thus has poor data processing reliability.

Description

Data processing method, device, computer readable storage medium and electronic device

Technical Field

The present invention relates to the field of distribution, and in particular to a data processing method, device, computer-readable storage medium and electronic device.

Background Art

With the rapid development of cloud computing technology, containerization technology has been widely used, especially in container orchestration platforms such as Kubernetes (K8s for short). Kubernetes is an open source container orchestration system that can automatically deploy, scale and manage containerized applications. In K8s, the container group Pod is the basic deployment unit, which is used to encapsulate one or more containers to implement application deployment and management.

Pods in K8s are often used to process data streams. After data processing is completed, the Pod writes the data to a file for use by other services. In related technologies, the K8s system includes multiple container group Pods, each of which processes the data separately and writes the processed data into the same file. When the data accumulates to a certain size, it is uploaded to the cloud storage service. The process is shown in Figure 1. When multiple container group Pods write data to the same file, write conflicts may occur, resulting in inconsistent or lost data. In high-concurrency scenarios, multiple Pods writing data to the same file at the same time may also cause I/O (Input/Output) performance bottlenecks, affecting the stability and response speed of the entire system.

In summary, for existing Pod-based data processing methods, multiple Pods writing processed data to the same file may cause data loss or confusion, thereby affecting the reliability of data processing.

Summary of the invention

The embodiments of the present invention provide a data processing method, device, computer-readable storage medium and electronic device to at least solve the technical problem in the related art that applications in each container group write processed data into the same file, which easily leads to data loss or confusion, thereby resulting in poor data processing reliability.

According to one aspect of an embodiment of the present invention, a data processing method is provided, which is applied to a distributed system. The distributed system includes multiple first container groups, and the first container group is a container group running a data processing application. The method includes: obtaining a matching relationship set and container group information of the first container group through an application in the first container group, wherein the matching relationship set includes matching relationships between the multiple first container groups and multiple storage directories, and different first container groups match different storage directories; determining a storage directory matched by the first container group according to the container group information and the matching relationship set through the application in the first container group to obtain a target storage directory; and storing data processed by the application in the first container group in the target storage directory through the application in the first container group.

Furthermore, the data processing method further includes: judging, based on the container group information, whether the matching relationship set includes the matching relationship corresponding to the first container group; if the matching relationship corresponding to the first container group is included, determining the storage directory in the matching relationship corresponding to the first container group as the target storage directory; if the matching relationship corresponding to the first container group is not included, determining the first storage directory, and determining the first storage directory as the target storage directory, wherein the first storage directory refers to a storage directory that is not matched with the first container group.

Furthermore, the data processing method further includes: after determining the first storage directory as the target storage directory, establishing a matching relationship between the first container group and the first storage directory in the matching relationship set.

Furthermore, the data processing method also includes: obtaining multiple preset directory numbers; for the Nth directory number among the multiple directory numbers, determining whether there is a second storage directory in the matching relationship set, wherein the second storage directory is a storage directory using the Nth directory number; if the second storage directory does not exist in the matching relationship set, determining the second storage directory as the first storage directory; if the second storage directory exists in the matching relationship set, updating N, and repeating the step of determining whether there is a new second storage directory in the matching relationship set until there is no new second storage directory in the matching relationship set, and determining the new second storage directory as the first storage directory.

Furthermore, the distributed system further includes a second container group, which is a container group running a directory management application. The data processing method further includes: determining, by the application in the second container group, a storage directory existing in the distributed system to obtain a plurality of third storage directories; for each third storage directory, determining, by the application in the second container group, whether the matching relationship set includes the third storage directory; if the third storage directory is included, determining, by the application in the second container group, whether a first container group matched by the third storage directory in the matching relationship set is in a running state; if the first container group matched by the third storage directory is not in a running state, deleting, by the application in the second container group, a matching relationship corresponding to the third storage directory from the matching relationship set.

Furthermore, the data processing method also includes: when the first container group matching the third storage directory is not in operation, uploading the data under the third storage directory to the target server through the application in the second container group; deleting the data under the third storage directory through the application in the second container group, and deleting the third storage directory.

Furthermore, the data processing method also includes: after determining whether the matching relationship set includes the third storage directory through the application in the second container group, if the third storage directory is not included, obtaining preset configuration information through the application in the second container group, wherein the configuration information indicates whether the third storage directory not recorded in the matching relationship set is allowed to be deleted; and determining, through the application in the second container group, a processing method for the third storage directory based on the configuration information.

According to another aspect of an embodiment of the present invention, a data processing device is provided, which is applied to a distributed system. The distributed system includes multiple first container groups, and the first container group is a container group running a data processing application. The device includes: a first acquisition module, which is used to acquire a matching relationship set and container group information of the first container group through the application in the first container group, wherein the matching relationship set includes matching relationships between multiple first container groups and multiple storage directories, and different first container groups match different storage directories; a first determination module, which is used to determine the storage directory matched by the first container group according to the container group information and the matching relationship set through the application in the first container group, and obtain a target storage directory; and a storage module, which is used to store data processed by the application in the first container group to the target storage directory.

According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned data processing method when running.

According to another aspect of an embodiment of the present invention, an electronic device is also provided, which includes one or more processors; a memory for storing one or more programs, so that when the one or more programs are executed by the one or more processors, the one or more processors are implemented to run the programs, wherein the programs are configured to execute the above-mentioned data processing method when running.

In the embodiment of the present invention, the data processed by the applications in different container groups are written into different directories, and the matching relationship set and the container group information of the first container group are obtained through the application in the first container group, and then the storage directory matched by the first container group is determined by the application in the first container group according to the container group information and the matching relationship set to obtain the target storage directory, so that the data processed by the application is stored in the target storage directory through the application in the first container group. The matching relationship set includes matching relationships between multiple first container groups and multiple storage directories, and different first container groups match different storage directories.

In the above process, by acquiring the matching relationship set and the container group information of the first container group, and determining the storage directory matched by the first container group according to the container group information and the matching relationship set, the target storage directory corresponding to the current first container group is accurately determined. Since different first container groups match different storage directories, the target storage directory is only used to store data processed by the application in the current first container group. When the data processed by the application is stored in the corresponding target storage directory through the application in the first container group, the data processed by the applications in different first container groups is written into different directories, that is, into different files, thereby avoiding data loss or confusion and improving data processing reliability.

It can be seen that the solution provided by the present application achieves the purpose of writing the data processed by applications in different container groups into different directories, thereby achieving the technical effect of improving the reliability of data processing, and further solves the technical problem that the applications in each container group in the related technology write the processed data into the same file, which easily leads to data loss or confusion, thereby resulting in poor data processing reliability.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

FIG1 is a schematic diagram of an optional data processing method in the related art;

FIG2 is a schematic diagram 1 of an optional data processing method according to an embodiment of the present invention;

FIG3 is a schematic diagram of an optional working application in a second container group according to an embodiment of the present invention;

FIG4 is a second schematic diagram of an optional data processing method according to an embodiment of the present invention;

FIG5 is a schematic diagram of an optional working application in a first container group according to an embodiment of the present invention;

FIG6 is a schematic diagram of an optional data processing device according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of an optional electronic device according to an embodiment of the present invention.

DETAILED DESCRIPTION

In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

Kubernetes: Kubernetes is an open source container orchestration engine that supports automated deployment, large-scale scalability, and application containerization management. Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services that facilitates declarative configuration and automation. Kubernetes has a large and rapidly growing ecosystem with a wide range of services, support, and tools.

Pod: In a kubernetes cluster, Pod is the basis of all business types and the smallest unit level of K8S management. It is a combination of one or more containers. These containers share storage, network, and namespace, as well as specifications for how to run. In a Pod, all containers are arranged and scheduled in the same way and run in a shared context. For specific applications, Pod is their logical host, and Pod contains multiple application containers related to the business.

Container: A container is a standardized software unit that packages code and all its dependencies so that applications can run reliably and quickly from one computing environment to another. A container image is a lightweight, independent executable software package. It contains everything a program needs to run: code, runtime dependencies, system tools, system libraries, and settings. The container is an abstraction at the application layer that packages code and dependencies together. Multiple containers can run on the same machine and share the operating system kernel with other containers, and each container runs as an independent process in user space.

Example 1

According to an embodiment of the present invention, an embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

FIG. 2 is a schematic diagram of an optional data processing method according to an embodiment of the present invention. As shown in FIG. 2 , the method is applied to a distributed system, the distributed system includes a plurality of first container groups, the first container group is a container group running a data processing application, and the method includes the following steps:

Step S201: obtaining a matching relationship set and container group information of the first container group through an application in the first container group, wherein the matching relationship set includes matching relationships between multiple first container groups and multiple storage directories, and different first container groups match different storage directories.

Optionally, in this embodiment, the distributed system may be a Kubernetes system, the container group may refer to a Pod, and the data processing application is at least used to process data, and the processing method includes but is not limited to parsing, format conversion, compression, etc. The codes and configurations of the multiple first container groups may be the same or different.

In some embodiments, the data processing application is different according to different actual application scenarios, and the processed data is also different. For example, in a financial scenario, the data processing application may be a payment application, which can be used to process the data in the user's payment request, and the processed data may be payment information and payment results. For another example, in a big data scenario, the data processing application may be an information analysis application, which can analyze the corresponding big data (for example, the user's browsing history, etc.) according to the user's needs, and generate corresponding analysis results (for example, the user's browsing preferences), and the processed data is the aforementioned analysis results. In this embodiment, there is no specific limitation on the application type of the data processing application and the data content of the data processed by the application.

Optionally, for any first container group, an application in the first container group may obtain a matching relationship set and container group information of the first container group. For example, the application obtains the matching relationship set and container group information of the first container group when the application is started, or for another example, the application obtains the matching relationship set and container group information of the first container group when receiving data to be processed.

In some embodiments, the matching relationship set includes matching relationships between multiple first container groups and multiple storage directories. If a first container group matches a storage directory, the storage directory is determined to be used to store data processed by the application in the first container group. Different first container groups match different storage directories, that is, each first container group stores the processed data in a different storage directory.

In some embodiments, the container group information may refer to an identifier of the container group. For example, the identifier of the container group is exposed as an environment variable POD_NAME, so that the application can obtain the value of POD_NAME from the environment variable, that is, obtain the container group identifier.

Step S202: Determine the storage directory that matches the first container group according to the container group information and the matching relationship set through the application in the first container group, and obtain the target storage directory.

For example, the matching relationship set includes the matching relationship corresponding to the first container group. In this case, for any first container group, the application in the first container group can determine the matching relationship corresponding to the first container group from the matching relationship set, and thus determine the target storage directory according to the matching relationship.

For another example, the matching relationship set may include the matching relationship corresponding to the first container group, or may not include the matching relationship corresponding to the first container group. In this case, for any first container group, the application in the first container group can first determine whether the matching relationship set includes the matching relationship corresponding to the first container group. If it does, the target storage directory is determined based on the matching relationship. If it does not, any storage directory that does not match the first container group can be determined as the target storage directory.

Step S203: The data processed by the application is stored in the target storage directory through the application in the first container group.

Optionally, for any first container group, when an application in the first container group completes processing after receiving the data, the data processed by the application can be stored in the target storage directory. For example, the target storage directory stores a target file, and the data processed by the application is stored in the target file in the target storage directory. The target files in different storage directories are also different.

Based on the scheme defined in the above steps S201 to S203, it can be known that in the embodiment of the present invention, the data processed by the applications in different container groups are written into different directories, and the matching relationship set and the container group information of the first container group are obtained through the application in the first container group, and then the storage directory matched by the first container group is determined by the application in the first container group according to the container group information and the matching relationship set to obtain the target storage directory, so that the data processed by the application is stored in the target storage directory through the application in the first container group. The matching relationship set includes matching relationships between multiple first container groups and multiple storage directories, and different first container groups match different storage directories.

It is easy to notice that, in the above process, by obtaining the matching relationship set and the container group information of the first container group, and determining the storage directory matched by the first container group according to the container group information and the matching relationship set, the target storage directory corresponding to the current first container group is accurately determined. Since different first container groups match different storage directories, the target storage directory is only used to store data processed by the application in the current first container group. When the data processed by the application is stored in the corresponding target storage directory through the application in the first container group, the data processed by the applications in different first container groups is written into different directories, that is, into different files, thereby avoiding data loss or confusion and improving data processing reliability.

It can be seen that the solution provided by the present application achieves the purpose of writing the data processed by applications in different container groups into different directories, thereby achieving the technical effect of improving the reliability of data processing, and further solves the technical problem that the applications in each container group in the related technology write the processed data into the same file, which easily leads to data loss or confusion, thereby resulting in poor data processing reliability.

In an optional embodiment, in the process of determining the storage directory matched by the first container group according to the container group information and the matching relationship set by the application in the first container group to obtain the target storage directory, the application in the first container group may determine whether the matching relationship set includes the matching relationship corresponding to the first container group according to the container group information; if the matching relationship corresponding to the first container group is included, the storage directory in the matching relationship corresponding to the first container group is determined as the target storage directory; if the matching relationship corresponding to the first container group is not included, the first storage directory is determined and determined as the target storage directory, wherein the first storage directory refers to a storage directory that is not matched with the first container group.

Optionally, the matching relationship set may record the matching relationships between multiple first container groups and multiple storage directories based on the container group information of the first container group and the identifier of the storage directory. The above container group information may refer to the identifier of the container group. The matching relationship set may be obtained by the application from the first storage area, and the first storage area may be a storage area in a cloud storage service, a storage area in a distributed coordination service, or other types of storage areas.

Optionally, for any first container group, an application in the first container group may determine whether the matching relationship set includes container group information of the first container group, thereby determining that the matching relationship set includes the matching relationship corresponding to the first container group when the matching relationship set includes the container group information of the first container group; and determining that the matching relationship set does not include the matching relationship corresponding to the first container group when the matching relationship set does not include the container group information of the first container group.

In some embodiments, the application of the first container group can obtain a matching relationship set when the startup is completed, and determine whether the matching relationship set includes the matching relationship corresponding to the first container group based on the container group information. If it is found in this process that the matching relationship set includes the matching relationship corresponding to the first container group, it means that the application in the first container group has been restarted abnormally. Therefore, the storage directory in the matching relationship corresponding to the first container group can be directly determined as the target storage directory, and the directory can be reused. The matching relationship corresponding to the first container group refers to the matching relationship that includes the container group information of the first container group.

In some embodiments, if the matching relationship set does not include the matching relationship corresponding to the first container group, it means that the application in the first container group may not have processed the data before. For example, the first container group is a newly created first container group. In this case, the application in the first container group can determine the first storage directory and determine the first storage directory as the target storage directory. The first storage directory refers to a storage directory that is not matched with the first container group.

In some embodiments, each storage directory is set in a second storage area, and the second storage area is different from the first storage area. The second storage area may refer to a disk or other types of storage areas in a distributed system.

For example, the application in the first container group may determine the storage directory that already exists in the second storage area, and determine the first storage directory from the storage directories that already exist in the second storage area, thereby determining the first storage directory as the target storage directory. For another example, the application in the first container group may directly create a new storage directory (that is, create the first storage directory) in the second storage area, thereby determining the storage directory as the target storage directory.

It should be noted that, by determining the storage directory in the matching relationship as the target storage directory when the matching relationship corresponding to the first container group is included, it is avoided to directly generate a new storage directory for use, thereby avoiding the unlimited increase in the number of directories, reducing fragmented files, upload and maintenance costs. By determining the first storage directory and determining the first storage directory as the target storage directory when the matching relationship corresponding to the first container group is not included, it is ensured that different first container groups will not write data to the same file, thereby improving the reliability of data processing.

In an optional embodiment, after the first storage directory is determined as the target storage directory, the application in the first container group may establish a matching relationship between the first container group and the first storage directory in the matching relationship set.

For example, the matching relationship set is stored in the first storage area, and the application in the first container group can establish a matching relationship between the first container group and the first storage directory in the matching relationship set, obtain an updated matching relationship set, and upload the updated matching relationship set to the first storage area to replace the old matching relationship set.

It should be noted that, through the above process, it is possible to share the update information of the matching relationship among multiple first container groups, thereby avoiding the phenomenon that a storage directory is used by multiple first container groups when the matching relationship set is not updated, thereby further improving the reliability of data processing.

In an optional embodiment, in the process of determining the first storage directory, the application in the first container group can obtain multiple preset directory numbers, and then for the Nth directory number among the multiple directory numbers, determine whether there is a second storage directory in the matching relationship set, so that if there is no second storage directory in the matching relationship set, the second storage directory is determined as the first storage directory, and if there is a second storage directory in the matching relationship set, N is updated, and the step of determining whether there is a new second storage directory in the matching relationship set is repeated until there is no new second storage directory in the matching relationship set, and the new second storage directory is determined as the first storage directory. The second storage directory is a storage directory using the Nth directory number.

Optionally, the preset multiple directory numbers may be stored in the first storage area, may be stored in the second storage area, or may be stored in an area other than the first storage area and the second storage area.

In some embodiments, different directory numbers are used as identifiers of different storage directories, that is, the application in the first container group can generate a storage directory corresponding to the directory number according to the directory number. When a directory number exists, it does not mean that there must be a storage directory corresponding to the directory number. For example, if the directory numbers are 01-10 and there are 10 directory numbers, then the number of storage directories may be less than or equal to 10.

Optionally, after obtaining the preset multiple directory numbers, the application in the first container group can start traversing from the Nth directory number among the multiple directory numbers, and for the Nth directory, determine whether there is a second storage directory in the matching relationship set. If not, it is determined that the second storage directory is not occupied by other first container groups and is an available storage directory, so the second storage directory can be determined as the first storage directory. Among them, the preset multiple directory numbers can be sorted according to the size of the directory numbers. For example, assuming that the directory numbers are 01-10, then 01 is the first directory number, 02 is the second directory number, and so on. Optionally, the preset multiple directory numbers can also be sorted in other ways.

If there is a second storage directory in the matching relationship set, it is determined that the second storage directory has been occupied by other first container groups. In this case, the application in the current first container group can update N. For example, assuming that the current N is 1, N can be added by one to obtain N=2. Then, the step of determining whether there is a new second storage directory in the matching relationship set is repeated until there is no new second storage directory in the matching relationship set, and the new second storage directory is determined as the first storage directory. The aforementioned new second storage directory refers to a storage directory using a new Nth directory number.

In some embodiments, each storage directory is set under the second storage area. In the process of determining the second storage directory as the first storage directory, the application in the first container group can determine whether the second storage directory exists in the second storage directory. If it exists, the second storage directory is directly determined as the first storage directory. If it does not exist, a second storage directory is created in the second storage area according to the current Nth directory number, and then the second storage directory is determined as the first storage directory.

It should be noted that by screening the storage directories not occupied by other first container groups according to the directory numbers, orderly screening of the storage directories is achieved, thereby improving the efficiency and accuracy of determining the first storage directories, and further improving the reliability of data processing.

In an optional embodiment, the distributed system further includes a second container group, where the second container group is a container group running a directory management application. The distributed system can determine the storage directories existing in the distributed system through the application in the second container group to obtain multiple third storage directories. For each third storage directory, the application in the second container group is used to determine whether the matching relationship set includes the third storage directory. If the third storage directory is included, the application in the second container group is used to determine whether the first container group matched by the third storage directory in the matching relationship set is in a running state. If the first container group matched by the third storage directory is not in a running state, the matching relationship corresponding to the third storage directory is deleted from the matching relationship set by the application in the second container group.

Optionally, the above-mentioned directory management application is used to manage the storage directory in this embodiment, and the management content includes but is not limited to deleting the storage directory, compressing the data in the storage directory, uploading the data in the storage directory to the target server, deleting the matching relationship corresponding to the storage directory in the matching relationship set, etc.

In some embodiments, each storage directory is set under the second storage area, and the second storage area can be a distributed disk. Figure 3 is a schematic diagram of the operation of an application in an optional second container group according to an embodiment of the present invention. As shown in Figure 3, the application in the second container group can traverse the disk (i.e., the second storage area) to determine the storage directories that already exist in the distributed system and obtain multiple third storage directories.

Optionally, after determining multiple third storage directories, as shown in FIG3, for each third storage directory, the application in the second container group can determine whether the matching relationship set includes the third storage directory, for example, determine whether the matching relationship set includes the directory identifier of the third storage directory, if the directory identifier is included, determine that the matching relationship set includes the third storage directory, if the directory identifier is not included, determine that the matching relationship set does not include the third storage directory. As shown in FIG3, before traversing to obtain multiple third storage directories, the application in the second container group can obtain the matching relationship set.

Optionally, when the matching relationship set does not include the third storage directory, the application in the second container group may delete the third storage directory or retain the third storage directory.

Optionally, in the case where the matching relationship set includes the third storage directory, the application in the second container group determines whether the first container group matched by the third storage directory in the matching relationship set is in a running state. For example, as shown in FIG3, before traversing to obtain multiple third storage directories, the application in the second container group can obtain the container group information of all first container groups in a running state, obtain a list of surviving containers, and then determine whether the first container group matched by the third storage directory in the matching relationship set is in a running state based on the surviving container list. For example, if the container group information of the first container group matched by the third storage directory exists in the surviving container list, it is determined that the first container group is in a running state. Conversely, if the container group information of the first container group matched by the third storage directory does not exist in the surviving container list, it is determined that the first container group is not in a running state.

In some embodiments, as shown in FIG3 , if the first container group that matches the third storage directory is in a running state, the third storage directory is not processed. If the first container group that matches the third storage directory is not in a running state, it means that the first container group has been run before and is now deleted. Therefore, as shown in FIG3 , the application in the second container group can delete the matching relationship corresponding to the third storage directory from the matching relationship set, so that other first container groups can use the third storage directory later.

It should be noted that, through the above process, the storage directory corresponding to the first container group that is no longer working is effectively released, which avoids the phenomenon that some storage directories are always occupied by the deleted first container group, resulting in the number of directories in the distributed system increasing over time, thereby avoiding the invalid occupation of distributed system resources.

In an optional embodiment, when the first container group matching the third storage directory is not in operation, the distributed system may upload the data under the third storage directory to the target server through the application in the second container group, delete the data under the third storage directory through the application in the second container group, and delete the third storage directory.

Optionally, as shown in FIG3 , when the first container group matching the third storage directory is not in operation, the application in the second container group can compress the target file under the third storage directory to obtain a compressed file, and then upload the compressed file to the target server. The target server can be the server where the cloud storage service is located, or other services.

Optionally, as shown in Figure 3, after uploading the compressed file to the target server, the application in the second container group may also delete the data in the third storage directory and delete the third storage directory, for example, by deleting the third storage directory from the second storage area.

It should be noted that, through the above process, the storage directory that is temporarily no longer needed in the distributed system is effectively cleaned up, and the files under the storage directory are effectively backed up, thereby further avoiding the ineffective occupation of distributed system resources.

In an optional embodiment, after determining whether the matching relationship set includes the third storage directory through the application in the second container group, the distributed system can obtain preset configuration information through the application in the second container group when the third storage directory is not included, and determine the processing method for the third storage directory through the application in the second container group according to the configuration information. The configuration information indicates whether the deletion of the third storage directory not recorded in the matching relationship set is allowed.

In some embodiments, configuration information may be preset in the distributed system, and the configuration information is used to indicate whether to allow deletion of the third storage directory not recorded in the matching relationship set.

In the case where the matching relationship set includes the third storage directory, the application in the second container group can obtain the configuration information, and thus determine the processing method for the third storage directory according to the configuration information. For example, as shown in FIG3 , if the configuration information indicates “delete the third storage directory not recorded in the matching relationship set”, the data under the third storage directory is uploaded to the target server, and the data under the third storage directory is deleted, and the third storage directory is deleted; on the contrary, if the configuration information indicates “do not delete the third storage directory not recorded in the matching relationship set”, the third storage directory is retained.

It should be noted that, through the above process, the flexibility of processing the third storage directory is improved, thereby improving the applicability of the present application.

In an optional embodiment, FIG4 is a schematic diagram of an optional data processing method according to an embodiment of the present invention. An optional working process of the distributed system in this embodiment is described according to FIG4. As shown in FIG4, the distributed system includes multiple first container groups (i.e., Pod1, Pod2, etc. in FIG4) and a second container group (i.e., the management Pod in FIG4). The applications in the first container groups such as Pod1 and Pod2 respectively obtain a matching relationship set from the first storage area. The matching relationships included in the matching relationship set are "01-Pod name1", "02-Pod name2", "03-Podname8", etc. shown in FIG4. Among them, "01-Pod name1" indicates that there is a matching relationship between storage directory No. 01 and Pod1, and other matching relationships are similar, so they are not repeated here.

Further, as shown in FIG4, the applications in Pod1 and Pod2 can respectively determine their matching storage directories according to the container group information and the matching relationship set to obtain the target storage directory. For example, in FIG4, the target storage directory corresponding to Pod1 is storage directory No. 01, and the target storage directory corresponding to Pod2 is storage directory No. 02. Afterwards, the applications in Pod1 and Pod2 can respectively store the processed data in the target files (i.e., "data.log" in FIG4) in the corresponding target storage directories in the second storage area.

Optionally, as shown in FIG4 , an application in the second container group (i.e., the management Pod in FIG4 ) in the distributed system can manage the matching relationship set, for example, deleting the matching relationship corresponding to the storage directory corresponding to the first container group that has stopped working or is abnormal from the matching relationship set. The second container group can also upload the data in the storage directory that needs to be deleted to the target server.

In an optional embodiment, Fig. 5 is a schematic diagram of the operation of an application in an optional first container group according to an embodiment of the present invention, and the process of determining the target storage directory by the application in the first container group is described according to Fig. 5. As shown in Fig. 5, for each first container group, the application in the first container group obtains the container group information of the current first container group, and then determines whether the matching relationship set includes the matching relationship corresponding to the first container group, and if it does, directly reuses the storage directory in the matching relationship, that is, determines the storage directory in the matching relationship as the target storage directory.

As shown in FIG4 , if the matching relationship set does not include the matching relationship corresponding to the first container group, the application in the first container group obtains multiple preset directory numbers and traverses the directory numbers in sequence. For the Nth directory number, it is determined whether there is a storage directory using the Nth directory number in the matching relationship set (that is, it is determined whether there is a second storage directory). If not, the storage directory is determined as the target storage directory, and a corresponding matching relationship is established in the matching relationship set. If there is a storage directory using the Nth directory number in the matching relationship set, the traversal is continued until there is no storage directory using the new Nth directory number in the matching relationship set, thereby determining the storage directory as the target storage directory.

It can be seen that the solution provided by the present application achieves the purpose of writing the data processed by applications in different container groups into different directories, thereby achieving the technical effect of improving the reliability of data processing, and further solves the technical problem that the applications in each container group in the related technology write the processed data into the same file, which easily leads to data loss or confusion, thereby resulting in poor data processing reliability.

Example 2

According to an embodiment of the present invention, an embodiment of a data processing device is provided, wherein FIG6 is a schematic diagram of an optional data processing device according to an embodiment of the present invention. As shown in FIG6 , the device is applied to a distributed system, and the distributed system includes a plurality of first container groups, and the first container group is a container group running a data processing application. The device includes:

A first acquisition module 601 is used to acquire a matching relationship set and container group information of the first container group through an application in the first container group, wherein the matching relationship set includes matching relationships between multiple first container groups and multiple storage directories, and different first container groups match different storage directories;

A first determination module 602 is used to determine the storage directory matched by the first container group according to the container group information and the matching relationship set through the application in the first container group, and obtain the target storage directory;

The storage module 603 is used to store the data processed by the application in the target storage directory through the application in the first container group.

It should be noted that the above-mentioned first acquisition module 601, first determination module 602 and storage module 603 correspond to steps S201 to S203 in the above-mentioned embodiment. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

Optionally, the first determination module 602 further includes: a judgment submodule, used to judge whether the matching relationship set includes the matching relationship corresponding to the first container group according to the container group information; a first determination submodule, used to determine the storage directory in the matching relationship corresponding to the first container group as the target storage directory if the matching relationship corresponding to the first container group is included; and a second determination submodule, used to determine the first storage directory if the matching relationship corresponding to the first container group is not included, and determine the first storage directory as the target storage directory, wherein the first storage directory refers to a storage directory that is not matched with the first container group.

Optionally, the data processing device further includes: a first processing module, configured to establish a matching relationship between the first container group and the first storage directory in the matching relationship set.

Optionally, the second determination submodule also includes: an acquisition unit, used to acquire a preset plurality of directory numbers; a judgment unit, used to judge whether there is a second storage directory in the matching relationship set for the Nth directory number among the plurality of directory numbers, wherein the second storage directory is a storage directory using the Nth directory number; a first determination unit, used to determine the second storage directory as the first storage directory if there is no second storage directory in the matching relationship set; and a second determination unit, used to update N if there is a second storage directory in the matching relationship set, and repeatedly perform the step of judging whether there is a new second storage directory in the matching relationship set, until there is no new second storage directory in the matching relationship set, and the new second storage directory is determined as the first storage directory.

Optionally, the data processing device further includes: a second determination module, used to determine, through the application in the second container group, a storage directory that already exists in the distributed system to obtain multiple third storage directories; a first judgment module, used to determine, for each third storage directory, through the application in the second container group, whether the third storage directory is included in the matching relationship set; a second judgment module, used to determine, through the application in the second container group, whether the first container group matched by the third storage directory in the matching relationship set is in a running state when the third storage directory is included; and a second processing module, used to delete, through the application in the second container group, the matching relationship corresponding to the third storage directory from the matching relationship set when the first container group matched by the third storage directory is not in a running state.

Optionally, the data processing device also includes: a third processing module, used to upload the data under the third storage directory to the target server through the application in the second container group; a fourth processing module, used to delete the data under the third storage directory through the application in the second container group, and delete the third storage directory.

Optionally, the data processing device also includes: a second acquisition module, used to obtain preset configuration information through the application in the second container group without including the third storage directory, wherein the configuration information indicates whether deletion of the third storage directory not recorded in the matching relationship set is allowed; and a third determination module, used to determine a processing method for the third storage directory according to the configuration information through the application in the second container group.

Example 3

According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned data processing method when running.

Example 4

According to another aspect of an embodiment of the present invention, an electronic device is further provided, wherein Figure 7 is a schematic diagram of an optional electronic device according to an embodiment of the present invention. As shown in Figure 7, the electronic device includes one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the above-mentioned data processing method when running.

The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims (10)

1. A data processing method, applied to a distributed system, the distributed system including a plurality of first container groups, the first container groups being container groups running data processing applications, the method comprising:

Acquiring a matching relation set and container group information of the first container group through an application in the first container group, wherein the matching relation set comprises matching relations between a plurality of first container groups and a plurality of storage catalogs, and different first container groups are matched with different storage catalogs;

determining a storage catalog matched with the first container group according to the container group information and the matching relation set by using the application in the first container group to obtain a target storage catalog;

and storing the data processed by the application in the first container group to the target storage catalog.

2. The method of claim 1, wherein determining, by the application in the first container group, a storage directory to which the first container group matches according to the container group information and the set of matching relationships, to obtain a target storage directory, comprises:

judging whether the matching relation set comprises the matching relation corresponding to the first container group according to the container group information;

If the matching relation corresponding to the first container group is included, determining a storage directory in the matching relation corresponding to the first container group as the target storage directory;

And if the matching relation corresponding to the first container group is not included, determining a first storage directory, and determining the first storage directory as the target storage directory, wherein the first storage directory refers to a storage directory which is not matched with the first container group.

3. The method of claim 2, wherein after determining the first storage directory as the target storage directory, the method further comprises:

And establishing a matching relation between the first container group and the first storage catalog in the matching relation set.

4. The method of claim 2, wherein determining the first storage directory comprises:

Acquiring a plurality of preset directory numbers;

Judging whether a second storage directory exists in the matching relation set for an Nth directory number in the plurality of directory numbers, wherein the second storage directory is a storage directory using the Nth directory number;

If the second storage catalogue does not exist in the matching relation set, determining the second storage catalogue as the first storage catalogue;

If the second storage catalogue exists in the matching relation set, updating the N, and repeatedly executing the step of judging whether a new second storage catalogue exists in the matching relation set or not until the new second storage catalogue does not exist in the matching relation set, and determining the new second storage catalogue as the first storage catalogue.

5. The method of claim 1, further comprising a second set of containers in the distributed system, the second set of containers being a set of containers running a catalog management application, the method further comprising:

Determining the existing storage catalogs in the distributed system through the application in the second container group to obtain a plurality of third storage catalogs;

Judging whether the matching relation set comprises the third storage catalogue or not through the application in the second container group for each third storage catalogue;

judging whether a first container group matched with the third storage catalog in the matching relation set is in an operation state or not through an application in the second container group under the condition that the third storage catalog is included;

and deleting the matching relation corresponding to the third storage catalog from the matching relation set through the application in the second container group under the condition that the first container group matched with the third storage catalog is not in an operation state.

6. The method of claim 5, wherein in the event that the third storage directory matched first container group is not in an operational state, the method further comprises:

uploading the data under the third storage directory to a target server through the application in the second container group;

Deleting the data under the third storage directory by the application in the second container group, and deleting the third storage directory.

7. The method of claim 5, wherein after determining, by the application in the second container group, whether the third storage directory is included in the set of matching relationships, the method further comprises:

Acquiring preset configuration information through an application in the second container group under the condition that the third storage catalogue is not included, wherein the configuration information represents whether deletion of the third storage catalogue which is not recorded in the matching relation set is allowed or not;

And determining the processing mode of the third storage catalogue according to the configuration information through the application in the second container group.

8. A data processing apparatus for use in a distributed system comprising a plurality of first container groups, the first container groups being container groups having data processing applications running thereon, the apparatus comprising:

the first acquisition module is used for acquiring a matching relation set and container group information of the first container group through an application in the first container group, wherein the matching relation set comprises matching relations between a plurality of first container groups and a plurality of storage catalogs, and different first container groups are matched with different storage catalogs;

the first determining module is used for determining a storage catalog matched with the first container group according to the container group information and the matching relation set through an application in the first container group to obtain a target storage catalog;

And the storage module is used for storing the data processed by the application in the first container group to the target storage catalog.

9. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program, wherein the computer program is arranged to execute the data processing method of any of the claims 1 to 7 when run.

10. An electronic device, the electronic device comprising one or more processors; a memory for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement a method for running a program, wherein the program is configured to perform the data processing method of any of claims 1 to 7 when run.