CN118118677A - Data and video processing method and device - Google Patents

Abstract

A data and video processing method and device. The data processing method is applied to a data processing system, the data processing system comprises a management component and a processing component, and the method comprises the following steps: determining a plurality of target processing components according to a task to be processed, wherein the task to be processed is used for processing target data; the management component loads the plurality of target processing components; and the loaded multiple target processing components process the data fragments of the target data in sequence, wherein each target processing component processes different data fragments at the same time, and at least one data fragment is processed by each target processing component in sequence. The method can simplify the management logic of the processing assembly and the processing process of the target video, is beneficial to improving the processing efficiency of the target data, and better meets the diversified processing requirements of the target data.

Description

Data and video processing method and device

Technical Field

The present invention relates to the field of electronic information technology, and in particular to a data and video processing method and device.

Background Art

When multiple processing methods are required to process data, at present, functional modules corresponding to various processing methods are usually used for processing, or general processing software is used for processing.

Taking the video to be processed as an example, when decoding, inserting frames, etc., it is usually necessary to develop discrete functional modules corresponding to each processing method for the video to be processed, and then use each functional module to process the video in turn. When there are many types or numbers of functional modules, this method often requires manual adjustment of the parameters of the functional modules or the intermediate data of the processed video. The overall processing logic and processing process are relatively complex, resulting in low processing efficiency. General video processing software usually only provides simple standardized functions, which often cannot meet diverse processing needs.

Summary of the invention

In view of this, an embodiment of the present invention proposes a data and video processing method and device to solve the deficiencies existing in the related art.

According to a first aspect of an embodiment of the present invention, a data processing method is provided, which is applied to a data processing system, wherein the data processing system includes a management component and a processing component, and the method includes:

Determine a plurality of target processing components according to the tasks to be processed, wherein the tasks to be processed are used to process target data;

The management component loads the multiple target processing components;

The multiple target processing components that have completed loading process the data segments of the target data in sequence, wherein each target processing component processes different data segments at the same time, and at least one data segment is processed in sequence by each target processing component.

According to a second aspect of an embodiment of the present invention, a video processing method is provided, which is applied to a video processing system. The video processing system includes a management component and a processing component. The method includes:

Determine a target processing component according to a task to be processed, wherein the task to be processed is used to process a target video, and the target processing component includes a decoding component, an encoding component, and at least one intelligent algorithm component;

The management component loads the target processing component;

The loaded intelligent algorithm component processes the video frames of the target video in sequence, wherein each target processing component processes different video frames at the same time, and at least one video frame is processed in sequence by each target processing component.

According to a third aspect of an embodiment of the present invention, a data processing device is provided. The device is applied to a data processing system. The data processing system includes a management component and a processing component. The device includes one or more processors. The processors are configured to:

Determine a plurality of target processing components according to the tasks to be processed, wherein the tasks to be processed are used to process target data;

The management component loads the multiple target processing components;

The multiple target processing components that have completed loading process the data segments of the target data in sequence, wherein each target processing component processes different data segments at the same time, and at least one data segment is processed in sequence by each target processing component.

According to a fourth aspect of an embodiment of the present invention, a video processing device is provided. The device is applied to a video processing system. The video processing system includes a management component and a processing component. The device includes one or more processors. The processors are configured to:

Determine a target processing component according to a task to be processed, wherein the task to be processed is used to process a target video, and the target processing component includes a decoding component, an encoding component, and at least one intelligent algorithm component;

The management component loads the target processing component;

The loaded intelligent algorithm component processes the video frames of the target video in sequence, wherein each target processing component processes different video frames at the same time, and at least one video frame is processed in sequence by each target processing component.

According to a fifth aspect of an embodiment of the present invention, an electronic device is proposed, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the data processing method described in any one of the first aspect or the second aspect.

According to a sixth aspect of an embodiment of the present invention, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps in the data processing method described in any one of the first aspect or the second aspect are implemented.

According to an embodiment of the present invention, a data processing system including a management component and a processing component is proposed. For tasks to be processed for processing target data, the system first determines multiple target processing components, and then the management component loads each target processing component; then, the multiple target processing components that have been loaded can process data fragments of the target data in sequence, wherein each target processing component processes different data fragments at the same time, and at least one data fragment is processed in sequence by each target processing component.

After the multiple processing components are loaded by the management component, each processing component can process the data fragments of the target data in order, thereby completing the task to be processed. On the one hand, the solution can be managed and scheduled by the management component in a unified manner, so that each target processing component can cooperate with each other to complete the processing process for the target data in an orderly manner. Even if the number of target processing components is large, each target processing component can process the data fragments in an orderly manner, thereby greatly simplifying the management logic for each processing component in the system and the processing process of the target data, which helps to improve the execution efficiency of the task to be processed. On the other hand, each target processing component can be determined and loaded according to the task to be processed, so that each target processing component can process the data fragments of the target data according to its own processing logic. Since the processing logic of each target processing component can be pre-defined according to actual needs, this method can implement a more complex task processing logic, so as to provide more processing functions, thereby facilitating the realization of diversified data processing needs.

In addition, the management component can determine the corresponding processing component according to the target data corresponding to the task to be processed, and then load each determined processing component into the data processing system. It is understandable that before the above-mentioned data processing system is started, the user can specify the corresponding processing component to the management component according to the target data, so as to realize the on-demand loading of the processing component.

According to an embodiment of the present invention, a video processing system including a management component and a processing component is also proposed. For a task to be processed, the system determines a target processing component according to the task to be processed, wherein the task to be processed is used to process a target video, and the target processing component includes a decoding component, an encoding component and at least one intelligent algorithm component; then the management component loads the target processing component; the loaded intelligent algorithm component processes video frames of the target video in sequence, wherein each target processing component processes different video frames at the same time, and at least one video frame is processed in sequence by each target processing component.

Similar to the aforementioned data processing system for data processing, on the one hand, the solution can be managed and scheduled by the management component in a unified manner, so that each target processing component (including a decoding component, an encoding component and at least one intelligent algorithm component) cooperates with each other to orderly complete the processing process for the target video. Even if the number of intelligent algorithm components is large, each intelligent algorithm component can process video frames in an orderly manner, thereby greatly simplifying the management logic for each intelligent algorithm component in the system and the processing process of the target data, which helps to improve the execution efficiency of the tasks to be processed. On the other hand, at least one intelligent algorithm component can be determined and loaded according to the tasks to be processed, so that each intelligent algorithm component can process the video frames of the target video according to its own processing logic. Since the processing logic of each intelligent algorithm component can be pre-defined according to actual needs, this method can realize a more complex video processing logic, so as to provide more processing functions, thereby facilitating the realization of diversified video processing needs. In addition, on-demand loading of intelligent algorithm components can be achieved.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG. 1 is a flow chart showing a data processing method according to an embodiment of the present invention.

FIG. 2 is a schematic diagram showing the architecture of a data processing system according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of a workflow of a management component according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a workflow of a processing component according to an embodiment of the present invention.

FIG. 5 is a schematic diagram showing a data transmission process between processing components according to an embodiment of the present invention.

FIG. 6 is a schematic diagram showing an algorithm service interface according to an embodiment of the present invention.

FIG. 7 is a flow chart showing a video processing method according to an embodiment of the present invention.

FIG. 8 is a schematic diagram showing the architecture of a video processing system according to an embodiment of the present invention.

FIG. 9 is a schematic diagram showing a video processing process according to an embodiment of the present invention.

FIG. 10 is a schematic diagram of a video processing system according to an embodiment of the present invention.

FIG. 11 is a schematic block diagram of a data processing device or a video processing device according to an embodiment of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

In view of the technical problems existing in the aforementioned related technologies, an embodiment of the present invention proposes a data processing method, which is described in detail below in conjunction with the accompanying drawings and corresponding embodiments.

The execution subject of the data processing method described in the embodiment of the present invention may be a data processing device, such as a server of an application or a terminal device where a client is located. The server may be a physical server including an independent host, or the server may be a virtual server carried by a host cluster, a cloud server, etc.; the terminal device may be a mobile phone, a tablet device, a laptop computer, a PDA (Personal Digital Assistants), a wearable device (such as smart glasses, smart watches, etc.), a VR (Virtual Reality) device, an AR (Augmented Reality) device, etc., and one or more embodiments of the present invention are not limited to this.

Fig. 1 is a flow chart of a data processing method according to an embodiment of the present invention. As shown in Fig. 1, the method is applied to a data processing system, the data processing system includes a management component and a processing component, and the method may include the following steps 102-106.

Step 102, determining a plurality of target processing components according to the tasks to be processed, wherein the tasks to be processed are used to process target data.

Step S104: the management component loads the multiple target processing components.

The management component described in the embodiment of the present invention can be used to manage various processing components in the data processing system, and the corresponding processing logic is predefined in the processing component. For the management component and the processing component, from the software level, any component can be implemented using any form of programming language, for example, it can be implemented using object-oriented programming languages such as C++. From the hardware level, the above-mentioned management component and processing component can both run in the CPU (Central Processing Unit) of the data processing device; or, the above-mentioned management component can run in the CPU of the data processing device, and the processing component can run in the GPU (Graphics Processing Unit) of the data processing device, which will not be repeated.

The pending tasks received by the data processing system are used to process target data, that is, the pending tasks can be used to indicate to the data processing system which data are the target data to be processed; accordingly, the process of the data processing system executing the pending tasks is the process of processing the data segments of the target data. Among them, the pending tasks can implement the above instructions in a variety of ways. For example, the pending tasks can include the target data, so that the first target processing component can obtain the target data to be processed when the pending tasks are obtained; for another example, the pending tasks can also include the storage address of the target data, so that the first target processing component can read the target data from the storage address after obtaining the pending tasks. For another example, the pending tasks can also include service information of other services, so that the first target processing component can use the processing results of the other services as the target data, thereby realizing task transfer between other tasks and the data processing system. The specific process will not be repeated.

The target data corresponding to the task to be processed in this solution can be in any form, and the corresponding data segments are different according to the different forms of the target data. For example, when the target data is a video, the data segment can be a video frame image or a video segment obtained by clipping the video; when the target data is audio, the data segment can also be an audio component extracted from the audio or an audio segment obtained by segmentation; when the target data is an image, the data segment can also be a color value of the image in the color space or a partial area cropped from the image, etc., which will not be repeated here.

In addition, it should be noted that although the algorithm task and the task to be processed are both called "tasks", their meanings are not the same. The algorithm task is a functional component (i.e., the processing component) used by the data processing system to implement a preset processing logic, while the task to be processed is a task that can be executed by the data processing system. Specifically, the management component can first determine and load the corresponding target processing component, and each loaded target processing component can process the data fragments of the target data according to its own processing logic to execute the processing component.

In the case of receiving the task to be processed, the management component may first determine multiple target processing components according to the task. Among them, the management component may determine multiple target processing components according to the demand information of the task to be processed, wherein the demand information is used to indicate the processing requirements of the target data, and the information may include at least one of the following: the meta information of the target data, the configuration information of the initiator of the task to be processed, and the default function information provided by the data processing system. For example, the management component may pre-acquire the mapping relationship between the meta information of the data and the processing component. The processing component may include at least one of the following: an encoding component, a decoding component, an interpolation component, a noise reduction component, a super-resolution component, a high dynamic range component, a detail repair component, a grayscale component, etc. The above mapping relationship can be pre-set or pre-acquired by the data processing system, and then after determining the meta information of the target data, the management component can determine the multiple processing components corresponding to the meta information according to the mapping relationship, and use it as the multiple target processing components corresponding to the task to be processed.

The meta information may be in enumerated form. For example, the enumerated members may refer to the following Table 1.

Table 1

For another example, the initiator of the task to be processed (such as the user of the data processing device) can set configuration information for the task (i.e., the initiator configuration information), so that the management component can determine the multiple target processing components based on the configuration information. For example, the initiator can specify the target processing component through the configuration information, so that the user can accurately specify the target processing component required to execute the task to be processed to the management component. Alternatively, the initiator can also specify the processing effect of the target data through the configuration information, so that the management component can determine the multiple processing components corresponding to the processing effect as the multiple target processing components corresponding to the target data. At this time, the user does not need to know what kind of processing component the data processing system needs to execute the task to be processed, but only needs to refer to the functional effect achieved by the target data after the task to be processed is completed, thereby realizing the functional encapsulation of each processing component, making the data processing system black-box for its users, which helps to simplify the user usage process and improve the execution efficiency of the task to be processed.

Taking the user of the data processing device as an example, the data processing device can provide the user with a graphical operation interface or programming interface for the data processing system, so that the user can implement operations in the graphical operation interface or edit instruction codes in the programming interface to interact with the data processing system. Taking the graphical operation interface as an example, the user can specify the target data in the interface and generate the corresponding tasks to be processed, so that the data processing system can know the tasks to be processed and the target data. Further, the user can initiate a loading instruction for the tasks to be processed, and the component identification contained in the instruction is used to indicate multiple target processing components to the data processing system. If the user can select multiple processing components from the processing components contained in the data processing system displayed in the aforementioned graphical interface, the loading instruction includes the component identification of each selected processing component. Accordingly, the management component can respond to the loading instruction and determine the processing components represented by each component identification contained in the instruction (i.e., the aforementioned selected processing components) as the multiple target processing components. In addition, the above trigger instruction can also include the loading order of the multiple target processing components, so that the management component obtains the loading order and loads each target processing component in sequence according to the order. For example, the user can edit the loading order in the aforementioned graphical operation interface or determine the loading order according to the selection order of the component identifiers, etc., which will not be repeated here. Alternatively, the user can also edit the program code for loading the multiple target processing components in the aforementioned programming interface, and the program code can be used to specify multiple target processing components. Of course, the above code can also be used to specify the loading/execution order of the multiple target processing components (such as the order in which the tasks of each target processing component are added can be used as the loading order), so that the management component can load the multiple target processing components to the data processing system in sequence according to the loading order during the execution of the above program code, so as to realize the orderly loading/execution of each processing component. In this way, the user can specify multiple target processing components for processing target data to the data processing system.

For another example, the data processing system may also be pre-set with default function information, wherein any default function information may correspond to at least one processing component, based on which, when the initiator of the task to be processed does not specify the target processing component, the management component may directly determine the corresponding processing component as the target processing component according to the default function information. The default function information may be pre-generated by the developer of the data processing system, or may be pre-set by the user of the data processing device before initiating the task to be processed, and the embodiment of the present invention is not limited to this.

In addition, a mapping relationship between data information and candidate processing components can be created in advance. Take a single mapping relationship between a data information and a candidate processing component as an example: video can correspond to a decoding component and an encoding component, 1280*720 resolution can correspond to an interpolation component, 4K resolution can correspond to a frame extraction component, etc. Of course, a one-to-many, many-to-one or many-to-many mapping relationship can also be established between the above data information and the candidate processing components, which will not be repeated. Based on this, the management component can query the mapping relationship after determining the video information of the target data to determine the at least one processing component. In this way, the management component can accurately and quickly query at least one processing component that matches the data information of the target data. Alternatively, the management component can also count the matching probabilities of different data information and each candidate processing component based on the processing records of historical data, so that the candidate processing component with the highest matching probability with the data information of the target data can be determined as the at least one processing component. In this way, the management component can make the determined at least one processing component as consistent with the user's usage habits as much as possible. Of course, the loading order of the at least one processing component can also be determined at the same time in the above process, which will not be repeated.

In one embodiment, the data processing system may include an algorithm service and an algorithm task. The algorithm service is a process management of the algorithm task, which is used to perform high-concurrency scheduling of all algorithm tasks to maximize resource utilization and achieve the most efficient processing efficiency. The algorithm task is a collection of all tasks that perform algorithmic processing on the target data corresponding to the task to be processed. The algorithm service may include the management component, the source component and the synchronization component, and the source component is used to generate the processing component. The algorithm task may include the processing component, that is, each processing component can be regarded as an algorithm task for implementing a preset processing logic in the data processing system. The specific generation and operation of the algorithm service and algorithm task can be seen in the description of the following embodiments, which will not be repeated here.

In one embodiment, the data processing system can be implemented by programming in an object-oriented programming language, such as C++, C#, java, python, etc. Using this type of programming language, the aforementioned algorithm services and algorithm tasks can be implemented through classes. For example, the algorithm service includes three parts: a process management class (corresponding to the management component), a task creation class (corresponding to the source component), and a data synchronization class (corresponding to the synchronization component). Among them, the task creation class is used to create algorithm tasks, such as creating a task set containing multiple algorithm functions; the data synchronization class includes configuring the input and output nodes of the algorithm task, which is used to synchronize data fragments between target processing components to ensure the transmission and synchronization of data fragments between different algorithm tasks; the process management class is used to perform task scheduling (such as adding algorithm tasks and/or deleting algorithm tasks), resource management (such as resource allocation and release for algorithm tasks), and status synchronization for algorithm tasks.

Monitoring (supervising the running status of algorithm tasks) and management, etc. Through the management of algorithm tasks by algorithm services, each data fragment of the target data can be processed by each algorithm task in turn. Each algorithm task only focuses on the processing steps corresponding to its own processing logic and judges its own resource occupancy. As long as the current resources are idle, it can accept the data of the adjacent previous computing task and activate the task for data processing. In other words, each algorithm task can be processed in parallel. Through the above three categories, the process management of algorithm tasks can be realized, so that the data processing system can be realized as a "pipeline" that processes at least part of the data fragments in the target data in a certain order.

In one embodiment, the data processing system may further include a source component, wherein the source component may include a predefined rule.

Then, the source component can generate the target processing component based on the predefined rules. The predefined rules may include predefined functions and/or rules. In this way, it can be ensured that the generated target processing component complies with the aforementioned predefined rules. In one embodiment, the target processing component is generated by the source component, and thus includes the predefined rules in the source component, and further executes the predefined rules to generate the target processing component that finally performs data processing.

5 Wherein, the task creation class may be a base class (or basic class), in which case the target processing component may be

The base class is derived, that is, the target processing component is a derived class corresponding to the base class. In this way, the target processing component as a derived class usually conforms to the data format, function, member variables, etc. predefined in the base class. For example, the member variables of the task creation class can contain the base class information of the target processing component, such as the configuration information of the target processing unit.

Information, forward nodes, backward nodes, data transfer mechanism and specific methods of data processing, etc. The derived class executes the pre-defined rules (such as member functions) to generate the target processing component that finally processes the data.

It should be noted that the generation process of the target processing component described above can be applied to each processing component in the data processing system, that is, any processing component in the processing component can be generated in the above manner. Of course, any processing component can also be generated by other devices and then provided to the data processing system. In addition, each processing component included in the data processing system

Components can be developed based on the system. For example, component parameters such as data format and data transmission mode of the processing component can meet the operation requirements of the data processing system, thereby ensuring that each processing component can be successfully loaded by the management component.

After determining multiple target processing components according to the tasks to be processed, the management component can load these processing components. Continuing from the above embodiment, the management component can add the multiple target processing components to a component list and start each target processing component in the component list. The management component can maintain a component list.

The list can be used to record the components of each target processing component corresponding to the pending task. Considering that the data processing system may process multiple pending tasks at the same time, the management component can simultaneously maintain component lists corresponding to each pending task, wherein the component list corresponding to any pending task can be used to record the target processing components determined according to the task. Alternatively, the management component can also maintain only one component list, which is used to record the target processing components corresponding to each pending task. In this case, the aforementioned "target processing components in the component list" that are activated should be understood as multiple pending tasks determined according to the pending tasks, and may not be all target processing components corresponding to all pending tasks at the current moment.

Wherein, the management component adds at least one of the multiple target processing components to the component list, which may include: the management component identifies whether the at least one target processing component is generated by the source component, and if so, the at least one target processing component may be added to the component list; if not, the at least one target processing component may be refused to be added to the component list. If the at least one target processing component is generated by the source component, these target processing components will follow the aforementioned preset rules of the source component, so the component after loading will be able to comply with the preset rules, so when the aforementioned judgment is passed, the target processing component is added and it can be ensured that each target processing component added to the component list complies with the preset rules, thereby ensuring that each loaded target processing component can be started normally.

In addition, any target processing component is recorded in the component list, and specifically, the component information of the component can be recorded in the list; based on this, the management component can obtain the component information of each target processing component in the component list, and start each target processing component based on the component information. The component information is used to indicate the identity of the target processing component (i.e., to indicate which processing component it is). For any recorded target processing component, the component information of the component can include member variables corresponding to the component, such as the component name, component identifier, forward node, backward node, etc. of the component.

The management component can obtain the member variables of any target processing component in a variety of ways. For example, the management component can load any target processing component into the corresponding thread, and can use the algorithm task thread function to provide a function pointer for the algorithm task thread creation, so that each algorithm task is processed in a separate thread, and the startup task function will call the pointer to create a thread. In the process of creating the thread based on the algorithm task thread function, the management component first needs to access the member function of any target processing component to obtain the member variables. In order to ensure that the management component can smoothly access the member function of any target processing component, the algorithm task thread function can be defined as a friend function in advance. At this time, the management component can access the friend function to obtain the member variables of any target processing component. Alternatively, in the case where the management component is a base class and any target processing component is a derived class corresponding to the base class (i.e., derived based on the base class), any target processing component can also be defined as a member variable of the management component, and the management component can directly access the member variables of the processing component.

Among them, in order to avoid adverse effects such as resource competition that may arise between different target processing components, each target processing component can be run in a different thread. It can be understood that each thread can belong to the same process. At this time, each thread is used to execute the subtask corresponding to each target processing component in the task to be processed, and the process can be regarded as the executor of the aforementioned task to be processed. Alternatively, in order to facilitate the unified management and scheduling of each thread, the aforementioned multiple target processing components can also run in the same thread, and the embodiment of the present invention does not limit this.

The data processing device can load the target processing component and execute the task to be processed by calling and executing the executable file. Taking the target data as a video in .mp4 format as an example, the data processing device can execute the following code:

./sr_sdk-i test_video/1080_25.mp4-m frc_1080:denoise_1080:sr1x_1080-c0-p 0-out.mp4

Among them, sr_sdk is the executable file to be executed. The string after -i (test_video/1080_25.mp4) is the file name of the target video to be processed. The string after -m (frc_1080:denoise_1080:sr1x_1080) represents the target processing component used to process the target video, where frc_1080 is the interpolation component, denoise_1080 is the small model noise reduction component, and sr1x_1080 is the single-frame super-resolution component. Of course, if other processing is required for the target video, the corresponding target processing component can be added here. Exemplarily, the optional target processing components can be shown in Table 2 below.

Table 2

The number after -c is used to indicate the video encoding mode of the target video, where 0 can be used to indicate h264 encoding. For example, the mapping relationship between the encoding mode and the digital identifier can be seen in Table 3 below.

Table 3

Digital ID Video encoding method 0 h264 1 h265 2 mpeg-1 3 mpeg-2 4 mpeg-4 5 wmv7 6 wmv8

The number after -P is used to indicate the pixel format when encoding the target video, such as 0 for yuv420 format, 1 for yuv422 format, etc., which will not be repeated here. The string after -o is used to indicate the file storage path and file name of the target video, such as the aforementioned out.mp4 indicates that the file name of the target video is "out.mp4". Of course, in addition to .mp4, the file format of the target video can also be .avi, .wmv, etc., which can be referred to the aforementioned records, which will not be repeated here.

In addition, the HDR standard information may be in enumeration form, and corresponding enumeration members may be seen in Table 4.

Table 4

As mentioned above, the management component can perform task scheduling, such as adding algorithm tasks and/or deleting algorithm tasks. Specifically, after loading the multiple target processing components, the management component can respond to the loading instruction for any other processing component and continue to load any other processing component. It can be understood that the other processing component is a processing component other than the multiple target processing components that are loaded. And/or the management component can also respond to the unloading instruction for any loaded target processing component to unload the any target processing component, that is, to restore any target processing component that has been loaded from a loaded state to an unloaded state (such as the component information of any target processing component described in the aforementioned component list, etc.). It should be noted that loading any other processing component or unloading any target processing component (already loaded) can be performed before executing the task to be processed, during the execution of the task to be processed, or after the execution of the task to be processed. This solution does not restrict the actual loading or unloading. In this way, the initiator of the task to be processed (such as the user of the data processing device) can flexibly control the management component to load the required target processing component according to the actual situation of the task to be processed, so as to facilitate the use of flexible and diverse processing methods to process the target data.

Step 106 , the multiple target processing components that have completed loading process the data segments of the target data in sequence, wherein each target processing component processes different data segments at the same time, and at least one data segment is processed in sequence by each target processing component.

It can be understood that, through the loading process of the aforementioned embodiment, each loaded target processing component can constitute at least one processing link (hereinafter referred to as a pipeline), wherein each target processing component in any processing link can respectively process different data fragments at the same time, and at least one data fragment corresponding to any target data can be processed in sequence by each target processing component in the processing link corresponding to the data. For example, in the case where the task to be processed includes an audio and video file, the target data may include a video file and an audio file obtained by decoding the file. At this time, the video file and the audio file can be processed by different processing links respectively, wherein any processing link can be regarded as a pipeline for processing the corresponding target data. In the embodiments below, unless otherwise specified, the multiple target processing components belong to any pipeline.

As mentioned above, the management component can perform resource management (such as resource allocation and release for algorithm tasks). Specifically, the management component can allocate corresponding processing resources to each target processing component. For any target processing component, it can use the processing resources allocated by the management component to itself to process the data fragments of the target data. Among them, the processing resources can include any form such as network resources, storage resources and computing resources, and the embodiment of the present invention does not limit this. And, after any target processing component completes processing the last data fragment of the target data, the management component can release the processing resources allocated to any target processing component, so that the data processing system can timely recycle the processing resources allocated to the component after any target processing component completes processing the pending task. Among them, since there are multiple target processing components processing the data fragments of the target data, the management component can immediately recycle the processing resources allocated to the component after any target processing component completes processing the data fragment; or, it can also be unified after all target processing components complete processing the data fragments (that is, the data processing system completes executing the pending task) and then recycle the processing resources allocated to all target processing components, which will not be repeated.

In addition, as mentioned above, each target processing component can run in a corresponding thread respectively. It can be understood that a process is the smallest unit of resource allocation by the operating system, and a thread is the smallest unit of task mobilization and execution. Based on this, the process of resource allocation and resource release by the aforementioned management component for any target processing component can be performed in units of threads where the component is located, that is, the management component can allocate resources for the thread where any target processing component is located, and release the resources of the thread when the thread is closed (when any target processing component completes processing the data fragment of the target data).

As mentioned above, the management component can also perform status monitoring (supervising the running status of algorithm tasks) and management, etc. In the process of the multiple target processing components sequentially processing the data fragments of the target data, the processing process may be abnormal due to certain reasons, such as data processing timeout, data transmission failure, etc. In this regard, the management component can output an abnormal reminder message when an abnormality occurs, so that the user of the data processing device can promptly know the above abnormal situation through the message. Alternatively, the management component can also record the relevant information of the abnormality when an abnormality occurs, so that the user can analyze the cause of the abnormality through the information and eliminate the abnormality. Alternatively, in order to avoid a certain abnormality that may have a wider range of adverse effects on the operation of the data processing system and reduce invalid processing after the abnormality occurs, the management component can also trigger the stop of the processing process for the target data when an abnormality occurs, thereby temporarily pausing or terminating the execution process of the task to be processed.

In one embodiment, the data processing system may further include a forward node and a backward node, the target processing component corresponds to one or more forward nodes, and the target processing component corresponds to one or more backward nodes; wherein the forward node is used to obtain a forward data segment for the target processing component to read, and the backward node is used to obtain a backward data segment processed by the target processing component. Wherein, the forward node and the backward node can be obtained by instantiating the aforementioned data synchronization class. In addition, the forward node and the backward node corresponding to any target processing component can be configured in a variety of ways. For example, after generating each node, when any target processing component is derived from the task creation class, the corresponding forward node and backward node can be configured for the target processing component based on the member variables of the task creation class. For another example, after generating any target processing component and each node, the management component can configure the corresponding forward node and backward node for the target processing component. After the above settings are completed, the component information of any target processing component includes the node identity information of its forward node and backward node. This information can be recorded in the component list maintained by the management component for query during the process of processing the data fragment of the target data.

As mentioned above, the multiple target processing components are used to process the data segments of the target data in sequence, so the multiple target processing components have a corresponding processing sequence. The first target processing component in the processing sequence may not have a forward node, or the forward node of the target processing component may be set to an empty node; similarly, the last target processing component in the data sequence may not have a backward node, or the backward node of the target processing component may be set to an empty node. In addition, for any two adjacent target processing components in the processing sequence, the backward node of the previous target processing component and the forward node of the next target processing component may be the same node.

Based on the forward node and backward node configured for the target processing component, data segments can be transferred between adjacent target processing components. For example, any target processing component can obtain the forward data segment when the forward node is in a readable state. Wherein, the forward data segment can be the data segment itself obtained by the previous target processing component of any target processing component, or it can be the storage address of the data segment, which is not limited here. The forward data segment can be transferred to the forward node by the previous target processing component (such as writing to the forward node), so that any target processing component can read the forward data segment from its forward node for data processing operations. When the forward data segment is the data segment itself, the target processing component can directly perform data processing. When the forward data segment is a storage address, the target processing component can obtain the data segment pointed to by the storage address and perform data processing. After any target processing component obtains the forward data segment from the forward node, the forward node can be updated from a readable state to a writable state. After that, any target processing component can perform data processing operations to obtain corresponding backward data segments, and specifically, can process the data segments according to its own preset processing logic. Furthermore, when the backward node is in a writable state, the backward node can obtain the backward data segment obtained by processing any target processing component; and, after obtaining the backward data segment, the backward node can be updated from a writable state to a readable state. Among them, the backward data segment can be the data segment itself obtained after the current target processing component performs data processing, or it can be the data storage address after processing, which is not limited here. It can be understood that after the backward node is updated to a readable state, if the backward node of any target processing component and the forward node of the latter target processing component are the same node, then the latter target processing component can obtain the backward data segment from the node when the node is in a readable state; or, if the backward node of any target processing component and the forward node of the latter target processing component are not the same node, the backward node of any target processing component and the forward node of the latter target processing component can be connected in sequence. In this regard, when the backward node of any target processing component is in a readable state, the forward node of the latter target processing component can obtain the backward processing data from the backward node and record it locally, so that the latter target processing component obtains the backward processing data from the forward node. It can be understood that the backward data segment is the forward data segment of the latter target processing component.

In addition, the forward data segment or the backward data segment can be stored in the format of integer, floating point (single precision floating point type float or double precision floating point type double), character (char), etc. The storage address can be the storage address of the data segment in the preset storage space, such as the storage address in the memory, etc. Specifically, the precision conversion can be achieved by calling a data type conversion function. The parameter list of the function can be seen in Table 5 below.

Table 5

parameter Input/Output Type Parameter Description Integer memory data Input/Output Parameters 8-bit integer data Floating point memory data Input/Output Parameters 32-bit floating point data

FIG. 2 is a schematic diagram of the architecture of a data processing system according to an embodiment of the present invention. As shown in FIG. 2 , n base classes are defined in the management component, namely, the base classes 1 to n, where n is an integer greater than 1. It should be noted that the correspondence between the base class and the target processing component shown in FIG. 2 is to indicate that the target processing component is a derived class created based on the corresponding base class. During the implementation of the solution, the base classes 1 to n can also be completely the same, that is, a base class can be predetermined in the management component, and multiple target processing components determined according to the task to be processed can be generated and loaded through the base class. After loading is completed, the target processing components 1 to n constitute a pipeline for executing the task to be processed (i.e., for processing the data fragment of the target data). Only one synchronization component is shown between adjacent target processing components shown in FIG. 2, such as synchronization component 1 can be used as a backward synchronization node of target processing component 1 and a forward synchronization node of target processing component 2, and synchronization component 2 can be used as a backward synchronization node of target processing component 2 and a forward synchronization node of target processing component 3, etc., which will not be repeated. Of course, for any two adjacent target processing components, the backward node of the previous target processing component and the forward node of the next target processing component may also be different from each other. For example, a synchronization component 12 and a synchronization component 21 (not shown in the figure) can be configured between target processing components 1 and 2. The synchronization component 12 can serve as the backward node of target processing component 1, and the synchronization component 21 can serve as the forward node of target processing component 2. No further details will be given.

In one embodiment, the target processing component may include a data waiting module, which may specifically be a data waiting function predefined in the target processing component. The process of any target processing component reading the forward data fragment from its forward node described in the aforementioned embodiment may be specifically implemented by the data waiting module. For example, the module may respond to the target processing component reading the forward data fragment when the forward node is in a readable state. When the forward data fragment is a storage address, the data waiting module may read the storage address, and then use the data acquisition function predefined in the target data component to obtain (such as read) the data fragment processed by the previous target data component recorded at the storage address. In one embodiment, the target processing component may include a data transfer module, which may specifically be a data waiting function predefined in the target processing component. The process of any target processing component in the aforementioned embodiment transferring the backward data fragment processed by itself to the backward node may be specifically implemented by the data transfer module. For example, the data transfer module may transfer the backward data fragment to the backward node when the backward node is in a writable state after the target processing component completes the data processing operation. The data transfer module can obtain a backward data segment (the data segment itself or the storage address) from any of the target processing components, and then write the backward data segment to the backward node. In one embodiment, the target data may include audio and/or video, and the multiple target processing components may include an encoding component, an intelligent algorithm component, and a decoding component. Among them, the number of the intelligent algorithm components may be one or more. In the case where the target data includes a video, the intelligent algorithm component includes at least one of the following sub-algorithm components: a frame insertion component, a noise reduction component, a super-resolution component, a high dynamic range component, and a detail repair component. Of course, during the implementation of the solution, the sub-algorithm component can be expanded according to actual conditions, such as including a grayscale component, etc., which will not be repeated. Taking video as an example, the decoding component is used to decode the video into individual video frames, the intelligent algorithm component is used to process the video frames based on a predefined AI algorithm logic (such as frame insertion processing, noise reduction processing, etc.), and the encoding is used to encode the processed video frames output after processing by the intelligent algorithm component, thereby generating a processed video corresponding to the video. It can be understood that since the processed video is generated based on the processed video frames processed in sequence by the intelligent algorithm component, the processed video can present the processed display effect corresponding to the intelligent algorithm component during the playback process.

In one embodiment, the CPU of the data processing device may have multiple processor cores, that is,

The CPU is a multi-core CPU. In the above multi-core scenario, if the number of the target processing components is greater than 5 and the data processing system runs in the multi-core CPU, the multiple target processing components can be configured as follows:

For example, the multiple target processing components may be run in the same processor core to facilitate unified management and scheduling of each thread and improve the resource utilization of the CPU core. For another example, when the number of the CPU cores is not less than the number of the target processing components, each target processing component may also be run in different CPU cores.

In the same CPU core, the interference between different target processing components can be avoided as much as possible, and the operation efficiency of each target processing component can be improved. For another example, the management component can also flexibly adjust the load of each target processing component according to the load of multiple CPU cores.

The CPU core where the target processing component is located is used to achieve load balancing, thereby avoiding interference with the normal operation of the CPU core as much as possible. For another example, any target processing component can also be run by multiple processor cores respectively. At this time, the multiple processor cores may have data interaction, so that the expected function of any target processing component is jointly achieved through the mutual cooperation between the multiple processor cores.

In another embodiment, the data processing device may be equipped with a plurality of GPUs, wherein the GPUs according to the

The multiple target processing components determined by the task to be processed are deployed in multiple GPUs. If the data processing system includes multiple pipelines, each pipeline is deployed in a different GPU; and/or, any pipeline can be deployed in multiple GPUs (in this case, any pipeline is deployed in each of the multiple GPUs).

The target processing component may also include a distribution component and a collection component. Each data fragment in the target data may have a corresponding serial number, and the serial numbers of any two data fragments are different. The serial number is used to characterize the

For example, each video frame may have its own frame number, and the frame number of any video frame can be used to determine the position of the video frame in the video (i.e., at which moment the video frame is located in the timeline of the video).

Based on the sequence number, the distribution component and the collection component can be used to process the data fragments. For example, the distribution component can distribute each data fragment output by the decoding component to the corresponding intelligent algorithm component. Specifically,

The data fragments corresponding to different pipelines can be distributed to the target processing components belonging to different pipelines, and the different data fragments corresponding to the same pipeline can also be distributed to the target processing components belonging to the pipeline deployed in different GPUs. Each intelligent algorithm component can process the data fragments it receives and send the processed data to the target processing components.

The data fragments are passed to the collection component respectively. The collection component can receive the processed data fragments respectively passed by each intelligent algorithm component, and send the received processed data fragments to the encoding component in sequence according to the sequence number. In this way, each data fragment of the target data can be accurately distributed to the corresponding intelligent algorithm component for processing, and each intelligent algorithm component can pass the processed data fragment to the collection component immediately after the processing is completed, and the latter will send each processed data fragment to the encoding component in sequence according to the sequence. Not only does it achieve the disordered reorganization of each target data fragment, thereby ensuring the accuracy of the execution result of the task to be processed, but it can also avoid the waiting of the intelligent algorithm component, which helps to improve its data processing efficiency.

In one embodiment, different intelligent algorithm components can be located in different GPUs, and the distribution component distributes each data fragment output by the decoding component to the corresponding intelligent algorithm component, that is, sends it to the intelligent algorithm components corresponding to different GPUs. The distribution rules can be distributed to different GPUs/intelligent algorithm components in advance in a certain order (such as serial number, order of receiving decoding components), and can be dynamically distributed according to different available resources/spaces of the GPU, which is not limited here. After the collection unit receives the data fragment, it sends it to the encoding component once according to the serial number. The sending process can be to identify whether it meets the sequence number order after receiving the data fragment. If it meets the sequence number order, it is immediately sent to the encoding component, or it can collect multiple data fragments and then send them in the sequence number order. The sequence number can be marked by the distribution component when distributing it.

The following is an explanation of the loading process of the target processing component in the data processing system and the process of each component cooperating with each other to execute the task to be processed. As mentioned above, the components in the data processing system can be divided into two categories: algorithm services and algorithm tasks. For the algorithm service, the data processing system can provide an algorithm service interface for task management and service calls between the algorithm service and the algorithm task; for the algorithm task, the data processing system can provide an algorithm task interface for calling specific algorithm tasks, such as decoding components, intelligent algorithm components, encoding components, etc. The algorithm service interface and the algorithm task interface can be implemented through interface functions and corresponding member variables. Exemplarily, the interface functions and member variables of the algorithm service interface can be seen in Table 6 below.

Table 6

CZ2214464-PX2231246JZCN

The data synchronization class can ensure data transfer and data synchronization between algorithm tasks by managing data nodes. The data node class is a member variable of the data synchronization class, which defines the specific operations required for algorithm task data transfer. The member list it contains can be found in Table 7 below.

Table 7

The data synchronization class is an algorithm task created by the task creation class based on the corresponding creation specification. After the algorithm task is created based on this specification, it can be added to the algorithm service for process management. The algorithm task created based on this specification can include the specification.

The forward node configuration function and the backward node configuration function can be used to configure the corresponding data synchronization class (such as the forward node and backward node instantiated by the class) for any target processing component to determine the position of any target processing component in the processing flow and ensure the synchronization of data transmission between algorithm tasks. Before starting all algorithm tasks, the forward and backward nodes of the algorithm tasks must be configured. The parameter list is shown in Table 8.

Table 8

Function name Input/Output Type Parameter Description Algorithm task forward/backward node configuration function Input Parameters Parameter type is data synchronization class

The data waiting function and data transfer function are used to enable the algorithm task to obtain and transfer data in the data node. The parameter list can be found in Table 9.

Table 9

Function name Input/Output Type Parameter Description Data waiting/transfer function parameter list Input Parameters The parameter type is data node class

Fig. 3 is a schematic diagram of a workflow of a management component according to an embodiment of the present invention. As shown in Fig. 3, a complete processing process for a task to be processed may include the following steps 302-308:

Step 302: Initialize the data processing system by executing the initialization function init().

The management component may include predefined member variables, such as the base class FrameProcessCla target data of the target processing component, the algorithm task thread function Process_thread() of the processing component, the error information acquisition function get_thread_e to-be-processed task or() of the processing component, etc. Of course, the management component may also include other predefined contents, which are not limited in the embodiment of the present invention. Before loading the target processing component, the management component may first execute the initialization function init() to initialize the predefined contents such as the above-mentioned member variables, function functions, etc., thereby completing the initialization process of the data processing system.

Step 304: Add the multiple target processing components to the component list maintained by the management component by executing the task adding function add_task().

After the initialization is completed, for the multiple target processing components determined according to the to-be-processed tasks, the management component can execute the task adding function add_task(), wherein the input parameter of the function can be the component information of each component in the multiple target processing components. Of course, after the above addition is completed, the management component can also respond to the uninstallation instruction to uninstall one or more target processing components recorded in the component list (such as deleting the component information of the target processing component from the list), which will not be repeated.

Step 306: Start each target processing component added to the task list by executing the task start function start_task().

After the above addition is completed, the management component can execute the task start function start_task() to start each target processing component. Correspondingly, after the target processing component is started, it can be regarded that the pipeline startup corresponding to the task to be processed in the data processing system is completed. Thereafter, each target processing component can process the data fragments of the target data in sequence, thereby completing the execution process of the task to be processed.

It can be understood that steps 304 to 306 are the process of loading the multiple target processing components. In other words, the loading process includes two sub-processes: adding and starting.

Step 308: Shut down the data processing system by executing the task shutdown function start_task().

After the above-mentioned pending tasks are completed, the management component can execute the task closing function start_task() to close the pipeline. It should be noted here that the embodiment of the present invention does not limit the number of target data corresponding to the pending tasks. For example, the pending tasks may correspond to only one video, or may correspond to multiple videos or multiple audios at the same time. The management component can execute the above-mentioned task closing function start_task() to close the pipeline after processing the last target data in the pending tasks. Of course, the management component can also continue to execute other tasks after executing the pending tasks, and execute the above-mentioned task closing function start_task() to close the pipeline after completing the execution of the other tasks, which will not be repeated.

It is understandable that step 106 should be performed between step 306 and step 308. The following describes the execution of step 306. Each target processing component in the data processing system can process the data segments of the target data in sequence. There are many ways to implement the sequential processing, which are described below:

In one embodiment, any target processing component running in the data processing system can process each data segment of the target data in sequence, wherein: when any target processing component is not the last target processing component in the data processing system, after processing any data segment, any target processing component can pass the corresponding processing result to the next target processing component for processing; and, when any target processing component is the last target processing component in the data processing system, any target processing component can output the corresponding processing result after processing the last data segment as the processing result of the data processing system for the target data, that is, the execution result of the task to be processed. Specifically, any target processing component can process each data segment received by itself according to its own predefined processing logic. Depending on the different functions implemented by the target processing component, the processing logic of the target processing component is also different. For example, when any target processing component is a decoding component (used to decode the video), its processing logic can be a preset decoding algorithm; when any target processing component is a super-resolution component (used to improve the resolution of the video frame image), its processing logic can be a preset upsampling algorithm, which will not be repeated.

As mentioned above, multiple target processing components are loaded in the data processing system, among which there is a one-way data transmission between any two adjacent target processing components, that is, according to the processing order corresponding to each target processing component, the previous target processing component will transmit the processing result obtained after processing the data segment itself (that is, the aforementioned processed data segment or backward data segment) to the next target processing component.

As shown in FIG2 , let us assume that n=4. In this case, the target processing component 4 is the last target processing component in the data processing system. After the target processing component 1 completes processing any data segment, it can pass the corresponding processing result (hereinafter referred to as the first-level data segment, the same below) to the target processing component 2; accordingly, the target processing component 2 can pass the second-level data segment obtained by processing the first-level data segment to the target processing component 3, and the target processing component 3 can pass the third-level data segment obtained by processing the second-level data segment to the target processing component 4. As the last target processing component in the data processing system, the target processing component 4 can directly output the fourth-level data segment obtained by processing the third-level data segment as part of the processing result of the target data; or, after completing the processing of the last data segment of the target data, it can generate the final processing result for the target data according to the fourth-level data segments of multiple data segments, and output the final processing result. It can be seen from this that from the perspective of any data segment of the target data, each target processing component in the data processing system processes the data segment in series. In other words, each target processing component processes the data segments of the target data in sequence, which can include each target processing component sequentially processing each data segment of the target data.

In another embodiment, in the process of any target processing component processing any data fragment, other target processing components can simultaneously process other data fragments of the target data, and the processing processes of each target processing component are independent of each other. For example, when target processing component 1 passes any primary data fragment to target processing component 2, target processing component 2 can start processing the primary data fragment according to its own processing logic, and target processing component 1 can start acquiring and processing the next data fragment. Similarly, when target processing component 2 passes any secondary data fragment to target processing component 3, target processing component 3 can start processing the secondary data fragment according to its own processing logic, and target processing component 2 can start acquiring and processing the next primary data fragment; when target processing component 3 passes any tertiary data fragment to target processing component 4, target processing component 4 can start processing the tertiary data fragment according to its own processing logic, and target processing component 3 can start acquiring and processing the next secondary data fragment. For another example, when target processing component 4 in FIG. 2 processes the i-th data fragment of the target data, target processing components 1, 2, and 3 can respectively process the i-3, i-2, and i-1 data fragments of the data fragment. It can be seen that from the perspective of each target processing component, each data processing system processes different data fragments in parallel. That is, each target processing component processes the data fragments of the target data in sequence, and each target processing component can also process the same data fragment of the target data in series in sequence, and each target processing component processes different data fragments in parallel.

Fig. 4 is a schematic diagram of a work flow of a target processing component according to an embodiment of the present invention. As shown in Fig. 4, the process includes steps 402-410:

Step 402, configure the forward node and the backward node.

Before loading any target processing component, the management component can first configure the forward node and backward node of the target processing component. The synchronization component between any target processing component and its previous target processing component can be regarded as the forward node of any target processing component, and the synchronization component between any target processing component and its subsequent target processing component can be regarded as the backward node of any target processing component. As shown in Figure 2, in the case of n=4, target processing component 1 does not have a forward node or its forward node is empty, and its backward node is synchronization component 1; the forward node and backward node of target processing component 2 are synchronization components 1 and 2 respectively, and the forward node and backward node of target processing component 3 are synchronization components 2 and 3 respectively; the forward node of target processing component 4 is synchronization component 3, and there is no backward node or its backward node is empty.

Specifically, the management component can determine the forward node and backward node of any target processing component according to the processing order, and then before executing the aforementioned task adding function add_task(), execute the forward node configuration function pre_node() and the backward node configuration function next_node() for the target processing component in sequence. Among them, the input parameter of the forward node configuration function pre_node() can include the node identification of any target processing component and its forward node, and the input parameter of the backward node configuration function next_node() can include the node identification of any target processing component and its backward node. In this way, the management component can complete the configuration of the forward component and the backward component of any target processing component, and then the aforementioned task adding function add_task() can be executed to load the component list of any target processing component. Among them, the management component can execute the task adding function add_task() for each target processing component, and when the processing order of each target processing component is determined, the management component can execute the task adding function for each target processing component in sequence according to the processing order. It can be seen that the execution order of the task adding function add_task() of each target processing component can be used as the loading order of each target processing component, thereby realizing the orderly loading of each target processing component and the orderly operation of each target processing component after startup. In addition, the processing order of each target processing component in this solution is the processing order of each target processing component for the same data fragment, that is, the multiple target processing components can process the same data fragment in sequence according to their corresponding processing order.

Step 404: When waiting for the forward node to be in a readable state, obtain the forward data segment from the forward node.

After the loading of each target processing component is completed, each target processing component loaded in the data processing system can participate in processing the data segment of the target data. Among them, the previous target processing component of any target processing component can pass the data segment processed by itself (the data segment is a backward data segment relative to the previous target processing component, and a forward data segment relative to any target processing component) to its own backward node. In the case where the backward node is the forward node of any target processing component, the any target processing component can wait for the forward node to be in a readable state; and in the case where the backward node is not the forward node of any target processing component, the forward node of any target processing component can obtain the forward data segment from the node when the backward node is in a readable state. After the forward node writes the forward data segment, the node can be updated to a readable state.

Thus, any of the target processing components can obtain the forward data segment from the forward node when the forward node is in a readable state. Specifically, the forward data segment can be directly read from the forward node, or the storage address of the segment can be read, and then the forward data segment can be obtained from the storage address. For details, please refer to the description of the aforementioned embodiment, which will not be repeated here. Of course, for the first target processing component in the data processing system, since it does not have a forward node, it can receive the forward data segment sent by other devices or read the forward data segment from a preset storage space.

The process of obtaining the forward data fragment from the forward node can be implemented by executing the data waiting function wait(), and the execution result of the function can be the forward data fragment.

Step 406: Process the acquired forward data segment according to its own processing logic.

After receiving the forward data segment, any target processing component may start to process the data segment according to its own processing logic, and the specific processing method will not be described in detail.

This is achieved by executing a data processing function Process(), in the function body of which the logic code of the aforementioned processing logic can be recorded, so that the execution result of the function is the processing result of the forward data segment.

Step 408: Output the processed data segments to the backward node.

After processing the forward data segment, any target processing component can pass the processed backward data 0 segment to its backward node. The process of passing the backward data segment can be implemented by calling the data transfer function pose(), and the input parameter of the function is the backward data segment or its storage address. After the transfer is completed, the backward node can update its own state to a readable state so that the next target processing component (or its forward node) can read the forward data segment or its storage address from the node.

Step 410, shut down the target processing component.

5. When any target processing component completes processing the last data segment of the target data,

Determine that the target processing component has completed the aforementioned subtask for the target data. At this point, the management component can trigger the closing of the target processing component; or it can wait for the last target processing component to complete processing the last data segment of the target data and then close the target processing component. Among them, any of the target processing components can trigger its own closing, or

Alternatively, the target processing component may be closed by the management component. Specifically, the component may be closed by executing a component closing function close() for any of the target processing components.

In one embodiment, a shared variable can be set for any synchronization component. The shared variable is used to store the transmitted data fragment. The value of the shared variable is the transmitted data fragment or its storage address, hash value, etc. For example, for the forward node and the backward node of any target processing component, the shared variable of the forward node is used to store the previous target node.

The forward data segment transmitted by the processing component to any target processing component, and the shared variable of the backward node is used to store the backward data segment transmitted by any target processing component to the next target processing component.

When a component is in a writable state, its shared variables are writable, that is, the data fragment to be transmitted can be written into the shared variable by assigning a value to the shared variable; correspondingly, when any synchronization component is in a readable state, its shared variables are readable, that is, the written data fragment can be obtained by reading the value of the shared variable.

For the sake of distinction, the shared variable of the forward node of any target processing component may be recorded as shared variable 1, and the shared variable of the backward node 0 may be recorded as shared variable 2. FIG5 is a schematic diagram of a data transmission process between target processing components according to an embodiment of the present invention. Taking the transmission of complete data of a certain data segment as an example, the process of transmitting intermediate data segments (i.e., the forward data segment or the backward data segment) by adjacent target processing components in the data processing system through the aforementioned shared variables is described in detail in conjunction with FIG5. As shown in FIG5, the process includes steps 502-514.

Step 502, waiting for shared variable 1 to be readable.

After the previous target processing component of the current target processing component completes processing any data segment of the target data, the corresponding processing result (hereinafter referred to as the target processing result) can be transmitted through the forward node between the current target processing component and the previous target processing component. Specifically, the target processing result can be written into shared variable 1, and after the writing is completed, the forward node is triggered to be set to a readable state, so that the shared variable 1 is readable, and the flag bit var1_readable_sem of shared variable 1 is 1. In this regard, the current target processing component can wait for the shared variable 1 to be readable by executing the state waiting function Sem_wait(), and determine that the shared variable 1 is readable when the value of var1_readable_sem is 1.

Step 504, read shared variable 1.

Step 506, releasing the shared variable 1 to be writable.

At this time, the current target processing component can read shared variable 1 to obtain the forward data segment to be written. After the reading is completed, the forward node can be updated to a writable state, that is, the shared variable 1 is released to be writable. Among them, the above update can be triggered by the current target processing component after the reading is completed, or the forward synchronization component itself can start the above update after listening to the completion of the reading. Specifically, the state update function Sem_post() can be executed to set the value of var1_writeable_sem to 1 to indicate that shared variable 1 is writable.

Of course, in steps 502-506, the forward node can also save the forward data segment in the memory of the data processing device, and write the corresponding storage address or index and other result retrieval information into the shared variable 1, and trigger the shared variable 1 to be readable after the writing is completed; accordingly, the current target processing component can read the result retrieval information from the shared variable 1 when it is determined that the shared variable 1 is readable, and further read the forward data segment from the corresponding storage space of the memory according to the information. In this way, a smaller storage space can be set for the shared variable 1, and it is avoided to write the forward data segment into the storage space or read the result therefrom, which helps to save storage space while improving the data transmission efficiency between components.

Step 508, process the target processing result.

After acquiring the target processing result, the current target processing component may process the target processing result according to its own processing logic, and further obtain a backward data segment.

Step 510, waiting for shared variable 2 to be writable.

After the backward node of the current target processing component completes reading data from the shared variable 2 of the backward node (i.e., the data obtained by the current target processing component after processing the previous processing result of the target processing result), it can trigger the backward node to be set to a writable state, thereby making the shared variable 2 writable, and at this time, the flag bit var1_writeable_sem of the shared variable 2 = 1. The current target processing component can continue to execute the state waiting function Sem_wait() to wait for the shared variable 2 to be writable, and determine that the shared variable 2 is writable when the value of var2_writeable_sem is 1.

Step 512, assign a value to shared variable 2.

Step 514, release the shared variable 2 to be readable.

When it is determined that shared variable 2 is writable, the current target processing component can assign a value to the variable. Specifically, the backward data segment (or the result retrieval information of the result) is written to the shared variable 2, and after the writing is completed, the backward node is triggered to be updated to a readable state, that is, the shared variable 2 is released to be readable. Specifically, the state update function Sem_post() can continue to be executed, so that the value of var2_readable_sem is set to 1 to characterize that shared variable 2 is readable. After the above update is completed, the backward node can obtain the backward data segment by reading the shared variable 2. The specific process is similar to the process in which the current target processing component obtains the target processing result through the shared variable 1, and will not be repeated.

Through the above-mentioned method, through the forward node between the current target processing component and the forward node, the current target processing component can obtain the target processing result obtained by the forward node processing the intermediate data fragment, that is, the transfer of the target processing result between the forward node and the current target processing component is realized; through the backward node between the current target processing component and the backward node, the backward node can obtain the backward data fragment obtained by the current target processing component processing the target processing result, that is, the transfer of the backward data fragment between the current target processing component and the backward node is realized. It can be seen that any two adjacent target processing components in the data processing system can realize the effective transfer of data through the above-mentioned method.

It is understandable that, because the current target processing component described in the embodiment of FIG. 5 has a forward node and a backward node, the target processing component can be any target processing component in the pipeline except the first and last target processing components. For the first target processing component in the pipeline, since there is no forward node, it can be applied to steps 508-514 of FIG. 5; and for the last target processing component in the pipeline, since there is no backward node, it can be applied to steps 502-508 of FIG. 5, which is hereby explained.

In one embodiment, in order to further improve the time synchronization effect of the synchronization component, a storage space as large as possible can be set for the synchronization component (ie, the data capacity of the synchronization component is increased) so that the synchronization component can store multiple intermediate data fragments at the same time.

In another embodiment, considering that the state update of the synchronization component still takes a certain amount of time, a message queue can be used as the above synchronization component, that is, the intermediate data fragment to be transmitted (or its storage address, hash value) is used as a message, and the target processing components other than the last target processing component in the pipeline connected by the synchronization component can be used as message producers, and the target processing components other than the first target processing component can be used as message consumers. Therefore, when the capacity of the message queue is met, the producer can immediately pass the processing result to its backward node after processing any data fragment without waiting for it to be in a readable state, thereby achieving more adequate asynchronous processing between adjacent target processing components.

In one embodiment, when a pipeline includes multiple target processing components, any target processing component can also obtain the associated information of any data segment from other target processing components during the process of processing any data segment of the target data, and process the any data segment according to the associated information, wherein the associated information is generated by the other target processing components when processing the associated data segments of any data segment. The associated data segments can be adjacent data segments of the current data segment being processed by any target processing component or a preset number of consecutive data segments (such as adjacent frames of the current frame or multiple consecutive frames, etc.); the associated information can be synchronization information, delay duration, parameter (such as grayscale value, etc.) difference between adjacent data segments, and other information. Using the above information, any target processing component can fully determine the relationship between the current data segment and its associated data segment, thereby helping to improve the processing accuracy of the current data segment by any target processing component.

Through the above-mentioned embodiments, after the multiple processing components are loaded by the management component, each processing component can process the data fragments of the target data in order, thereby completing the task to be processed. On the one hand, the solution can be managed and scheduled by the management component in a unified manner, so that each target processing component can cooperate with each other to complete the processing process for the target data in an orderly manner. Even if the number of target processing components is large, each target processing component can process the data fragments in an orderly manner, thereby greatly simplifying the management logic for each processing component in the system and the processing process of the target data, which helps to improve the execution efficiency of the task to be processed. On the other hand, each target processing component can be determined and loaded according to the task to be processed, so that each target processing component can process the data fragments of the target data according to its own processing logic. Since the processing logic of each target processing component can be pre-defined according to actual needs, this method can implement a more complex task processing logic, so as to provide more processing functions, thereby facilitating the realization of diversified processing needs.

In addition, the management component can determine the corresponding processing component according to the target data corresponding to the task to be processed, and then load each determined processing component into the data processing system. It is understandable that before the above-mentioned data processing system is started, the user can specify the corresponding processing component to the management component according to the target data, so as to realize the on-demand loading of the processing component.

Figure 6 is a schematic diagram of the algorithm service interface described in this solution. As shown in Figure 6, by deriving the task creation class, specific algorithm tasks can be created, namely, algorithm task 1, algorithm task 2, algorithm task 3 and algorithm task, etc. in the figure. After the algorithm task is created, it can be added to the process management class (specifically, it can be added to the component list) for management. The process management class can create corresponding threads for each algorithm task and allocate processing resources. By instantiating the data synchronization class, objects passed between algorithm tasks can also be created, and the forward nodes and backward nodes corresponding to each algorithm task can be configured by calling the forward node configuration function and the backward node configuration function. The data synchronization class can ensure that the algorithm task can transmit data fragments in an orderly manner under high concurrency.

After all algorithm tasks are added and forward nodes and backward nodes are configured, the process management class can call the task start function to start all tasks. After the start, each computing task constitutes a pipeline for executing the task to be processed. During the execution of the task, the process management class can monitor each algorithm task in the pipeline, and when it is determined that the data output by the output node of the pipeline is empty, it closes each algorithm task in the pipeline (i.e., manages the pipeline) and releases the corresponding processing resources.

Corresponding to the aforementioned data processing method, the present invention also proposes a video processing method, which is applied to a video processing system, wherein the video processing system includes a management component and a processing component. FIG7 is a flow chart of a video processing method according to an embodiment of the present invention. As shown in FIG7 , the method includes steps 702-706.

Step 702, determining a target processing component according to a task to be processed, wherein the task to be processed is used to process a target video, and the target processing component includes a decoding component, an encoding component, and at least one intelligent algorithm component;

Step 704: the management component loads the target processing component.

Step 706, the loaded intelligent algorithm component processes the video frames of the target video in sequence, wherein each target processing component processes different video frames at the same time, and at least one video frame is processed in sequence by each target processing component.

In the embodiment of the present invention, the decoding component is used to decode the target video to obtain multiple video frames, and send each video frame to the intelligent algorithm component in sequence. The intelligent algorithm component can process the video frames of the target video in sequence, wherein the last intelligent algorithm component can send the processed video frames corresponding to each video frame to the encoding component, so that the latter can encode each processed video frame to obtain a processed video.

In one embodiment, the intelligent algorithm component may include at least one of the following sub-algorithm components: a frame insertion component, a noise reduction component, a super-resolution component, a high dynamic range component, and a detail restoration component. Of course, during the implementation of the solution, the sub-algorithm component may be expanded according to actual conditions, such as including a grayscale component, etc., which will not be described in detail.

In one embodiment, the video processing system may further include a forward node and a backward node, the target processing component corresponds to one or more forward nodes, and the target processing component corresponds to one or more backward nodes; wherein the forward node is used to obtain forward video frame information for the target processing component to read, and the backward node is used to obtain backward video frame information processed by the target processing component. The forward node and the backward node may be regarded as the forward synchronization component and the backward synchronization component described in the aforementioned embodiment, and the configuration and startup methods of the forward node and the backward node may refer to the records of the aforementioned related embodiments, which will not be repeated here.

In one embodiment, any target processing component can obtain forward video frame information when the forward node is in a readable state. Wherein, the forward video frame information is obtained by processing the previous target processing component of any target processing component, and the forward video frame information (video frame data or storage address) can be transmitted by the previous target processing component to the forward node (such as writing to the forward node), so that any target processing component can directly read the video frame data from its forward node, or after reading the storage address from the forward node, further read the video frame data from the storage address. After any target processing component obtains the forward video frame information from the forward node, the forward node can be updated from a readable state to a writable state. Thereafter, any target processing component can perform data processing operations to obtain corresponding backward video frame information, and specifically, the forward video frame information can be processed according to its own preset processing logic. Furthermore, when the backward node is in a writable state, the backward node can obtain the backward video frame information (video frame data or storage address) processed by any target processing component; and after obtaining the backward video frame information, the backward node can be updated from a writable state to a readable state. For example, when any target processing component obtains the backward video frame information through processing, the backward video frame information can be written into the backward node.

It is understandable that after the backward node is updated to a readable state, if the backward node of any target processing component is the same node as the forward node of the latter target processing component, the latter target processing component can obtain the backward video frame information from the node when the node is in a readable state; or, in the case where the backward node of any target processing component is not the same node as the forward node of the latter target processing component, the backward node of any target processing component and the forward node of the latter target processing component can be connected in sequence. In this regard, when the backward node of any target processing component is in a readable state, the forward node of the latter target processing component can read the backward processing data from the backward node and record it locally, so that the latter target processing component reads the backward processing data from the forward node. It is understandable that the backward video frame information is its forward video frame information for the latter target processing component.

In one embodiment, the video processing device may be equipped with multiple GPUs, wherein each intelligent algorithm component determined according to the task to be processed may be deployed in multiple GPUs respectively. For example, in the case where the video processing system includes multiple pipelines, each pipeline is deployed in a different GPU respectively; and/or, any one of the pipelines may be deployed in multiple GPUs respectively (in this case, any one of the pipelines is deployed in the multiple GPUs respectively). The target processing component may also include a distribution component and a collection component. Each video frame in the target video may have a corresponding serial number, and the serial numbers of any two video frames are different from each other. The serial number is used to characterize the position of each video frame in the timeline of the target video. For example, the frame number of any video frame can be used to determine at which moment the video frame is located in the timeline of the video.

Based on the sequence number, the distribution component and the collection component can be used to process video frames. For example, the distribution component can distribute each video frame output by the decoding component to the corresponding intelligent algorithm component. Specifically, the video frames corresponding to different pipelines can be distributed to the intelligent algorithm components belonging to different pipelines, or the different video frames corresponding to the same pipeline can be distributed to the intelligent algorithm components belonging to the pipeline deployed in the different GPUs. Each intelligent algorithm component can process the video frames received by itself, and pass the processed video frames obtained by processing to the collection component. The collection component can receive the processed video frames respectively passed by each intelligent algorithm component, and send the received processed video frames to the encoding component in sequence according to the sequence number. In this way, each video frame of the target video can be accurately distributed to the corresponding intelligent algorithm component for processing, and each intelligent algorithm component can pass the processed video frame to the collection component immediately after the processing is completed, and the latter sends each processed video frame to the encoding component in sequence according to the sequence. It not only achieves the random reorganization of each target video clip, thereby ensuring the accuracy of the execution results of the tasks to be processed, but also avoids the waiting of intelligent algorithm components, which helps to improve its data processing efficiency.

It is understandable that the processed video can present the processed display effect corresponding to the intelligent algorithm component during the playback process. For example, when the intelligent algorithm component includes an interpolation component, the processed video will have a higher frame rate relative to the target video before processing, and accordingly, the processed video can be played at a higher frame rate relative to the target video. For another example, when the intelligent algorithm component includes a detail restoration component, the detail display effect of the processed video is relative to the target video before processing, such as higher clarity of the details.

According to the aforementioned embodiment, on the one hand, the management component can uniformly manage and schedule the target processing components, so that each target processing component (including a decoding component, an encoding component and at least one intelligent algorithm component) cooperates with each other to orderly complete the processing process for the target video. Even if the number of intelligent algorithm components is large, each intelligent algorithm component can process the video frames in an orderly manner, thereby greatly simplifying the management logic for each intelligent algorithm component in the system and the processing process of the target data, which helps to improve the execution efficiency of the tasks to be processed. On the other hand, at least one intelligent algorithm component can be determined and loaded according to the tasks to be processed, so that each intelligent algorithm component can process the video frames of the target video according to its own processing logic. Since the processing logic of each intelligent algorithm component can be pre-defined according to actual needs, this method can realize a more complex video processing logic, so as to provide more processing functions, thereby facilitating the realization of diversified video processing needs. In addition, on-demand loading of intelligent algorithm components can be achieved.

Fig. 8 is a schematic diagram of the architecture of a video processing system according to an embodiment of the present invention. As shown in Fig. 8, when the above data processing system is implemented using an object-oriented programming language, the management component (i.e., VideoProcessFrameWork) may pre-define a basic implementation class (referred to as the base class) for video pipeline processing, and each target processing component loaded in the data processing system may be derived based on the base class, i.e., each target processing component may be a derived class corresponding to the base class.

The processing order of each target processing component in the pipeline shown in Figure 8 is: decoding component (i.e., VideoReaderUnit), distribution component (i.e., OneToManyUnit), intelligent algorithm component (TensorrtProcessUnit[]), collection component (i.e., ManyToOneUnit) and encoding component (VideoWriterImgUnit). In addition, a synchronization component (i.e., UnitSynArg1-4) can be set between any two adjacent target processing components, such as UnitSynArg1 between the decoding component and the distribution component, UnitSynArg3 between the intelligent algorithm component and the collection component, etc., which will not be repeated here. Among them, any target processing component can be set with a forward node and a backward node. Taking the distribution component as an example, "pre_node(synArg_videoReaderunit_out)" indicates that the forward node of the component is a decoding component, and the distribution component receives the processing result (i.e., the primary video frame) transmitted by the decoding component through the synchronization component UnitSynArg1; "next_node(synArg_OneToManyUnit_out[])" indicates that the backward node of the component is an intelligent algorithm component, and the processing result (i.e., the secondary video frame) after the component processes any video frame is transmitted to the subsequent collection component through the synchronization component UnitSynArg2. It should be noted that, as the first target processing component, the decoding component has an empty forward node (Pre_node(NULL)); as the last target processing component, the encoding component has an empty backward node (next_node(NULL)).

After the decoding component, the intelligent algorithm component and the encoding component are loaded, the loaded target processing components together constitute a pipeline for the task to be processed, and the management component can start the pipeline to execute the task, that is, to process the target video. Specifically, the management component can start each target processing component in the pipeline by executing the aforementioned task start function start_task().

Furthermore, after each target processing component is started, the decoding component can obtain the target video specified by the task to be processed. Specifically, the target video can be directly extracted from the task to be processed, or the target video can be read from the storage space specified by the storage address, index and other video retrieval information contained in the task to be processed.

Furthermore, the component can decode the acquired target video to obtain multiple video frames. Then, the decoding component can pass the multiple video frames obtained by decoding to the intelligent algorithm component, and then, at least one intelligent algorithm component loaded in the pipeline can process the multiple video frames in sequence to obtain corresponding processed video frames.

The intelligent algorithm component shown in Figure 8 is only illustrated by one TensorrtProcessUnit[]. In fact, the number of loaded sub-algorithm components can be one or more. For example, the intelligent algorithm component described in this solution may include one of the following sub-algorithm components: interpolation component, noise reduction component, super-resolution component, high dynamic range component, detail restoration component, and may also include other components shown in Table 1 above, which will not be repeated here. The above-mentioned intelligent algorithm component can be a pre-trained AI (Artificial Intelligence) algorithm model. In the practice of the solution, any of the above-mentioned AI algorithm models can use Tensorrt as the AI acceleration framework, CUDA (Compute Unified Device Architecture, graphics processor development environment) as the algorithm acceleration framework, and ffmpeg as the encoding and decoding framework.

In addition, the encoding format, encoding pixel format and video file encapsulation format of the target video in this solution can be selected according to actual conditions, and the embodiment of the present invention does not limit this. For example, the encoding format can be h264, the encoding pixel format can be yuv420, and the video file encapsulation format can be .mp4. Among them, the video stream encoding format information and video stream pixel format information of the video can both be in enumeration form, and the corresponding enumeration members can be seen in Table 10.

Table 10

Wherein, similar to the aforementioned data processing method, in the process of the at least one sub-algorithm component sequentially processing the multiple video frames, each sub-algorithm component can also follow the aforementioned pipeline working method. For example, after processing any video frame, any sub-algorithm component can pass the corresponding processing result to the next target processing component for processing. Wherein, depending on the difference of any of the sub-algorithm components, the corresponding next target processing component can be other sub-algorithm components, or can also be the encoding component.

For another example, when the pipeline further includes a synchronization component, a forward node may be connected before any first-category target processing component, and the first-category target processing component is any target processing component other than the decoding component in the processing sequence of the multiple target processing components; and/or, a backward node may be connected after any second-category target processing component, and the second-category target processing component is any target processing component other than the encoding component in the processing sequence. The forward node of any first-category target processing component and the backward node of the previous second-category target processing component may be the same node, or may be different nodes.

Among them, when any first-category target processing component is in a readable state, it can obtain (such as read) the target video frame from the first storage address indicated by the forward node, and after the acquisition is completed, the forward node is updated to a writable state; wherein, the target video frame is processed by the previous target processing component of any first-category target processing component and written to the first storage address when the forward node is in a writable state.

In addition, when the backward node is in a writable state, any second-category target processing component can write the processed video frame obtained by itself into the second storage address indicated by the backward node, and trigger the backward node to be updated to a readable state after writing is completed. The process of realizing video frame exchange by the first-category target processing component and the second-category target processing component through the synchronization component is essentially the same as the process of realizing video frame exchange by the synchronization component in the data processing scheme. The specific implementation method can be found in the records of the corresponding embodiments above, which will not be repeated here.

In one embodiment, the image processing device may be equipped with multiple independent GPU boards; or, any GPU board may be logically divided into multiple GPU modules, and the independent GPU board or GPU module is referred to as a GPU in the embodiment of the present invention. In the above case, in order to improve the processing efficiency of the task to be processed, the multiple GPUs may be used to implement parallel processing of different video frames. For example, on the one hand, the intelligent algorithm component may be deployed in multiple GPUs respectively, and on the other hand, the distribution component and the collection component may be loaded by the management component.

Among them, there may be multiple pipelines in the food processing system. At this time, the sub-algorithm components in each pipeline can be deployed in different GPUs respectively. For example, when the target video is an audio and video file, the video processing system may include an audio pipeline and a video pipeline. At this time, each sub-algorithm component in the audio pipeline can be deployed in any GPU, and each sub-algorithm component in the video pipeline can be deployed in another GPU. Based on this, after the decoding component in the pipeline decodes the audio and video file to obtain audio clips and video frames, the distribution component can distribute all audio clips to the first sub-algorithm component in the audio pipeline for processing in sequence, and send all video frames to the first sub-algorithm component in the video pipeline for processing in sequence. Among them, the audio stream encoding format information of the audio shown can be in enumerated form, and the corresponding enumeration members can be seen in Table 11.

Table 11

Alternatively, for any pipeline, all sub-algorithm components in the pipeline can be deployed in multiple GPUs in the same deployment mode, such as deploying all sub-algorithm components in each GPU in the same loading order, so that each GPU can implement exactly the same processing logic. Alternatively, in the case where the intelligent algorithm component includes multiple sub-algorithm components, the multiple sub-algorithm components can also be deployed in multiple GPUs in a dispersed manner, such as deploying a certain sub-algorithm component in a certain GPU, and deploying another sub-algorithm component in another GPU; in particular, when the number of GPUs is not less than the number of sub-algorithm components, each sub-algorithm component can be deployed in a different GPU. Based on this deployment method, the distribution component can distribute the multiple video frames output by the decoding component to the corresponding GPUs according to preset rules. After any GPU receives any video frame, the sub-algorithm components deployed in the GPU can process the video frame respectively.

Exemplarily, in the case where all sub-algorithm components in any pipeline are deployed in multiple GPUs in the same deployment mode, in order to simplify the distribution logic of the distribution component, the preset rule can adopt a polling rule, that is, the GPU component can send the multiple video frames to each GPU in sequence; or, considering that there may be performance differences between different GPUs, in order to maximize the overall processing efficiency of video frames, the preset rule can also adopt a load balancing rule to distribute each video frame to the GPU with the best performance at the current moment. At this time, different GPUs can receive different video frames respectively, and all sub-algorithm components deployed in any GPU can process each video frame received by the GPU. In the case where multiple sub-algorithm components are dispersedly deployed in multiple GPUs, in order to ensure that each video frame can be processed by all sub-algorithm components, the distribution component can send any video frame to each GPU respectively. At this time, each GPU can obtain all video frames of the target video respectively, and each sub-algorithm component deployed in any GPU can process the video frame received by the GPU according to the corresponding processing logic.

Among them, the GPU distribution unit can maintain an available GPU resource information container, which is used to store the resource information of the GPU provided by the video processing device. For example, the container may include an array, which can be used to record the GPU identification of each GPU, and any GPU identification can be an 8-bit or 16-bit integer. For another example, the container may also include a structure, which can be used to record the GPU identification, model information, performance parameters, data format, available resource type, processable task type and other GPU-related information of each GPU, which will not be repeated. In the process of distributing video frames according to the aforementioned preset rules, the distribution component can accurately determine to which GPU any video frame should be sent based on the GPU identification or GPU-related information stored in the above container, thereby achieving accurate distribution of video frames.

It should be noted that the management component, distribution component and collection component of the data processing system, except for the intelligent algorithm component, can run in the CPU of the video processing device. The decoding component and the encoding component run in different locations according to different encoding and decoding methods. For example, if the target video is decoded by hardware, the decoding component can run in the GPU; if the software decoding method is used, the decoding component can run in the CPU. Similarly, if the hardware encoding method is used for each processed video frame, the encoding component can run in the GPU; if the software encoding method is used, the encoding component can run in the CPU.

The management component can configure and run the distribution component and the collection component by calling functions. The functions corresponding to the distribution component are:

Initialization function, used to initialize GPU resource distribution information. Function return value: if the call is successful, it returns 0; if the call fails, it returns non-zero.

Data processing function, used to distribute data to each GPU. Function return value: 0 if the call is successful; non-zero if the call fails.

Resource release function, used to release the resources allocated to the GPU. Function return value: 0 if the call is successful; non-zero if the call fails.

Correspondingly, the functions corresponding to the collection component are:

Initialization function, used to initialize GPU resource collection information. Function return value: if the call is successful, it returns 0; if the call fails, it returns non-zero.

Data processing function, used to collect GPU data in an ordered manner. Function return value: 0 if the call is successful; non-zero if the call fails.

Resource release function, used to release the resources allocated to the GPU. Function return value: 0 if the call is successful; non-zero if the call fails.

In addition, the parameter lists of the initialization functions corresponding to the above distribution components and collection components can be found in Table 12 below.

Table 12

For any GPU, each sub-algorithm component deployed therein can obtain a corresponding processing result after processing any video frame. In the case where all sub-algorithm components are deployed in each GPU in the same deployment mode, the processing result is the processed video frame corresponding to any video frame. In the case where multiple sub-algorithm components are dispersedly deployed in multiple GPUs, the processing result output by any GPU is only the result after processing by some sub-algorithm components in the GPU. Therefore, the collection component also needs to combine and process the processing results corresponding to the same serial number output by each GPU, so as to obtain the processed video frame corresponding to the serial number. Thereafter, the collection component can send each processed video frame to the encoding component in sequence according to the serial number order of the multiple video frames. In the above manner, on the basis of multiple target processing components processing video frames in sequence, the parallel processing capability of the GPU can be used to run multiple pipelines concurrently, thereby further improving the execution efficiency of the data processing system for the tasks to be processed.

In addition, after receiving a video frame, any GPU may pre-process the data, and then the intelligent algorithm component may process the pre-processed video frame according to its own algorithm logic. Accordingly, the GPU may also post-process the processed video frame or processed data obtained by the intelligent algorithm component by processing the video frame. The embodiment of the present invention does not limit the specific methods of the aforementioned pre-processing and post-processing, and may include data type conversion and normalization, etc., which will not be described in detail.

Among them, due to the performance differences of different GPUs, the processing time of different video frames may be different. Therefore, the order in which the collection component receives each processed video frame may be different from the order in which the GPU sends each video frame, that is, each processed video frame may be out of order. At this time, in order to ensure that the encoding component can successfully complete the video encoding work, the collection component can reorganize the received processed video frames in a disordered order, such as temporarily storing the disordered processed video frames locally, and when receiving the processed video frames with the expected sequence number, send the processed video frames to the encoding component in sequence according to the aforementioned order. Of course, the embodiment of the present invention does not limit the specific disordered reorganization logic and implementation process. Alternatively, the above-mentioned out-of-order reorganization can also be completed by the encoding component. For example, the collection component can output the received processed video frames to the encoding component in sequence according to the receiving order. At this time, the processed video frames received by the encoding component may be out of order, that is, the receiving order of the processed video frames is inconsistent with the size order of the corresponding serial numbers. Therefore, when the code component encodes each processed video frame, it can encode each processed video frame in sequence according to the size order of the corresponding serial numbers, thereby realizing out-of-order reorganization of each processed video frame.

Figure 9 is a schematic diagram of a video processing process according to an embodiment of the present invention. In conjunction with Figure 8, taking the video ultra-high-definition processing scenario as an example, the video processing process in which all sub-algorithm components are deployed in each GPU in the same deployment manner is described. As shown in Figure 9, the same intelligent algorithm components are deployed in m (m is greater than 2) GPUs, and the connection method of each sub-algorithm component in the GPU (corresponding to the processing order of each sub-algorithm component) remains consistent; in other words, at least one sub-algorithm component is deployed in the same manner in m GPUs. Among them, the sub-algorithm component included in the intelligent algorithm component can be any of the components shown in Table 3 above. The following embodiment is described by taking three sub-algorithm components: noise reduction component, super-resolution component and HDR component as examples.

As shown in FIG9 , the target video to be processed is a video file with a resolution of 1080P. It is assumed that the target video name is 1080p.mp4, the frame rate is 25, the resolution is 1080*1920, the encapsulation format is mp4, the encoding format is libx264, and the bit rate is 25fps.

In addition, it is assumed that the decoder used by the decoding component is libx264 corresponding to the target video, the decoding pixel format is RGB24, the decoded data type is 8-bit integer data, the data range is 0-255, and the data size is 1080*1920*3. After the decoding component obtains the video file, it decodes it to obtain multiple video frame images. Specifically, the ffmpegAPI (Application Programming Interface) can be called to implement decoding, and then the distribution component distributes each video frame image decoded by the decoding component to the corresponding GPU in sequence. For example, in the case of distribution according to the polling rule, video frame images 1~m can be distributed to GPU1~m in sequence, and then video frame images m+1~2m video frame images can continue to be distributed to GPU1~m in sequence, which will not be repeated. For any GPU, each intelligent algorithm component deployed therein processes multiple video frame images received by the core in sequence. In this way, each GPU can process the video frame images received by itself in parallel, and each intelligent algorithm component deployed in any GPU processes each video frame image received by the GPU in sequence. It can be seen that this method can use the parallel processing capability of the GPU to run multiple pipelines concurrently on the basis of multiple target processing components processing video frame images in sequence, thereby further improving the execution efficiency of the data processing system for the tasks to be processed.

Among them, the intelligent algorithm component can be implemented based on the CUDA framework and the TensorRT reasoning engine (a deep learning reasoning SDK) to make full use of the computing resources of the GPU where the intelligent algorithm component is located. In the case where the intelligent algorithm component includes multiple sub-algorithm components, each sub-algorithm component can process the video frame image separately by sharing the GPU video memory, so that there is no need to transmit data between the TensorRT reasoning engines where each sub-algorithm component is located, thereby reducing the redundancy and performance overhead caused by data transmission. In the process of processing any video frame image by the intelligent algorithm component in any GPU, the video frame image (that is, the data of the image) is transmitted from the host end of the GPU to the device end, and can be transmitted once, and the algorithm logic of each sub-algorithm component can be implemented on the device end, and the final processed data (such as the processed image or the processed data) can be transmitted from the device end to the host end for use by the encoding unit.

The specific processing process may include: the host side copies the original data of any video frame image to the device side, and the CUDA framework performs pre-processing such as normalization and data type conversion on the original data. Among them, the above type conversion may include converting uint8 data in the range of 0-255 to float32 data in the range of 0-1. Then call the TensorRT inference engine of the denoising component to perform model inference, that is, perform denoising on the original data after pre-processing, and pass the storage address of the denoised data to the TensorRT inference engine of the super-resolution component for model inference, that is, perform super-resolution processing on the denoised data, and pass the storage address of the obtained super-resolution data to the HDR component for model inference, that is, perform HDR processing on the super-resolution data, and the final processed data constitutes the processed image. Among them, the above storage address can be the storage address of the corresponding data in the video memory (Video Memory, graphics card memory) of the GPU.

Thereafter, the CUDA framework may also perform post-processing on the processed image, such as data clipping, data mapping and/or data type conversion. Data clipping may include: truncating data beyond the range of 0-1, replacing data less than 0 with 0, and replacing data greater than 1 with 1. Data mapping may include: mapping data in the range of 0-1 to 0-255 (or 0-65535), the mapping range depends on whether HDR output is performed, if not, it is mapped to 0-255, and if so, it is mapped to 0-65535. The data type conversion may be the opposite of the data type conversion process in the pre-processing. For example, when converting uint8 data in the range of 0-255 to float32 data in the range of 0-1 during pre-processing, the float32 data may be converted to uint8 data during post-processing to ensure that the data format and precision of the GPU input data and output data remain consistent.

In the above process, for the processed video frame image, the original data of the decoded video frame image can be 8-bit integer RGB24 data, which can be stored in the memory of the image processing device and transmitted to any GPU video memory. Thereafter, the CUDA framework can be called to convert and normalize the above 8-bit integer data into 32-bit floating point data, and the data range can be 0-1. After being processed by the denoising component in the GPU, the output denoised data can be maintained as 32-bit floating point data (range 0-1), the storage location is the GPU video memory, and the data size is 1080*1920*3. The denoised data can be further passed to the super-resolution component, and the super-resolution data processed by the component can be maintained as 32-bit floating point data (range 0-1), and the corresponding data size can be 1080*2*1920*2*3. The super-resolution data can be further passed to the HDR component, and the HDR data processed by the component can continue to be 32-bit floating point data (range 0-1), and the output size is 1080*2*1920*2*3. Thereafter, the CUDA framework can be called to post-process the HDR data, such as performing data range mapping and data type conversion, so as to convert the data into a 16-bit integer with a range of 0-65535, and the data can be regarded as the data of the processed image.

The processed image can be passed to the CPU by the GPU, and the GPU combined component and the encoding component are respectively assembled and encoded. Among them, the encoding component can create an encoder and encapsulate the encoded file according to the encoder format (such as libx264, libx265, mpeg, etc.) and encoded pixel type (yuv420p, yuv444p, etc.) configured by the user, as well as the file encapsulation format (.mov, .mp4, etc.) and other information. For example, the above encoding can be implemented in a ffmpeg hard-coded manner (h264_nvenc, hevc_nvenc, etc.) to increase the encoding speed. In addition, when the GPU hardware does not support encoding, the encoding component can use soft editing to implement the above encoding processing. The specific process can be referred to the records in the relevant technology, which will not be repeated here.

Furthermore, the HDR component deployed in each GPU (i.e., the last intelligent algorithm component) can pass the processed images to the collection component in sequence. The collection component can send the processed images (which can be reorganized in a disordered order) respectively passed by the HDR components in each GPU to the encoding component. It can be understood that because the interpolation component is used, the number of processed images passed by the HDR component in any GPU to the collection component may be greater than the number of video frame images distributed to the GPU by the distribution component; accordingly, the number of processed images received by the collection component may also be greater than the number of video frame images distributed by the GPU distribution unit. Of course, the number of processed images used by the encoding component for encoding processing is also greater than the number of video frame images decoded by the decoding component. Accordingly, the encoding component can encode the multiple processed images received to obtain the processed images. For example, when the target video is a video file with a resolution of 1080P, the processed image finally output can be a 4K HDR video file.

As shown in Figure 10, the management component in the data processing system is the algorithm service, and the target processing component is the corresponding decoding task, intelligent algorithm task and encoding task. For the data processing system, a video processing algorithm interface can be provided to the user. The interface can be generated based on the polymorphic specification of C++, and multi-threading technology can be used to allow users to fully schedule the hardware resources of the video processing device, so that the algorithm task can process the video efficiently with high concurrency. Among them, the decoding task is to decode the input audio and video files to obtain the media data (including video frames and audio) of the audio and video files. AI model processing tasks include but are not limited to video noise reduction, super resolution, frame rate improvement, HDR and other ultra-high-definition video processing algorithm functions. Among them, the AI model information corresponding to the AI model processing task can be in enumeration form, and the corresponding enumeration members can be seen in Table 13.

Table 13

In addition, the encoding task is to encode the media data processed by the algorithm task and output a video file or a video data stream. The encapsulation type information of the video file can be in enumeration form, and the corresponding enumeration members can be seen in Table 14.

Table 14

Wherein, the video processing algorithm interface may include an algorithm service interface and an algorithm task interface. Wherein, the algorithm service interface is used for task management and service calls between the video processing system and the algorithm task. The interface corresponds to the algorithm service shown in FIG10. The user can initiate configuration instructions for the data processing system to the algorithm service through the interface, so as to build a data processing system, specify the processing order of each target processing component, or specify the specific parameters of any processing component. The algorithm task interface can be used to call specific algorithm tasks, such as decoding components, intelligent algorithm components, encoding components, etc. The interface corresponds to the intelligent algorithm task shown in FIG10. The user can specify a specific intelligent algorithm task to the management component through the interface so as to load it into the corresponding pipeline.

Among them, the configuration information of the algorithm task can be in the form of a string, and the corresponding algorithm task configuration information can be seen in Table 15.

Table 15

In addition, the function list of the algorithm task interface can be found in Table 16.

Table 16

CZ2214464-PX2231246JZCN

During the construction phase of the above data processing system, the user can specify the various AI video processing tasks that need to be loaded through the algorithm service interface. Thereafter, the AI video processing task can be run by calling the interface function of the AI model processing task. For example, its interface function is as follows:

Model initialization function, used to initialize the AI inference engine. Parameters are shown in Table 28. Function return value: If the call is successful, it returns 0; if the call fails, it returns non-zero.

Data pre-processing function, used to convert the integer data into floating-point data required by the model and pass it to the GPU. Function return value: 0 if the call is successful; non-zero if the call fails.

Data processing function, used to process data. Function return value: 0 if the call is successful; non-zero if the call fails.

Data post-processing function, used to convert floating-point data into integer data and pass GPU data to host memory. Function return value: 0 if the call is successful; non-zero if the call fails.

Model resource release function, used to release the AI inference engine and other resources. Function return value: 0 if the call is successful; non-zero if the call fails.

AI model processing task initialization function, used to initialize AI model processing tasks. Function return value: 0 if the call is successful; non-zero if the call fails.

The AI model processing task data processing function is used to start the AI model processing task for data processing and transfer data with other algorithm tasks. Function return value: If the call is successful, it returns 0; if the call fails, it returns non-zero.

AI model processing task resource release function, the purpose is to notify the algorithm service that the task has been completed and release resources. Function return value: if the call is successful, it returns 0; if the call fails, it returns non-zero.

After the data processing system is started, the user can specify the video to be processed to the data processing system through the aforementioned algorithm task interface.

The following predefined functions of the video decoding task can be called through the video decoding task interface to implement the corresponding functions. In the case of inputting the target video through the path indication, the decoding component can call:

The video file function is used to open the video according to the video file path and obtain the decoding environment information. The video frame acquisition function is used to decode and obtain the video frame sequence of the target video. The video frame rate acquisition function is used to obtain the frame rate of the target video. The video encoding information acquisition function is used to obtain the encoding environment information of the target video. The video width acquisition function is used to obtain the width of the target video. The video height acquisition function is used to obtain the input video height. The decoding task initialization function is used to initialize the decoding task. The decoding task data processing function is used to understand the decoding of each frame of data and pass the data to other algorithm tasks. The decoding task resource release function is used to release the processing resources allocated to the algorithm task. The parameter list of the above functions can be found in Table 17.

Table 17

Assuming that the video to be processed is a 2K video, the video processing system can upsample the video by 2 times to process it into a 4K HDR video. Specifically, by executing the decoding task, the video to be processed can be decoded into multiple video frame images. Furthermore, by executing the AI model processing task, the above-mentioned video frame images can be processed in sequence such as noise reduction, super-resolution, interpolation, HDR up-conversion, etc. Finally, by executing the encoding task, the processed video frame images can be encoded and packaged into files to obtain and output the processed video.

The encoding component may call a video file creation function to initialize the audio and video encoder and create a video file based on the processing results of each algorithm task pair. The return value of the function: if the call is successful, it returns 0; if the call fails, it returns non-zero. The parameter list of the function can be found in Table 18.

Table 18

parameter Input/Output Type Parameter Description Video stream encoding format type Input Parameters / Video stream pixel format Input Parameters / HDR Standard Type Input Parameters / Audio stream encoding format type Input Parameters / Video file packaging type Input Parameters / Video file path Output Parameters Output video file path and file name

In addition, the encoding component can also call the video encoding function to encode each processed image frame, and the return value of the function is: if the call is successful, it returns 0; if the call fails, it returns non-zero. And, call the audio encoding function to encode the audio frame corresponding to each processed image frame. The parameter list of the video encoding function and the audio encoding function can be found in Table 19.

Table 19

parameter Input/Output Type Parameter Description Memory Data Input Parameters Memory data of video stream Memory Data Input Parameters Memory data of audio stream

In addition, the encoding component can also call the following functions to implement corresponding functions:

Set the HDR parameter function, which is used for each image frame after encoding. Function return value: if the call is successful, it returns 0; if the call fails, it returns non-zero.

The write file tail function is used to write the output video file tail to ensure the integrity of the output video file. Function return value: if the call is successful, it returns 0; if the call fails, it returns non-zero.

The video encoding task initialization function is used to initialize the video encoding task. Function return value: if the call is successful, it returns 0; if the call fails, it returns non-zero.

The video encoding task data processing function is used to encode each image frame after processing and transfer data with other algorithm tasks. Function return value: if the call is successful, it returns 0; if the call fails, it returns non-zero.

The video encoding task resource release function is used to write the file tail and release the processing resources. Function return value: if the call is successful, it returns 0; if the call fails, it returns non-zero.

The audio encoding task initialization function is used to initialize the video encoding task. Function return value: if the call is successful, it returns 0; if the call fails, it returns non-zero.

The audio encoding task data processing function is used to encode each image frame after processing and transfer data with other algorithm tasks. Function return value: if the call is successful, it returns 0; if the call fails, it returns non-zero.

The audio encoding task resource release function is used to write the file tail and release the processing resources. Function return value: if the call is successful, it returns 0; if the call fails, it returns non-zero.

It is understandable that the processed video outputted in the above manner is a 4K HDR video, which can be decoded by a display terminal and further rendered for display. In addition to transmitting intermediate data for processing video frame images, the target processing components of each adjacent task can also transmit corresponding static metadata and/or dynamic metadata so that subsequent components can smoothly process the intermediate data.

In the above process, the algorithm service can manage the decoding tasks, AI model processing tasks and encoding tasks in a process-based manner, and perform high-concurrency scheduling of the data processing system for each task to maximize the utilization of resources and achieve the most efficient processing efficiency. For example, the algorithm service can manage and schedule algorithm tasks, control the transmission and synchronization of data between algorithm tasks, allocate resources to each algorithm task, monitor the task execution status of each algorithm task, and release the resources occupied by the algorithm task after the task is completed.

Through this solution, users can be provided with a calling interface for algorithm services and algorithm tasks, so that users can conveniently and efficiently call the corresponding algorithm functions through the above interface to achieve processing of target data. Obviously, this solution can simplify the user's configuration and calling operations, thereby further improving video processing efficiency.

Corresponding to the above-mentioned data processing method embodiment, the present invention also provides an embodiment of a data processing device. The device is applied to a data processing system, the data processing system includes a management component and a processing component, the device includes one or more processors, and the processor is configured to:

Determine a plurality of target processing components according to the tasks to be processed, wherein the tasks to be processed are used to process target data;

The management component loads the multiple target processing components;

The multiple target processing components that have completed loading process the data segments of the target data in sequence, wherein each target processing component processes different data segments at the same time, and at least one data segment is processed in sequence by each target processing component.

In one embodiment, the data processing system further comprises a source component, the source component comprising predefined rules, and the processor is further configured to:

The source component generates the target processing component based on the predefined rule.

In one embodiment, the source component is a base class, and the processor is further configured to:

The base class is derived to generate the target processing component.

In one embodiment, the processor is further configured to:

The management component adds the multiple target processing components to a component list, and starts each target processing component in the component list.

In one embodiment, the processor is further configured to:

The management component identifies whether the at least one target processing component is generated by the source component, and if so, adds the at least one target processing component to a component list.

In one embodiment, the processor is further configured to:

The management component obtains component information of each target processing component in the component list, and starts each target processing component based on the component information.

In one embodiment, the data processing system also includes a forward node and a backward node, the target processing component corresponds to one or more forward nodes, and the target processing component corresponds to one or more backward nodes; wherein the forward node is used to obtain forward data segments for the target processing component to read, and the backward node is used to obtain backward data segments processed by the target processing component.

In one embodiment, the processor is further configured to:

After acquiring the forward data fragment, the forward node is updated to a writable state;

The target processing component performs a data processing operation to obtain a corresponding backward data segment;

When the backward node is in a writable state, the backward node obtains the backward data segment;

After acquiring the backward data segment, the backward node is updated to a readable state.

In one embodiment, the target processing component includes a data waiting module, and the processor is further configured to:

In response to the target processing component acquiring the forward data segment when the forward node is in a readable state.

In one embodiment, the target processing component includes a data transfer module, and the processor is further configured to:

The backward data segment is transferred to the backward node when the backward node is in a writable state.

The forward node of the target processing component and the backward node of its previous target processing component are the same node.

In one embodiment, the processor is further configured to: after the management component completes loading the multiple target processing components,

In response to a load instruction for any other processing component, continue to load any other processing component; and/or,

In response to an uninstall instruction for any loaded target processing component, the any loaded target processing component is uninstalled.

In one embodiment, the processor is further configured to:

The management component allocates corresponding processing resources to each target processing component; and, after any target processing component completes processing the last data fragment of the target data, releases the processing resources allocated to any target processing component; or,

If an exception occurs when the multiple target processing components sequentially process the data segments of the target data, an exception reminder message is output, relevant information of the exception is recorded, and/or the processing process for the target data is stopped.

In one embodiment, the target data includes audio and/or video, and the multiple target processing components include an encoding component, an intelligent algorithm component, and a decoding component.

In one embodiment, when the target data includes a video, the intelligent algorithm component includes at least one of the following sub-algorithm components:

Interpolation component, noise reduction component, super resolution component, high dynamic range component, detail restoration component.

In one embodiment, the number of the intelligent algorithm components is multiple, the target processing component further includes a distribution component and a collection component, each data fragment in the target data has a corresponding sequence number, and the sequence number is used to characterize the order of each data fragment in the target data, and the processor is further configured to:

The distribution component distributes each data segment output by the decoding component to the corresponding intelligent algorithm component, so that each intelligent algorithm component processes the data segment received by itself, and transmits the processed data segment to the collection component;

The collection component receives the processed data fragments respectively transmitted by each intelligent algorithm component, and sends the received processed data fragments to the encoding component in sequence according to the serial number.

In one embodiment, the processor is further configured to:

A plurality of target processing components are determined according to the requirement information of the task to be processed, wherein the requirement information includes at least one of the following: meta information of the target data, configuration information of the initiator of the task to be processed, and default function information provided by the data processing system.

Corresponding to the above-mentioned embodiment of the video processing method, the present invention also provides an embodiment of a video processing device. The device is applied to a video processing system, the video processing system includes a management component and a processing component, the device includes one or more processors, and the processor is configured to:

Determine a target processing component according to a task to be processed, wherein the task to be processed is used to process a target video, and the target processing component includes a decoding component, an encoding component, and at least one intelligent algorithm component;

The management component loads the target processing component;

The loaded intelligent algorithm component processes the video frames of the target video in sequence, wherein each target processing component processes different video frames at the same time, and at least one video frame is processed in sequence by each target processing component.

In one embodiment, the video processing system also includes a forward node and a backward node, the target processing component corresponds to one or more forward nodes, and the target processing component corresponds to one or more backward nodes; wherein the forward node is used to obtain forward video frame information for the target processing component to read, and the backward node is used to obtain backward video frame information processed by the target processing component.

In one embodiment, the processor is further configured to:

The target processing component acquires forward video frame information when the forward node is in a readable state;

After acquiring the forward video frame information, the forward node is updated to a writable state;

The target processing component processes and performs data processing operations to obtain corresponding backward video frame information;

When the backward node is in a writable state, the backward node obtains the backward video frame information;

After acquiring the backward video frame information, the backward node is updated to a readable state.

In one embodiment, the intelligent algorithm component includes at least one of the following sub-algorithm components:

Interpolation component, noise reduction component, super resolution component, high dynamic range component, detail restoration component.

In one embodiment, the number of the intelligent algorithm components is multiple, the target processing component further includes a distribution component and a collection component, each video frame in the target video has a corresponding serial number, and the serial number is used to characterize the position of each video frame in the timeline of the target video, and the processor is further configured to:

The distribution component distributes each video frame output by the decoding component to the corresponding intelligent algorithm component, so that each intelligent algorithm component processes the video frame received by itself, and transmits the processed video frames to the collection component respectively;

The collection component receives the processed video frames respectively transmitted by each intelligent algorithm component, and sends each received processed video frame to the encoding component in sequence according to the sequence number.

An embodiment of the present invention further proposes an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the data processing method or data processing method described in any of the above embodiments.

An embodiment of the present invention further provides a non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method or steps in the data processing method described in any of the above embodiments.

Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the relevant method, and will not be elaborated here.

11 is a schematic block diagram of an apparatus 1100 for data storage or driving mode determination according to an embodiment of the present invention. For example, the apparatus 1100 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

11 , the device 1100 may include one or more of the following components: a processing component 1112, a memory 1104, an electrical first power component 1106, a multimedia component 1108, an audio component 1110, an input/output (I/O) interface 1112, a sensor component 1114, and a communication component 1116. It should be noted that the processing component 1112 here is a functional component of the device 1100, and is not any processing component or target processing component described in the foregoing embodiments of the present invention.

The processing component 1112 generally controls the overall operation of the device 1100, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1112 may include one or more processors 1120 to execute instructions to complete all or part of the steps of the above-mentioned data processing method or video processing method. In addition, the processing component 1112 may include one or more modules to facilitate the interaction between the processing component 1112 and other components. For example, the processing component 1112 may include a multimedia module to facilitate the interaction between the multimedia component 1108 and the processing component 1112.

The memory 1104 is configured to store various types of data to support operations on the device 1100. Examples of such data include instructions for any application or method operating on the device 1100, contact data, phone book data, messages, pictures, videos, etc. The memory 1104 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

The electrical first source component 1106 provides power to the various components of the device 1100. The electrical first source component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 1100.

The multimedia component 1108 includes a screen that provides an output interface between the device 1100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1108 includes a front camera and/or a rear camera. When the device 1100 is in an operating mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 1110 is configured to output and/or input audio signals. For example, the audio component 1110 includes a microphone (MIC), and when the device 1100 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1104 or sent via the communication component 1116. In some embodiments, the audio component 1110 also includes a speaker for outputting audio signals.

The I/O interface 1112 provides an interface between the processing component 1112 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 1114 includes one or more sensors for providing various aspects of the status assessment of the device 1100. For example, the sensor assembly 1114 can detect the open/closed state of the device 1100, the relative positioning of components, such as the display and keypad of the device 1100, the sensor assembly 1114 can also detect the position change of the device 1100 or a component of the device 1100, the presence or absence of user contact with the device 1100, the orientation or acceleration/deceleration of the device 1100, and the temperature change of the device 1100. The sensor assembly 1114 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1114 can also include an optical sensor, such as a CMOS or processing component D image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1114 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 1116 is configured to facilitate wired or wireless communication between the device 1100 and other devices. The device 1100 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 6G NR or a combination thereof. In an exemplary embodiment, the communication component 1116 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1116 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, the device 1100 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned data processing method or video processing method.

In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, and the above instructions can be executed by the processor 1120 of the device 1100 to complete the above data processing method or data processing method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the embodiments disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprises a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

The method and device provided in the embodiments of the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of the present invention should not be understood as a limitation on the present invention.

Claims (26)

1. A data processing method, applied to a data processing system, wherein the data processing system comprises a management component and a processing component, and the method comprises: Determine a plurality of target processing components according to the tasks to be processed, wherein the tasks to be processed are used to process target data; The management component loads the multiple target processing components; The multiple target processing components that have completed loading process the data segments of the target data in sequence, wherein each target processing component processes different data segments at the same time, and at least one data segment is processed in sequence by each target processing component. 2. The method according to claim 1, wherein the data processing system further comprises a source component, wherein the source component comprises a predefined rule, and the method further comprises: The source component generates the target processing component based on the predefined rule. 3. The method according to claim 2, wherein the source component is a base class, and the source component generates the target processing component based on the predefined rule, comprising: The base class is derived to generate the target processing component. 4. The method according to claim 2, wherein the management component loads the plurality of target processing components, comprising: The management component adds the multiple target processing components to a component list, and starts each target processing component in the component list. 5. The method according to claim 4, wherein the management component adds at least one target processing component among the plurality of target processing components to the component list, comprising: The management component identifies whether the at least one target processing component is generated by the source component, and if so, adds the at least one target processing component to a component list. 6. The method according to claim 4, wherein the management component starts each target processing component in the component list, comprising: The management component obtains component information of each target processing component in the component list, and starts each target processing component based on the component information. 7. According to the method according to claim 1, the data processing system also includes a forward node and a backward node, the target processing component corresponds to one or more forward nodes, and the target processing component corresponds to one or more backward nodes; wherein the forward node is used to obtain forward data segments for the target processing component to read, and the backward node is used to obtain backward data segments processed by the target processing component. 8. The method according to claim 7, wherein the plurality of target processing components that have completed loading process the data segments of the target data in sequence, comprising: The target processing component acquires the forward data segment when the forward node is in a readable state; After acquiring the forward data fragment, the forward node is updated to a writable state; The target processing component performs a data processing operation to obtain a corresponding backward data segment; When the backward node is in a writable state, the backward node obtains the backward data segment; After acquiring the backward data segment, the backward node is updated to a readable state. 9. The method according to claim 8, wherein the target processing component comprises a data waiting module, wherein the data waiting module is used to: In response to the target processing component acquiring the forward data segment when the forward node is in a readable state. 10. The method according to claim 8, wherein the target processing component comprises a data transfer module, wherein the data transfer module is configured to: The backward data segment is transferred to the backward node when the backward node is in a writable state. 11. The method according to claim 8, wherein the forward node of the target processing component and the backward node of its previous target processing component are the same node. 12. The method according to claim 1, further comprising: after the management component completes loading the plurality of target processing components, In response to a load instruction for any other processing component, continue to load any other processing component; and/or, In response to an uninstall instruction for any loaded target processing component, the any target processing component is uninstalled. 13. The method according to claim 1, further comprising: The management component allocates corresponding processing resources to each target processing component; and, after any target processing component completes processing the last data fragment of the target data, releases the processing resources allocated to any target processing component; or, If an exception occurs when the multiple target processing components sequentially process the data segments of the target data, an exception reminder message is output, relevant information of the exception is recorded, and/or the processing process for the target data is stopped. 14. The method according to claim 1, wherein the target data comprises audio and/or video, and the multiple target processing components comprise an encoding component, an intelligent algorithm component and a decoding component. 15. The method according to claim 14, wherein when the target data includes a video, the intelligent algorithm component includes at least one of the following sub-algorithm components: Interpolation component, noise reduction component, super resolution component, high dynamic range component, detail restoration component. 16. The method according to claim 14, wherein the number of the intelligent algorithm components is multiple, the target processing component further comprises a distribution component and a collection component, each data fragment in the target data has a corresponding sequence number, and the sequence number is used to characterize the order of each data fragment in the target data, and the method further comprises: The distribution component distributes each data segment output by the decoding component to the corresponding intelligent algorithm component, so that each intelligent algorithm component processes the data segment received by itself, and transmits the processed data segment to the collection component; The collection component receives the processed data fragments respectively transmitted by each intelligent algorithm component, and sends the received processed data fragments to the encoding component in sequence according to the serial number. 17. The method according to claim 1, wherein determining a plurality of target processing components according to the tasks to be processed comprises: A plurality of target processing components are determined according to the requirement information of the task to be processed, wherein the requirement information includes at least one of the following: meta information of the target data, configuration information of the initiator of the task to be processed, and default function information provided by the data processing system. 18. A video processing method, applied to a video processing system, the video processing system comprising a management component and a processing component, the method comprising: Determine a target processing component according to a task to be processed, wherein the task to be processed is used to process a target video, and the target processing component includes a decoding component, an encoding component, and at least one intelligent algorithm component; The management component loads the target processing component; The loaded intelligent algorithm component processes the video frames of the target video in sequence, wherein each target processing component processes different video frames at the same time, and at least one video frame is processed in sequence by each target processing component. 19. According to the method of claim 18, the video processing system further includes a forward node and a backward node, the target processing component corresponds to one or more forward nodes, and the target processing component corresponds to one or more backward nodes; wherein the forward node is used to obtain forward video frame information for the target processing component to read, and the backward node is used to obtain backward video frame information processed by the target processing component. 20. The method according to claim 19, wherein the loaded intelligent algorithm component processes the video frames of the target video in sequence, comprising: The target processing component acquires forward video frame information when the forward node is in a readable state; After acquiring the forward video frame information, the forward node is updated to a writable state; The target processing component processes and performs data processing operations to obtain corresponding backward video frame information; When the backward node is in a writable state, the backward node obtains the backward video frame information; After acquiring the backward video frame information, the backward node is updated to a readable state. 21. The method according to claim 18, wherein the intelligent algorithm component comprises at least one of the following sub-algorithm components: Interpolation component, noise reduction component, super resolution component, high dynamic range component, detail restoration component. 22. The method according to claim 18, wherein the number of the intelligent algorithm components is multiple, the target processing component further comprises a distribution component and a collection component, each video frame in the target video has a corresponding sequence number, and the sequence number is used to characterize the position of each video frame in the timeline of the target video, and the method further comprises: The distribution component distributes each video frame output by the decoding component to the corresponding intelligent algorithm component, so that each intelligent algorithm component processes the video frame received by itself, and transmits the processed video frames to the collection component respectively; The collection component receives the processed video frames respectively transmitted by each intelligent algorithm component, and sends the received processed video frames to the encoding component in sequence according to the serial number. 23. A video processing device, the device being applied to a data processing system, the data processing system comprising a management component and a processing component, the device comprising one or more processors, the processors being configured to: Determine a plurality of target processing components according to the tasks to be processed, wherein the tasks to be processed are used to process target data; The management component loads the multiple target processing components; The multiple target processing components that have completed loading process the data segments of the target data in sequence, wherein each target processing component processes different data segments at the same time, and at least one data segment is processed in sequence by each target processing component. 24. A video processing device, the device being applied to a video processing system, the video processing system comprising a management component and a processing component, the device comprising one or more processors, the processors being configured to: Determine a target processing component according to a task to be processed, wherein the task to be processed is used to process a target video, and the target processing component includes a decoding component, an encoding component, and at least one intelligent algorithm component; The management component loads the target processing component; The loaded intelligent algorithm component processes the video frames of the target video in sequence, wherein each target processing component processes different video frames at the same time, and at least one video frame is processed in sequence by each target processing component. 25. An electronic device, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to implement the data processing method described in any one of claims 1 to 17 or the video processing method described in any one of claims 18 to 22. 26. A non-volatile computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the steps of the data processing method described in any one of claims 1 to 17 or the video processing method described in any one of claims 18 to 22.