CN113938634B - Multi-channel video call processing method and display device - Google Patents

Abstract

The present invention discloses a multi-channel video call processing method and display device, the method comprising: obtaining the current call number of a video call from a server; in response to determining that the current call number is greater than a preset number, calling an interface layout template corresponding to the preset number, and displaying a preset number of video windows in a first area on the video call interface; and displaying a number of voice windows equal to the difference between the current call number and the preset number in a second area; in response to determining that the current call number is less than or equal to the preset number, calling an interface layout template corresponding to the current call number, and displaying the current call number of video windows on the video call interface without displaying a voice window. This application makes multi-channel video calls no longer limited by the configuration and model of the display device, and improves the user's video call application experience.

Description

Multi-channel video call processing method and display device

Technical Field

The present invention relates to the technical field of display devices, and in particular to a multi-channel video call processing method and a display device.

Background technique

The display device can be installed with a video call application. When a video call is established, multiple members can join the corresponding virtual room. This requires multiple video windows to be displayed on the call interface, thus forming a multi-way video call. However, due to the configuration of the display device, especially for some older models, the number of video calls that can be supported is generally low. When the number of members joining the virtual room is large, it cannot meet the needs of multi-way video calls.

Summary of the invention

In order to solve the above technical problems, the present invention provides a multi-channel video call processing method and a display device.

In a first aspect, the present invention provides a display device, comprising:

monitor;

Sound player;

A communicator, used to enable the display device to communicate with the server;

A user interface is used to receive user input operations;

A controller connected to the display, the sound player, the communicator and the user interface respectively, for executing:

Get the current call number of the video call from the server;

In response to determining that the current call number is greater than the preset number, calling the interface layout template corresponding to the preset number to control the display to display the preset number of video windows in the first area of the video call interface; and displaying the voice windows in the second area equal to the difference between the current call number and the preset number;

In response to determining that the current call number is less than or equal to the preset number, the interface layout template corresponding to the current call number is called to control the display to display video windows of the current call number on the video call interface without displaying the voice window.

In a second aspect, the present invention provides a method for processing a multi-channel video call, comprising:

Get the current call number of the video call from the server;

In response to determining that the current call number is greater than the preset number, calling the interface layout template corresponding to the preset number, displaying the preset number of video windows in the first area of the video call interface; and displaying the voice windows in the second area, the number of which is the difference between the current call number and the preset number;

In response to determining that the current call number is less than or equal to the preset number, the interface layout template corresponding to the current call number is called, and video windows of the current call number are displayed on the video call interface, and the voice window is not displayed.

In the technical solution provided by the present application, when initiating a video call, the server can establish a virtual room, and the display device obtains the current call number from the server, that is, the number of call members currently connected in the virtual room. By comparing the current call number with the preset number of the display device, the display mode of the video call interface can be determined, wherein the preset number is the number of video call channels that the display device itself can support, and the preset number depends on the following factors such as the configuration and model of the device. When the current call number is less than or equal to the preset number, the current call number of video windows can be displayed according to the interface layout template. At this time, the call members who join the virtual room are all in video access mode, and the video window of each member can be displayed on the video call interface.

When the current number of calls is greater than the preset number, it is obvious that the display device cannot support video calls with the current number of calls. In this regard, the present application sets a first area and a second area on the video call interface. The upper limit of the video windows displayed in the first area is the preset number. Other calls exceeding the preset number can be accessed by voice, that is, automatically switched to voice windows in the second area. In this way, the number of voice windows in the second area is the difference between the current number of calls and the preset number. The total number of windows in the first area and the second area is the current number of calls. In this way, the local user can chat in voice mode with the call object in the second area, so that multi-channel video calls are no longer limited by the configuration and model of the display device, thereby improving the user's video call application experience.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram showing an operation scenario between a display device 200 and a control apparatus 100;

FIG. 2 exemplarily shows a hardware configuration block diagram of the display device 200 in FIG. 1 ;

FIG3 exemplarily shows a hardware configuration block diagram of the control device 100 in FIG1 ;

FIG4 exemplarily shows a schematic diagram of software configuration in the display device 200 in FIG1 ;

FIG5 exemplarily shows a schematic diagram of an icon control interface display of an application in a display device 200;

FIG6 a exemplarily shows a schematic diagram of an interface layout template;

FIG6b exemplarily shows a schematic diagram of a video call interface when the current call number and the preset call number are 3;

FIG6c exemplarily shows a schematic diagram of a video call interface when the current call number is 4 and the preset call number is 3;

FIG. 7 exemplarily shows a schematic diagram of a video call interface when the actual number of channels is 6 and the preset number of channels is 3;

FIG8 is a schematic diagram showing an example of a prompt pop-up window displayed when a video call is started;

FIG9 exemplarily shows a schematic diagram of a video call interface when the control list is opened;

FIG10 exemplarily shows a schematic diagram of a video call interface when the first object is selected;

FIG11 exemplarily shows a schematic diagram of a video call interface when a pop-up window for selecting a second object is displayed;

FIG12 exemplarily shows a schematic diagram of the interface when the audio and video window switching is completed;

FIG13 exemplarily shows a flow chart of a method for processing switching audio and video windows in a multi-channel video call;

FIG14 exemplarily shows a processing logic diagram when inviting a third party to join a video call;

FIG. 15 exemplarily shows a processing logic diagram when the call partner hangs up a video call.

Detailed ways

In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this application.

In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components explicitly listed but may include other components not explicitly listed or inherent to such products or devices.

The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and/or software code that is capable of performing the functions associated with that element.

The term "remote control" as used in this application refers to a component of an electronic device (such as the display device disclosed in this application), which can usually wirelessly control the electronic device within a short range. It is generally connected to the electronic device using infrared and/or radio frequency (RF) signals and/or Bluetooth, and may also include functional modules such as WiFi, wireless USB, Bluetooth, motion sensors, etc. For example: a handheld touch remote control replaces most of the physical built-in hard keys in a general remote control device with a user interface in a touch screen.

The term "gesture" used in this application refers to a user's behavior of expressing an intended idea, action, purpose, or result through a change in hand shape or hand movement.

Fig. 1 is a schematic diagram showing an operation scenario between a display device and a control apparatus according to an embodiment. As shown in Fig. 1 , a user can operate a display device 200 through a mobile terminal 300 and a control apparatus 100 .

In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-range communication methods, etc., to control the display device 200 wirelessly or in a wired manner. The user may input user commands through buttons on the remote controller, voice input, control panel input, etc. to control the display device 200. For example, the user may input corresponding control commands through the volume plus and minus keys, channel control keys, up/down/left/right movement keys, voice input keys, menu keys, power on/off keys, etc. on the remote controller to realize the function of controlling the display device 200.

In some embodiments, a mobile terminal, a tablet computer, a computer, a laptop computer, and other smart devices may also be used to control the display device 200. For example, an application running on the smart device is used to control the display device 200. The application may be configured to provide the user with various controls in an intuitive user interface (UI) on a screen associated with the smart device.

In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and realize connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. For example, it is possible to establish a control instruction protocol with the mobile terminal 300 and synchronize the remote control keyboard to the mobile terminal 300, and realize the function of controlling the display device 200 by controlling the user interface on the mobile terminal 300. It is also possible to transmit the audio and video content displayed on the mobile terminal 300 to the display device 200 to realize the synchronous display function.

As also shown in FIG. 1 , the display device 200 also communicates data with the server 400 through a variety of communication methods. The display device 200 may be allowed to communicate and connect via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200. For example, the display device 200 may receive software program updates or access a remotely stored digital media library by sending and receiving information and interacting with an electronic program guide (EPG). The server 400 may be a cluster or multiple clusters, and may include one or more types of servers. Other network service contents such as video on demand and advertising services are provided by the server 400.

The display device 200 may be a liquid crystal display, an OLED display, or a projection display device. The specific display device type, size, and resolution are not limited. It is understood by those skilled in the art that the display device 200 may be modified in performance and configuration as required.

In addition to providing the broadcast receiving television function, the display device 200 may also provide an intelligent network television function with computer support functions, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

FIG. 2 exemplarily shows a hardware configuration block diagram of a display device 200 according to an exemplary embodiment.

In some embodiments, the display device 200 includes at least one of a controller 250, a tuner and demodulator 210, a communicator 220, a detector 230, an input/output interface 255, a display 275, an audio output interface 285, a memory 260, a power supply 290, a user interface 265, and an external device interface 240.

In some embodiments, the display 275 is a component for receiving the image signal outputted from the first processor and displaying video content and images as well as a menu control interface.

In some embodiments, the display 275 includes a display component for presenting images and a driving component for driving image display.

In some embodiments, the displayed video content may be from broadcast television content, or may be various broadcast signals received through wired or wireless communication protocols, or may be various image content sent from a network server via a network communication protocol.

In some embodiments, the display 275 is used to present a user manipulation UI interface generated in the display device 200 and used to control the display device 200 .

In some embodiments, depending on the type of display 275, a driving component for driving the display is also included.

In some embodiments, the display 275 is a projection display, and may also include a projection device and a projection screen.

In some embodiments, the communicator 220 is a component for communicating with an external device or an external server according to various communication protocol types. For example, the communicator may include at least one of a Wifi chip, a Bluetooth communication protocol chip, a wired Ethernet communication protocol chip, other network communication protocol chips or a near field communication protocol chip, and an infrared receiver.

In some embodiments, the display device 200 may establish control signal and data signal transmission and reception with the external control apparatus 100 or the content providing apparatus through the communicator 220 .

In some embodiments, the user interface 265 may be used to receive infrared control signals from the control device 100 (eg, an infrared remote controller, etc.).

In some embodiments, the detector 230 is a signal used by the display device 200 to collect external environment or interact with the outside.

In some embodiments, the detector 230 includes a light receiver, a sensor for collecting ambient light intensity, and can adaptively display parameter changes by collecting ambient light.

In some embodiments, the detector 230 may also include an image collector, such as a camera, a camera, etc., which can be used to collect external environment scenes, as well as to collect user attributes or gestures for interacting with the user. It can adaptively change display parameters and recognize user gestures to achieve the function of interacting with the user.

In some embodiments, the detector 230 may also include a temperature sensor or the like, such as by sensing the ambient temperature.

In some embodiments, the display device 200 can adaptively adjust the display color temperature of the image. For example, when the temperature is high, the display device 200 can be adjusted to display a cooler color temperature of the image, or when the temperature is low, the display device 200 can be adjusted to display a warmer color temperature of the image.

In some embodiments, the detector 230 may also be a sound collector, such as a microphone, which can be used to receive the user's voice, for example, including a voice signal of a control instruction of the user to control the display device 200, or collect environmental sounds to identify the type of environmental scene, so that the display device 200 can adaptively adapt to the environmental noise.

In some embodiments, as shown in FIG2 , the input/output interface 255 is configured to perform data transmission between the controller 250 and other external devices or other controllers 250 , such as receiving video signal data and audio signal data, or command instruction data from external devices.

In some embodiments, the external device interface 240 may include, but is not limited to, any one or more interfaces such as a high-definition multimedia interface HDMI interface, an analog or digital high-definition component input interface, a composite video input interface, a USB input interface, an RGB port, etc. It is also possible that the above multiple interfaces form a composite input/output interface.

In some embodiments, as shown in FIG. 2 , the tuner-demodulator 210 is configured to receive broadcast television signals by wired or wireless reception, and may perform modulation and demodulation processing such as amplification, mixing, and resonance, and demodulate audio and video signals from multiple wireless or wired broadcast television signals. The audio and video signals may include television audio and video signals carried in the television channel frequency selected by the user, as well as EPG data signals.

In some embodiments, the frequency demodulated by the tuner-demodulator 210 is controlled by the controller 250, which can send a control signal based on the user's selection to make the modem respond to the TV signal frequency selected by the user and modulate and demodulate the TV signal carried by the frequency.

In some embodiments, the broadcast television signal can be classified into terrestrial broadcast signals, cable broadcast signals, satellite broadcast signals, or Internet broadcast signals according to different broadcasting formats of the television signal. Or it can be classified into digital modulation signals, analog modulation signals, etc. according to different modulation types. Or it can be classified into digital signals, analog signals, etc. according to different signal types.

In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box. In this way, the set-top box outputs the TV audio and video signals modulated and demodulated from the received broadcast TV signals to the main device, and the main device receives the audio and video signals through the first input/output interface.

In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 can control the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 275, the controller 250 can perform operations related to the object selected by the user command.

In some embodiments, the object may be any one of the selectable objects, such as a hyperlink or an icon. Operations related to the selected object may include, for example, displaying a page, document, image, etc. connected to the hyperlink, or executing a program corresponding to the icon. User commands for selecting UI objects may be commands inputted through various input devices (e.g., a mouse, keyboard, touchpad, etc.) connected to the display device 200 or voice commands corresponding to voices spoken by the user.

As shown in FIG2 , the controller 250 includes at least one of a random access memory 251 (Random Access Memory, RAM), a read-only memory 252 (Read-Only Memory, ROM), a video processor 270, an audio processor 280, other processors 253 (for example: a graphics processor (Graphics Processing Unit, GPU), a central processing unit 254 (Central Processing Unit, CPU), a communication interface (Communication Interface), and a communication bus 256 (Bus). The communication bus connects the various components.

In some embodiments, RAM 251 is used to store temporary data of an operating system or other running programs. In some embodiments, ROM 252 is used to store various system startup instructions.

In some embodiments, ROM 252 is used to store a basic input/output system, called Basic Input Output System (BIOS), which is used to complete the power-on self-test of the system, the initialization of each functional module in the system, the basic input/output driver of the system, and the booting operating system.

In some embodiments, when receiving the power-on signal, the display device 200 starts to power on, and the CPU runs the system startup instruction in the ROM 252, and copies the temporary data of the operating system stored in the memory to the RAM 251, so as to start or run the operating system. After the operating system is started, the CPU copies the temporary data of various application programs in the memory to the RAM 251, and then starts or runs various application programs.

In some embodiments, the CPU processor 254 is used to execute operating system and application instructions stored in the memory, and to execute various applications, data and content according to various interactive instructions received from external input, so as to finally display and play various audio and video contents.

In some exemplary embodiments, the CPU processor 254 may include multiple processors. The multiple processors may include a main processor and one or more sub-processors. The main processor is used to perform some operations of the display device 200 in the pre-power-on mode and/or the operation of displaying the picture in the normal mode. The one or more sub-processors are used for an operation in the standby mode and the like.

In some embodiments, the graphics processor 253 is used to generate various graphic objects, such as icons, operation menus, and user input command display graphics, etc. It includes an operator that performs operations by receiving various interactive instructions input by the user and displays various objects according to display attributes. It also includes a renderer that renders various objects obtained based on the operator, and the rendered objects are used to be displayed on the display.

In some embodiments, the video processor 270 is configured to receive an external video signal and perform video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc. according to the standard codec protocol of the input signal, and obtain a signal that can be directly displayed or played on the display device 200.

In some embodiments, the video processor 270 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, etc.

Among them, the demultiplexing module is used to demultiplex the input audio and video data stream. For example, if MPEG-2 is input, the demultiplexing module demultiplexes it into video signals and audio signals.

The video decoding module is used to process the demultiplexed video signal, including decoding and scaling.

The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator according to the user input or itself with the scaled video image to generate an image signal for display.

The frame rate conversion module is used to convert the input video frame rate, such as converting the 60Hz frame rate to 120Hz frame rate or 240Hz frame rate. The usual format is implemented by interpolation.

The display formatting module is used to receive the frame rate converted video output signal and change the signal to conform to the display format signal, such as outputting RGB data signal.

In some embodiments, the graphics processor 253 can be integrated with the video processor or can be separately configured. When integrated, it can perform processing of graphics signals output to the display. When separately configured, it can perform different functions respectively, such as a GPU+FRC (Frame Rate Conversion) architecture.

In some embodiments, the audio processor 280 is used to receive an external audio signal, and perform decompression and decoding, as well as noise reduction, digital-to-analog conversion, and amplification processing according to the standard codec protocol of the input signal to obtain a sound signal that can be played in a speaker.

In some embodiments, the video processor 270 may include one or more chips. The audio processor may also include one or more chips.

In some embodiments, the video processor 270 and the audio processor 280 may be separate chips, or may be integrated with the controller into one or more chips.

In some embodiments, the audio output receives the sound signal output by the audio processor 280 under the control of the controller 250, such as the speaker 286, and in addition to the speaker carried by the display device 200 itself, can be output to the external audio output terminal of the generating device of the external device, such as an external audio interface or a headphone interface, etc., and can also include a short-range communication module in the communication interface, for example, a Bluetooth module for Bluetooth speaker sound output.

The power supply 290, under the control of the controller 250, uses the power input from the external power supply to provide power supply support for the display device 200. The power supply 290 may include a built-in power supply circuit installed inside the display device 200, or may be an external power supply installed on the display device 200 to provide a power interface for an external power supply in the display device 200.

The user interface 265 is used to receive a user input signal, and then send the received user input signal to the controller 250. The user input signal may be a remote control signal received by an infrared receiver, and may receive various user control signals through a network communication module.

In some embodiments, the user inputs a user command through the control device 100 or the mobile terminal 300 , the user input interface is based on the user input, and the display device 200 responds to the user input through the controller 250 .

In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 275, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

In some embodiments, the "user interface" is a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display of an electronic device, where the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

The memory 260 includes storing various software modules for driving the display device 200. For example, the various software modules stored in the first memory include at least one of a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules.

The basic module is a bottom-level software module used for signal communication between various hardware in the display device 200 and sending processing and control signals to the upper-level modules. The detection module is a management module used to collect various information from various sensors or user input interfaces, and perform digital-to-analog conversion and analysis management.

For example, the speech recognition module includes a speech analysis module and a speech instruction database module. The display control module is a module used to control the display to display image content, and can be used to play multimedia image content and UI interface information. The communication module is a module used to perform control and data communication with external devices. The browser module is a module used to perform data communication between browsing servers. The service module is a module used to provide various services and various applications. At the same time, the memory 260 is also used to store and receive external data and user data, images of various items in various user interfaces, and visual effects of focus objects.

Fig. 3 exemplarily shows a block diagram of a configuration of a control device 100 according to an exemplary embodiment. As shown in Fig. 3 , the control device 100 includes a controller 110, a communication interface 130, a user input/output interface, a memory, and a power supply.

The control device 100 is configured to control the display device 200, and can receive the user's input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200. For example, the user operates the channel increase and decrease keys on the control device 100, and the display device 200 responds to the channel increase and decrease operations.

In some embodiments, the control device 100 may be a smart device, for example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

In some embodiments, as shown in FIG1 , the mobile terminal 300 or other intelligent electronic device can play a similar function to the control device 100 after installing the application for controlling the display device 200. For example, the user can install the application to realize the functions of the physical buttons of the control device 100 through various function keys or virtual buttons of the graphical user interface provided on the mobile terminal 300 or other intelligent electronic device.

The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller is used to control the operation and operation of the control device 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

The communication interface 130, under the control of the controller 110, realizes the communication of control signals and data signals with the display device 200. For example, the received user input signal is sent to the display device 200. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

The user input/output interface 140 includes at least one of other input interfaces such as a microphone 141, a touch panel 142, a sensor 143, and a button 144. For example, a user can input a user command through voice, touch, gesture, pressing, and other actions. The input interface converts the received analog signal into a digital signal, and the digital signal into a corresponding command signal, and sends it to the display device 200.

The output interface includes an interface for sending received user commands to the display device 200. In some embodiments, it can be an infrared interface or a radio frequency interface. For example, in the case of an infrared signal interface, the user input command needs to be converted into an infrared control signal according to an infrared control protocol, and then sent to the display device 200 via an infrared sending module. For another example, in the case of a radio frequency signal interface, the user input command needs to be converted into a digital signal, and then modulated according to a radio frequency control signal modulation protocol, and then sent to the display device 200 by a radio frequency sending terminal.

In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input/output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, which can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the display device 200.

The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller, and can be a battery and related control circuits.

In some embodiments, the system may include a kernel, a command parser (shell), a file system, and an application. The kernel, shell, and file system together form the basic operating system structure, which allows users to manage files, run programs, and use the system. After power-on, the kernel starts, activates the kernel space, abstracts the hardware, initializes hardware parameters, etc., runs and maintains virtual memory, schedulers, signals, and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. After startup, the application is compiled into machine code to form a process.

Referring to Figure 4, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

In some embodiments, at least one application is running in the application layer, and these applications may be window programs, system settings programs, clock programs, camera applications, etc. provided by the operating system; or they may be applications developed by third-party developers, such as hi-view programs, karaoke programs, magic mirror programs, etc. In specific implementation, the application packages in the application layer are not limited to the above examples, and may actually include other application packages, which are not limited in the embodiments of the present application.

The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

As shown in Figure 4, the application framework layer in the embodiment of the present application includes managers, content providers, and view systems, etc., wherein the manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit of the application (including switching the user interface currently displayed in the display window to the system desktop), opening, and backing (including switching the user interface currently displayed in the display window to the parent user interface of the currently displayed user interface), etc.

In some embodiments, the window manager is used to manage all window programs, such as obtaining the display size, determining whether there is a status bar, locking the screen, capturing the screen, controlling display window changes (for example, shrinking the display window, shaking the display, distorting the display, etc.), etc.

In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C/C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

In some embodiments, the kernel layer is a layer between hardware and software. As shown in FIG4 , the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, touch sensor, pressure sensor, etc.), etc.

In some embodiments, the core layer further includes a power driver module for performing power management.

In some embodiments, the software programs and/or modules corresponding to the software architecture in FIG. 4 are stored in the first memory or the second memory shown in FIG. 2 or FIG. 3 .

In some embodiments, taking the magic mirror application (photographing application) as an example, when the remote control receiving device receives the remote control input operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the input operation into a raw input event (including the value of the input operation, the timestamp of the input operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer, identifies the control corresponding to the input event according to the current position of the focus, and the input operation is a confirmation operation, and the control corresponding to the confirmation operation is the control of the magic mirror application icon. The magic mirror application calls the interface of the application framework layer to start the magic mirror application, and then starts the camera driver by calling the kernel layer to realize the capture of static images or videos through the camera.

In some embodiments, for a display device with a touch function, taking a split-screen operation as an example, the display device receives an input operation (such as a split-screen operation) performed by a user on the display, and the kernel layer can generate a corresponding input event according to the input operation, and report the event to the application framework layer. The activity manager of the application framework layer sets the window mode (such as a multi-window mode) and the window position and size corresponding to the input operation. The window management of the application framework layer draws the window according to the settings of the activity manager, and then sends the drawn window data to the display driver of the kernel layer, and the display driver displays the corresponding application interface in different display areas of the display.

In some embodiments, as shown in Figure 5, the application layer includes at least one application that can display a corresponding icon control in the display, such as: a live TV application icon control, a video on demand (Video On Demand, VOD) application icon control, a media center application icon control, an application center icon control, a game application icon control, etc.

In some embodiments, the live TV application can provide live TV through different signal sources. For example, the live TV application can use input from cable TV, wireless broadcast, satellite service, or other types of live TV services to provide TV signals. And, the live TV application can display video of the live TV signal on the display device 200.

In some embodiments, the video on demand application can provide videos from different storage sources. Unlike the live TV application, the video on demand provides video display from certain storage sources. For example, the video on demand can come from the server side of the cloud storage, from the local hard disk storage containing the stored video programs.

In some embodiments, the media center application can provide an application for playing various multimedia contents. For example, the media center can be a service provided by a user to access various images or audios through the media center application, which is different from live TV or video on demand.

In some embodiments, the application center can provide and store various applications. The application can be a game, an application, or some other application related to a computer system or other device but can be run on a smart TV. The application center can obtain these applications from different sources, store them in a local storage, and then run them on the display device 200.

The hardware configuration, software configuration and function implementation of the display device are introduced above. When the initiator initiates a video call, the server will establish a corresponding virtual room, and set up a sound collector and an image collector on the display device. The image collector is used to collect the video of the local user in real time during the video call, and the sound collector is used to collect the audio of the local user in real time during the video call. The audio and video data of the local user is then sent to the server. The audio and video data of each member in the virtual room can be uploaded to the server in this way, so that during the video call, the display device can obtain the audio/video data of other members from the server for display and playback on the local side.

After the video call is started, the virtual room will record the current call number, that is, the number of callers currently connected to the virtual room. As the number of call members increases (such as new calls, invitations to new members to join the video call, etc.) or decreases (such as a member hanging up the video call), the current call number recorded in the virtual room also needs to be updated accordingly.

In some embodiments, the display device needs to obtain the current call number from the server to compare the current call number with the preset number, so as to execute the corresponding multi-channel video call logic. The display device can send a query request to the server, and the server will query the current call number recorded in the virtual room in response to the query request, and send the current call number to the display device. Alternatively, if the server detects that the current call number recorded in the virtual room has changed, it needs to send the latest current call number to the display device, so as to synchronize the current call number recorded in the virtual room to the display devices of each call member.

Based on the configuration and model of the display device itself, different display devices have different preset numbers of channels. The preset number of channels is the number of video call channels that the display device can support, that is, the upper limit of the number of video windows that can be displayed on the video call interface. The preset number of channels is determined based on the processing power and experience value of the display device. The number of channels supported by each display device is fixed with the hardware model. In some embodiments, a display device with a higher configuration can support up to 9 video calls. When the display device supports 2-way video calls, it only supports one-to-one video calls. When the display device supports 2-9-way video calls, many-to-many video calls can be realized.

The display device generally has an interface layout template, which is used to set the size and position of each video call window when making a call. When the current call route is less than or equal to the preset route, the interface layout template can be called to display the current call route video window. The interface layout template has a corresponding relationship with the route of the video call. In some embodiments, as shown in FIG6a, different interface layout templates corresponding to the routes are shown when the route is 2-9. When the route is 2, the video screen of the friend can be displayed in a full-screen window and the video screen of the local user can be displayed in a small window, or the size of the two video call windows can be the same; when the route is 3-9, the video windows can be arranged in an array. For example, if the preset route of a display device is 6, it can support 2-6 video calls. For example, if the current call route is 4, the video windows of 4 call members can be displayed according to the interface layout template with a route of 4, that is, the 2*2 window layout in FIG6a. The interface layout templates corresponding to the above-mentioned call routes are only a display layout setting for implementing the call method. Those skilled in the art can also give other designs by adjusting the size and position of the video window. The size and position setting of the display window does not affect the implementation of this solution.

However, in some embodiments, different display devices may support different numbers of call routes. When two display devices supporting different numbers of call routes start a call, or when a new invited party is added during a call and the number of call routes supported by the display device is exceeded, the display device supporting the lower number of call routes still needs to be retained in the call process, and the user cannot be forced to exit. The solution of the present application can allow the display device supporting the lower number of call routes to remain in the call process when the number of calls exceeds the preset number, thereby improving the user experience.

In this application, the basic processing logic of multi-channel video calls is: obtain the current call number of video calls from the server; in response to determining that the current call number is greater than the preset number, call the interface layout template corresponding to the preset number to control the display to display the preset number of video windows in the first area of the video call interface; and display the voice windows equal to the difference between the current call number and the preset number in the second area; in response to determining that the current call number is less than or equal to the preset number, call the interface layout template corresponding to the current call number to control the display to display the current call number of video windows on the video call interface, and do not display the voice window. The basic processing logic of multi-channel video calls is first described in detail below.

In some embodiments, when there are many members in the virtual room that joins the video call, the current call number may be greater than the preset number. For example, there are 10 members in the communication group A, and user B initiates a video chat in the communication group. When all the other 9 members answer the video call, the current call number is 10, but the preset number of the display device is 9, that is, the video call interface displays a maximum of 9 video windows, so there must be 1 member who cannot access the video. In this case, the member can access the video by voice, that is, the mode of combining video window + voice window is adopted, and 9 video windows and 1 voice window are displayed on the video call interface. For another example, when the display device A supporting two-way video is conducting a two-way video call with the display device B supporting three-way video, if any of the two parties in the call invites a third party, the display device A supporting two-way video can set the audio and video data of the third-party display device C that was connected last to the voice call by determining the access call time, so that the current call can continue, and the display device B supporting three-way video can call the interface layout template of the three-way video call to conduct the video call. For another example, when display device A supporting two-way video, display device B supporting three-way video, and display device C supporting four-way video are conducting a three-way video call, if any of the three parties in the call invites display device D, display device A supporting two-way video can set the audio and video data of display device D, which was connected last, to a voice call by determining the time of the call, so that the current call can continue; display device B supporting three-way video can set the audio and video data of display device D, which was connected last, to a voice call by determining the time of the call, so that the current call can continue; display device C supporting four-way video can use the interface layout template of the four-way video call to display the call window.

In some embodiments, the display device determines the audio and video data of the opposite display device for video display and the audio and video data of the opposite display device for audio display according to the access time of the opposite display device participating in the call. The opposite display device that accesses later only displays audio data when the preset number of routes is exceeded.

In some embodiments, the display device sets the audio and video data of the display device that performs video display and the audio and video data of the display device that performs only audio display according to the user's switching operation.

In some embodiments, as shown in FIG6b and FIG6c, a display device supporting 3-way video calls is used as an example to illustrate the interface implementation during the call, and the switching process of other channels is similar to this example. In the window displayed on the interface, user identifiers are added to the voice window and the video window to represent the different display devices corresponding to the audio and video streams. In some embodiments, the identifier can be generated based on the address book or notes of the device. In some embodiments, the identifier can be generated based on the device identifier or account identifier of the other end.

In some embodiments, the display device of user 1 is display device A, the display device of user 2 is display device B, the user's own display device is display device C, and the display device of user 4 is display device D.

In some embodiments, in the scenario shown in FIG. 6b, user 1, user 2 and themselves are making a video call, the server establishes a virtual room for the video call, and display device A, display device B and display device C respectively access the virtual room to upload the locally collected audio and video data to the server, and pull the audio and video data uploaded by the opposite display device according to the device ID and/or account ID. In some embodiments, the audio and video data of the opposite end can be distinguished by the server according to the device ID and/or account ID, and then the display device is notified to pull.

In some embodiments, in the scenario shown in Figure 6c, any one of the three parties previously participating in the call invites user 4 to join the call through the invitation control in the interface, or there are 4 call members joining the call initially, that is, the current call number is 4, but user 4 joins the call later than user 1, user 2 and himself, or, when joining the call through other automatic call joining means, the display device adds a floating layer (that is, the second area) to the original interface layout, and a voice window is set in the second area. In the voice window, the user 4's identifier can be displayed on the control according to the preset characters and the user identifier corresponding to the later joined pronunciation video data obtained from the server, as well as other prompt information.

In some embodiments, if user 1 exits the call, the video window in the display interface that originally displayed the audio and video data of user 1 can be used to display the audio and video data of user 4, and the second area containing the voice window of user 4 is cancelled.

In some embodiments, when the current number of call routes is greater than the preset number, that is, when a video call interface adopts a video window + voice window mode, the interface may mainly include a first area and a second area. The first area displays a preset number of video windows, and the second area displays voice windows whose number is the difference between the actual number and the preset number.

For example, the actual number of video calls is 6, and the preset number of display devices is 3. As shown in FIG7, since the preset number is 3, the first area includes 3 video windows, each of which displays the video screen of the corresponding member, and a voice control is also provided in the video window for playing the audio of the corresponding member. The user ID can also be marked in the video window to identify the call member corresponding to each video window. The user ID mentioned here can be a user name, account or remark name, etc. For example, the user IDs corresponding to the three video windows in FIG7 are user 1, user 2 and user 3 respectively (this is only exemplary and does not represent the actual user ID).

The second area in FIG. 7 includes three voice windows for playing the audio of the corresponding members. Since the real video screen of the friend cannot be seen in the voice window, a preset image can be displayed. The preset image here can be a unified fictional avatar, or an avatar set by the user in the video call application; the voice window can display prompt information such as "voice access" to prompt that the friend has accessed the voice and is having a voice call with the local user; the voice window can also identify the user ID to identify the call member corresponding to each voice window. The user ID here can be a user name, account or note name, etc. For example, the user IDs corresponding to the three voice windows in FIG. 7 are user 4, user 5 and oneself respectively. The first area and the second area have no intersection, overlap or occlusion in position.

In the video call interface, when the current number of call channels is greater than the preset number, the number of video windows in the first area is equal to the preset number, and the number of voice windows in the second area is not limited, mainly depending on the number of call channels connected to the virtual room during the call, the number of new members invited, and the number of voice calls hung up. Due to the limitations of the video call interface and voice window size in the second area, the voice window may not be fully displayed. For example, there are 6 members of the voice call, but the user can only see 4 at most in the second area. Then the user can swipe left and right in the second area to view the 2 voice windows hidden in the second area. When the current number of call channels is less than or equal to the preset number, the video window only needs to be displayed according to the interface layout template corresponding to the current number of call channels. In this case, the second area and voice window will not be displayed.

In actual applications, the video window and the voice window are relatively fixed. For example, if user 4 is in voice access, the local user can only have a voice chat with user 4 but not a video chat. Similarly, if user 1 is in video access, the local user can only have a video chat with user 1 but not a voice chat. The user can only see the call object corresponding to the video window in the first area, but cannot see the call object corresponding to the voice window in the second area. Since the user cannot see the video images of all members in the virtual room, there is a need to autonomously choose which member's video image to watch.

In this regard, in the mode where the video call interface is reflected as a video call in the first area + a voice call in the second area, this application provides a solution for switching audio and video windows during a video call. The following first introduces the processing logic of switching audio and video windows when a multi-channel video call is started and the corresponding UI change display, specifically taking the preset number of channels as 3 as an example.

In some embodiments, when a video call is started, it is first determined whether the current call number is greater than the preset number. If so, it means that the number of video channels participating in the call exceeds the upper limit of the number of video windows in the first area. It is necessary to first call the interface layout template corresponding to the maximum number (i.e., the preset number) in the call interface, and play the audio and video data of the connected display device on the video window of the call interface according to the time of call access. After the video windows of the call interface all display the audio and video data corresponding to the audio and video streams, the call members who join later can only switch to voice access, which is displayed through the voice window in the second area; if the current call number does not exceed the preset number, it means that the number of video channels participating in the call does not exceed the upper limit of the number of video windows in the first area, and the interface layout template corresponding to the current call number can be called in the call interface, and the audio and video data of the display device connected to the video call can be displayed in the video window in the first area. The call members who join later can also access the video until the number of video windows in the first area reaches the upper limit; if the actual number of video call channels is equal to the preset number, it means that the first area has just reached the upper limit after the local user accesses, and the call members who join later can only switch to voice access.

In some embodiments, when a new party joins the call, it is first determined whether the current number of call channels is greater than the preset number. If so, it means that the number of video windows in the first area has reached the upper limit. At this time and later, the call members who join can only switch to voice access and display through the voice window in the second area; if it is not greater than, it means that the first area is not full, and the call members who join later can also access the video, and the audio and video data of the display device after access are displayed by calling different interface layout templates until the number of video windows in the first area reaches the upper limit; if the current number of call channels is equal to the preset number, it means that the first area has just reached the upper limit of the display device on this side after access, so the newly joined call member can only switch to voice access.

In some embodiments, for example, the preset number of channels of the local display device is 3, but there are 5 members currently connected to the video call. When a new user accesses, the new user generally defaults to the voice access mode in the local display device, that is, a new voice window is added in the second area; the new access situation is shown in Figures 6b and 6c. After receiving the notification of new video access sent by the server, the corresponding display device C adds a second area and sets a voice window in the second area. The controller of the display device C pulls the audio and video data uploaded by the newly connected display device D in the virtual room of the server, and separates the audio data and video data of the received audio and video data. In some embodiments, the controller of the display device C only parses the separated audio data and sends it to the sound output device of the display device C for audio output, and does not parse the separated video data. In some embodiments, the controller of the display device C parses the separated audio data and the separated video data, but sends the parsed audio data to the sound output device of the display device C for audio output, and discards the parsed video data, thereby realizing the voice access of the display device D.

In some embodiments, the user ID of the voice window can be displayed according to the user ID carried in the notification of new video access sent by the server, or according to the user ID in the pulled audio and video data.

In some embodiments, in order to provide certain prompts and guidance for the user to switch audio and video windows, when the current call number is greater than the preset number, a prompt pop-up window can be displayed on the upper layer of the video call interface. The pop-up window can be displayed when the video call is started or during the call. For example, the preset number of channels of the display device is 3, and the current call number is 3. During the call, either party invites a new member to join the call. After the new member joins the virtual room, the current call number will exceed the preset number. After the display device receives the invitation success message, a prompt pop-up window can be popped up. For another example, the local device supports 4-way calls. Before the local user answers the 8-person video call, 5 people have already joined the virtual room. The local user is the 6th to join. When the local user joins, the preset number of channels supported by his device has been exceeded. A prompt pop-up window can be popped up after the local user successfully answers the call. When the current video call number is less than or equal to the preset number, the prompt pop-up window is not displayed.

In some embodiments, a display interface of a prompt pop-up window is shown in FIG8 , in which a prompt message such as “Dear, due to the limitations of your TV configuration, a video call with more than X parties joining will automatically switch to voice access” may be displayed, where X is the value of a preset number of channels. In this way, the local user can be informed that the number of channels after the user has connected or the new caller has connected has exceeded the preset number of channels, and members who connect to the virtual room after exceeding the preset number of channels will automatically switch to voice access to the video call.

Secondly, the prompt pop-up window can also display guidance information such as "Press the key to open the operation button - select 'Switch audio and video' to switch voice to video". The user can click the first control in the interface of Figure 7 to open the operation button. The first control can be the "down key" in Figure 7 or other forms, so as to expand the operation button. The operation buttons include but are not limited to microphone status controls, camera status controls, hang up controls, invitation controls, switch audio and video window controls and small window call controls, etc. The user clicks on the switch audio and video window control to switch the voice chat mode of a certain object to the video chat mode.

Operation controls such as "Don't remind me again" and "I understand" can also be set at the bottom of the prompt pop-up window. When the user clicks "I understand", the current prompt pop-up window can be destroyed. The next time the user starts a video call, if the number of routes exceeds the preset number, the prompt pop-up window will need to be started again; if the user clicks "Don't remind me again", the current prompt pop-up window can be destroyed, but the prompt pop-up window will only be displayed this time. In the future, even if the number of routes exceeds the preset number when connecting a call, the prompt pop-up window will no longer be displayed.

After the user clicks "I understand" or "Don't remind me again" on the prompt pop-up window, the prompt pop-up window is destroyed, and then the user can enter the video call interface as shown in Figure 7. After the user clicks the first control, the operation button can be opened, as shown in Figure 9. The operation buttons include but are not limited to microphone status control, camera status control, hang up control, invitation control, switch audio and video window control and small window call control. The microphone control is used to turn on or off the microphone when triggered; the camera control is used to turn on or off the camera when triggered; the hang up control is used to hang up the current video call when triggered; the invitation control is used to invite new members to join the current video call when triggered; the small window call control is used to switch the current full-screen video call interface to a small window when triggered, so as to support users to make video calls while watching other signal sources or applications, that is, to achieve watching and chatting. On the basis of these conventional operation controls, this solution adds a switch audio and video window control to realize the start of audio and video switching logic when triggered.

Since there are many operation buttons in the video call setting, they are displayed in the form of a control list. In Figure 9, the control list is displayed in the second area in a row-by-row floating layer, and the operation controls can be switched by sliding left and right. Of course, the display form of the control list is not limited in the specific implementation, such as display in columns or arrays, and the display position of the control list floating layer is also not limited. The user can input some operation instructions by operating a remote control or mouse and other devices, aiming the cursor at the corresponding control and clicking the OK key. In the specific implementation, when the cursor moves on the control list, different controls can be focused. For example, when the cursor moves to the switch audio and video window control, the switch audio and video window control obtains the focus and is relatively magnified to a certain extent, and displays tips information at the same time, such as "Click here to switch voice to video". When the user does not click on any operation control in the control list, the control list can also be hidden through certain operations. For example, the control list can be opened when the first control is clicked once, and the control list can be hidden when the first control is clicked a second time; or, the user can hide the control list by single-clicking/double-clicking at other locations on the video call interface except the control list floating layer; or, a time threshold can be set. When the display time of the control list exceeds the time threshold and the user does not click on any operation control, the control list can be automatically hidden. The hiding and displaying schemes of the control list can be flexibly set according to actual applications.

In response to the operation of switching audio and video windows, the target voice window corresponding to the first object is obtained from the second area, and the target video window corresponding to the second object is obtained from the first area. That is, after starting the audio and video switching logic, the two parties that need to be replaced are first determined.

Specifically, in response to the click operation on the first control, the control display is controlled to display a control list on the floating layer of the video call interface. When the current call number is greater than the preset number, in response to the user clicking the operation of switching the audio and video window control in the control list, the control list is automatically hidden, and the audio and video switching logic is started in the display device. The user can select the target voice window from the second area. The call member corresponding to the target voice window is the replacement object, that is, the first object. By replacing a member in the first area to switch to the video window, when reflected at the UI level, as shown in Figure 10, the user can move the focus left and right in the second area by clicking the direction key on the remote control, or moving the remote control/mouse, etc. When a certain voice window obtains the focus, the voice window is relatively enlarged to a certain extent, and tips information is displayed at the same time, such as "Press the confirmation key O to select the friend to switch to video". In Figure 10, the voice window corresponding to user 4 obtains the focus. If the user clicks the confirmation key, that is, confirms that the voice window corresponding to user 4 is the target voice window, the display device responds to the confirmation operation of the target voice window, and records the first object as the first user ID corresponding to the target voice window, that is, records the first object as user 4.

After the first object is confirmed, the user can select a target video window from the first area. The call member corresponding to the target video window is the replaced object, that is, the second object. The call is switched to the voice window by being replaced by the first object in the second area. In an implementation method of selecting the second object, the display device responds to the confirmation operation of the target voice window and can also control the display to display a second object selection pop-up window on the upper layer of the call interface. When reflected at the UI level, as shown in FIG11, the second object selection pop-up window is displayed on the upper layer of the video call interface. The second object selection pop-up window can display prompt information, such as "Please select the friend to be replaced with voice". The second object selection pop-up window displays the user ID corresponding to all the video windows included in the first area. In the examples of FIG7 to FIG10, the first area includes 3 video windows, corresponding to user 1, user 2 and user 3, respectively. Then user 1, user 2 and user 3 are displayed on the second object selection pop-up window, and the user ID and application avatar can be displayed for user selection.

The display device obtains the second user ID selected by the user in the second object selection pop-up window, records the second object as the second user ID, and the video window corresponding to the second user ID is the target video window. In Figure 11, user 1 is selected, that is, the second user ID is user 1, then the second object is recorded as user 1, which is equivalent to replacing the first object and the second object call windows. After the switching is completed, as shown in Figure 12, user 4 switches to the previous video window of user 1 (i.e., the target video window), and user 1 switches to the previous voice window of user 4 (i.e., the target voice window), so that user 4 switches from voice chat to video chat, and user 1 switches from video chat to voice chat. At the same time, information to prompt the local user that the switch is successful can be displayed on the interface, such as "video window switching successful", and the audio and video window switching process is completed.

The display device simultaneously pulls the audio and video data of user 4 from the virtual room of the server, and associates the audio and video data of user 4 with the target video window for playback. At the same time, the display device stops pulling the video data of user 1 from the virtual room, only pulls the audio data of user 1, and associates the audio data of user 1 with the target voice window for playback; or, the display device still pulls the audio and video data of user 1 from the virtual room at the same time, but only parses and outputs the audio data of user 1, but does not parse and output the video data of user 1, so that the voice access effect of user 1 can also be achieved.

Of course, the method of selecting the second object is not limited to the embodiment of the present application, and other implementation methods may be used, such as directly selecting the target video window in the first area in a similar manner to the selection method of the first object. Before the video call is hung up, the local user can choose to switch any voice-connected friend or himself to the video window mode at any time according to personal wishes. The user operation, UI display and audio and video switching logic during the switching can refer to the above description.

The display device provided in some embodiments may include the display devices in some of the above embodiments, or other display devices that can implement the present application. The display device includes: a display 275 for displaying the video call interface and the aforementioned UI, a sound player for playing the audio of each call member, a communicator 220 for connecting the display device 200 to the server 400, a user interface 265 for receiving user input operations, and a controller 250 for processing multiple video calls and switching audio and video windows. The sound player may be a speaker 286, or an external audio device, etc. As shown in FIG. 13 , the controller 250 is configured to execute the following multiple video call processing method:

Step S0, obtaining the current call number of the video call from the server.

The display device can send a query request to the server. In response to the query request, the server will query the current call number recorded in the virtual room and send the current call number to the display device. Alternatively, if the server detects that the current call number recorded in the virtual room has changed, it will send the latest current call number to the display device, thereby synchronizing the current call number recorded in the virtual room to the display devices of each call member.

In some embodiments, after the initiator initiates a video call, since the invited members respond and answer at different times, the current call number recorded in the virtual room continues to increase before all members answer the call. Every time a new member joins the virtual room, the current call number accumulates and increases by 1 until the number of callers specified by the initiator is finally reached, at which point the current call number will no longer change temporarily.

During a video call, since an invitation control is set in the control list, any party in the call can click the invitation control to invite new members to join the call. The new members mentioned here can be users who were not invited when the initiator initiated the call. For example, user A invited user B and user C to a video call, that is, a three-person call was initiated. During the call, user B invited user D to join the video call, then user D is the new member. After user D answers the invitation call, user D is connected to the virtual room, and the recorded current call number is increased by 1, that is, the current call number is changed to 4. Each time a new member is successfully invited, the current call number needs to be accumulated and increased by 1 once.

For another example, user A on the local end invites user B and user C to make a video call, that is, a three-person call is initiated. Before the video call ends, user C clicks the hang-up control and hangs up the video call. Then user C exits the virtual room, and the current call number recorded in the virtual room needs to be reduced by 1, that is, the current call number is changed to 2. Every time a member hangs up the call, the current call number needs to be reduced by 1 once. After user C hangs up the video call, user A or user B can also re-invite user C to join the video call again through the invitation control. In this case, the new member is included in the invitation list when the initiator initiates the call. Step S10, determine whether the current call number is greater than the preset number. If the judgment result is yes, execute steps S20 to S40; otherwise, if the judgment result is no, execute step S50.

In response to determining that the current number of call routes is greater than the preset number, in step S20, the interface layout template corresponding to the preset number is called to control the display to display a preset number of video windows in the first area of the video call interface; and to display a number of voice windows equal to the difference between the current number of call routes and the preset number in the second area.

When the actual number of video calls is greater than the preset number, the video call interface is presented as a mode combining video calls in the first area and voice calls in the second area, where the preset number is the number of video call channels that the display device can support, and the first area displays a maximum of the preset number of video windows, and the remaining channels can be switched to voice windows in the second area, that is, the sum of the number of windows in the first area and the second area is equal to the actual number of channels.

Each video window in the first area and each voice window in the second area may have a preset window sorting rule. Referring to the layout of Figure 6, the smaller the video window number, the higher the order, and the larger the video window number, the lower the order. The sorting rules may include multiple ones, such as random sorting, user-specified priority, or sorting by the time of joining the virtual room, etc.

In some embodiments, the controller is also used to execute: after the video call is initiated, the video windows are first displayed in the first area in order according to the time sequence in which the call members join the virtual room; in response to the total number of video windows in the first area reaching the preset number, the mode for subsequent call members to access the virtual room is switched to voice access, and the voice windows are displayed in the second area in order according to the time sequence in which they join the virtual room.

After the initiator initiates a video call, since the time for each call member to answer and respond is different, the time for each call member to join the virtual room is also different. For example, the preset number of channels of the local device is 3, and user 1 initiates a video call with 6 people. After that, the virtual room is joined in the order of user 2, user 3, user 4, user 5, and local user 6 (i.e., himself). Since user 1 is the initiator (i.e., the first person to join the virtual room), and then user 2 and user 3 join the virtual room in turn, the video call interface displayed by the local device is sorted in the first area according to user 1, user 2, and user 3, and the corresponding video windows are displayed; when user 3 accesses, the preset number of channels is reached, and user 4, who starts after user 3, automatically switches to voice access mode, and then user 5 and user 6 (himself) access by voice in turn, and the corresponding voice windows are displayed in the second area according to user 4, user 5, and himself, thereby presenting the sorting state as described in Figure 7.

In some embodiments, the local user hopes to be able to see his or her own video screen in the first area. For example, the sorting rules can be set to set the local user's video window at the last position in the first area and the initiator's video window at the first position in the first area. Other video windows in the first area and audio windows in the second area can be sorted according to the time sequence of connected calls.

Step S30, in response to the operation of switching the audio and video windows, obtaining a target voice window corresponding to the first object from the second area, and obtaining a target video window corresponding to the second object from the first area.

In some embodiments, referring to Figure 8, the controller is also used to execute: when a video call is started, if the current number of call routes is greater than the preset number, the display is controlled to display a prompt pop-up window; the prompt pop-up window is used to prompt the user to display the control list by clicking the first control, and to switch the voice call to a video call by clicking the audio and video window control, and at the same time prompt the user to execute the corresponding processing logic by clicking other operation controls; if the current number of call routes is less than or equal to the preset number, the prompt pop-up window is not displayed.

When the current number of call routes is greater than the preset number, a prompt pop-up window as shown in Figure 8 can be displayed, and after the user clicks "I understand" or "Don't prompt again", the prompt pop-up window is destroyed and the user returns to the video call interface. When the current number of call routes is less than or equal to the preset number, it means that the user is still supported to access the video call via video access, so there is no need to display the prompt pop-up window.

In some embodiments, referring to Figure 9, the controller is also used to execute: in response to a click operation on the first control, control the display to display a control list on the floating layer of the video call interface; wherein, the control list includes operation controls such as microphone status controls, camera status controls, hang up controls, invitation controls, switching audio and video window controls, and small window call controls.

In some embodiments, referring to Figures 10 and 11, for step S30, the controller is specifically used to execute: when the current number of calls is greater than the preset number, in response to a click operation on the switching audio and video window control, obtaining the target voice window selected by the user in the second area; in response to a confirmation operation on the target voice window, recording the first object as the first user ID corresponding to the target voice window, and controlling the display to display a second object selection pop-up window on the upper layer of the video call interface; the second object selection pop-up window displays the user IDs corresponding to all video windows included in the first area; obtaining the second user ID selected by the user in the second object selection pop-up window, and recording the second object as the second user ID.

After the control list is displayed, the user clicks on the control for switching audio and video windows, that is, inputs the operation of switching audio and video windows. According to the user's selection and confirmation operation in the second area, the controller can obtain and record the first object and its corresponding target voice window. For example, the target voice window in Figure 10 is the voice window with a sequence number of 1 in the second area, and the first user ID identified on the target voice window is user 4, that is, the first object is recorded as user 4; according to the user's selection operation on the second object selection pop-up window, the controller can obtain and record the second object and its corresponding target video window. For example, the target video window in Figure 10 is the video window with a sequence number of 1 in the first area, and the second user ID identified on the target video window is user 1, that is, the second object is recorded as user 1.

Step S40, associating the audio and video data of the first object with the target video window for playback, and associating the audio data of the second object with the target voice window for playback.

Referring to Figure 12, the audio and video data of user 4 are associated with the target video window for playback, so the local user can not only hear the voice of user 4, but also see the video screen of user 4 in the target video window; and as user 1 is the replaced object, the local user can no longer see the video screen of user 1, but can listen to the voice of user 1 in the target voice window, thereby switching user 4 from the original voice access to video access, and switching user 1 from the original video access to voice access, thereby realizing the switching of audio and video windows.

For the video windows of each call object in the first area, since it is necessary to play video and audio at the same time, it is necessary to obtain the audio and video data of each call object in the first area from the server, that is, to pull the video stream and audio stream from the server at the same time. For each voice window in the second area, since only audio needs to be played, two methods can be involved. The first is to pull only the audio stream but not the video stream; the second is to pull the video stream and audio stream at the same time, but not decode and render the video data, so that the unparsed video data cannot be played on the display, and the effect of separate voice access can be achieved. Among them, the call object is the other call members in the virtual room except the local user. This is because the local user can collect audio and video data through the local sound/image collector, so there is no need to pull the audio/video stream of the local user from the server, so what is pulled is the audio/video stream of the call object on the other end except the local user.

In some embodiments, for the first situation mentioned above, the controller is also used to execute: obtaining audio and video data of all call objects in the virtual room from the server; associating the audio and video data of the call objects in the first area to the corresponding video windows for playback; not parsing the video data of the call objects in the second area, but only associating the audio data of the call objects in the second area to the corresponding voice windows for playback; after responding to the operation of switching the audio and video windows, obtaining the audio and video data of the first object from the server, and stopping obtaining the video data of the second object from the server.

For the voice window in the second area, only the audio data of the call object is obtained, but not the video data. When the user chooses to switch the voice window of the first object to the video window mode, since the first object has been in the voice access state before, that is, it has been pulling the audio stream of the first object, it is only necessary to open the transmission channel of the video data of the first object to start pulling the video stream of the first object from the virtual room of the server, so that the controller can obtain the audio and video data of the first object and associate it with the target video window for playback. The second object is replaced with voice access, so it is only necessary to stop obtaining the video data of the second object from the server and keep obtaining the audio data of the second object from the server, that is, it is only necessary to close the video data transmission channel of the second object to stop pulling the video stream of the second object, and then associate the audio data of the second object with the target voice window for playback.

In some embodiments, for the second situation mentioned above, the controller is also used to execute: obtaining audio and video data of the call objects included in the first area from the server, and associating them with corresponding video windows for playback; obtaining only audio data of the call objects in the second area from the server without obtaining video data, and associating the audio data of the call objects in the second area with corresponding voice windows for playback; after responding to the operation of switching the audio and video windows, starting to parse the video data of the first object, and stopping to parse the video data of the second object.

In this embodiment, for the voice window in the second area, the video stream and audio stream are still pulled, but the video data is not parsed. When the user chooses to switch the voice window of the first object to the video window mode, the audio and video streams of the first object and the second object are kept pulled, and the parsing of the video data of the first object is started, so that the video of the first object can be played in the target video window after parsing; the video data of the second object needs to stop parsing, so that the video data of the second object that has not been parsed cannot be displayed on the display, and the audio data of the second object is associated with the target voice window for playback, thereby switching the second object to the voice access mode.

In some embodiments, associating audio data with voice window playback may mean: determining not to display the user's video on the display according to the user corresponding to the voice window, and only playing the user's audio data through the sound player. In some embodiments, the user's audio and video data may be pulled from the server/collected locally, but the user's video data is not parsed, the audio data is parsed, or only the audio data is pulled from the server without pulling the video data.

In some embodiments, associating audio data with voice window playback may also mean: determining not to display the user's video on the display based on the user corresponding to the voice window, only playing the user's audio data through a sound player, and loading a dynamic control in the voice window at the same time, the dynamic control is configured to dynamically display the dynamic control when the audio data of the user corresponding to the voice window meets preset conditions, and display a still image of the dynamic control when the user's audio data does not meet the preset conditions.

In some embodiments, the audio and video data of the first object is associated with the target video window for playing, which may refer to playing the video data in the audio and video data of the first object in the target video window and playing the audio data in the audio and video data of the first object through a sound player. In some embodiments, the audio and video data of the user may be pulled from a server/collected locally.

In some embodiments, the audio and video data of the first object is associated with the target video window for playing, which may also refer to playing the video data in the audio and video data of the first object in the target video window, and playing the audio data in the audio and video data of the first object through a sound player. At the same time, a dynamic control is loaded in the target video window, and the dynamic control is configured to dynamically display the dynamic control when the audio data of the user corresponding to the target video window meets a preset condition, and to display a still image of the dynamic control when the audio data of the user does not meet the preset condition.

In some embodiments, the purpose of the association is to enable the user to see which video window corresponds to which connected user, and which voice window corresponds to which connected user.

In some embodiments, a user identification control is provided on each window (video window or voice window), and the name displayed on the user identification control may be the ID of the user accessing the video call. In some embodiments, the user identification control may also be displayed in combination with an address book pre-stored in the display device. If the user accessing the video call has a name with a note in the address book, it will be displayed on the user identification control according to the name of the note. If not, the note customized by the other user will be displayed, and the note is uniquely corresponding to or identical to the user ID.

In some embodiments, the text displayed by the user identification control in the window representing the user itself is any one of "self", "myself", and "I", and is not displayed according to the user ID, or the remark name, or the custom remark, but is mapped to the user ID. The user identification control displays the name of the access device at the other end.

In some embodiments, there is a mapping relationship between the window serial number and the user ID. After the user pulls the audio and video stream from the server according to the user ID, the parsed video data is displayed in the corresponding video window according to the mapping relationship.

In some embodiments, the audio and video data that need to be parsed and that do not need to be parsed are determined based on the mapping relationship between the serial number of the video window and the user ID, or the audio and video data that need to be parsed and that do not need to be parsed are determined based on the mapping relationship between the serial number of the voice window and the user ID, so that the audio and video data mapped to the video window are parsed, and the audio and video data mapped to the voice window are not parsed.

In some embodiments, based on the mapping relationship between the serial number of the video window and the user ID, the audio data corresponding to the window is detected, and the dynamic control is controlled based on the audio data.

In response to determining that the current call number is less than or equal to the preset number, in step S50, the interface layout template corresponding to the current call number is called to control the display to display the current call number of video windows on the video call interface without displaying the voice window.

When the current call number is less than or equal to the preset number, the interface layout template shown in FIG. 6a can be referred to, and the corresponding interface layout template can be selected according to the current call number to display each video window, and the audio and video data of each passing object pulled from the server is associated with the corresponding video window for playback. Since the voice window is not displayed, there is no division between the first area and the second area on the video call interface at this time, or it can also be understood that there is only the first area but no second area.

The present application can improve the ability of a display device to handle multiple video calls. When the current number of calls is greater than the preset number, the free and flexible switching of audio and video windows can be achieved by adding a switching audio and video window control and corresponding audio and video switching logic in the control list, so that users can see their friends' video images according to their personal wishes, and multiple video calls are no longer limited by the configuration and model of the display device, thereby improving the user's video call application experience.

In the above embodiments of the present application, considering that the video screen of the member who has voice access cannot be seen, the first object is selected first from the perspective of switching the first object in the second area from audio to video, and then the second object whose video access is replaced by the first object is selected. In practical applications, for example, if the current video screen quality of user 1 in the first area is poor or a black screen is displayed, the local user may not want to watch the video screen of user 1. Therefore, it is also possible to switch user 1 to audio access from the perspective of switching a certain object in the first area from video to audio. After the user clicks the switch audio and video window control, user 1 can be selected as the replacement object in the first area first, and then the replaced user 4 can be selected in the second area, thereby switching user 1 from video to voice and user 4 from voice to video.

It should be noted that, although the display devices used by each call member in the virtual room can be configured with the multi-channel video call processing logic described in the above embodiments, due to the different configurations, performances and models of the display devices themselves, the UI and window layouts of the video call interfaces on these display devices may be different. For example, the actual number of channels of the current video call is 6, the preset number of channels of the display device of user 1 is 9, and the preset number of channels of the display device of user 2 is 3. Then, the video call interface watched by user 1 displays 6 video windows but no voice window, and the video call interface watched by user 2 displays 3 video windows and 3 voice windows. It can be seen that each display device processes the multi-channel video call logic based on its own configuration; in addition, each call member switches the audio and video windows according to his or her own viewing intention. The UI changes caused by the local user switching the audio and video windows are only displayed on the local device, and will not be synchronized to the display devices of other call members. Therefore, it will not affect other call members, and other call members cannot perceive the local user's operation of switching audio and video windows.

In summary, although the display device of each member in the virtual room is configured with the same multi-channel video call processing logic, the process of each device processing multi-channel video calls is independent, without influence or interference with each other. Each display device can adaptively process the multi-channel video calls on its own end according to the aforementioned processing logic based on factors such as its own configuration and the viewing intention of the user on this end.

In some embodiments, during a video call, the user can choose to invite new members to join the current call as desired. In this regard, as shown in Figure 14, the controller is also used to execute: in response to a click operation on the invitation control, obtaining a third user ID of a third object invited by the user; sending the third user ID to the server, so that the server sends an invitation message to the display device of the third object after receiving the third user ID.

After the user clicks the invitation control, he can select the third party to be invited in the contact list, or enter the account of the third party, the mobile phone number or email address bound to the video call application and other information to find the third party, so as to invite the third party to join the current video call. At this time, the controller can automatically obtain the third user ID of the third party, or the local user can manually enter the third user ID of the third party, so that the controller can obtain the third user ID. The controller sends the third user ID to the server, and the server receives the third user ID and sends the invitation information to the display device used by the third party according to the third user ID. The invitation information can carry the information of the inviter, the relevant information of the video call (such as including XX chat group, the members of the current call in the chat group) and other relevant contents; the display device of the third party can control the video call application to generate an incoming call interface and prompt tone in response to the invitation information. The incoming call interface can display "User A invites you to join XX chat group", and the incoming call interface can also be provided with answering control and cancel control. If the third object clicks the cancel control, the invitation fails and the video call maintains its current state; if the third object clicks the answer control, the third object answers the incoming video call and joins the virtual room, so that the invitation is successful; the server responds to the message that the third object has answered the call and the third object has been newly added to the virtual room, then the current call number is increased by 1 to obtain the updated current call number, and then an invitation success message is sent to the controller.

In some embodiments, as shown in FIG14 , the controller is further configured to execute: in response to receiving the invitation success information sent by the server, when the updated current call number is greater than the preset number, controlling the display to display a newly added voice window in the second area, obtaining the audio data of the third object from the server, and associating the audio data of the third object with the newly added voice window for playback. The indication information is sent by the server after the current call number recorded in the virtual room is increased by 1 when the display device of the third object responds to the invitation information and answers the incoming video call.

The controller receives and responds to the invitation success information, and needs to compare the updated current call number with the preset number. In this embodiment, the updated current call number is greater than the preset number, indicating that when the third object is not connected, the number of video windows in the first area has reached the upper limit. For example, the current call number is 3, and the preset number of the local device is 3, that is, the number of video windows in the first area is just the upper limit. At this time, the second area is not displayed. When the third object invites successfully, the updated current call number is changed to 4, then the local device needs to switch the third object to voice access, and add a voice window of the third object to display in the second area, only receive the audio data of the third object from the server, or receive the audio and video data of the third object, but do not parse and output the video data of the third object, and associate the audio data of the third object with the newly added voice window for playback, so that the local user can hear the voice message sent by the third object. For another example, the current number of call paths is 5, and the preset number of paths of the local device is 3, that is, the number of video windows in the first area has reached the upper limit, and the second area includes 2 voice windows. Since the third object is the last to access the virtual room, the voice window of the newly added third object can be sorted at the end of the second area for display, and the updated second area includes 3 voice windows. In this embodiment, the newly invited third object is voice access. If the local user needs to switch the third object from voice access to video access, the aforementioned audio and video switching logic can be used, which will not be repeated here.

In some embodiments, as shown in FIG14 , the controller is further configured to execute: in response to receiving the invitation success message sent by the server, when the updated current call number is less than or equal to the preset number, controlling the display to display the newly added video window in the first area, calling the interface layout template corresponding to the updated current call number and using the preset window sorting rules to refresh the video call interface. Acquire the audio and video data of the third object from the server, and associate the audio and video data of the third object with the newly added video window for playback; wherein the indication information is sent by the server after the current call number recorded in the virtual room is increased by 1 when the display device of the third object responds to the invitation message and answers the incoming video call.

The controller receives and responds to the invitation success information, and needs to compare the updated current call number with the preset number. In this embodiment, the updated current call number is less than or equal to the preset number, indicating that when the third object is not connected, the number of video windows in the first area has not reached the upper limit. For example, the current call number is 5, and the preset number of the local device is 6, that is, the number of video windows in the first area is 5. At this time, the second area is not displayed. When the third object is invited successfully, the updated current call number is changed to 6, that is, the number of video windows in the first area just reaches the upper limit after the third object is connected, then the local device can support the third object video access; for another example, the current call number is 6, and the preset number of the local device is 9. When the third object is invited successfully, the updated current call number is changed to 7, that is, the number of video windows in the first area has not reached the upper limit after inviting new members to join, so the local device supports the third object video access.

When the third object video is connected, it is necessary to add a new video window of the third object. At this time, since the number of video windows in the first area has increased, the corresponding window layout may need to be appropriately adjusted. For example, the preset number of channels of the local device is 9, and the original first area displays 8 video windows. After the third object is connected, the number increases to 9 video windows. Referring to Figure 6a, the interface layout template used in the first area needs to be changed from 2 rows and 4 columns to an array of 3 rows and 3 columns. Then, according to the preset window sorting rules, for example, according to the time sequence of the 9 call members in the virtual room accessing the call, the video windows of each member are sorted. Since the third object is the last to access the virtual room, the video window of the newly added third object can be sorted and displayed at the end of the first area. Refresh the video call interface to obtain the updated 3*3 interface after sorting. At the same time, the audio and video data of the third object are received from the server, and the audio and video data of the third object are associated with the newly added video window for playback. In this way, the local user can see the video screen of the third object and hear the voice message of the third object.

The above takes the example of inviting a new member to join the virtual room to illustrate the processing logic and UI adjustment when inviting new members. In some embodiments, the inviter can invite one or more new members to join the call at the same time. When inviting multiple new members at one time, since the time for each new member to respond and answer the invitation call may be inconsistent, it is equivalent to adding a new member at intervals until all invited new members are connected to the virtual room. Then, according to the order in which each new member answers the call, the changes in the current call route and the UI interface can be adaptively adjusted. The processing logic is basically similar to the aforementioned invitation logic and will not be repeated here.

In some embodiments, before the video call ends, a caller may click the hang-up control to exit the video call in advance, which may include but is not limited to the following situations: situation (A), the caller who hangs up is connected via video, and the current call number updated after hanging up is less than the preset number; situation (B), the caller who hangs up is connected via video, and the current call number updated after hanging up is equal to the preset number; situation (C), the caller who hangs up is connected via video, and the current call number updated after hanging up is greater than the preset number; situation (D), the caller who hangs up is connected via voice, and the current call number updated after hanging up is equal to the preset number; situation (E), the caller who hangs up is connected via voice, and the current call number updated after hanging up is greater than the preset number.

In some embodiments, as shown in FIG15 , for scenario (A), the controller is further used to execute: receiving a hang-up indication message sent by the server, wherein the hang-up indication message is sent by the server after the current call number recorded in the virtual room is reduced by 1 when the fourth object in the virtual room clicks the hang-up control; in response to the hang-up indication message, if the fourth object is a video access, controlling the display to display the hang-up prompt message of the fourth object and destroying the video window of the fourth object; when determining that the updated current call number is less than the preset number, calling the interface layout template corresponding to the updated current call number and using the preset window sorting rules to refresh the video call interface.

When any call object (i.e., the fourth object) in the virtual room other than the user at the local end clicks the hang-up control to exit the video call in advance, the server responds to the message that the fourth object has hung up, and learns that a user has exited the virtual room. The user is the fourth object, and then it is necessary to reduce the recorded current call number by 1 to obtain the updated current call number, and then send a hang-up indication message to the controller of the display device. The hang-up indication message may carry the fourth user ID of the fourth object, which is used to indicate to the controller that the call object corresponding to the ID has hung up the video call. Since the fourth object hangs up in the video access mode relative to the local end, the local end device cannot continue to pull the audio and video stream of the fourth object, so the video window of the fourth object cannot display a valid video screen, and the video window may be displayed in the form of a black screen or a gray screen. The controller receives and responds to the hang-up indication message, and controls the display to display a hang-up prompt message, such as "the fourth object has hung up the call", and the display position of the hang-up prompt message is not limited, such as it can be displayed on the black screen video window of the fourth object, and then it is necessary to destroy the video window of the fourth object so that the video window of the hang-up party is no longer displayed on the video call interface of the local end.

When the fourth object hangs up the call in the video access mode, the video window of the fourth object needs to be destroyed. At this time, since the number of video windows has been reduced, the corresponding window layout may need to be appropriately adjusted. If the updated current call number is less than the preset number, it means that the number of video windows when the fourth object is not hung up is less than or equal to the preset number. For example, the preset number of channels of the local device is 4, and the original video call interface displays 4 video windows. After the fourth object hangs up, the number is reduced to 3 video windows. Referring to Figure 6a, the interface layout template used by the video call interface needs to be changed from 2 rows and 2 columns to an array of 1 row and 3 columns, and then according to the preset window sorting rules, for example, according to the time order of the remaining 3 call members in the virtual room accessing the call, the 3 video windows in the new interface layout template are sorted to obtain the updated sorted 1*3 interface.

In some embodiments, if the video windows are sorted in order of access time when the fourth object is not hung up, then after the fourth object is hung up, the other video windows sorted after the fourth object can be raised by one position in sequence, and the sequence of the video windows sorted before the fourth object remains unchanged. For example, when the fourth object is not hung up, the video windows are sorted in order of access time as user 1, user 2, user 3 and user 4 (himself). When user 2 hangs up the video call, that is, user 2 is the fourth object, user 1 still remains in the first place in the new template, user 3 is raised to the second place in the new template, and user 4 is raised to the third place in the new template. After refreshing, the sequence is user 1, user 3 and user 4.

In some embodiments, as shown in FIG15 , for situation (B), the controller is further used to execute: receiving a hang-up indication message sent by the server, wherein the hang-up indication message is sent by the server after the current call number recorded in the virtual room is reduced by 1 when the fourth object in the virtual room clicks the hang-up control; in response to the hang-up indication message, if the fourth object is a video access, controlling the display to display the hang-up prompt message of the fourth object and destroying the video window of the fourth object; when it is determined that the updated current call number is equal to the preset number, obtaining the audio and video data of the fifth object in the second area from the server, canceling the display of the second area on the video call interface, and generating a new video window based on the audio and video data association of the fifth object; and sorting and displaying the video windows in the current video call interface according to the preset window sorting rules.

If the updated current call route is equal to the preset route, it means that the call route when the fourth object is not hung up is the preset route + 1, that is, the preset route video windows in the first area (including the fourth object) + 1 voice window in the second area, then the 1 video window in the second area corresponds to the fifth object, then after the fourth object hangs up, the fifth object can be automatically switched from voice access to video access, that is, the voice window of the fifth object is destroyed, the display of the second area is canceled on the video call interface, and the video stream of the fifth object is started to be pulled from the server, plus the audio stream that has been in the pulling state, that is, the audio and video data of the fifth object is obtained at the same time, and a new video window is generated according to the audio and video data of the fifth object, so that the local user can see the video screen of the fifth object according to the new video window, and hear the voice message of the fifth object. At this time, the current video call interface displays no fourth object but includes the preset route video windows of the fifth object.

What is provided above is that after the fourth object hangs up, the video window of the fourth object is destroyed, and a new video window is created for the fifth object that originally had the only voice access to play the audio and video data. In some embodiments, since the local device cannot pull the audio and video stream of the fourth object after the fourth object hangs up, the video window of the fourth object may not be destroyed. Instead, the audio and video data of the fifth object received from the server is associated with the video window of the fourth object for playback. In this way, there is no need to create a new video window for the fifth object, that is, the fourth object is replaced by the fifth object.

When the fourth object has not hung up, the number of video windows in the first area is the preset number. After the fourth object hangs up, the updated number of current call routes is equal to the preset number. Therefore, the interface layout template used in the video call interface does not need to be changed. They are all interface layout templates corresponding to the preset number. It is only necessary to sort and display the video windows of the current video call interface according to the preset window sorting rules. For example, the window sorting rules are based on the time order of access to the virtual room. The video windows previously sorted after the fourth object are raised by one position one by one, and the position of the video windows previously sorted before the fourth object remains unchanged. The video window of the fifth object is placed at the end of the template. After the sorting is completed, the update of the video call interface is completed.

In some embodiments, as shown in FIG. 15 , for scenario (C), the controller is further used to execute: receiving a hang-up indication message sent by the server, wherein the hang-up indication message is sent by the server after the current call number recorded in the virtual room is reduced by 1 when the fourth object in the virtual room clicks the hang-up control; in response to the hang-up indication message, if the fourth object is a video access, controlling the display to display the hang-up prompt message of the fourth object and destroying the video window of the fourth object; when it is determined that the updated current call number is greater than the preset number, selecting the sixth object from the second area according to the preset filtering rule, obtaining the audio and video data of the sixth object from the server, and generating a new video window in the first area according to the audio and video data of the sixth object, and destroying the voice window of the sixth object in the second area; and sorting and displaying the video windows in the current first area and the voice windows in the current second area according to the preset window sorting rule.

If the updated current call number is greater than the preset number, it means that the call number when the fourth object is not hung up is the preset number + N, where N≥2, that is, the preset number of video windows in the first area (including the fourth object) + N voice windows in the second area. Since the fourth object has been hung up, and there are multiple voice windows in the second area, it involves selecting one of the objects (named the sixth object in this embodiment) from the second area to switch to the video access mode. The sixth object can be selected from the second area according to the preset screening rules. The preset screening rules are, for example, according to the order of the windows in the second area, the call member corresponding to the voice window ranked first in the second area is preferentially switched to the first area, or, if the local user (oneself) is voice access, the local user can be automatically switched to video access. The preset screening rules can be set according to actual applications and are not limited in this embodiment.

After the sixth object is selected, the video stream of the sixth object is started to be pulled, and the audio stream of the sixth object is always in the pulling state, so that the audio and video data of the sixth object can be obtained, and a new video window is generated in the first area according to the audio and video data of the sixth object, and the voice window of the sixth object in the second area is destroyed at the same time, so that the user can see the video screen of the sixth object in the new video window and hear the voice message of the sixth object. At this time, the current video call interface is presented as follows: the current first area displays the preset number of video windows without the fourth object but including the sixth object, the current second area displays N-1 voice windows, and the current call number is the preset number + N-1, N≥2.

What is provided above is that after the fourth object hangs up, the video window of the fourth object is destroyed, and a new video window is created for the sixth object that was originally voice-accessed to play the audio and video data. In some embodiments, since the local device cannot pull the audio and video stream of the fourth object after the fourth object hangs up, the video window of the fourth object may not be destroyed. Instead, the audio and video data of the sixth object received from the server is associated with the video window of the fourth object for playback. In this way, there is no need to create a new video window for the sixth object, that is, the fourth object is replaced by the sixth object.

When the fourth object has not hung up, the number of video windows in the first area is the preset number. After the fourth object has hung up, the updated number of current call routes is still greater than the preset number. Therefore, the interface layout template used in the video call interface does not need to be changed. They are all interface layout templates corresponding to the preset number of routes. It is only necessary to sort and display the video windows of the current video call interface according to the preset window sorting rules. For example, the window sorting rules are based on the time sequence of access to the virtual room. The video windows previously sorted after the fourth object are raised by one position one by one, and the position of the video windows previously sorted before the fourth object remains unchanged. The video window of the sixth object is placed at the end of the template. After the sorting is completed, the update of the first area is completed.

Since the second area destroys the voice window of the sixth object, that is, the number of voice windows in the second area is reduced, it is also necessary to reorder and display the voice windows in the second area. Referring to Figure 7, for example, the preset screening rule is to switch user 4, who is ranked first in the second area, to video access, that is, user 4 is the sixth object. After destroying the voice window of user 4, the voice windows of user 5 and oneself are raised by one rank in turn, that is, the voice window of user 5 is ranked first, and one's own voice window is ranked second, which is equivalent to moving the voice windows after the sixth object forward by one rank in turn. For another example, the preset screening rule is to switch the local user to video access, that is, oneself is the sixth object, then the ranks of user 4 and user 5 ranked before the sixth object remain unchanged. If there is no voice window after oneself, then destroy one's own voice window. If there are other voice windows after oneself, then raise the ranks of the voice windows after oneself by one rank in turn.

In some embodiments, referring to FIG. 15 , for situation (D), the controller is further used to execute: receiving a hang-up indication message sent by the server, wherein the hang-up indication message is sent by the server after the current call number recorded in the virtual room is reduced by 1 when the fourth object in the virtual room clicks the hang-up control; in response to the hang-up indication message, if the fourth object is a voice access, when it is determined that the updated current call number is equal to the preset number, the display is controlled to display the hang-up prompt message of the fourth object, the display of the second area is canceled, and the current display state of the first area is maintained unchanged.

The fourth object is hung up in voice access mode, which means that the number of call lines when the fourth object is not hung up is greater than the preset number. Let M = the number of call lines when the fourth object is not hung up - the preset number, then M≥1, that is, the second area includes at least one voice window when the fourth object is not hung up, that is, the number of call lines when the fourth object is not hung up = the preset number + M, and the updated current number of call lines is equal to the preset number, which means M=1.

Please refer to Figure 6c when M=1. When user 4 hangs up the call as the fourth object, the hang-up prompt information of the fourth object can be displayed and the voice window of the fourth object can be destroyed. At this time, there is no voice window displayed in the second area, so the display of the second area is cancelled. No one hangs up in the first area, so the first area maintains the current display state unchanged, that is, the interface layout template, the audio and video playback of each call member in the first area (user 1, user 2 and himself), and the window sorting state remain unchanged, and the video call interface update is completed.

In some embodiments, referring to FIG. 15 , for situation (E), the controller is further used to execute: receiving a hang-up indication message sent by the server, wherein the hang-up indication message is sent by the server after the current call number recorded in the virtual room is reduced by 1 when the fourth object in the virtual room clicks the hang-up control; in response to the hang-up indication message, if the fourth object is a voice access, when it is determined that the updated current call number is greater than the preset number, controlling the display to display the hang-up prompt message of the fourth object and destroying the voice window of the fourth object; and sorting and displaying the updated voice windows in the second area according to the preset window sorting rules, and maintaining the current display state of the first area unchanged.

For situation (E), the updated current call number is greater than the preset number, indicating that M is greater than 1. When M is greater than 1, see Figure 7. When user 5 hangs up the call, that is, user 5 is the fourth object, the local user can be prompted "user 5 has hung up the call", and the voice window of the fourth object is destroyed. Since M is greater than 1, after the voice window of the fourth object is destroyed, the second area still has at least one voice window. In the example of Figure 7, after the voice window of user 5 is destroyed, there are still two voice windows of user 4 and himself, so it is necessary to reorder according to the window sorting rules. If user 5 does not hang up, user 4, user 5 and himself are sorted according to the time of access to the virtual room, then after the voice window of user 5 is destroyed, the order of the voice window of user 4, which is ranked before user 5, remains unchanged, and the voice window of himself, which is ranked after user 5, is moved up by one order, that is, moved to the second place for display, thereby completing the update of the second area. As in the above situation (D), the fourth object in the second area hangs up in the voice access mode, which has no effect on the display status in the first area.

It should be noted that if the fourth object hangs up the call in voice access mode, the updated current call number is greater than or equal to the preset number, and there is no situation where the updated current call number is less than the preset number.

In some embodiments, after the video call is initiated, the interface layout template of the video call and the window display in the first area/second area can be adaptively changed and adjusted according to the number of people currently connected to the virtual room. For example, the preset number of routes of the device of user 1 (as the initiator, local user) is 3. After user 1 initiates a video call with 5 people, the current interface first displays the video window of the initiator. Then, after user 2 accesses, the video call interface is displayed according to the arrangement order of user 1 and user 2 and the interface layout template of 1 row and 2 columns; if user 3 accesses after user 2, the video call interface is adjusted to the interface layout template of 1 row and 3 columns corresponding to the preset number of routes according to the arrangement order of user 1, user 2 and user 3; if user 4 accesses after user 3, the first area maintains the display state when user 3 accesses unchanged, and displays the voice window of user 4 in the second area; if user 5 accesses after user 4, the display state of the first area remains unchanged, and the voice window of user 5 is displayed in the next position after the voice window of user 4, then all 5 people have accessed, and the video call interface is adjusted accordingly according to the access of each person.

In some embodiments, the initiator initiates a video call with a target number of channels, and the server will establish a virtual room that supports the target number of channels. Then, when the server sends a call request to other members, the call request will carry the target number of channels, so that when each call member's display device starts the video call, it can directly establish a window layout of the video call interface corresponding to the target number of channels: if the target number of channels is less than the preset number of channels, the interface layout template corresponding to the target number of channels is called to display the target number of video windows; if the target number of channels is equal to the preset number of channels, the interface layout template corresponding to the preset number of channels is called to display the preset number of video windows; if the target number of channels is greater than the preset number of channels, the interface layout template corresponding to the preset number of channels is called to display the preset number of video windows in the first area, and generate and display the voice windows of the difference between the target number of channels and the preset number of channels in the second area.

In some embodiments, the initiator's initial video call interface can see the video screen of the local user himself first, and the video windows of other call objects are displayed as gray screens and a prompt message such as "Waiting for access" is displayed on the window. Afterwards, the audio and video data of the call objects are pulled in sequence according to the time sequence in which each call object accesses the virtual room, and associated with the corresponding video window for playback, and/or the audio data of call objects after exceeding a preset number of channels are pulled and associated with the corresponding voice window for playback, until all windows on the initial video call interface have completed data loading. For example, user 1 initiates a video call with 4 people, and the preset number of channels of user 1's display device is 3. When user 1 initiates the video call, the initial video call interface established is presented as a video window with a 1*3 layout in the first area + a voice window in the second area, and the first video window displays the video screen of user 1; then user 2 connects, the audio and video data of user 2 is obtained from the server, and the audio and video data of user 2 is associated with the second video window for playback; user 3 connects after user 2, then the audio and video data of user 3 is pulled, and the audio and video data of user 3 is associated with the third video window for playback; user 4 connects after user 3, and user 4 is automatically switched to voice access, the audio data of user 4 is pulled, and the audio data of user 4 is associated with the first voice window in the second area for playback, and all the video calls of the target number are connected.

In some embodiments, for example, user 1 initiates a video call with 5 people and invites user 2, user 3, user 4 and user 5 to join the video call. The preset number of channels of user 3's display device is 4. When user 3 joins the video call, the initial video call interface constructed is presented as a video window with a 2*2 layout in the first area + a voice window in the second area. Before user 3, user 1 initiates the call, user 4 joins first, then user 2 joins, and then it is user 3's turn to join. Then user 3 can receive the audio and video data of user 1, user 4 and user 2 from the server, then the first video window plays the audio and video data of user 1, the second video window plays the audio and video data of user 4, the third video window plays the audio and video data of user 2, and the fourth video window plays the video screen collected by user 3. The voice window in the second area displays a gray screen and displays a prompt message such as "waiting for access". User 5 joins after user 3, and receives the audio data of user 5, so that the first voice window plays the audio data of user 5, and then all the video calls of the target number of channels are connected.

It should be noted that, during the process of accessing the video call of the target number, the specific UI presentation form and processing logic are not limited to those described in the embodiments of the present application.

In some embodiments, the controller is further configured to execute: in response to a click operation on the hang-up control, control the display to destroy the current video call interface. When the local user clicks the hang-up operation, that is, the local user actively hangs up the video call and exits the virtual room, the local device will no longer collect and upload the local user's audio and video data, nor will it receive audio and video data of other call objects from the server, the video call interface is destroyed, and the video call is terminated.

In some embodiments, the controller is also used to execute: in response to receiving a call object hang-up message sent by the server, control the display to destroy the current video call interface. The local user did not actively hang up the call, but all the call objects that the local user was talking to have hung up, that is, there is only one local user in the virtual room, then the video call does not need to be maintained, which is equivalent to the local user passively hanging up the video call. When the server receives hang-up messages from all other call objects in the virtual room except the local user, it sends the call object hang-up message to the display device of the local user. The controller of the local device receives and responds to the call object hang-up message, and then starts the hang-up logic to destroy the call interface.

A video call application (APP) can be pre-installed in the controller. In this APP, the creation and transformation of the UI interface for initiating, answering, inviting, hanging up the other end, and hanging up the local end of the video call can be realized. It supports user operation of the UI interface and controls the processing logic such as the streaming status of each call. Other functions of the video call can also be realized through the APP, which will not be repeated here. It should be noted that the interface layout template of multi-channel video calls and the UI display of related processing logic are not limited to those shown in the attached figure. The processing logic when the user operates based on the UI and related controls can be adaptively changed.

In some embodiments, the present application also provides a method for processing a multi-channel video call, the method comprising the program steps that the controller in each embodiment of the above display device is configured to execute, which will not be repeated here. The same and similar parts between the various embodiments in this specification can be referenced to each other.

Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium can store a program, and when the computer storage medium is located in a display device, the program can include the program steps included in the multi-channel video call processing method that the controller 250 is configured to execute when the program is executed. The computer storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The description and examples are to be considered merely as exemplary and do not constitute limitations on the scope of the present invention. The true scope and spirit of the present invention are indicated by the appended claims.

Claims (14)

1. A display device, characterized by comprising:

A display;

a sound player;

a communicator for communicatively connecting the display device with the server;

a user interface for receiving an input operation;

a controller coupled to the display, the sound player, the communicator, and the user interface, respectively, for performing:

acquiring the current call path number of the video call from a server;

In response to determining that the current call path number is greater than a preset path number, invoking an interface layout template corresponding to the preset path number to control a display to display a preset path number video windows in a first area on a video call interface; displaying voice windows with the number of differences between the current conversation path number and the preset path number in a second area;

And in response to determining that the current call path number is less than or equal to the preset path number, calling an interface layout template corresponding to the current call path number to control a display to display a plurality of video windows of the current call path number on a video call interface, and not displaying voice windows.

2. The display device of claim 1, wherein the controller is further configured to perform:

When the current call path number is greater than the preset path number, responding to the operation of switching the audio and video windows, acquiring a target voice window corresponding to the first object from the second area, and acquiring a target video window corresponding to the second object from the first area;

And associating the audio and video data of the first object to the target video window for playing, and associating the audio data of the second object to the target voice window for playing.

3. The display device of claim 2, wherein the controller is further configured to perform:

responding to clicking operation of the first control, and controlling a display to display a control list on a floating layer of a video call interface;

The control list comprises a microphone state control, a camera state control, a hang-up control, an invitation control, an audio and video window switching control and a small window call control.

4. A display device according to claim 3, wherein the controller is specifically configured to perform:

when the current call path number is greater than the preset path number, responding to clicking operation of the audio/video window switching control, and acquiring the target voice window selected by the user in the second area;

Responding to the confirmation operation of the target voice window, recording the first user ID of the first object corresponding to the target voice window, and controlling a display to display a second object selection popup window on the upper layer of a video call interface; the second object selection popup window displays user IDs corresponding to all video windows included in the first area;

And acquiring a second user ID clicked by the user in the second object selection popup window, and recording the second object as the second user ID.

5. A display device as claimed in claim 3, wherein the controller is further adapted to obtain the current number of call legs for the video call from the server as follows:

Sending a query request to the server, wherein the query request is used for enabling the server to query the current call path number of the virtual room record of the video call; receiving the current call path number sent by the server;

Or receiving the current call path number sent by the server, wherein the current call path number is sent by the server when the server detects that the current call path number recorded in the virtual room is changed;

Wherein, the change of the current call path number comprises: when any call object in the virtual room hangs up the video call, subtracting 1 from the current call path number recorded in the virtual room, wherein the call object is a call member except a local end user in the virtual room; when any call member in the virtual room invites a new member to answer the video call, the current call path number recorded in the virtual room is added with 1.

6. The display device of claim 5, wherein the controller is further configured to perform:

responding to clicking operation of the invitation control, and acquiring a third user ID of a third object invited by a user;

transmitting the third user ID to a server, so that the server transmits invitation information to the display equipment of the third object after receiving the third user ID;

responding to the successful invitation information sent by the server, when the updated current call route number is greater than the preset route number, controlling the display to display a newly added voice window in a second area, acquiring audio data of a third object from the server, and associating the audio data of the third object to the newly added voice window for playing;

or in response to receiving successful invitation information sent by the server, when the updated current call path number is smaller than or equal to the preset path number, controlling the display to display a newly added video window in the first area, calling an interface layout template corresponding to the updated current call path number, and refreshing a video call interface by utilizing a preset window ordering rule; acquiring audio and video data of a third object from a server, and associating the audio and video data of the third object to the newly added video window for playing;

the invitation success information is sent by the server after adding 1 to the current call route number recorded in the virtual room when the display device of the third object responds to the invitation information and receives the video call incoming call.

7. The display device of claim 5, wherein the controller is further configured to perform:

Receiving hang-up indication information sent by a server, wherein the hang-up indication information is sent by the server after subtracting 1 from the current call route number recorded in a virtual room when a fourth object in the virtual room clicks a hang-up control;

Responding to the hang-up indication information, and if the fourth object is video access, controlling a display to display hang-up prompt information of the fourth object, and destroying a video window of the fourth object;

and when the updated current call path number is determined to be smaller than the preset path number, calling an interface layout template corresponding to the updated current call path number, and refreshing a video call interface by utilizing a preset window ordering rule.

8. The display device of claim 5, wherein the controller is further configured to perform:

Receiving hang-up indication information sent by a server, wherein the hang-up indication information is sent by the server after subtracting 1 from the current call route number recorded in a virtual room when a fourth object in the virtual room clicks a hang-up control;

Responding to the hang-up indication information, and if the fourth object is video access, controlling a display to display hang-up prompt information of the fourth object, and destroying a video window of the fourth object;

When the updated current conversation path number is equal to the preset path number, acquiring audio and video data of a fifth object in a second area from a server, canceling display of the second area on a video conversation interface, and generating a new video window according to audio and video data association of the fifth object;

And according to a preset window ordering rule, ordering and displaying all video windows in the current video call interface.

9. The display device of claim 5, wherein the controller is further configured to perform:

Receiving hang-up indication information sent by a server, wherein the hang-up indication information is sent by the server after subtracting 1 from the current call route number recorded in a virtual room when a fourth object in the virtual room clicks a hang-up control;

Responding to the hang-up indication information, and if the fourth object is video access, controlling a display to display hang-up prompt information of the fourth object, and destroying a video window of the fourth object;

when the updated current call path number is determined to be greater than the preset path number, selecting a sixth object from the second area according to a preset screening rule, acquiring audio and video data of the sixth object from a server, generating a new video window in a first area in a related manner according to the audio and video data of the sixth object, and destroying a voice window of the sixth object in the second area;

according to a preset window ordering rule, ordering and displaying all video windows in a current first area, and ordering and displaying all voice windows in a current second area.

10. The display device of claim 5, wherein the controller is further configured to perform:

Receiving hang-up indication information sent by a server, wherein the hang-up indication information is sent by the server after subtracting 1 from the current call route number recorded in a virtual room when a fourth object in the virtual room clicks a hang-up control;

responding to the hang-up indication information, if the fourth object is voice access, controlling a display to display hang-up prompt information of the fourth object when the updated current conversation path number is equal to the preset path number, canceling the display of the second area, and maintaining the current display state of the first area unchanged;

Or responding to the hang-up indication information, if the fourth object is voice access, controlling a display to display hang-up prompt information of the fourth object and destroying a voice window of the fourth object when the updated current conversation path number is determined to be larger than the preset path number; and according to a preset window ordering rule, ordering and displaying each voice window in the updated second area, and maintaining the current display state of the first area unchanged.

11. The display device of any one of claims 2-10, wherein the controller is further configured to perform:

Acquiring audio and video data of all call objects in the virtual room from a server; respectively associating the audio and video data of the call object in the first area to the corresponding video window for playing; the video data of the call object in the second area is not analyzed, and only the audio data of the call object in the second area are respectively associated to the corresponding voice window for playing;

Or acquiring audio and video data of the call object in the first area from the server, and respectively associating the audio and video data with the corresponding video window for playing; only acquiring the audio data of the call object in the second area from the server, but not acquiring the video data, and respectively associating the audio data of the call object in the second area to the corresponding voice window for playing;

the call object is other call members except the home terminal user in the virtual room.

12. The display device according to any one of claims 1-10, wherein the controller is further preset with a window ordering rule as follows:

after the video call is started, according to the time sequence of joining the call members into the virtual room, firstly ordering and displaying video windows in a first area;

And switching the mode of the following call members accessing the virtual room into voice access in response to the total number of the video windows in the first area reaching the preset number of paths, and sequencing and displaying the voice windows in the second area according to the time sequence of joining the virtual room.

13. A display device according to claim 3, wherein the controller is further configured to perform:

If the current call route number is greater than the preset route number, controlling a display to display a prompt popup window; the prompt popup is used for prompting a user to display a control list by clicking a first control and switching a voice window of a first object into a video window by clicking a switching audio and video window control;

and if the current video call path number is smaller than or equal to the preset path number, the prompt popup window is not displayed.

14. The multipath video call processing method is characterized by comprising the following steps of:

acquiring the current call path number of the video call from a server;

In response to determining that the current call path number is greater than a preset path number, invoking an interface layout template corresponding to the preset path number, and displaying a preset path number video window in a first area on a video call interface; displaying voice windows with the number of differences between the current conversation path number and the preset path number in a second area;

and in response to determining that the current call path number is less than or equal to the preset path number, calling an interface layout template corresponding to the current call path number, displaying a plurality of video windows of the current call path number on a video call interface, and not displaying a voice window.