CN114297435A - A communication terminal and a multi-screen interactive video browsing method - Google Patents

Abstract

The application provides a communication terminal and a multi-screen interactive video browsing method. The video sharing interface comprises video directory entries corresponding to a plurality of videos to be shared, and each video directory entry comprises a plurality of key frame images extracted from the videos to be shared so that a user can browse and view the key frame images. Through the plurality of key frame images, the video content in the preset time period can be displayed, a user can conveniently know the video content and select a video file to be shared to push to the display equipment, and the problem that a traditional video file browsing method is not beneficial to the user to select the video resource file to be shared is solved.

Description

A communication terminal and a multi-screen interactive video browsing method

technical field

The present application relates to the field of communication technologies, and in particular, to a communication terminal and a multi-screen interactive video browsing method.

Background technique

The communication terminal based on the network connection relationship can establish a multi-screen interactive relationship with the display device. For example, a communication terminal such as a mobile phone can establish a connection relationship with a display device such as a smart TV through a wireless network, so as to synchronously display the screen on the mobile phone through the smart TV, or share resources between the mobile phone and the smart TV. Since the display screen of the display device is larger, it is more suitable for use scenarios such as conference presentations than the communication terminal. Therefore, the communication terminal can push local video resources to the display device for display after establishing a connection relationship.

Generally, the operation of the communication terminal to push video resources to the display device is as follows: the user enters the file management interface through an interactive operation, and the file management interface can display a video list composed of multiple video resource icons for the user to select. After the user selects the icon of any video resource file, the video file is sent to the display device through the UI interface or a preset sharing operation, so that the display device can play the shared video resource.

The video resource pattern displayed in the file management interface is generally a video thumbnail and a video name, that is, the video resource content is represented by the video picture of the first frame or a specific frame. However, under some video resources, the user cannot effectively identify the video resource through the video thumbnail, that is, it is impossible to know which video resource the thumbnail corresponds to, and the names of some video resources, such as the name of the video recorded by the mobile phone, usually cannot express their The significance of the specific video content makes it impossible for the user to accurately select the video resource file to be shared when pushing the video, which reduces the operation efficiency.

SUMMARY OF THE INVENTION

The present application provides a communication terminal and a multi-screen interactive video browsing method, so as to solve the problem that the traditional video file browsing method is not conducive for users to select video resource files to be shared.

In a first aspect, the present application provides a communication terminal, including: a display unit, a communication circuit, and a processor. Wherein, the display unit is configured to present various user interfaces and to present a video sharing interface. The communication circuit is configured to establish a communication connection with the display device to push the shared video data to the display device. The processor is configured to perform the following program steps:

Obtain the multi-screen interactive instructions entered by the user;

In response to the multi-screen interactive instruction, the display unit is controlled to display a video sharing interface.

Wherein, the video sharing interface includes a plurality of video catalog items, and each of the video catalog items includes a plurality of key frame images extracted from the video to be shared.

The communication terminal provided by the first aspect of the present application can display a video sharing interface in the display unit according to the multi-screen interactive instruction input by the user. The video sharing interface includes a plurality of video catalog items corresponding to the videos to be shared, and each video catalog item includes a plurality of key frame images extracted from the videos to be shared for the user to browse and view. Through multiple key frame images, the video content within a preset time period can be displayed, which is convenient for users to understand the video content and select the video file to be shared and push it to the display device, solving the problem that the traditional video file browsing method is not conducive to the user to select the video to be shared. Problems with resource files.

In a second aspect, the present application further provides a display device including a display, a communicator and a controller. Wherein, the display is configured to display the user interface and the video content shared by the communication terminal, and the communicator is configured to establish a communication connection with the communication terminal, so as to obtain the shared video data at the communication terminal. The controller is configured to perform the following procedural steps:

After establishing a communication connection with the communication terminal, obtain the video file pushed by the communication terminal;

Parse the video file, and control the display to display the video content of the video file.

Wherein, the video file is a video file selected by the user in the video sharing interface of the communication terminal; the video sharing interface includes a plurality of video catalog items, and each of the video catalog items includes a plurality of video catalog items extracted from the video to be shared keyframe image.

It can be seen from the above technical solutions that the display device provided by the second aspect of the present application can receive, in real time, the video file pushed by the communication terminal through the communicator after the communication connection is established, and parse the received video file to generate the specific video image content, Finally, the content of the video image is displayed on the display. Since the video file received by the display device is the video file selected by the user in the video sharing interface, and in the video sharing interface, the video file is represented by multiple video directory items, so the received video can directly display the video to be shared by the user video content, reducing the number of repeated user operations and improving user experience.

In a third aspect, the present application further provides a multi-screen interactive video browsing method, which is applied to a communication terminal that establishes a communication connection with a display device, and the method includes:

Obtain the multi-screen interactive instructions entered by the user;

In response to the multi-screen interactive instruction, a video sharing interface is displayed through the communication terminal.

Wherein, the video sharing interface includes a plurality of video catalog items, and each of the video catalog items includes a plurality of key frame images extracted from the video to be shared. The multi-screen interactive video browsing method provided by the third aspect of the present application may be configured in the processor of the communication terminal, so that the video sharing interface can be displayed after the communication terminal obtains the multi-screen interactive instruction input by the user. And the video sharing interface includes multiple video catalog items, and each video catalog item includes multiple key frame images extracted from the video to be shared. The method can display the video file through multiple key frame images, so that the user can fully understand the video content, and it is convenient to accurately select the video file to be shared.

In a fourth aspect, the present application further provides a multi-screen interactive video browsing system, including a communication terminal and a display device, wherein the communication terminal and the display device establish a communication connection;

The communication terminal is configured to: acquire a multi-screen interactive instruction input by a user, and in response to the multi-screen interactive instruction, display a video sharing interface, where the video sharing interface includes a plurality of video catalog items, each of the video The catalog item includes a plurality of keyframe images extracted from the video to be shared; and,

Pushing a video file to the display device, where the video file is the video to be shared selected by the user in the video sharing interface of the communication terminal;

The display device is configured to: acquire a video file pushed by the communication terminal, parse the video file, and display video content of the video file.

It can be known from the above technical solutions that the multi-screen interactive video browsing system provided by the fourth aspect of the present application includes a communication terminal and a display device that establish a communication connection with each other. The communication terminal may display a video sharing interface after acquiring the multi-screen interactive instruction, wherein the video sharing interface includes a plurality of video catalog items, and each of the video catalog items includes a plurality of key items extracted from the video to be shared. frame image. After the user selects any video to be shared, the communication terminal also pushes the selected video file to the display device, so that the display device can display the specific video content of the video file after acquiring the video file. The system displays video files through a plurality of key frame images, so that the user can fully understand the video content, and it is convenient to accurately select the video file to be shared.

Description of drawings

In order to illustrate the technical solutions of the present application more clearly, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, for those of ordinary skill in the art, without creative work, the Additional drawings can be obtained from these drawings.

1 is a schematic diagram of an operation scenario between a display device and a control device in an embodiment of the present application;

2 is a block diagram of a hardware configuration of a display device in an embodiment of the application;

3 is a schematic diagram of a software configuration diagram of a display device in an embodiment of the application;

4 is a schematic diagram of displaying an icon control interface of a device application in an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a communication terminal provided in an embodiment of the present application;

6 is a schematic diagram of a software architecture of a communication terminal provided in an embodiment of the present application;

7 is a schematic diagram of a user interface of a communication terminal provided in an embodiment of the present application;

8 is a schematic diagram of a communication connection between a communication terminal and a display device provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of a screen projection connection between a communication terminal and a display device provided in an embodiment of the application;

10 is a schematic diagram of data flow of the multi-screen interactive video browsing method provided in the embodiment of the application;

11 is a schematic flowchart of a multi-screen interactive video browsing method provided in an embodiment of the application;

12 is a schematic diagram of a video sharing interface provided in an embodiment of the application;

13 is a schematic flowchart of a display browsing interface provided in an embodiment of the present application;

14 is a schematic diagram of a browsing interface provided in an embodiment of the present application;

FIG. 15 is a schematic flowchart of switching video files on a browsing interface provided in an embodiment of the application;

FIG. 16 is a schematic diagram of a display device provided in an embodiment of the application.

Detailed ways

The technical solutions in the embodiments of the present application will be cleared and described in detail below with reference to the accompanying drawings. Wherein, in the description of the embodiments of the present application, unless otherwise specified, “/” means or, for example, A/B can mean A or B; “and/or” in the text is only a description of an associated object The association relationship indicates that there can be three kinds of relationships, for example, A and/or B can indicate that A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of this application , "plurality" means two or more than two.

Hereinafter, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as implying or implying relative importance or implying the number of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, the "multiple" The meaning is two or more.

FIG. 1 exemplarily shows a schematic diagram of an operation scenario between a display device and a control apparatus according to an embodiment. As shown in FIG. 1 , the user can operate the display apparatus 200 through the communication terminal 100A and the control apparatus 100 (eg, the remote controller 100B).

In some embodiments, the communication terminal 100A, tablet computer, computer, notebook computer, and other smart devices may also be used to control the display device 200 . For example, the display device 200 is controlled using an application running on the smart device. The app can be configured to provide users with various controls in an intuitive user interface (UI) on the screen associated with the smart device.

In some embodiments, the communication terminal 100A may install a software application with the display device 200 to realize connection communication through a network communication protocol, so as to achieve the purpose of one-to-one control operation and data communication. For example, the communication terminal 100A can establish a control command protocol with the display device 200, synchronize the remote control keyboard to the communication terminal 100A, and control the display device 200 by controlling the user interface on the communication terminal 100A. The audio and video content displayed on the communication terminal 100A may also be transmitted to the display device 200 to implement a synchronous display function.

As also shown in FIG. 1 , the display device 200 also performs data communication with the server 400 through various communication methods. The display device 200 may be allowed to communicate via local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200 . For example, the display device 200 interacts by sending and receiving information, and electronic program guide (EPG), receiving software program updates, or accessing a remotely stored digital media library. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers. Other network service contents such as video-on-demand and advertising services are provided through the server 400 .

The display device 200 may be a liquid crystal display, an OLED display, or a projection display device. The specific display device type, size and resolution are not limited. Those skilled in the art can understand that the display device 200 can make some changes in performance and configuration as required.

The display device 200 may additionally provide a smart IPTV function that provides computer-supported functions, including but not limited to, IPTV, smart TV, Internet Protocol TV (IPTV), and the like, in addition to the broadcast receiving TV function.

FIG. 2 exemplarily shows a block diagram of the hardware configuration of the display device 200 according to the exemplary embodiment.

In some embodiments, the display device 200 includes a controller 250, a tuner 210, a communicator 220, a detector 230, an input/output interface 255, a display 275, an audio output interface 285, a memory 260, a power supply 290, At least one of the user interface 265 and the external device interface 240 .

In some embodiments, the display 275, for receiving the image signal from the output of the first processor, performs components for displaying video content and images and a menu manipulation interface.

In some embodiments, the display 275 includes a display screen component for presenting pictures, and a driving component for driving image display.

In some embodiments, the video content displayed may be from broadcast television content or various broadcast signals that may be received via wired or wireless communication protocols. Alternatively, various image contents sent from the network server side can be displayed and received from the network communication protocol.

In some embodiments, display 275 is used to present a user-manipulated UI interface generated in display device 200 and used to control display device 200 .

In some embodiments, depending on the type of display 275, a driving component for driving the display is also included.

In some embodiments, display 275 is a projection display, and may also include a projection device and projection screen.

In some embodiments, communicator 220 is a component for communicating with external devices or external servers according to various communication protocol types. For example, the communicator may include at least one of a Wifi chip, a Bluetooth communication protocol chip, a wired Ethernet communication protocol chip and other network communication protocol chips or a near field communication protocol chip, and an infrared receiver.

In some embodiments, the display apparatus 200 may establish control signal and data signal transmission and reception between the communicator 220 and the external control apparatus 100 or the content providing apparatus.

In some embodiments, the user interface 265 can be used to receive infrared control signals from the control device 100 (eg, the infrared remote control 100B, etc.).

In some embodiments, the detector 230 is a signal used by the display device 200 to collect the external environment or interact with the outside.

In some embodiments, the detector 230 includes a light receiver, a sensor for collecting ambient light intensity, and can adaptively display parameter changes and the like by collecting ambient light.

In some embodiments, the detector 230 may also include an image collector, such as a camera, a camera, etc., which can be used to collect external environment scenes, and used to collect user attributes or interactive gestures with the user, and can adaptively change display parameters, User gestures can also be recognized to implement functions that interact with users.

In some embodiments, detector 230 may also include a temperature sensor or the like, such as by sensing ambient temperature.

In some embodiments, the display device 200 can adaptively adjust the display color temperature of the image. For example, when the temperature is relatively high, the display device 200 can be adjusted to display a relatively cool color temperature of the image, or when the temperature is relatively low, the display device 200 can be adjusted to display a warmer color of the image.

In some embodiments, the detector 230 may also be a sound collector or the like, such as a microphone, which may be used to receive the user's voice. Exemplarily, a voice signal including a control instruction of the user to control the display device 200, or collecting ambient sounds, is used to identify the type of the environment scene, so that the display device 200 can adaptively adapt to the ambient noise.

In some embodiments, as shown in FIG. 2 , the input/output interface 255 is configured to enable data transfer between the controller 250 and other external devices or other controllers 250 . Such as receiving video signal data and audio signal data of external equipment, or command instruction data, etc.

In some embodiments, the external device interface 240 may include, but is not limited to, the following: any one or more of a high-definition multimedia interface HDMI interface, an analog or data high-definition component input interface, a composite video input interface, a USB input interface, an RGB port, etc. interface. It is also possible to form a composite input/output interface by a plurality of the above-mentioned interfaces.

In some embodiments, as shown in FIG. 2 , the tuner and demodulator 210 is configured to receive broadcast television signals through wired or wireless reception, and can perform modulation and demodulation processing such as amplification, frequency mixing, and resonance, from multiple wireless Or demodulate the audio and video signal from the cable broadcast TV signal, the audio and video signal may include the TV audio and video signal carried in the frequency of the TV channel selected by the user, and the EPG data signal.

In some embodiments, the frequency demodulated by the tuner-demodulator 210 is controlled by the controller 250, and the controller 250 can send a control signal according to the user's selection, so that the modem responds to the user-selected TV signal frequency and modulates and demodulates the frequency. The frequency carried by the television signal.

In some embodiments, broadcast television signals may be classified into terrestrial broadcast signals, cable broadcast signals, satellite broadcast signals, or Internet broadcast signals, etc. according to different broadcast formats of the television signals. Or according to different modulation types, it can be divided into digital modulation signal, analog modulation signal, etc. Or, it can be divided into digital signals, analog signals, etc. according to different types of signals.

In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box Wait. In this way, the set-top box outputs the modulated and demodulated television audio and video signals of the received broadcast television signals to the main device, and the main device receives the audio and video signals through the first input/output interface.

In some embodiments, the controller 250, through various software control programs stored on the memory, controls the operation of the display device and responds to user operations. The controller 250 may control the overall operation of the display apparatus 200 . For example, in response to receiving a user command for selecting a UI object to be displayed on the display 275, the controller 250 may perform an operation related to the object selected by the user command.

In some embodiments, the object may be any of the selectable objects, such as a hyperlink or an icon. Operations related to the selected object, such as displaying operations linked to hyperlinked pages, documents, images, etc., or executing operations corresponding to the icon. The user command for selecting the UI object may be an input command through various input devices (eg, a mouse, a keyboard, a touchpad, etc.) connected to the display device 200 or a voice command corresponding to a voice spoken by the user.

As shown in FIG. 2 , the controller 250 includes a random access memory 251 (Random Access Memory, RAM), a read-only memory 252 (Read-Only Memory, ROM), a video processor, an audio processor, and other processors (for example: At least one of a graphics processing unit (Graphics Processing Unit, GPU), a central processing unit (Central Processing Unit, CPU), a communication interface (Communication Interface), and a communication bus 256 (Bus), wherein the communication bus connects various components.

In some embodiments, RAM 251 is used to store temporary data for the operating system or other running programs. In some embodiments, ROM 252 is used to store various system startup instructions.

In some embodiments, ROM 252 is used to store a basic input output system, referred to as a Basic Input Output System (BIOS). It is used to complete the power-on self-check of the system, the initialization of each functional module in the system, the driver program of the basic input/output of the system, and the boot operating system.

In some embodiments, when a power-on signal is received, the power supply of the display device 200 starts, and the CPU executes the system start-up instruction in the ROM 252, and copies the temporary data of the operating system stored in the memory to the RAM 251, so as to facilitate the start-up or running operation system. After the startup of the operating system is completed, the CPU copies the temporary data of various application programs in the memory to the RAM 251, so as to facilitate the startup or operation of various application programs.

In some embodiments, a processor for executing operating system and application program instructions stored in memory. And various application programs, data and content are executed according to various interactive instructions received from the external input, so as to finally display and play various audio and video content.

In some exemplary embodiments, the processor may include multiple processors. The plurality of processors may include a main processor and one or more sub-processors. The main processor is used to perform some operations of the display device 200 in the pre-power-on mode, and/or an operation of displaying a picture in the normal mode. One or more sub-processors for an operation in a state such as standby mode.

In some embodiments, the graphics processor is used to generate various graphic objects, such as: icons, operation menus, and user input instructions to display graphics, etc. It includes an operator, which performs operations by receiving various interactive instructions input by the user, and displays various objects according to the display properties. and includes a renderer, which renders various objects obtained based on the operator, and the rendered objects are used for displaying on a display.

In some embodiments, the video processor is configured to receive the external video signal, perform decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc. according to the standard codec protocol of the input signal For video processing, a signal that can be directly displayed or played on the display device 200 can be obtained.

In some embodiments, the video processor includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, and the like.

Wherein, the demultiplexing module is used for demultiplexing the input audio and video data stream. For example, if MPEG-2 is input, the demultiplexing module demultiplexes it into video signals and audio signals.

The video decoding module is used to process the demultiplexed video signal, including decoding and scaling.

The image synthesizing module, such as an image synthesizer, is used for superimposing and mixing the GUI signal generated by the graphics generator according to the user's input or itself, and the zoomed video image, so as to generate an image signal that can be displayed.

The frame rate conversion module is used to convert the input video frame rate, such as converting 60Hz frame rate to 120Hz frame rate or 240Hz frame rate. The usual format is implemented by means of frame insertion.

The display formatting module is used for converting the received frame rate into the video output signal, and changing the signal to conform to the display format signal, such as outputting the RGB data signal.

In some embodiments, the graphics processor and the video processor can be integrated and set up, or can be set up separately. When the integrated setting is performed, the processing of the graphics signal output to the display can be performed, and when the separate setting is performed, different functions can be performed respectively, for example GPU+FRC (Frame Rate Conversion)) architecture.

In some embodiments, the audio processor is configured to receive an external audio signal, and perform decompression and decoding according to a standard codec protocol of the input signal, as well as noise reduction, digital-to-analog conversion, and amplification, etc. The sound signal played in the loudspeaker.

In some embodiments, the video processor may include one or more chips. The audio processor may also include one or more chips.

In some embodiments, the video processor and the audio processor may be separate chips, or may be integrated into one or more chips together with the controller.

In some embodiments, the audio output, under the control of the controller 250, receives the sound signal output by the audio processor, such as the speaker 286, and in addition to the speaker carried by the display device 200 itself, can be output to a generating device of an external device The external audio output terminal of the device, such as an external audio interface or a headphone interface, etc., may also include a short-range communication module in the communication interface, such as a Bluetooth circuit for outputting sound from a Bluetooth speaker.

The power supply 290, under the control of the controller 250, provides power supply support for the display device 200 with the power input from the external power supply. The power supply 290 may include a built-in power supply circuit installed inside the display device 200 , or may be an external power supply installed in the display device 200 to provide an external power supply interface in the display device 200 .

The user interface 265 is used for receiving user input signals, and then sending the received user input signals to the controller 250 . The user input signal may be a remote controller 100B signal received through an infrared receiver, and various user control signals may be received through a network communication circuit.

In some embodiments, the user inputs a user command through the control apparatus 100 or the communication terminal 100A, the user input interface is based on the user's input, and the display device 200 responds to the user's input through the controller 250 .

In some embodiments, the user may input user commands on a graphical user interface (GUI) displayed on the display 275, and the user input interface receives the user input commands through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

In some embodiments, a "user interface" is a medium interface for interaction and information exchange between an application program or an operating system and a user, which enables conversion between an internal form of information and a form acceptable to the user. A commonly used form of user interface is a graphical user interface (Graphic User Interface, GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It can be an icon, window, control and other interface elements displayed on the display screen of the electronic device, wherein the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc. visual interface elements.

The memory 260 includes storing various software modules for driving the display device 200 . For example, various software modules stored in the first memory include at least one of a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules.

The basic module is used for signal communication between various hardwares in the display device 200, and is a low-level software module that sends processing and control signals to the upper-layer modules. The detection module is a management module used to collect various information from various sensors or user input interfaces, perform digital-to-analog conversion, and analyze and manage.

For example, the speech recognition module includes a speech parsing module and a speech instruction database module. The display control module is a module used to control the display to display image content, and can be used to play information such as multimedia image content and UI interface. Communication module, a module for control and data communication with external devices. The browser module is a module for performing data communication between browsing servers. Service modules are used to provide various services and modules including various applications. At the same time, the memory 260 is also used to store and receive external data and user data, images of various items in various user interfaces, and visual effect diagrams of focal objects.

Referring to FIG. 4 , in some embodiments, the system is divided into four layers, from top to bottom, the applications layer (referred to as “application layer”), the application framework layer (referred to as “framework layer”) "), the Android runtime (Android runtime) and the system library layer (referred to as "system runtime library layer"), and the kernel layer.

In some embodiments, at least one application program runs in the application program layer, and these application programs may be a Window program, a system setting program, a clock program, a camera application, etc. built into the operating system; they may also be developed by a third party The application programs developed by the author, such as the Hijian program, the K song program, the magic mirror program, etc. During specific implementation, the application package in the application layer is not limited to the above examples, and may actually include other application packages, which is not limited in this embodiment of the present application.

The framework layer provides an application programming interface (application programming interface, API) and a programming framework for the applications of the application layer. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center, which decides to let the applications in the application layer take action. Through the API interface, the application can access the resources in the system and obtain the services of the system during execution

As shown in FIG. 3 , the application framework layer in the embodiment of the present application includes managers (Managers), content providers (Content Provider), etc., wherein the manager includes at least one of the following modules: the activity manager (ActivityManager) is used with Interact with all activities running in the system; Location Manager (Location Manager) is used to provide system services or applications with access to system location services; File Package Manager (Package Manager) is used to detect currently installed on the device. Various information related to the application package; Notification Manager (Notification Manager) is used to control the display and clearing of notification messages; Window Manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers and desktops on the user interface part.

In some embodiments, the activity manager is used to: manage the life cycle of each application and the usual navigation and fallback functions, such as controlling the exit of the application (including switching the user interface currently displayed in the display window to the system desktop), opening the , back (including switching the currently displayed user interface in the display window to the upper-level user interface of the currently displayed user interface), and the like.

In some embodiments, the window manager is used to manage all window programs, such as obtaining the size of the display screen, judging whether there is a status bar, locking the screen, taking screenshots, and controlling the change of the display window (such as shrinking the display window, shaking the display, twisting the display) deformation display, etc.) and so on.

In some embodiments, the system runtime layer provides support for the upper layer, that is, the framework layer. When the framework layer is used, the Android operating system will run the C/C++ library included in the system runtime layer to implement the functions to be implemented by the framework layer.

In some embodiments, the kernel layer is the layer between hardware and software. As shown in Figure 4, the kernel layer at least includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, touch sensors, pressure sensors, etc.).

In some embodiments, the kernel layer further includes a power driver module for power management.

In some embodiments, taking the magic mirror application (photography application) as an example, when the remote control receiving device receives the input operation of the remote control 100B, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the input operation into the original input event (including the value of the input operation, the timestamp of the input operation, etc.). Raw input events are stored at the kernel layer. The application framework layer obtains the original input event from the kernel layer, identifies the control corresponding to the input event according to the current position of the focus, and regards the input operation as a confirmation operation, and the control corresponding to the confirmation operation is the control of the magic mirror application icon. The mirror application calls the interface of the application framework layer, starts the mirror application, and then starts the camera driver by calling the kernel layer to capture still images or videos through the camera.

In some embodiments, for a display device with a touch function, taking a split-screen operation as an example, the display device receives an input operation (such as a split-screen operation) performed by the user on the display screen, and the kernel layer can generate corresponding input operations according to the input operation. Enter an event and report the event to the application framework layer. The window mode (such as multi-window mode) and window position and size corresponding to the input operation are set by the activity manager of the application framework layer. The window management of the application framework layer draws the window according to the settings of the activity manager, and then sends the drawn window data to the display driver of the kernel layer, and the display driver displays the corresponding application interface in different display areas of the display screen.

In some embodiments, as shown in FIG. 4, the application layer contains at least one application that can display a corresponding icon control on the display, such as: a live TV application icon control, a video-on-demand application icon control, a media center application Program icon controls, application center icon controls, game application icon controls, etc.

In some embodiments, the live TV application may provide live TV from different sources. For example, a live TV application may provide a TV signal using input from cable, over-the-air, satellite services, or other types of live TV services. And, the live TV application may display the video of the live TV signal on the display device 200 .

In some embodiments, a video-on-demand application may provide video from various storage sources. Unlike live TV applications, video-on-demand provides a display of video from certain storage sources. For example, video-on-demand can come from a server-side cloud storage, from a local hard drive storage containing existing video programs.

In some embodiments, the media center application may provide various multimedia content playback applications. For example, a media center may provide services other than live TV or video-on-demand, where users can access various images or audio through a media center application.

In some embodiments, the application center may provide storage of various applications. An application can be a game, an application, or some other application that is related to a computer system or other device but can be run on a Smart TV. The application center can obtain these applications from various sources, store them in local storage, and then run them on the display device 200 .

FIG. 5 shows a schematic structural diagram of the communication terminal 100A.

The embodiment will be specifically described below by taking the communication terminal 100A as an example. It should be understood that the communication terminal 100A shown in FIG. 1 is only an example, and that the communication terminal 100A may have more or fewer components than those shown in FIG. 1, two or more components may be combined, or Different component configurations are possible. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

FIG. 5 exemplarily shows a hardware configuration block diagram of the communication terminal 100A according to the exemplary embodiment. As shown in FIG. 5 , the communication terminal 100A includes: a radio frequency (RF) circuit 110 , a memory 120 , a display unit 130 , a camera 140 , a sensor 150 , an audio circuit 160 , and a Wireless Fidelity (Wi-Fi) circuit 170 , the processor 180 , the Bluetooth circuit 181 , and the power supply 190 and other components.

The RF circuit 110 can be used for receiving and sending signals during sending and receiving information or during a call, can receive downlink data from the base station and then hand it over to the processor 180 for processing; and can send uplink data to the base station. Typically, RF circuits include, but are not limited to, antennas, at least one amplifier, transceivers, couplers, low noise amplifiers, duplexers, and the like.

The memory 120 may be used to store software programs and data. Processor 180 executes various functions of communication terminal 100A and data processing by executing software programs or data stored in memory 120 . Memory 120 may include high speed random access memory, and may also include nonvolatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. The memory 120 stores an operating system that enables the communication terminal 100A to operate. In this application, the memory 120 may store an operating system and various application programs, and may also store codes for executing the methods described in the embodiments of this application.

The display unit 130 can be used to receive input digital or character information, and generate signal input related to user settings and function control of the communication terminal 100A. Touch operations on or near it, such as clicking a button, dragging a scroll box, etc.

The display unit 130 may also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the communication terminal 100A. Specifically, the display unit 130 may include a display screen 132 disposed on the front of the communication terminal 100A. The display screen 132 may be configured in the form of a liquid crystal display, a light emitting diode, or the like. The display unit 130 may be used to display various graphical user interfaces described in this application.

The touch screen 131 may cover the display screen 132, or the touch screen 131 and the display screen 132 may be integrated to realize the input and output functions of the communication terminal 100A, which may be referred to as touch screen after integration. In this application, the display unit 130 can display application programs and corresponding operation steps.

Camera 140 may be used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 180 for conversion into a digital image signal.

The communication terminal 100A may further include at least one sensor 150 , such as an acceleration sensor 151 , a distance sensor 152 , a fingerprint sensor 153 , and a temperature sensor 154 . The communication terminal 100A may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

The audio circuit 160, the speaker 161, and the microphone 162 may provide an audio interface between the user and the communication terminal 100A. The audio circuit 160 can transmit the electrical signal converted from the received audio data to the speaker 161, and the speaker 161 converts it into a sound signal and outputs it. The communication terminal 100A may also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 160 and then converted into audio data, and then outputs the audio data to the RF circuit 110 for transmission to, for example, another terminal, or outputs the audio data to memory 120 for further processing. In this application, the microphone 162 can acquire the user's voice.

Wi-Fi is a short-range wireless transmission technology, and the communication terminal 100A can help users to send and receive emails, browse web pages, access streaming media, etc. through the Wi-Fi circuit 170, which provides users with wireless broadband Internet access.

The processor 180 is the control center of the communication terminal 100A, uses various interfaces and lines to connect various parts of the entire terminal, and executes the communication terminal by running or executing the software programs stored in the memory 120 and calling the data stored in the memory 120. 100A's various functions and processing data. In some embodiments, the processor 180 may include one or more processing units; the processor 180 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the baseband The processor mainly handles wireless communication. It can be understood that, the above-mentioned baseband processor may not be integrated into the processor 180 . In this application, the processor 180 may run an operating system, an application program, a user interface display and a touch response, as well as the processing methods described in the embodiments of this application. In addition, the processor 180 is coupled with the input unit 130 and the display unit 140 .

The bluetooth circuit 181 is used to exchange information with other bluetooth devices having bluetooth circuits through the bluetooth protocol. For example, the communication terminal 100A may establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth circuit through the Bluetooth circuit 181, so as to perform data interaction.

The communication terminal 100A also includes a power source 190 (eg, a battery) for powering the various components. The power supply can be logically connected to the processor 180 through a power management system, so that functions such as managing charging, discharging, and power consumption are implemented through the power management system. The communication terminal 100A may also be configured with a power button, which is used to power on and off the terminal, and lock the screen and other functions.

FIG. 6 is a block diagram of the software structure of the communication terminal 100A according to the embodiment of the present invention.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, which are, from top to bottom, an application layer, an application framework layer, an Android runtime (Android runtime) and system libraries, and a kernel layer.

The application layer can include a series of application packages.

As shown in Figure 6, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and so on.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for the applications of the application layer. The application framework layer includes some predefined functions.

As shown in Figure 6, the application framework layer may include window managers, content providers, view systems, telephony managers, resource managers, notification managers, and the like.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc.

Content providers are used to store and retrieve data and make these data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone book, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. View systems can be used to build applications. A display interface can consist of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The telephony manager is used to provide the communication function of the communication terminal 100A. For example, the management of call status (including connecting, hanging up, etc.).

The resource manager provides various resources for the application, such as localization strings, icons, pictures, layout files, video files and so on.

The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages, and can disappear automatically after a brief pause without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also display notifications in the status bar at the top of the system in the form of graphs or scroll bar text, such as notifications of applications running in the background, and notifications on the screen in the form of dialog windows. For example, text information is prompted in the status bar, a prompt tone is issued, the communication terminal vibrates, and the indicator light flashes.

Android Runtime includes core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

The core library consists of two parts: one is the function functions that the java language needs to call, and the other is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, safety and exception management, and garbage collection.

A system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL) and so on.

The Surface Manager is used to manage the display subsystem and provides a fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display drivers, camera drivers, audio drivers, and sensor drivers.

In the following, the workflow of the software and hardware of the communication terminal 100A is exemplarily described in conjunction with the capturing and photographing scene.

When the touch screen 131 receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into raw input events (including touch coordinates, timestamps of touch operations, etc.). Raw input events are stored at the kernel layer. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon, for example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer. Camera 140 captures still images or video.

The communication terminal 100A in the embodiment of the present application may be a mobile phone, a tablet computer, a wearable device, a notebook computer, a TV, and the like.

FIG. 7 is a schematic diagram for illustrating a user interface on a communication terminal such as the communication terminal 100A of FIG. 1 . In some specific implementations, the user can open a corresponding application program by touching an application icon on the user interface, or can open a corresponding folder by touching a folder icon on the user interface.

In the embodiment of the present application, as shown in FIG. 8 , the communication terminal 100A may establish a communication connection with the display device 200 to realize the multi-screen interaction function. The display device 200 refers to a device capable of presenting a display screen and capable of data interaction with other devices, for example, a smart TV, a tablet computer, a smart presentation screen, a smart projector, and the like.

In order to display a specific user interface and establish a communication connection with the communication terminal 100A, the display device 200 at least includes a display 275 , a communicator 220 and a controller 250 . The display 275 is used to present a specific user interface, the communicator 220 is used to establish a communication connection with the communication terminal 100A, and the controller 250 is used to receive, send, and process related data or control instructions in the display process and the communication process.

In some embodiments, the display device 200 further includes at least one of a tuner, a detector, an input/output interface, an audio output interface, a memory, a power supply, a user interface, and an external device interface for auxiliary display process and communication process. For example, the external environment or signals interacting with the outside are collected by detectors such as light receivers, and parameter changes can be adaptively displayed by collecting ambient light.

The communication connection established between the communication terminal 100A and the display device 200 may be wired or wireless. For example, the communicator 220 may include at least one of a Wifi chip, a Bluetooth communication protocol chip, a wired Ethernet communication protocol chip and other network communication protocol chips or a near field communication protocol chip, and an infrared receiver. Correspondingly, the connection between the communication terminal 100A and the display device 200 may be realized by means of wireless local area network, Bluetooth, on-site communication, and the like.

After the communication connection is established between the communication terminal 100A and the display device 200 , the communication terminal 100A may establish different data transmission channels with the display device 200 using different transmission protocols to realize the data transmission function. For example, in order to realize the screen projection function, a data transmission channel based on the Digital Living Network Alliance (DLNA) protocol can be established between the communication terminal 100A and the display device 200, and the channel can be used to transmit the screen projection data stream, so that the The display device 200 can synchronously display the content in the communication terminal 100A.

The communication terminal 100A and the display device 200 can communicate data to each other using the established communication connection relationship. For example, the communication terminal 100A can push the video file stored in the communication terminal 100A to the display device 200, and the display device 200 performs the video playback function according to the received video file, so that in the scenarios of conference presentations, movie projections, etc. A large-scale display 275 displays the video content. In this embodiment of the present application, the process of pushing the video file from the communication terminal 100A to the display device 200 is called video sharing. It should be understood that the files pushed by the communication terminal 100A to the display device 200 include not only video files, but also document files, picture files and other files that can be played in the communication terminal 100A and the display device 200 .

In order to realize the video sharing process, the user can first select the video file to be shared in the file management interface of the communication terminal 100A, and then execute the corresponding interactive action of "push" through the UI interface of the communication terminal 100A. At this time, the communication terminal 100A can send the video file through the communication device, and the display device 200 can receive the video file through the communicator 220, and perform playback-related operations such as decoding on the received video file according to a predetermined playback method, and Finally, the specific content of the video is presented on the display 275 .

In some embodiments, as shown in FIG. 9 , in order to be able to present the video content on the display 275 of the display device 200 , the transmission of the video content may also be completed through the established screen projection data channel. Among them, the screen projection function can include two types, one is that the display device 200 displays all the pictures displayed by the communication terminal 100A, including the UI interface and the playback interface on the communication terminal 100A, and the content displayed by the display device 200 follows that of the communication terminal 100A. The other is that the display device 200 only displays part of the screen in the communication terminal 100A, for example, the screen projection display device 200 can only display the playback interface, and during the screen projection process, the communication terminal has nothing to do with playback. Other operations do not affect the content displayed by the display device 200 .

For the screen projection function, after the communication terminal 100A establishes a screen projection connection with the display device 200, the communication terminal 100A may send the screen projection data to the display device 200 through the screen projection data channel, and the screen projection data may include the data displayed on the communication terminal 100A. The data corresponding to the picture may also include data corresponding to the video to be shared. The corresponding user operation process is as follows: the user first selects a video file on the communication terminal 100A to play, and after the communication terminal 100A displays the playing interface, the user activates the screen-casting function through the "cast screen" button on the playback interface, and the played video is displayed. The video-related data is sent to the display device 200 so that the display device 200 can display the corresponding video content on the display 275 . After the above operations are completed, the communication terminal 100A may no longer display the video content being played, and maintain the screen projection function running in the background, so as to avoid the impact of the user operation on the projection screen.

It can be seen that in the above two video sharing modes, the user needs to first select a video file to be shared in the communication terminal 100A, and then push the video file to the display device 200 . The process of user selection of video files can be done in the file management interface or in an application with file sharing function. For example, the user can run the media asset player program, such as "XX video", etc., call the "screen projection" function in the player program, select local resource projection or network resource projection, and select the corresponding video file to start projection. screen function.

On the communication terminal 100A, the video file can usually be displayed in the form of a file icon and a file name. For example, in the file management interface, a plurality of video file icons stored in the communication terminal 100A can be displayed in sequence, and a file name, such as "Video 1.AVI", is displayed on the right or below each icon. Since different video files correspond to different specific video contents, the method of displaying the video files through icons and file names cannot display the specific contents of the video files, so that when the number of video files stored in the communication terminal 100A is large, the user cannot Determining which video file is to be shared requires the user to determine the video content through multiple open/close operations.

To this end, the content of the video file can be displayed through thumbnails instead of file icons. The thumbnail image may be generated from the first frame of the video file or the video image of a specific frame. However, due to the diversity of video file content, some videos cannot be effectively distinguished by a single thumbnail. For example, for a movie-type video file, the first frame of the video image is usually a public-screening license screen, which will cause the thumbnails of the movie-type video files to be the public-screening license screen, so the content displayed by the thumbnail of the screen cannot be effectively distinguished. With multiple movie and video files, the user still cannot accurately select the video file to be shared.

In order to facilitate the user to select the video file to be shared, as shown in FIG. 10 and FIG. 11 , some embodiments of the present application provide a multi-screen interactive video browsing method, which can be applied to the communication of establishing a communication connection with the display device 200 Terminal 100A, the method includes the following steps:

Get multi-screen interactive instructions entered by the user.

The multi-screen interactive instruction refers to an instruction input according to the user's interactive action to activate the video sharing function. For example, when the user starts an application related to the screen projection function, it is determined that the user has input a multi-screen interaction instruction. For some communication terminals, multi-screen interactive instructions can also be generated by specific operation actions. For example, inputting a three-finger swipe up instruction on the communication terminal 100A that supports touch operation will activate the screen projection function of the communication terminal 100A. Therefore, when the user inputs a three-finger swipe up instruction, it means that the user has entered a multi-screen interactive instruction .

In some embodiments, in order to input the multi-screen interactive instruction, the communication terminal 100A may first detect the application startup action input by the user, and then parse the application program specified to run in the application startup action. If the specified running application is a multi-screen interactive application, a multi-screen interactive instruction is generated.

Wherein, the application startup action may have different expression forms according to different UI interaction modes of the communication terminal 100A. For example, the application startup action may be a click touch instruction input by the user in the application program interface of the communication terminal 100A, and the position is within the range of the application icon. After the user inputs the application startup action, the communication terminal 100A can analyze the application program specified to run in the application startup action, and at the same time determine the type of the application program to be started. The multi-screen interactive instruction enables the processor 180 to obtain the multi-screen interactive instruction input by the user.

Since the implementation of the multi-screen interaction function is based on the established communication connection relationship between the communication terminal 100A and the display device 200 , in some embodiments, the multi-screen interaction instruction may further include an operation instruction related to establishing a communication connection. For example, when the user inputs a multi-screen interactive instruction on the communication terminal 100A, the communication terminal 100A can detect whether the display device 200 is currently connected. If the communication terminal 100A has been connected to the display device 200, a corresponding screen-casting data channel is established so as to perform subsequent actions related to file sharing; if the communication terminal 100A has not established a communication connection with the display device 200, the established connection can be displayed on the communication terminal Related setting interface to guide the user to establish a communication connection relationship.

In addition, if the communication terminal 100A establishes a wireless connection relationship with multiple display devices 200, after inputting the multi-screen interactive command, the device list can also be displayed on the communication terminal 100A, and the user can further select the target to be pushed in the list, so that the video The file is pushed to the target display device 200 .

A video sharing interface is displayed through the communication terminal.

After the user inputs the multi-screen interactive instruction, the communication terminal 100A may display a video sharing interface. As shown in FIG. 8 , the video sharing interface includes a plurality of video catalog items, and each of the video catalog items includes a plurality of key frame images extracted from the video to be shared. In this embodiment, the video sharing interface is used to display each shareable video file for the user to select. Each shareable video file can be displayed in the form of a video catalog item in the video sharing interface. The user can move the focus mark to any video catalog item through the interactive UI, and then select the video catalog item, that is, select the desired video catalog item. shared video files.

Among them, the focus mark is a mark form used to represent the selected object, which can specifically be expressed as a pointer, a box, a circle and other logo graphics, and can also be in the form of interface display effects such as highlighting and zooming in. For example, when the user clicks in the area where the video catalog item corresponding to video 2 is located in the video sharing interface, the focus mark can be set on the video catalog item corresponding to video 2, and the UI interface of the communication terminal 100A can set the video catalog item corresponding to video 2 at this time. The directory item is enlarged and highlighted to indicate where the current focus marker is located.

Depending on the shape and area of the display area of the communication terminal 100A, different display effects may be presented in the video sharing interface. For example, as shown in FIG. 12 , when the communication terminal 100A is a mobile phone placed vertically, since its display unit 130 is in a vertical display state with a width smaller than its height, the video sharing interface can also be presented in a vertical state with a width smaller than its height. , and arrange each video catalog item to be shared from top to bottom in order to form a list effect.

Obviously, in order to find the specified video file conveniently, in the video sharing interface, the video directory items can be arranged according to the video name or according to the modification date of the video file. It is also possible to set a multi-level menu in the video sharing interface to display the video file list according to the storage location of the video file in the memory 120 . For example, multiple folders can be set in the video sharing interface according to the source of the video file, including "movie, short video, camera", etc., and the user can choose from different folders according to the type of video to be shared.

When there are multiple video files stored in the communication terminal 100A, in order to effectively distinguish the video files in the communication terminal 100A, the video directory item may include multiple images, that is, in the video sharing interface, each video directory item may include multiple images. Consists of multiple images. where images are keyframes extracted from this video file. Obviously, when the contents of the video files are different, the contents of the extracted key frame images are also different. Therefore, the video files can be effectively distinguished by using multiple key frame images, so that the user can know the contents of the video files in time.

For example, for a certain video file, its video directory entry may consist of 10 key frame images in the first 1-10s of the video file. For video files with different contents, the probability that the 10 key frame images are identical or similar is extremely low, so different video contents can be effectively distinguished through multiple key frame images. In addition, for a general video file, the user can understand the video content contained in the video file through its key frame images within 1-10s, so as to facilitate the user to complete the selection and improve the efficiency of the video sharing operation.

It can be seen from the above technical solutions that the multi-screen interactive video browsing method provided by the above embodiments can be configured in the processor 180 of the communication terminal 100A, so that the video sharing interface can be displayed after the communication terminal 100A obtains the multi-screen interactive instruction input by the user. . And the video sharing interface includes multiple video catalog items, and each video catalog item includes multiple key frame images extracted from the video to be shared. The method can display the video file through multiple key frame images, so that the user can fully understand the video content, and it is convenient to accurately select the video file to be shared.

In the above embodiments, since the video sharing interface and the file management interface display different video files, in order to present the video sharing interface, in some embodiments, the step of controlling the display unit to display the video sharing interface further includes the following steps :

Traverse video resource files.

After acquiring the multi-screen interaction instruction input by the user, the communication terminal may traverse the video resource files that can be shared in the communication terminal 100A before displaying the video sharing interface. Wherein, the video resource files include local video files stored in the communication terminal and network resource files displayed through the resource interface of the communication terminal. That is, during the video sharing process, the user can not only share the video files stored locally, but also share the video files in the network.

Obviously, for different forms of video files, the data transmitted in the subsequent video sharing process are different. For example, for a local video file, the communication terminal 100A needs to send the video file to the display device 200 , so a video data stream can be formed between the communication terminal 100A and the display device 200 . For the network resource file, the communication terminal 100A needs to send the address information of the network resource file to the display device 200, such as a URL address, so that the display device 200 can obtain the video file from the network server according to the address information. A video data stream is formed with the server 400 .

Video data within a preset time period is decoded in each video resource file.

By traversing the video resource files in the communication terminal 100A, the communication terminal 100A can parse some segments of each video file traversed, that is, decode video data within a preset time period in each video resource file. Since the communication terminal 100A can only parse a part of the video clips, after traversing the video resource files, the communication terminal 100A can complete the video decoding in a relatively short time.

After traversing the video files, the communication terminal 100A can first judge the video length, and if the video length exceeds the preset video length value, it can parse the video content within a specific time period. For example, when the video length exceeds 10s, the communication terminal 100A can parse the video content within 1-10s. If the video resource is a shorter video with a length lower than the preset video length value, the entire video can be parsed. For example, for a resource file whose total video length is less than 10s, it can be directly extracted according to the entire video length.

Extracts a set number of keyframe images from video data, and caches keyframe images.

After partially parsing the video file, the communication terminal 100A may also extract key frames from the video content to obtain key frame images, thereby forming a video directory item. For example, after entering the application, the communication terminal 100A can execute the above program in the background, and traverse the video files that can be shared in the communication terminal 100A to obtain five video files. Then cache the key frames of the video clips from the 1st second to the 10th second of the 5 videos, each with a frame of 5 × 10 frames of images. Since the background program does not directly cache all video keyframes, unnecessary memory usage can be reduced.

The key frame images are arranged according to the time sequence of the key frame images in the video file to generate a video directory entry.

The cached key frame images can be grouped according to the extraction source of the key frame images, that is, the key frame images originating from the same video file are regarded as a group. Then, according to the time sequence of the key frame images in the video file, the key frame images in the same group are arranged to generate a video directory item.

For example, in the 1-10s content of video 1, the times of the extracted key frames in the video file are: 00:01:30, 00:01:80, ..., 00:10:00, then you can use The extracted key frame images are arranged in this order, so that these key frame images are displayed in the first row. Similarly, the same arrangement is also adopted for video 2, and the key frame image is displayed in the second row.

According to the arrangement sequence of the video files, the display unit is controlled to display the video directory item corresponding to each video file.

After extracting the key frame images, the communication terminal 100A may display the extracted key frame images in groups according to the arrangement sequence of the video files, thereby forming a video sharing interface. For example, the key frame images corresponding to 5 videos are displayed in the video sharing interface.

Due to the limitation of the display area of the communication terminal 100A, not all the key frame images contained in each video catalog item can be displayed, so only part of the key frame images may be displayed in each video catalog item, while other undisplayed images are displayed. It can be displayed by further operations, such as paging, sliding, etc. For example, in the video sharing interface, each video directory item area can display the entire contents of the first three frames of images, and by swiping, you can continue to view all the key frame images of the video.

It should be noted that since different video file compression methods have different differences, the number of key frame images extracted is also different. Therefore, in this embodiment of the present application, the key frame images extracted from video data can be larger than After the preset number, it is dynamically set according to the processing capability of the communication terminal 100A and the number of videos that can be shared in the communication terminal 100A. For example, for the communication terminal 100A with strong processing capability and a small number of shareable video files, the number of key frame images in each video directory item can be increased, so that the user can learn more video content information; The communication terminal 100A with weak processing capability or a large number of shareable video files can set the number of key frame images in each video directory item to be equal to the preset number, so as to cache key frames corresponding to multiple video files as soon as possible image.

In addition, in order to present the video sharing interface in time, the communication terminal 100A can display the sharing interface in real time during the process of caching the key frame image, and further display the video of the corresponding video file according to the cache progress of the key frame image and the operation process of the user. directory entry. For example, when a maximum of 5 video directory items can be displayed in the video sharing interface on one page, the communication terminal 100A can display the sharing interface after caching the key frame images of the first 5 video files. That is, in the initial display state, only the video directory items of the first five video files may be included in the video sharing interface for the user to operate. At the same time, the communication terminal 100A also maintains a related program for buffering key frame images in the background, and then buffers the key frame images of the sixth and subsequent video files in real time, so as to improve page display efficiency.

In the above embodiment, the key frame image refers to an image that can represent the video content, and is usually an uncompressed original image image in the video file. Taking the IDR-based image compression method as an example, the key frame is the I frame, and the non-key frame is the P frame or B frame. Therefore, in order to extract key frame images, the step of extracting a set number of key frame images from the video data further includes:

Parse the video file to obtain the video stream within a preset time period;

extracting key bytes from the video code stream;

Converting the code stream value of the key byte bit into a binary value, and converting the preset bit value of the binary value into a decimal value;

If the decimal value is equal to the key frame judgment value, extract the image corresponding to the video stream to generate a key frame image.

Wherein, the key byte is a specific length byte after the start code in the video stream. For example, if a video stream is: 00 00 00 01 41 E6 60..., then "00 00 00 01" in the video stream is the start code, and the next byte after the start code can be detected The type of this frame, that is, the key byte bit is 0x41, which is converted into a binary value of 0100 0001. According to the conversion of the last 5 digits of the binary value to the decimal value of 1, it can be determined that this code stream is a slice of a non-partitioned, non-IDR image, that is, a P frame.

If another code stream is: 00 00 00 01 65 E8..., then the key byte is 0x65, and the binary value is 0110 0101. Then, converting the decimal value to 5 according to the last 5 digits of the binary value, it can be determined that this code stream is a slice in the IDR image and is a key frame, that is, an I frame.

The above key frame extraction method may be configured in the memory 120 of the communication terminal 100A in the form of a control program for the processor 180 to call. The specific control program expression form can be compiled according to the operating system type of the communication terminal 100A. Taking the Android platform as an example, the configured keyframe program can be expressed as:

MediaMetadataRetriever retriever=new MediaMetadataRetriever();

retriever.setDataSource(dataPath);

That is to get the key frame of the ith second of the video:

bitmap=retriever.getFrameAtTime(i, MediaMetadataRetriever.OPTION_CLOSEST);

Among them, bitmap is the key frame image of the ith second.

It can be seen that by extracting the value of the key bytes in the video stream and converting the binary and decimal values, the key frame image contained in the video stream can be determined, and the key frame image extraction can be completed.

It can be seen from the above technical solutions that the communication terminal in the above embodiment can first extract key frame images from the shareable video file after acquiring the multi-screen interactive instruction input by the user, and then use the key frame images to form a video directory item and display it in the video directory. in the video sharing interface. Therefore, in the above embodiment, multiple key frame images may be used instead of monotonous icons or thumbnails to display video files, so that users can obtain video content in time, and it is convenient to select video files to be shared to complete video sharing.

Since the area occupied by the video directory item is limited, the communication terminal 100A cannot display all of the extracted key frame images, but can only display part of the key frame images. For example, in a line of video catalog item area, the display unit of communication terminal 100A can only display the first three key frame images. In some video files, the first three key frame images are not enough to distinguish the video files. Therefore, in some embodiments, further operations may be performed to enable the communication terminal 100A to display other undisplayed or undisplayed images. The fully displayed keyframe image. That is, the method further includes the following steps:

Get the page turning instruction entered by the user.

The page-turning instruction refers to an instruction input according to a user's interactive action to perform a page-turning function, which may be a sliding instruction input by the user in the video catalog item area under the current focus mark. For example, if the user wants to know the video content of the video 2, he can first click on the video catalog item corresponding to the video 2 to obtain the focus mark. At this time, the communication terminal 100A can highlight the video catalog item of the video 2 . After highlighting, if the user still cannot know the video content through the displayed key frame image, he or she can input a sliding instruction to drag to the left in the video catalog item area to input an instruction for turning pages.

Obviously, according to the different interaction modes that the communication terminal 100A can provide, the page turning instructions input by the user are also different. For the communication terminal 100A that does not support touch operation, a page turning button may be set in the interactive UI of the communication terminal 100A under the video directory item with the focus mark, and the page turning instruction can be input when the user clicks the page turning button. For the communication terminal 100A that only supports physical case operations, the page turning instruction can also be input when the focus mark is on the video directory item and the communication terminal 100A presses the "left" key or the "right" key.

The display unit is controlled to scroll and display a plurality of key frame images in the video catalog item area according to the sliding direction of the sliding instruction.

After the user inputs a page turning instruction, the communication terminal 100A can scroll and display the positions of multiple key frame images in response to the page turning instruction, so that other key frame images can be displayed in the video catalog item. For example, after the user inputs the sliding instruction of dragging to the left in the video catalog item area, the communication terminal 100A can follow the action of dragging to the left, and sequentially display the fourth key frame image, the fifth key frame image, etc. until the user Stop entering page turning commands when you know the video content, or stop after all key frame images have been displayed.

In some embodiments, if the user completes the display of all the key frame images through the page turning instruction, the video content still cannot be known. The communication terminal 100A may also continue to extract key frame images from the video file after displaying a preset number of key frame images, and display them in the video directory item in real time. For example, after the user drags the 10th key frame image in the area of video 2 through the touch action of dragging to the left, and still cannot distinguish the video content, he can continue to input the touch command of dragging to the left to trigger the communication The terminal 100A continues the function of extracting key frame images. At this time, the communication terminal 100A can obtain a key frame image in the video content corresponding to 10-20s of video 2 by using the above-mentioned key frame extraction method. And after a new key frame image is obtained, it is displayed behind the 10th key frame image, that is, the 11th key frame image. And so on, until the user knows the video content corresponding to the video 2 through the key frame image.

Because the corresponding area of video directory items is limited, and in order to present video directory items of multiple video files, it is usually necessary to display multiple key frame images in the video sharing interface. Therefore, in the video sharing interface, it is necessary to simplify the processing of key frame images. , that is, displayed by thumbnails. However, since some details will be lost after the thumbnails are reduced, in order to facilitate users to view, as shown in FIG. 14 , in some embodiments of the present application, the video catalog items can be further displayed through the browsing interface, that is, the method described above Also includes the following steps:

Get the browsing instructions entered by the user.

Wherein, the browsing instruction is an instruction input by the user's interactive action, and is used to control the communication terminal 100A to display the complete key frame image. For example, as shown in FIG. 13 , when the user inputs a long-press instruction on the video directory item with the focus mark, the browsing instruction input by the user is acquired. For another example, the user may also input a browsing instruction by clicking the "Browse" button in the video sharing interface.

Extract the key frame image contained in the video directory item under the current focus cursor.

After acquiring the browsing instruction input by the user, the communication terminal 100A may extract the selected key frame image from the cache in response to the browsing instruction. For example, after the user inputs a long-press command in the video directory item area corresponding to video 2, it means that the user wants to browse video 2, so all key frame images corresponding to video 2 can be extracted from the cache for display in the browsing interface .

The display unit is controlled to display a browsing interface.

After extracting the key frame images, the communication terminal 100A can switch from the video sharing interface to the browsing interface, and display a plurality of key frame images one by one in the browsing interface. Wherein, as shown in FIG. 10 , the extracted key frame images are displayed in sequence according to preset time intervals in the browsing interface. In order to display the complete original keyframe image, in the browsing interface, you can set a content area where the user displays the keyframe image, and the keyframe image can be displayed cyclically at preset time intervals in this area. The time interval of cyclic display can be customized as required, for example, the display time interval between two adjacent key frame images is 1-5s.

It can be seen that, in this embodiment, the key frame images in the video are displayed in detail through the browsing interface, so as to overcome the lost details in the thumbnail images, so as to facilitate the user to identify the video content.

In some embodiments, the key frame image displayed in the browsing interface may also be related to the browsing instruction input by the user. That is, after acquiring the browsing instruction, the communication terminal 100A may also parse the initial key frame image specified in the browsing instruction, and control the display unit to sequentially display the initial key frame and the key frame images after the initial key frame. Wherein, the starting key frame image is a key frame image in which the user performs a long-press action in the video sharing interface.

For example, for movie video files, since the key frame images at the beginning of the file tend to be the same, such as the license screen for public screening, the publisher's logo, etc., the user can choose to skip these repeated content and directly start from the third key frame image. Browse. That is, the user can input a long-press command on the third key frame image to input a browsing command. However, after receiving the long-press instruction, the communication terminal 100A can obtain the third key frame image corresponding to the long-press instruction. Therefore, when displaying the browsing interface, the third key frame image can be displayed in the content area. And according to the preset time interval, the 4th, 5th and subsequent key frame images are displayed.

In addition to the content area, other controls may also be included in the browsing interface for controlling the display effect in the browsing interface. For example, multiple controls can be displayed below the content area, namely: "Previous", "Next", "Play", "Pause", "Push". The user can click any control to implement the corresponding function. For example, the user can switch the video catalog items displayed in the browsing interface by clicking the "previous" and "next" button controls. Therefore, as shown in FIG. 15 , in some embodiments, after displaying the browsing interface, the method may further include:

Get the switch instruction entered by the user.

The switching instruction can be input by clicking a button control in the browsing interface, or can be input by a specific action gesture. For example, the user may input a switching instruction by inputting a sliding instruction of dragging up or down in the content area.

In response to the switching instruction, keyframe images of adjacent directory entries are extracted.

After the user inputs the switching instruction, the communication terminal 100A can display the Linji directory item in the browsing interface in the same manner as in the above-mentioned embodiment. Wherein, the adjacent catalog item is the previous or next video catalog item of the video catalog item under the current focus mark. For example, video 2's adjacent directory entry is either video 1 or video 3. Which adjacent directory item to switch to can be determined according to the switching instruction input by the user. For example, when the user clicks the "previous" button, the communication terminal 100A extracts the key frame image corresponding to the video 1; when the user clicks the "next" button, the The communication terminal 100A extracts the key frame image corresponding to the video 3 .

Controlling the display unit to display the key frame image of the adjacent resource on the browsing interface.

After extracting the key frame images of the adjacent directory items, the communication terminal 100A may display the extracted key frame images in the browsing interface to complete the video browsing switching. For example, after browsing the key frame images of video 2, the user determines that video 2 is not the video that you want to share, and can directly click the "Next" button in the browsing interface to switch to the key frame images of video 3. The communication terminal 100A can display the key frame image of the video 3 in the content area. The display mode of image 3 can be the same as that of video 2.

It can be seen that in this embodiment, by performing the switching operation in the browsing interface, it is possible to quickly switch to the directory items of different videos, which is convenient for the user to understand the video content in a clearer environment, so as to determine whether the current video file is to be shared. video file.

In addition, after viewing the video content through the above-mentioned video sharing interface and browsing interface, if it is determined that a certain video file is a file to be shared, the user may further input a push instruction, so that the communication terminal 100A sends the corresponding video file to the display The device 200 completes video sharing.

The specific push method may be as follows: on the video sharing interface, the user can input a push instruction by double-clicking the video catalog item, and the communication terminal 100A sends the video resource file corresponding to the catalog item to the display device 200; on the browsing interface, the user can click the " Push" button control to input a push instruction, the communication terminal 100A also sends the currently browsed video resource file to the display device 200 to complete the video sharing.

Based on the above multi-screen interactive video browsing method, some embodiments of the present application further provide a communication terminal 100A, including: a display unit 130 , a communication circuit, and a processor 180 . The display unit 130 is configured to present various user interfaces and to present a video sharing interface. The communication circuit may include communication circuits such as the Bluetooth circuit 181 , the Wi-Fi circuit 170 , etc., and is configured to establish a communication connection with the display device 200 to push the shared video data to the display device 200 . The processor 180 is configured to perform the following program steps:

Obtain the multi-screen interactive instructions entered by the user;

In response to the multi-screen interactive instruction, the display unit is controlled to display a video sharing interface.

Wherein, the video sharing interface includes a plurality of video catalog items, and each of the video catalog items includes a plurality of key frame images extracted from the video to be shared.

It can be known from the above technical solutions that the communication terminal 100A provided in the above embodiments can display a video sharing interface in the display unit 130 according to the multi-screen interactive instruction input by the user. In the video sharing interface, a plurality of video catalog items corresponding to the videos to be shared may be included for the user to browse and view. In addition, each video catalog item includes multiple key frame images extracted from the video to be shared, and through multiple key frame images, the video content within a preset time period can be displayed, so that the user can understand the video content and select it. The video file to be shared is pushed to the display device 200, which solves the problem that the traditional video file browsing method is not conducive to the user selecting the video resource file to be shared.

Based on the above communication terminal 100A, some embodiments of the present application further provide a display device 200 , including a display 275 , a communicator 220 and a controller 250 . The display 275 is configured to display the user interface and the video content shared by the communication terminal 100A, and the communicator 220 is configured to establish a communication connection with the communication terminal 100A to obtain the shared video data at the communication terminal 100A.

As shown in Figure 16, the controller 250 is configured to perform the following procedural steps:

After establishing a communication connection with the communication terminal, obtain the video file pushed by the communication terminal;

Parse the video file, and control the display to display the video content of the video file.

Wherein, the video file is a video file selected by the user in the video sharing interface of the communication terminal; the video sharing interface includes a plurality of video catalog items, and each of the video catalog items includes a plurality of video catalog items extracted from the video to be shared keyframe image.

It can be seen from the above technical solutions that the display device 200 provided by the above embodiments can receive the video file pushed by the communication terminal in real time through the communicator 220 after the communication connection is established, and parse the received video file to generate specific video screen content, Finally, the video screen content is displayed through the display 275 . Since the video file received by the display device 200 is the video file selected by the user in the video sharing interface, and in the video sharing interface, the video file is represented by a plurality of video directory items, so the received video can directly display the user's The shared video content reduces the number of repeated user operations and improves user experience.

Based on the above-mentioned communication terminal 100A and the display device 200, some embodiments of the present application further provide a multi-screen interactive video browsing system, including the communication terminal 100A and the display device 200, and the communication terminal 100A and the display device 200 establish communication connect;

The communication terminal 100A is configured to obtain a multi-screen interactive instruction input by a user, and in response to the multi-screen interactive instruction, display a video sharing interface, where the video sharing interface includes a plurality of video catalog items, each of the The video catalog item includes a plurality of key frame images extracted from the video to be shared; and,

Pushing a video file to the display device 200, where the video file is the video to be shared selected by the user in the video sharing interface of the communication terminal;

The display device 200 is configured to: acquire a video file pushed by the communication terminal 100A, parse the video file, and display the video content of the video file.

It can be known from the above technical solutions that the multi-screen interactive video browsing system provided by the fourth aspect of the present application includes a communication terminal and a display device that establish a communication connection with each other. The communication terminal may display a video sharing interface after acquiring the multi-screen interactive instruction, wherein the video sharing interface includes a plurality of video catalog items, and each of the video catalog items includes a plurality of key items extracted from the video to be shared. frame image. After the user selects any video to be shared, the communication terminal also pushes the selected video file to the display device, so that the display device can display the specific video content of the video file after acquiring the video file. The system displays video files through a plurality of key frame images, so that the user can fully understand the video content, and it is convenient to accurately select the video file to be shared.

Similar parts between the embodiments provided in the present application may be referred to each other. The specific embodiments provided above are just a few examples under the general concept of the present application, and do not constitute a limitation on the protection scope of the present application. For those skilled in the art, any other implementations expanded according to the solution of the present application without creative work fall within the protection scope of the present application.

Claims (10)

1.A communication terminal, comprising:

a display unit;

a communication circuit configured to establish a communication connection with a display device;

a processor configured to:

acquiring a multi-screen interaction instruction input by a user;

responding to the multi-screen interaction instruction, and controlling the display unit to display a video sharing interface, wherein the video sharing interface comprises a plurality of video directory entries, and each video directory entry comprises a plurality of key frame images extracted from the video to be shared.

2. The communication terminal according to claim 1, wherein in the step of obtaining the multi-screen interaction instruction input by the user, the processor is further configured to:

detecting an application starting action input by a user;

analyzing the application program appointed to run in the application starting action;

and if the specified running application program is a multi-screen interaction application, generating the multi-screen interaction instruction.

3. The communication terminal according to claim 1, wherein in the step of controlling the display unit to display the video sharing interface, the processor is further configured to:

traversing video resource files, wherein the video resource files comprise local video files stored in the communication terminal and network resource files displayed through a resource interface of the communication terminal;

decoding video data in a preset time period in each video resource file;

extracting a set number of key frame images from the video data;

and caching the key frame image.

4. The communication terminal according to claim 3, wherein in the step of controlling the display unit to display the video sharing interface, the processor is further configured to:

arranging the key frame images according to the time sequence of the key frame images in the video file to generate a video directory entry;

and controlling a display unit to display the video directory entries corresponding to the video files according to the arrangement sequence of the video files.

5. The communication terminal according to claim 3, wherein in the step of extracting a set number of key frame images from the video data, the processor is further configured to:

analyzing the video file to obtain a video code stream in a preset time period;

extracting key byte bits from the video code stream, wherein the key byte bits are byte bits behind a start code in the video code stream;

converting the code stream numerical value of the key byte bit into a binary numerical value, and converting a preset bit numerical value of the binary numerical value into a decimal numerical value;

and if the decimal value is equal to the key frame judgment value, extracting the image corresponding to the video code stream to generate a key frame image.

6. The communication terminal of claim 1, wherein the processor is further configured to:

acquiring a page turning instruction input by a user, wherein the page turning instruction is a sliding instruction input by the user in the video directory entry area under the current focus mark;

and responding to the page turning instruction, controlling the display unit to scroll and display a plurality of key frame images in the video directory entry area according to the sliding direction of the sliding instruction.

7. The communication terminal of claim 1, wherein the processor is further configured to:

acquiring a browsing instruction input by a user;

in response to the browsing instruction, extracting a key frame image contained in the video directory item under the current focus cursor;

and controlling the display unit to display a browsing interface, wherein the extracted key frame images are sequentially displayed in the browsing interface according to a preset time interval.

8. A multi-screen interactive video browsing method is applied to a communication terminal which establishes communication connection with display equipment, and comprises the following steps:

acquiring a multi-screen interaction instruction input by a user;

responding to the multi-screen interaction instruction, and displaying a video sharing interface through the communication terminal, wherein the video sharing interface comprises a plurality of video directory items, and each video directory item comprises a plurality of key frame images extracted from the video to be shared.

9. A display device, comprising:

a display;

a communicator configured to establish a communication connection with a communication terminal;

a controller configured to:

after communication connection is established with a communication terminal, a video file pushed by the communication terminal is obtained, wherein the video file is a video to be shared selected by a user in a video sharing interface of the communication terminal; the video sharing interface comprises a plurality of video directory entries, and each video directory entry comprises a plurality of key frame images extracted from videos to be shared;

and analyzing the video file, and controlling the display to display the video content of the video file.

10. A multi-screen interactive video browsing system is characterized by comprising a communication terminal and display equipment, wherein the communication terminal is in communication connection with the display equipment;

the communication terminal is configured to: acquiring a multi-screen interaction instruction input by a user, responding to the multi-screen interaction instruction, and displaying a video sharing interface, wherein the video sharing interface comprises a plurality of video directory items, and each video directory item comprises a plurality of key frame images extracted from a video to be shared; and the number of the first and second groups,

pushing a video file to the display equipment, wherein the video file is a video to be shared selected by a user in a video sharing interface of the communication terminal;

the display device is configured to: the method comprises the steps of obtaining a video file pushed by a communication terminal, analyzing the video file, and displaying video content of the video file.

Images