KR102679418B1 - Electronic device for improving quality of call and method for the same - Google Patents

Abstract

In an electronic device and a method of operating the electronic device according to various embodiments, the electronic device includes a communication module that transmits or receives data using a call channel established through a call connection with an external electronic device; and a processor, wherein the processor transmits a call connection request message for establishing a call channel between an external electronic device and the electronic device, receives a response message from the external electronic device, and connects the call included in the response message. Determining whether to perform a pre-processing operation of the content using a transmission filter that changes the quality of the content transmitted through the call connection based on capability information of the external electronic device related to the content, and pre-processing the content A call connection confirmation message indicating whether to perform an operation is transmitted to the external electronic device, the communication module is controlled to complete the call channel establishment, and the connection request message includes performance information of the electronic device related to the filter. It can be included.
In addition, various embodiments are possible.

Description

Electronic devices and methods of operating electronic devices that improve the quality of calls {ELECTRONIC DEVICE FOR IMPROVING QUALITY OF CALL AND METHOD FOR THE SAME}

Various embodiments of the present invention relate to electronic devices and methods of operating the electronic devices.

Various electronic devices such as smart phones, tablet PCs, portable multimedia players (PMPs), personal digital assistants (PDAs), laptop personal computers, and wearable devices are becoming popular. there is.

Recent electronic devices can provide voice calls or video calls based on IMS (IP multimedia subsystem) or RCS (rich communication suite). Processing may be performed on the audio or video data to reduce the size of the transmitted or received audio or video data while providing high quality audio or video. Processing of audio data or video data can be largely implemented in two steps. A processing step for audio raw data generated using a microphone or video raw data generated using a camera. A processing step that reduces the size of the data while minimizing quality degradation using a filter and audio data or video While performing compression on the data, there may be processing steps that reduce the size of the data.

When a call connection is performed, a negotiation process based on SDP (session description protocol) may be performed between electronic devices to set the environment for data to be transmitted and received, and during the negotiation process, voice data or video data that can be supported between electronic devices may be negotiated. Compression format, sampling rate of audio data, or resolution information of video data may be exchanged.

In order to improve the quality of transmitted and received voice data or video data, a preprocessing operation may be performed on the voice data or video data using a filter before the voice data or video data is encoded. Recently, due to the development of technologies related to artificial intelligence, machine learning, or deep learning, and the emergence of neural processing units, the processing of voice data or video data using filters has increased. Quality improvement is being studied.

In the SDP-based negotiation process that occurs when a call connection is performed, the transmitted and received data may not include information about the filter. Therefore, the electronic device cannot know which filter the external electronic device on the other end of the call connection is using to preprocess the voice data or video data, and quickly performs operations to improve voice or video quality using the filter. You may not be able to do it.

Alternatively, in a situation where information about a filter is exchanged after a call connection, the call may be temporarily interrupted as the settings of components of the electronic device are changed due to activation of the filter.

An electronic device according to various embodiments of the present invention includes a communication module that transmits or receives data using a call channel established through a call connection with an external electronic device; and a processor, wherein the processor transmits a call connection request message for establishing a call channel between an external electronic device and the electronic device, receives a response message from the external electronic device, and connects the call included in the response message. Determining whether to perform a pre-processing operation of the content using a transmission filter that changes the quality of the content transmitted through the call connection based on capability information of the external electronic device related to the content, and pre-processing the content A call connection confirmation message indicating whether to perform an operation is transmitted to the external electronic device, the communication module is controlled to complete the call channel establishment, and the connection request message includes performance information of the electronic device related to the filter. It can be included.

An electronic device according to various embodiments of the present invention includes a communication module for transmitting or receiving data using a call channel established through a call connection with an external electronic device, and a processor, the processor Receives a call connection request message for establishing a call channel between an external electronic device and the electronic device, and sends a response message containing performance information of the electronic device related to a reception filter for preprocessing content received through the call connection. Transmit to an external electronic device, receive a call connection confirmation message from the external electronic device including an indicator indicating whether to perform a pre-processing operation of the content using a transmission filter corresponding to the reception filter, and based on the indicator, It may be set to perform an operation of activating the reception filter.

A method of operating an electronic device according to various embodiments of the present invention includes transmitting a call connection request message to establish a call channel between an external electronic device and the electronic device; Receiving a response message from the external electronic device, changing the quality of content transmitted through the call connection based on capability information of the external electronic device related to the call connection included in the response message. An operation of determining whether to perform a pre-processing operation of the content using a transmission filter, an operation of transmitting a call connection confirmation message indicating whether to perform a pre-processing operation of the content to the external electronic device, and completing the establishment of the call channel. It includes an operation, and the call connection request message may include performance information of the electronic device related to the transmission filter.

Electronic devices and operating methods of electronic devices according to various embodiments of the present invention include sending a call connection request message or response message containing information instructing processing of voice data or video data using a filter during the negotiation process when connecting a call. It can transmit or receive, allowing you to check the filter used by an external electronic device. Accordingly, high quality voice data or video data can be obtained by using a filter corresponding to the filter used by the external electronic device.

Electronic devices and operating methods of electronic devices according to various embodiments of the present invention include sending a call connection request message or response message containing information instructing processing of voice data or video data using a filter during the negotiation process when connecting a call. It can transmit or receive, enabling filters and related components to be activated before the call connection is complete. Accordingly, after the call connection is completed, the filter is activated, thereby preventing possible degradation of the quality of the call channel and improving the quality of the call through the call channel.

Electronic devices and operating methods of electronic devices according to various embodiments of the present invention include determining whether to support video calls based on metadata and generating model information or metadata to be used for video calls based on metadata in the negotiation process performed when connecting a call. By exchanging standard information used to make a call, components that support video calls based on metadata can be activated before the call connection is completed. Therefore, it may be possible to enter the video call mode in a relatively short period of time.

Electronic devices and operating methods of electronic devices according to various embodiments of the present invention can support video calls using metadata that has a size lower than that of video or image streams, so that a network smaller than a typical video call that exchanges video or image streams Video calls using bandwidth can be implemented.

1 is a block diagram of an electronic device according to various embodiments of the present invention.
Figure 2 is a block diagram of a program according to various embodiments.
3A and 3B are block diagrams of an example in which a first electronic device and a second electronic device transmit or receive audio or video content, according to various embodiments of the present invention.
4A is a block diagram of an electronic device according to various embodiments of the present invention.
Figure 4B is a block diagram of an electronic device according to various embodiments of the present invention.
5 is a block diagram of an electronic device according to various embodiments of the present invention.
FIG. 6 is an operation flowchart illustrating an operation in which a first electronic device and a second electronic device perform a call connection according to various embodiments of the present invention.
FIG. 7 is a diagram illustrating a screen for performing settings for content transmission or reception in a first electronic device and a second electronic device according to various embodiments of the present invention.
Figure 8 is an operational flowchart illustrating a method of operating an electronic device according to various embodiments of the present invention.
Figure 9 is an operation flowchart showing a method of operating an electronic device according to another embodiment of the present invention.
10A is a block diagram of an electronic device according to various embodiments of the present invention.
10B is a block diagram of an electronic device according to various embodiments of the present invention.
FIG. 11A is a diagram illustrating an example of extracting metadata from an acquired image in an electronic device according to various embodiments of the present invention.
FIG. 11B is a diagram illustrating an example of rendering an image based on a model and metadata in an electronic device according to various embodiments of the present invention.
FIG. 12 is an operation flowchart illustrating an operation in which a first electronic device and a second electronic device perform a call connection according to various embodiments of the present invention.
Figure 13 is an operation flowchart showing a method of operating an electronic device according to various embodiments of the present invention.
Figure 14 is an operation flowchart showing a method of operating an electronic device according to various embodiments of the present invention.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments. Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, and a sensor module ( 176), interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 ) may include. In some embodiments, at least one of these components (eg, the display device 160 or the camera module 180) may be omitted, or one or more other components may be added to the electronic device 101. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, the sensor module 176 (e.g., a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented while being embedded in the display device 160 (e.g., a display).

The processor 120, for example, executes software (e.g., program 140) to configure at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or operations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (e.g., a central processing unit or an application processor), and an auxiliary processor 123 that can operate independently or together (e.g., a graphics processing unit, an image signal processor). , sensor hub processor, or communication processor). Additionally or alternatively, the auxiliary processor 123 may be set to use less power than the main processor 121 or to specialize in a designated function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), along with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display device 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, coprocessor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or application 146.

The input device 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input device 150 may include, for example, a microphone, mouse, keyboard, or digital pen (eg, stylus pen).

The sound output device 155 may output sound signals to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display device 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display device 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display device 160 may include touch circuitry configured to detect a touch, or a sensor circuit configured to measure the intensity of force generated by the touch (e.g., a pressure sensor). there is.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input device 150, the sound output device 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through an electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 388 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module 190 provides a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, WiFi direct, or IrDA (infrared data association)) or a second network 199 (e.g., a cellular network, the Internet, or It can communicate with external electronic devices through a computer network (e.g., a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed and authenticated.

The antenna module 197 may transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module may include one antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected to the plurality of antennas by, for example, the communication module 190. can be selected. Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, RFIC) in addition to the radiator may be additionally formed as part of the antenna module 197.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the electronic devices 102 and 104 may be the same or different type of device from the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology. This can be used.

Figure 2 is a block diagram 200 illustrating program 140 according to various embodiments. According to one embodiment, the program 140 includes an operating system 142, middleware 144, or an application 146 executable on the operating system 142 for controlling one or more resources of the electronic device 101. It can be included. Operating system 142 may include, for example, Android ^TM , iOS ^TM , Windows ^TM , Symbian ^TM , Tizen ^TM , or Bada ^TM . At least some of the programs 140 are preloaded into the electronic device 101, for example, at the time of manufacture, or are stored in an external electronic device (e.g., the electronic device 102 or 104, or a server) when used by a user. It can be downloaded or updated from 108)).

The operating system 142 may control management (eg, allocation or retrieval) of one or more system resources (eg, process, memory, or power) of the electronic device 101 . Operating system 142 may additionally or alternatively operate on other hardware devices of electronic device 101, such as input device 150, audio output device 155, display device 160, audio module 170. , sensor module 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or It may include one or more driver programs for driving the antenna module 197.

The middleware 144 may provide various functions to the application 146 so that functions or information provided from one or more resources of the electronic device 101 can be used by the application 146. The middleware 144 includes, for example, an application manager 201, a window manager 203, a multimedia manager 205, a resource manager 207, a power manager 209, a database manager 211, and a package manager 213. ), connectivity manager (215), notification manager (217), location manager (219), graphics manager (221), security manager (223), call manager (225), or voice recognition manager (227). You can.

The application manager 201 may, for example, manage the life cycle of the application 146. The window manager 203 may, for example, manage one or more GUI resources used on the screen. For example, the multimedia manager 205 identifies one or more formats required for playing media files, and encodes or decodes the corresponding media file using a codec suitable for the selected format. It can be done. The resource manager 207 may, for example, manage the source code of the application 146 or the memory space of the memory 130. The power manager 209 manages, for example, the capacity, temperature, or power of the battery 189, and may use this information to determine or provide related information necessary for the operation of the electronic device 101. . According to one embodiment, the power manager 209 may interface with a basic input/output system (BIOS) (not shown) of the electronic device 101.

Database manager 211 may create, search, or change a database to be used by application 146, for example. The package manager 213 may, for example, manage the installation or update of applications distributed in the form of package files. The connectivity manager 215 may manage, for example, a wireless connection or direct connection between the electronic device 101 and an external electronic device. For example, the notification manager 217 may provide a function for notifying the user of the occurrence of a designated event (eg, an incoming call, message, or alarm). The location manager 219 may, for example, manage location information of the electronic device 101. The graphics manager 221 may, for example, manage one or more graphic effects to be provided to the user or a user interface related thereto.

Security manager 223 may provide, for example, system security or user authentication. The telephony manager 225 may manage, for example, a voice call function or a video call function provided by the electronic device 101. For example, the voice recognition manager 227 transmits the user's voice data to the server 108 and provides a command corresponding to a function to be performed in the electronic device 101 based at least in part on the voice data, Alternatively, text data converted based at least in part on the voice data may be received from the server 108. According to one embodiment, the middleware 244 may dynamically delete some existing components or add new components. According to one embodiment, at least a portion of the middleware 144 may be included as part of the operating system 142 or may be implemented as separate software different from the operating system 142.

The application 146 includes, for example, home 251, dialer 253, SMS/MMS (255), instant message (IM) 257, browser 259, camera 261, and alarm 263. , Contacts (265), Voice Recognition (267), Email (269), Calendar (271), Media Player (273), Album (275), Watch (277), Health (279) (such as exercise amount or blood sugar) It may include applications that measure biometric information) or environmental information 281 (e.g., measure atmospheric pressure, humidity, or temperature information). According to one embodiment, the application 146 may further include an information exchange application (not shown) that can support information exchange between the electronic device 101 and an external electronic device. The information exchange application may include, for example, a notification relay application configured to deliver designated information (e.g., calls, messages, or alarms) to an external electronic device, or a device management application configured to manage the external electronic device. there is. The notification relay application, for example, transmits notification information corresponding to a specified event (e.g., mail reception) generated in another application (e.g., email application 269) of the electronic device 101 to an external electronic device. You can. Additionally or alternatively, the notification relay application may receive notification information from an external electronic device and provide it to the user of the electronic device 101.

The device management application may, for example, control the power (e.g., turn-on or turn-on) of an external electronic device or some component thereof (e.g., display device 160 or camera module 180) that communicates with the electronic device 101. -off) or functions (e.g., brightness, resolution, or focus of the display device 160 or the camera module 180) can be controlled. A device management application may additionally or alternatively support installation, deletion, or update of applications running on external electronic devices.

3A and 3B are block diagrams of an example in which a first electronic device and a second electronic device transmit or receive audio or video content, according to various embodiments of the present invention.

According to various embodiments of the present invention, the first electronic device 310 (e.g., the electronic device 101 of FIG. 1) and the second electronic device 320 (e.g., the electronic device 101 of FIG. 1) have various Voice or video calls can be made based on a method (e.g., IP multimedia subsystem (IMS) or rich communication suite (RCS)). For convenience of explanation, the first electronic device 310 is defined as a sending terminal, and the second electronic device 320 is defined as a receiving terminal. The first electronic device 310 and the second electronic device 320 determine the bitrate, compression format (codec), or quality (e.g., sampling in the case of voice data) of voice data or video data transmitted and received when a call is connected. In order to determine the rate (in the case of video data, it may mean various variables representing the quality of voice data or video data, including resolution), the method defined in the session description protocol (SDP) is used. You can use this to conduct mutual negotiations. The first electronic device 310 and the second electronic device 320 may determine characteristics of voice data or video data to be transmitted through mutual negotiation, and transmit or receive voice data or video data using the determined characteristics. . Specific embodiments of the mutual negotiation process for connection between the first electronic device 310 and the second electronic device 320 will be described later with reference to FIGS. 4 and 5.

FIG. 3A is a diagram illustrating an embodiment of processing voice data between the first electronic device 310 and the second electronic device 320.

According to various embodiments of the present invention, the first electronic device 310 includes a microphone 311 (e.g., input device 150 of FIG. 1) that receives the voice of the user of the first electronic device 310, and a transmission filter. (313), an encoder (315) that performs encoding on the voice data to compress the voice data transmitted by the transmission filter (313), and a packetizer (317) that converts the encoded voice data into voice packet data in the form of packets. ) may include.

According to various embodiments of the present invention, the transmission filter 313 may be a component that performs a filtering operation on the user's voice data transmitted by the microphone 311. The transmission filter 313 performs a preprocessing operation (e.g., filtering operation) before the voice data generated by the microphone 311 is input to the encoder 315, so that the sampling rate is lower than the sampling rate of the voice data generated by the microphone 311. Voice data with a sampling rate (e.g. downscaled) can be generated. The transmission filter 313 can generate voice data with a low sampling rate and transmit the generated voice data to the encoder 315, thereby reducing the size of voice data to be transmitted to the second electronic device 320.

According to various embodiments of the present invention, the transmission filter 313 may be implemented using various algorithms, and may maintain the quality of the voice content generated by the microphone 311 while maximizing the quality of the voice content with a relatively low sampling rate. can be created.

According to various embodiments of the present invention, the second electronic device 320 includes a depacketizer 321 and a depacketizer 321 that convert voice data packets received through the network 330 into encoded voice data. a decoder 323 that decodes the encoded voice data converted by ), a reception filter 325 that changes the sampling rate of the decoded voice data, and a speaker 327 that outputs the voice data generated by the reception filter (e.g. It may include the audio output device 155 of FIG. 1).

According to various embodiments of the present invention, the reception filter 325 may be a component that performs a filtering operation on the decoded voice data generated by the decoder 323. The transmission filter 313 can generate voice data having a higher sampling rate (upscale) than the sampling rate corresponding to the decrypted voice data by performing a post-processing operation (e.g., filtering operation) on the decrypted voice data. there is. The reception filter 325 may generate voice content having a higher sampling rate than the sampling rate corresponding to the decrypted voice data, and transmit the generated voice content to the speaker 327.

According to various embodiments of the present invention, the reception filter 325 can be implemented using various algorithms and can generate voice content with a quality as similar as possible to the voice content generated by the microphone 311.

According to various embodiments of the present invention, the transmission filter 313 and the reception filter 325 are filters learned to maintain the quality of the original image as much as possible, and the neural processing unit of the electronic device 101 is It may be a filter used to process voice data or video data.

FIG. 3B is a diagram illustrating an embodiment of processing image data between the first electronic device 310 and the second electronic device 320.

According to various embodiments of the present invention, the first electronic device 310 includes a camera 341 (e.g., the camera module 180 of FIG. 1) that captures image data including at least one still image, and a transmission filter ( 343), an encoder 345 that encodes the video data to compress the video data transmitted by the transmission filter 343, and a packetizer 347 that converts the encoded video data into video packet data in the form of packets. may include.

According to various embodiments of the present invention, the transmission filter 343 may be a component that performs a filtering operation on image data. The transmission filter 343 performs a preprocessing operation (e.g., filtering operation) before the image data generated by the camera 341 is input to the encoder 345, thereby having a lower resolution than the image data generated by the camera 341. (e.g. downscale) image data can be generated. The transmission filter 343 generates image data with a relatively low resolution and transmits the generated image data to the encoder 345, thereby reducing the size of the image data to be transmitted to the second electronic device 320. .

According to various embodiments of the present invention, the transmission filter 343 can be implemented using various algorithms and generates image data with relatively low resolution while maintaining the quality of the image data generated by the camera 341 as much as possible. can do.

According to various embodiments of the present invention, the second electronic device 320 includes a depacketizer 351 and a depacketizer 351 that convert video data packets received through the network 330 into encoded video data. a decoder 353 that decodes the encoded video data converted by ), a reception filter 355 that changes the resolution of the decoded video data, and a display 357 that outputs the video data generated by the reception filter 355 ( Example: the display device 160 of FIG. 1) may be included.

According to various embodiments of the present invention, the reception filter 355 may be a component that performs a filtering operation on the decoded video data generated by the decoder 353. The reception filter 355 may generate image data with a higher resolution (e.g., upscale) than the resolution of the decoded image data by performing a post-processing operation (e.g., filtering operation) on the decoded image data. The reception filter 355 can transmit image data with relatively high resolution to the display 357.

According to various embodiments of the present invention, the reception filter 355 can be implemented using various algorithms and can generate image data with a quality as similar as possible to the image data generated by the camera 341.

According to various embodiments of the present invention, the transmission filter 343 and the reception filter 355 are filters learned to maintain the quality of the original voice as much as possible, and the neural processing unit of the electronic device 101 is It may be a filter used to process voice data or video data.

According to various embodiments of the present invention, the transmission filters 313 and 343 and the reception filters 325 and 355 may be filters implemented using the same algorithm, and filter voice data or video data as a pair. can be performed. The fact that the transmission filters 313 and 343 and the reception filters 325 and 355 operate as a pair may mean filtering voice data or video data using filters implemented using the same algorithm. there is. The first electronic device 310 and the second electronic device 320 must use the transmission filters 313 and 343 and reception filters 325 and 355 implemented using the same algorithm, so that the second electronic device 320 can 1 Content with quality similar to that of content (e.g., voice data or video data) generated by the electronic device 310 can be obtained.

According to various embodiments of the present invention, the first electronic device 310 and the second electronic device 320 provide information about whether the transmission filters 313 and 343 and the reception filters 325 and 355 are supported, or transmission filter ( Only by obtaining the information of the reception filters 313 and 343 and the reception filters 325 and 355 before the call connection can the preparation for transmitting or receiving content be completed after the call connection and before starting the call. When the first electronic device 310 and the second electronic device 320 obtain information about the transmission filters 313 and 343 and the reception filters 325 and 355 after a call connection, the transmission filters 313 and 343 In order to perform a call using the reception filters 325 and 355, the settings of hardware-implemented components may be changed, and the settings of the components of the electronic device 101 may be changed during a call, thereby preventing the call connection from being disconnected. phenomenon may occur.

According to various embodiments of the present invention, various components included in the first electronic device 310 and the second electronic device 320 (e.g., transmission filters 313 and 343, encoders 315 and 345, and packet Tizers (317, 347), depacketizers (321, 351), decoders (323, 3532), and reception filters (325, 355) are implemented in software or hardware (e.g., implemented in a circuit or chip). It can be implemented as:

Below, a specific embodiment in which the first electronic device 310 and the second electronic device 320 obtain information about whether the transmission filter 313 and the reception filter 325 are supported before a call connection is described. .

Figure 4A is a block diagram of an electronic device according to various embodiments of the present invention.

According to various embodiments of the present invention, the electronic device 400 shown in FIG. 4A (e.g., the electronic device 101 in FIG. 1 or the first electronic device 310 in FIGS. 3A and 3B) includes a communication module 410. ) (e.g., the wireless communication module 192 of FIG. 1) and a processor 420 (e.g., the processor 120 of FIG. 1).

According to various embodiments of the present invention, the communication module 410 is a communication channel (e.g., cellular communication channel) established through a call connection with an external electronic device (e.g., the second electronic device 320 of FIGS. 3A and 3B). You can transmit voice data or video data using .

According to various embodiments of the present invention, the processor 420 uses a method defined in the session technology protocol to determine the characteristics of data transmitted and received when making a call between the external electronic device and the electronic device 400. You can negotiate with.

Hereinafter, the electronic device 400 corresponds to the transmitting terminal (e.g., the first electronic device 310 in FIG. 3A), and the external electronic device corresponds to the second electronic device (e.g., the second electronic device 320 in FIG. 3A). ), and a specific embodiment in which information on whether the transmission filter 313 and the reception filter 325 are supported is obtained before call connection will be described.

According to various embodiments of the present invention, the processor 420 uses the communication module 410 to transmit a call connection request message to the external electronic device 320 to establish a call channel between the external electronic device 320 and the electronic device 400. ) can be controlled. For example, the call connection request message can be implemented in the SDP Invite message format defined in SDP.

According to various embodiments of the present invention, the call connection request message may include quality information of voice data or video data that the electronic device 400 can transmit. For example, the quality information of voice data includes the sampling rate of voice data that can be generated using a microphone (e.g., microphone 311 in FIG. 3A) and transmitted through a network (e.g., network 320 in FIG. 3A). It may include a sampling rate of voice data that can be received or a sampling rate of voice data that an external electronic device can receive. The quality information of image data includes the resolution of image data that can be generated using a camera (e.g., camera 341 in FIG. 3B), the resolution of image data that can be transmitted through the network 320, or the resolution of image data that can be received by an external electronic device. The resolution of the video data may be included.

According to various embodiments of the present invention, the processor 420 performs a transmission function in order to check whether the external electronic device 320 and the transmission filters 313 and 343 and the reception filters 325 and 355 are supported before completing the call connection. A call connection request message containing information about the filters 313 and 343 and performance information of the electronic device 400 related to the transmission filters 313 and 343 may be transmitted to the external electronic device 320.

According to various embodiments of the present invention, the call connection message may include identification information of the transmission filters 313 and 343. The processor 420 may include identification information of the transmission filters 313 and 343 in the call connection request message. The identification information of the transmission filters 313 and 343 includes model information and version information of the transmission filters 313 and 343, and information about the maximum sampling rate of voice data or the maximum resolution of video data that the transmission filters 313 and 343 can process. It may include at least one of capability information including.

According to various embodiments of the present invention, the call connection request message may include performance information of the electronic device 400 related to the transmission filters 313 and 343. The processor 420 may include performance information of the electronic device 400 related to the transmission filters 313 and 343 in the call connection request message. Performance information of the electronic device 400 related to the transmission filters 313 and 343 may include information indicating whether preprocessing of voice data or video data is possible using the transmission filters 313 and 343. For example, the processor 420 may include an indicator indicating that it can support pre-processing of voice data or video data using a specific algorithm in the call connection request message. Pre-processing of voice data or video data using a specific algorithm is performed using a transmission filter (e.g., the transmission filter 313 in FIG. 3A or FIG. 3B) implemented to be processed by a neural network processor (NPU) provided in the electronic device 400. This may refer to a pre-processing operation before encoding voice data or video data using the transmission filter 343).

According to various embodiments of the present invention, the performance information of the electronic device 400 related to the transmission filters 313 and 343 is obtained by the electronic device 400 using the transmission filters 313 and 343 for voice data or video data. It may include information indicating whether it is equipped with a component that can perform preprocessing (e.g., a central processing unit (CPU), graphics processing unit (GPU), or neural network processing unit (NPU)). For example, the call connection request message may include an indicator indicating that it includes a neural network processing device capable of performing preprocessing on voice data or video data using the transmission filters 313 and 343. The call connection request message may include information about components that can perform preprocessing on voice data or video data using the transmission filters 313 and 343. For example, information about components that can perform preprocessing on audio data or video data may include performance and identification information of the components that can perform preprocessing on audio data or video data.

According to various embodiments of the present invention, the processor 420 may transmit a call connection request message including performance information of the electronic device related to the transmission filters 313 and 343 to the external electronic device 320. The processor 420 may receive a response message corresponding to the call connection request message from the external electronic device 320. For example, the response message may be implemented in the SDP 200 OK message format defined in the SDP protocol.

According to various embodiments of the present invention, the response message transmitted by the external electronic device 320 may include performance information of the external electronic device 320 related to the call connection. The external electronic device 320 selects a data transmission method (e.g., sampling method, sampling rate, or video of voice data) that the electronic device 400 can support based on the performance information of the electronic device 400 included in the call connection request message. The performance information of the external electronic device 320 may select a data transmission method supported by the external electronic device 320 among the data compression format and resolution. can do.

According to various embodiments of the present invention, the performance information of the external electronic device 320 is processed through pre-processed voice data or pre-processed video data generated by pre-processing of voice data or video data supported by the electronic device 400. Components that can do this (e.g., components implemented in software (reception filter 325 in FIG. 3A, reception filter 355 in FIG. 3B)) or components implemented in hardware (e.g., neural network processing) may include information about the unit)).

According to various embodiments of the present invention, the processor 420 may determine whether to use the transmission filters 313 and 343 based on performance information of the external electronic device 320 included in the response message. The processor 420 may perform a preprocessing operation on data to be transmitted using the transmission filters 313 and 343.

According to various embodiments of the present invention, the processor 420 may transmit a call connection confirmation message including an indicator indicating whether to use the transmission filters 313 and 343 to the external electronic device 320.

According to various embodiments of the present invention, the call connection confirmation message may be a message instructing to perform a call connection using a transmission method of voice data or video data included in the response message.

According to various embodiments of the present invention, the electronic device 400 performs various operations (e.g., using the microphone 311 to generate voice data) for call connection with the external electronic device 320 that has received the call connection confirmation message. An operation of activating, an operation of activating the camera 341 to generate image data, an operation of activating a component (e.g., NPU) that performs preprocessing on audio data or image data using the transmission filters 313 and 343. Alternatively, an operation of controlling the communication module 410 to transmit or receive data for establishing a call channel may be performed.

According to various embodiments of the present invention, the processor 420 establishes a call connection with the external electronic device 320 by including performance information of the electronic device 400 related to the transmission filters 313 and 343 in the call connection request message. Before completion, the transmission filters 313 and 343 can be activated. Accordingly, after the call connection is completed, it is possible to prevent deterioration in the quality of the call channel that may occur when the transmission filters 313 and 343 are activated, or to improve the quality of a call using the call channel.

Figure 4B is a block diagram of an electronic device according to various embodiments of the present invention.

Referring to FIG. 4B, the electronic device 400 according to various embodiments of the present invention includes a call connection management unit 430 that performs or manages a call connection, a processing operation for transmitting voice data, or processing received voice data. an audio data processing unit 440 that performs an operation to transmit image data or an image data processing unit 450 that performs an operation to process received image data, and a memory (e.g., the memory of FIG. 1 ( 130)) 461, a camera (e.g., the camera module 180 of FIG. 1) 462, a microphone (e.g., the input device 150 of FIG. 1) 463, a display (e.g., the display device of FIG. 1) It may include (160)) (464), a speaker (e.g., the sound output device 155 of FIG. 1) (465), and a neural processing unit (neural processing unit) (466).

According to various embodiments of the present invention, the call connection management unit 430 parses a message received from an external electronic device (e.g., the second electronic device 320 of FIGS. 3A and 3B) while performing a call connection ( An SDP parser 431 that parses, an SDP writer 432 that generates a message to be transmitted to an external electronic device 320 to perform a call connection, and a SDP writer 432 that manages connection or termination of a call session. A media processing decision unit that determines whether to perform data processing using a specific filter based on the performance information of the external electronic device 320 included in the response message received from the call session management unit 433 and the external electronic device 320. It may include (434).

According to various embodiments of the present invention, the voice data processing unit 440 from the viewpoint of transmitting voice data includes a voice acquisition unit 441 that acquires voice data using a microphone 463, and a filtering operation for voice data. A voice transmission filter (e.g., the transmission filter 313 in FIG. 3A) 442, which is a component that performs a voice encoder ( Example; it may include an encoder 315 in FIG. 3A) 443 and a packetizer (e.g. packetizer 317 in FIG. 3A) 444 that converts the encoded voice data into voice packet data in packet form. You can.

According to various embodiments of the present invention, in terms of receiving voice data, the voice data processing unit 440 includes a depacketizer that converts voice packet data received from the external electronic device 320 into encoded voice data. Example: Depacketizer 321) 445 in FIG. 3A, voice decoder for decoding encoded voice data (e.g. decoder 323 in FIG. 3A) 446, changing the sampling rate of decoded voice data. The command may include a voice reception filter (e.g., reception filter 325 in FIG. 3A) 447 and a voice output unit 448 that outputs voice through a speaker 465.

According to various embodiments of the present invention, the image data processing unit 450 from the viewpoint of transmitting image data includes an image acquisition unit 451 that acquires an image captured by the camera 462 and a filtering operation on the image data. Components that perform the audio transmission filter (e.g., the transmission filter 343 in FIG. 3B) 452, and a video encoder that performs encoding on video data to compress the video data (e.g., the encoder 345 in FIG. 3B) ) 453 and a packetizer (e.g., packetizer 347 in FIG. 3B) 454 that converts the encoded video data into video packet data in the form of packets.

According to various embodiments of the present invention, the image data processing unit 450 from the viewpoint of receiving image data includes a depacketizer (depacketizer) that converts the image data packet received from the external electronic device 320 into encoded image data. Example: Depacketizer 351) 455 in FIG. 3B, video decoder (e.g. decoder 353 in FIG. 3B) 456 that decodes the encoded video data converted by the depacketizer 455. , an image reception filter that changes the resolution of the decoded image data (e.g., the reception filter 355 in FIG. 3B) 457, and an image that outputs the image data generated by the reception filter 457 through the display 464. It may include an output unit 458.

According to various embodiments of the present invention, the SDP writer 432 generates a call connection request message based on the control of the call session manager 433 and connects the call to the external electronic device 320 through the communication module 410. A request message can be sent. For example, the call connection request message can be implemented in the SDP Invite message format defined in SDP.

According to various embodiments of the present invention, the call connection message includes information about the voice transmission filter 442 and the video transmission filter 452, and preprocessing of voice data or video data using a specific algorithm (e.g., preprocessing algorithm using NPU). An indicator indicating whether processing can be supported and performance information of the electronic device 400 related to the voice transmission filter 442 and the video transmission filter 452 may be included.

According to various embodiments of the present invention, the SDP parser 431 may parse the response message received through the communication module 410. The response message is related to the voice reception filter (e.g., reception filter 325 in FIG. 3A) and the video reception filter (e.g., reception filter 355 in FIG. 3B) of the external electronic device 320. May contain performance information. For example, the response message includes an indicator indicating whether a post-processing operation corresponding to a pre-processing operation for voice data or video data using a specific algorithm can be supported, and an indicator indicating whether the external electronic device 500 can support the voice reception filter 325. ) and a component that can perform post-processing on audio data or video data using the video reception filter 355 (e.g., central processing unit (CPU), graphics processing unit (GPU), or neural network processing unit (NPU) ) may include an indicator indicating whether it is provided.

According to various embodiments of the present invention, the performance information of the external electronic device 320 extracted by the SDP parser 431 through parsing of the response message is transmitted to the media processing decision unit 434 through the call session management unit 433. It can be.

According to various embodiments of the present invention, the media processing decision unit 434 performs a pre-processing operation of voice data using the voice transmission filter 442 or the video transmission filter 452 based on the performance information of the external electronic device 320. It is possible to decide whether to perform a preprocessing operation on the used image data.

According to various embodiments of the present invention, the media processing decision unit 434 checks an indicator indicating whether a post-processing operation of voice data or video data using a specific algorithm included in the response message is supported, and uses a specific algorithm. It may be decided to perform a preprocessing operation on the used audio data or video data.

According to various embodiments of the present invention, the media processing decision unit 434 checks performance information of the external electronic device 320 and performs a preprocessing operation of audio data or video data by the external electronic device 320 using a specific algorithm. You can decide to support it.

According to various embodiments of the present invention, the media processing decision unit 434, in response to determining to perform a preprocessing operation of audio data or video data, transmits the audio transmission filter 442, the video transmission filter 452, and the audio A component (eg, NPU 466) that performs preprocessing of audio data or video data using the transmission filter 442 and the video transmission filter 452 may be activated.

According to various embodiments of the present invention, the call session management unit 433 sends a call connection confirmation message indicating whether to use the voice transmission filter 442 or the video transmission filter 452 to the external electronic device through the communication module 410. It can be sent to (320).

Hereinafter, when the electronic device 400 corresponds to a receiving terminal (e.g., the second electronic device 320 in FIG. 3A), information on whether the transmission filter 313 and the reception filter 325 are supported is provided. An embodiment of acquisition before connection will be described.

According to various embodiments of the present invention, components implemented on the processor 420 of the electronic device 400 may be components implemented in software, but some components may be implemented in hardware depending on the design method. It may be possible.

5 is a block diagram of an electronic device according to various embodiments of the present invention.

According to various embodiments of the present invention, the electronic device 500 shown in FIG. 5 (e.g., the electronic device 101 in FIG. 1 or the second electronic device 320 in FIGS. 3A and 3B) includes a communication module 510. ) (e.g., the wireless communication module 192 of FIG. 1) and a processor 520 (e.g., the processor 120 of FIG. 1).

According to various embodiments of the present invention, the communication module 510 is a communication channel (e.g., cellular communication channel) established through a call connection with an external electronic device (e.g., the first electronic device 310 of FIGS. 3A and 3B). You can transmit voice data or video data using .

According to various embodiments of the present invention, the processor 520 uses a method defined in the session description protocol to determine the characteristics of data transmitted and received when making a call between the external electronic device 310 and the electronic device 500. Negotiation can be performed with external electronic devices.

According to various embodiments of the present invention, the processor 520 may receive a call connection request message from the external electronic device 310 to establish a call channel between the external electronic device 310 and the electronic device 500. For example, the call connection request message can be implemented in the SDP Invite message format defined in SDP.

According to various embodiments of the present invention, the call connection request message may be the same message as the call connection request message described in FIG. 4. As described in FIG. 4, the call connection request message includes quality information of voice data or video data that can be transmitted by the external electronic device 310, model information of the transmission filters 313 and 343 of the external electronic device 310, Version information, identification information of the transmission filters 313 and 343, including at least one of performance information of the transmission filters 313 and 343, and performance information of the external electronic device 310 related to the transmission filters 313 and 343. can do.

According to various embodiments of the present invention, the processor 520 may check the performance information of the external electronic device 310 included in the call connection request message in response to receiving the call connection request message. The performance information of the external electronic device 310 is information on voice data or video data using a transmission filter (e.g., the transmission filter 313 in FIG. 3A or the transmission filter 343 in FIG. 3B) implemented in the external electronic device 310. It may include information indicating whether preprocessing can be performed.

According to various embodiments of the present invention, the processor 520 transmits based on information indicating whether preprocessing of voice data or video data is possible using the transmission filters 313 and 343 included in the call connection request message. It is possible to determine whether post-processing of voice data or video data is possible using the reception filters 325 and 355 corresponding to the filters 313 and 343. The processor 520 checks information about the transmission filters 313 and 343 included in the call connection request message and receives the reception filters 325 and 355 corresponding to the transmission filters 313 and 343 implemented on the processor 520. ) is present, and the processor 520 uses the receive filters 325 and 355 in response to confirming that the receive filters 325 and 355 corresponding to the transmit filters 313 and 343 exist. It is possible to determine whether post-processing is possible for audio data or video data.

According to various embodiments of the present invention, the processor 520 determines a data transmission method (e.g., voice) that the external electronic device 310 can support based on the performance information of the external electronic device 310 included in the call connection request message. A data transmission method that the electronic device 500 can support can be selected among the data sampling method, sampling rate, or video data compression format and resolution.

According to various embodiments of the present invention, the processor 520 may transmit a response message containing information about the selected data transmission method to the external electronic device 310. For example, the response message can be implemented in the SDP 200 OK message format defined in the SDP protocol.

According to various embodiments of the present invention, the information about the data transmission method included in the response message is a reception method that performs post-processing on the voice data or received data to adjust the quality of the voice data or received data after the call connection. It may include performance information of the electronic device 500 related to a filter (eg, the reception filter 325 in FIG. 3A or the reception filter 355 in FIG. 3B).

According to various embodiments of the present invention, the response message may include an indicator indicating that post-processing of voice data or video data using a specific algorithm can be processed.

According to various embodiments of the present invention, the response message may include performance information of the electronic device 500 related to the reception filters 325 and 355. The performance information of the electronic device 500 related to the reception filters 325 and 355 refers to a component that allows the electronic device 500 to perform post-processing on voice data or video data using the reception filters 325 and 355. It may include an indicator indicating whether it is equipped with a central processing unit (CPU), graphics processing unit (GPU), or neural network processing unit (NPU). For example, the response message may include an indicator indicating that it includes a neural network processing device capable of performing post-processing on voice data or video data using the reception filters 325 and 355. The response message may include information about components that can perform post-processing on voice data or video data using the reception filters 325 and 355. For example, information about components that can perform post-processing on audio data or video data may include performance and identification information of components that can perform post-processing on audio data or video data. .

According to various embodiments of the present invention, the processor 520 receives from the external electronic device 310 a call connection confirmation message including an indicator indicating whether to use the transmission filters 325 and 355 of the external electronic device 310. You can receive it. In response to confirming that the external electronic device 310 uses the transmission filters 313 and 343, the processor 520 performs various operations for call connection (activation of the reception filters 325 and 355 and related components). , an operation of controlling the communication module 510 to transmit or receive data for establishing a call channel) may be performed.

According to various embodiments of the present invention, the processor 520 may transmit performance information of the electronic device 500 to the external electronic device 320 during a call connection negotiation process. Accordingly, the processor 520 may activate the reception filters 325 and 355 before the call connection with the external electronic device 320 is completed. Accordingly, after the call connection is completed, it is possible to prevent deterioration in the quality of the call channel that may occur due to the operation of the reception filters 325 and 355, or to improve the quality of the call through the call channel.

FIG. 6 is an operation flowchart illustrating an operation in which a first electronic device and a second electronic device perform a call connection according to various embodiments of the present invention.

In operation 610, a first electronic device (e.g., the first electronic device 310 of FIGS. 3A and 3B or the electronic device 400 of FIG. 4) according to various embodiments of the present invention operates a second electronic device (e.g., A call connection request message (eg, an SDP Invite message defined in the SDP session protocol) may be transmitted to the second electronic device 320 of FIGS. 3A and 3B or the electronic device 500 of FIG. 5.

According to various embodiments of the present invention, the first electronic device 310 determines whether the second electronic device 320 supports the transmission filters 313 and 343 and the reception filters 325 and 355 before completing the call connection. To check, information about the transmission filters 313 and 343, an indicator indicating whether preprocessing of voice data or video data using a specific algorithm (e.g., a preprocessing algorithm using NPU) can be supported, and a transmission filter 313 A call connection request message containing performance information of the electronic device 400 related to , 343) may be transmitted to the second electronic device 320. For example, the call connection request message may be implemented in the form shown in Table 1 below.

[-->] INVITE
m=audio 7010 RTP/AVP 110 100 98
b=AS:42
a=rtpmap:110EVS/16000
a=mediaproc:110 upmodel=voice_up_coeff_1.1; downmodel=voice_down_coeff_1.1
a=audioattr:110 source [32000] send [16000] recv [16000] target [32000]
a=npu:ver2.0
a=rtpmap:100 AMR-WB/16000/1
a=mediaproc:100 upmodel=voice_up_coeff_1.1; downmodel=voice_down_coeff_1.1
a=audioattr:100 source [32000] send [16000] recv [16000] target [32000]
a=npu:ver2.0
a=rtpmap:98 AMR/8000/1

......

m=video 7020 RTP/AVP 112 102 34
b=AS:1280
a=rtpmap:112 H265/90000
a=mediaproc:112 upmodel=video_up_coeff_1.02; downmodel=video_down_coeff_1.02
a=npu:ver2.0
a=imageattr:112 source [x=1440, y=2560] send [x=720, y=1280]
recv [x=720, y=1280] target [x=1440, y=2560]
a=rtpmap:102 H264/90000
a=mediaproc:102 upmodel=video_up_coeff_1.02; downmodel=video_down_coeff_1.02
a=npu:ver2.0
a=imageattr:102 source [x=960, y=1280] send [x=480, y=640]
recv [x=480, y=640] target [x=960, y=1280]
a=rtpmap:34 H263/90000
a=imageattr:34 send [x=176,y=144] recv [x=176,y=144]

Referring to Table 1, the call connection request message includes an indicator (a=mediaproc) indicating whether the first electronic device 310 supports pre-processing of voice data using a specific algorithm, and a neural network processing unit (NPU). Whether the processing operation is supported and NPU information (a=npu:ver2.0), identification information of the transmission filters (313, 343) (upmodel=voice_up_coeff_1.1, upmodel=video_up_coeff_1.1), and reception filters (325, 355) It may include identification information (downmodel=voice_down_coeff_1.1, downmodel=video_down_coeff_1.1) and quality information of voice data or video data.

According to various embodiments of the present invention, in operation 620, the second electronic device 320 is connected to a reception filter (e.g., the reception filter 325 in FIG. 3A or the reception filter 355 in FIG. 3B). You can check the performance of (320).

According to various embodiments of the present invention, the second electronic device 320 checks the transmission methods of voice data or video data supported by the first electronic device 310 included in the call connection request message, and 320 may select at least one transmission method supported by the second electronic device 320 among the confirmed transmission methods.

According to various embodiments of the present invention, the second electronic device 320 may check the performance of the second electronic device 320 related to the reception filters 325 and 355 to support the selected transmission method.

According to various embodiments of the present invention, the performance of the second electronic device 320 related to the reception filters 325 and 355 corresponds to a preprocessing operation for voice data or video data using a specific algorithm included in the call connection request message. Whether the second electronic device 320 can support a post-processing operation, a component (e.g., a central processing unit) capable of performing post-processing on voice data or video data using the reception filters 325 and 355 This may include whether it is equipped with a CPU (CPU), graphics processing unit (GPU), or neural network processing unit (NPU).

According to various embodiments of the present invention, in operation 630, the second electronic device 320 may transmit a response message (e.g., an SDP 200 OK message defined in the SDP protocol) to the first electronic device 310.

According to various embodiments of the present invention, the response message may include the performance of the second electronic device 320 related to the reception filters 325 and 355. For example, the response message includes an indicator indicating whether a post-processing operation corresponding to a pre-processing operation for voice data or video data using a specific algorithm can be supported, and the second electronic device 320 supports the reception filter 325. , 355) is equipped with components that can perform post-processing on audio or video data (e.g., central processing unit (CPU), graphics processing unit (GPU), or neural network processing unit (NPU)) An indicator indicating availability may be included. For example, the response message may be implemented in the form shown in Table 2 below.

[<--] SIP/2.0 200 OK
m=audio 12350 RTP/AVP 100
b=AS:41
a=rtpmap:100 AMR-WB/16000/1
a=mediaproc:100 upmodel=voice_up_coeff_1.0; downmodel=voice_down_coeff_1.0
a=audioattr:100 source [32000] send [16000] recv [16000] target [32000]
a=npu:ver1.0

m=video 15490 RTP/AVP 102
b=AS:640
a=rtpmap:102 H264/90000
a=mediaproc:102 upmodel=nobias1.01; downmodel=basedown1.01
a=npu:ver1.0
a=imageattr:102 source [x=960, y=1280] send [x=480, y=640]
recv [x=480, y=640] target [x=960, y=1280]

Referring to Table 2, the response message includes an indicator (a=mediaproc) indicating whether the second electronic device 320 supports post-processing operations of voice data or video data using a specific algorithm, and a neural network processing unit (NPU). Whether to support processing operations using and NPU information (a=npu:ver2.0), identification information of the reception filter (325, 355) (upmodel=voice_up_coeff_1.1, upmodel=video_up_coeff_1.1), reception filter (325, 355) It may include identification information (downmodel=voice_down_coeff_1.1, downmodel=video_down_coeff_1.1) and quality information of voice data or video data.

According to various embodiments of the present invention, in operation 640, the first electronic device 310 transmits voice data or video data using the transmission filters 313 and 343 based on the performance information of the second electronic device 320. You can decide whether to perform preprocessing operations.

According to various embodiments of the present invention, whether to perform a preprocessing operation of audio data or video data may mean performing a preprocessing operation of audio data or video data using the transmission filters 313 and 343.

According to various embodiments of the present invention, the first electronic device 310 checks an indicator indicating whether a post-processing operation of voice data or video data using a specific algorithm included in the response message is supported, and uses a specific algorithm. It may be decided to perform a preprocessing operation on the used audio data or video data.

According to various embodiments of the present invention, in operation 650, the first electronic device 310 may transmit a call connection confirmation message (e.g., an SDP ACK message defined in the SDP session protocol) to the second electronic device 320. .

According to various embodiments of the present invention, the call connection confirmation message may be a message instructing to perform a call connection using a transmission method of voice data or video data included in the response message.

According to various embodiments of the present invention, in operation 661, the first electronic device 310 may perform a setting operation for the components of the first electronic device 310 for call connection.

According to various embodiments of the present invention, the first electronic device 310 includes information about at least one transmission method supported by the second electronic device 320 included in the response message (e.g., sampling method of voice data, sampling method) Components for call connection can be set based on the rate, compression method of video data, or resolution of video data). The first electronic device 310 is configured to transmit or receive data for activating the microphone 311 to generate voice data, activating the camera 341 to generate video data, or establishing a call channel. Components for call connection can be set through various operations including controlling the communication module 410.

According to various embodiments of the present invention, in operation 663, the second electronic device 320 may perform a setting operation for the components of the second electronic device 320 for call connection.

According to various embodiments of the present invention, the second electronic device 320 includes information about at least one transmission method supported by the second electronic device 320 (e.g., sampling method of voice data, sampling method) included in the response message. Components for call connection can be set based on the rate, compression method of video data, or resolution of video data). The second electronic device 320 is configured to transmit or receive data for activating the microphone 311 to generate voice data, activating the camera 341 to generate image data, or establishing a call channel. Components for call connection can be set through various operations including controlling the communication module 510.

According to various embodiments of the present invention, in operation 670, the first electronic device 310 and the second electronic device 320 complete the call connection and perform a call using the call channel.

According to various embodiments of the present invention, the first electronic device 310 and the second electronic device 320 may transmit or receive voice data or video data through a call channel. The first electronic device 310 and the second electronic device 320 transmit preprocessed voice data or video data using the transmission filters 313 and 343, or transmit the received voice data or video data using the reception filters 325 and 320. By post-processing using 355), audio data or video data having a quality similar to that of the original audio data or video data can be obtained.

FIG. 7 is a diagram illustrating a screen for performing settings for content transmission or reception in an electronic device according to various embodiments of the present invention.

According to various embodiments of the present invention, a first electronic device (e.g., the first electronic device 310 of FIG. 6) and a second electronic device (e.g., the second electronic device 320 of FIG. 6) perform voice processing using a specific algorithm. Voice data or video data can be transmitted to the other electronic device using a data or video data processing method. Filters that perform pre-processing or post-processing of audio data or video data (e.g., the transmission filters 313 and 343 in FIGS. 3A and 3B or the reception filters 325 and 355 in FIGS. 3A and 3B) exhibit high performance. To do this, it can be generated through machine learning. Filters created through machine learning can be used in neural network processing devices in electronic devices. The operation of processing voice data or video data using a filter generated through machine learning can improve sound quality or image quality, but may increase power consumption.

An electronic device (e.g., the electronic device 400 of FIG. 4 or the electronic device 500 of FIG. 5) according to various embodiments of the present invention performs an operation of pre-processing or post-processing voice data according to user settings. Whether or not it can be decided.

Referring to FIG. 7, the electronic device 400 displays a screen 710 (e.g., 710) that allows you to select whether to apply the sound quality improvement function during a call connection (711) or whether to apply the picture quality improvement function during a call connection (713). It can be displayed on the display device 160 of FIG. 1. The electronic device 400 may determine whether to activate the sound quality improvement function or the picture quality improvement function based on user input.

According to various embodiments of the present invention, the electronic device 400 may determine whether to activate the sound quality improvement function or the image quality improvement function depending on the state of the electronic device 400. For example, the electronic device 400 may determine to disable the sound quality improvement function or the image quality improvement function in response to confirming that the remaining battery capacity of the electronic device 400 is less than a preset value. For another example, the electronic device 400 may decide to activate a sound quality improvement function or an image quality improvement function in response to confirming that the quality of cellular communication used for call connection is below a preset value.

An electronic device according to various embodiments of the present invention includes a communication module that transmits or receives data using a call channel established through a call connection with an external electronic device; and a processor, wherein the processor transmits a call connection request message for establishing a call channel between an external electronic device and the electronic device, receives a response message from the external electronic device, and connects the call included in the response message. Determining whether to perform a pre-processing operation of the content using a transmission filter that changes the quality of the content transmitted through the call connection based on capability information of the external electronic device related to the content, and pre-processing the content A call connection confirmation message indicating whether to perform an operation is transmitted to the external electronic device, the communication module is controlled to complete the call channel establishment, and the connection request message includes performance information of the electronic device related to the filter. It can be included.

In an electronic device according to various embodiments of the present invention, the performance information of the electronic device may include an indicator indicating whether a component of the electronic device for processing the transmission filter exists.

In an electronic device according to various embodiments of the present invention, components of the electronic device may include a neural processing unit (NPU).

In an electronic device according to various embodiments of the present invention, the call connection request message may be set to include identification information of the NPU.

In an electronic device according to various embodiments of the present invention, the call connection request message may be set to include identification information of the transmission filter.

In electronic devices according to various embodiments of the present invention, performance information of the electronic device may include resolution or sampling rate of content that can be processed using the transmission filter.

In an electronic device according to various embodiments of the present invention, the connection request message may include an indicator indicating whether a processing operation of the content based on the transmission filter is supported.

In an electronic device according to various embodiments of the present invention, the processor determines whether the reception filter corresponding to the transmission filter exists in the external electronic device based on the response message, and determines whether the reception filter corresponding to the transmission filter exists in the external electronic device based on the confirmation result. It can be set to determine whether to use a transmission filter.

In an electronic device according to various embodiments of the present invention, in response to determining to use the transmission filter, the processor is set to activate the transmission filter and components for controlling the transmission filter before the call connection is completed. It can be.

An electronic device according to various embodiments of the present invention includes a communication module that transmits or receives data using a call channel established through a call connection with an external electronic device; and a processor, wherein the processor receives a call connection request message for establishing a call channel between an external electronic device and the electronic device, and the performance of the electronic device related to a reception filter that preprocesses content received through the call connection. A response message containing information is transmitted to the external electronic device, and a call connection confirmation message including an indicator indicating whether to perform a pre-processing operation of the content using a transmission filter corresponding to the reception filter is sent from the external electronic device. It may be set to receive and activate the reception filter based on the indicator.

In an electronic device according to various embodiments of the present invention, the performance information of the electronic device may include an indicator indicating whether a component of the electronic device for processing the reception filter exists.

In an electronic device according to various embodiments of the present invention, components of the electronic device may include a neural processing unit (NPU).

In an electronic device according to various embodiments of the present invention, the response message may be set to include identification information of the reception filter.

In an electronic device according to various embodiments of the present invention, the response message may include an indicator indicating whether a processing operation of the content based on the reception filter is supported.

Figure 8 is an operational flowchart showing a method of operating an electronic device according to various embodiments of the present invention.

According to various embodiments of the present invention, in operation 810, an electronic device (e.g., the electronic device 400 of FIG. 4) sends a call connection request message to establish a call channel to an external electronic device (e.g., the electronic device 400 of FIG. 5). 500)).

According to various embodiments of the present invention, before completing a call connection, the electronic device 400 transmits data to determine whether the external electronic device 500 supports a post-processing operation of voice data or video data using a specific algorithm. Information about the filters 313 and 343, an indicator indicating whether preprocessing of voice data or video data using a specific algorithm (e.g., preprocessing algorithm using NPU) can be supported, and information related to the transmission filters 313 and 343. A call connection request message containing performance information of the electronic device 400 may be transmitted to the external electronic device 500.

According to various embodiments of the present invention, in operation 820, the electronic device 400 may receive a response message containing performance information of the external electronic device 500 related to the call connection from the external electronic device 500.

According to various embodiments of the present invention, the response message may include the performance of the external electronic device 500 related to the reception filters 325 and 355. For example, the response message includes an indicator indicating whether a post-processing operation corresponding to a pre-processing operation for voice data or video data using a specific algorithm can be supported, and an indicator indicating whether the external electronic device 500 can support the reception filter 325, 355) whether it is equipped with components that can perform post-processing on audio or video data (e.g., central processing unit (CPU), graphics processing unit (GPU), or neural network processing unit (NPU)) It may include an indicator indicating .

According to various embodiments of the present invention, in operation 830, the electronic device 400 determines whether to perform a preprocessing operation of content (e.g., voice data or image data) based on performance information of the external electronic device 500. You can.

According to various embodiments of the present invention, the content preprocessing operation uses transmission filters 313 and 343 before the content is input to the encoder (e.g., the encoder 315 in FIG. 3A or the encoder 345 in FIG. 3B). This may mean performing a preprocessing operation on audio data or video data.

According to various embodiments of the present invention, the electronic device 400 checks an indicator indicating whether a post-processing operation of voice data or video data using a specific algorithm included in the response message is supported, and determines whether the electronic device 400 supports a post-processing operation of voice data or video data using a specific algorithm included in the response message. It may be decided to perform a pre-processing operation on data or image data.

According to various embodiments of the present invention, the electronic device 400 checks the performance information of the external electronic device 500, and the external electronic device 500 is expected to support a preprocessing operation of voice data or video data using a specific algorithm. You can decide.

According to various embodiments of the present invention, in operation 840, the electronic device 400 may transmit a call connection confirmation message indicating whether to use a transmission filter to the external electronic device 500.

According to various embodiments of the present invention, the call connection confirmation message may be a message instructing to perform a call connection using a transmission method of voice data or video data included in the response message.

Figure 9 is an operation flowchart showing a method of operating an electronic device according to another embodiment of the present invention.

According to various embodiments of the present invention, in operation 910, an electronic device (e.g., the electronic device 500 of FIG. 5) sends a call connection request message for establishing a call channel to an external electronic device (e.g., the electronic device 500 of FIG. 4). 400)).

According to various embodiments of the present invention, the call connection request message includes quality information of voice data or video data that can be transmitted by the external electronic device 500, and model information of the transmission filters 313 and 343 of the external electronic device 500. , version information, identification information of the transmission filters 313 and 343, including performance information of the transmission filters 313 and 343, and performance information of the external electronic device 500 related to the transmission filters 313 and 343. there is.

According to various embodiments of the present invention, in operation 920, the electronic device 500 sends a response message containing performance information related to a reception filter for adjusting the quality of content (e.g., voice data or video data) to the external electronic device. It can be sent to (400).

According to various embodiments of the present invention, in operation 930, the electronic device 500 may receive a call connection confirmation message indicating whether to use the transmission filters 313 and 343 of the external electronic device 400.

According to various embodiments of the present invention, in operation 940, the electronic device 500 may set components of the electronic device 500 for call connection.

According to various embodiments of the present invention, the electronic device 500 includes information about at least one transmission method supported by the electronic device 500 (e.g., sampling method of voice data, sampling rate, video data) included in the response message. Components for call connection can be set based on the compression method or resolution of the video data. The electronic device 500 has a communication module to transmit or receive data for activating the microphone 311 to generate voice data, activating the camera 341 to generate image data, or establishing a call channel. Components for call connection can be set through various operations, including the operation of controlling 510.

The embodiments described above obtain information about whether the transmission filter 313 and the reception filter 325 are supported before the call connection between the first electronic device 310 and the second electronic device 320. These are examples. However, various embodiments of the present invention can obtain various information, including whether the transmission filter 313 and the reception filter 325 are supported, prior to call connection. According to one embodiment, when performing a video call connection, the first electronic device 310 and the second electronic device 320 provide information including whether or not various methods of performing a video call are supported before connecting a video call. It can be obtained before connecting the call. For example, among various methods of performing a video call, the first electronic device 310 and the second electronic device 320 exchange data (or information) for rendering the video without transmitting the video. Information on whether a method of performing a video call (hereinafter defined as a metadata-based video call) is supported can be obtained before connecting the call.

According to various embodiments of the present invention, a metadata-based video call method may refer to a video call method that exchanges metadata, which is data for creating an emoji containing a video. A metadata-based video call method may be a method of exchanging metadata that is smaller in size than video data, rather than exchanging video data output during a video call. The electronic device that received the metadata applies (reflects) the metadata to a model (e.g., 3D modeling data) of the emoji stored in the electronic device, and at least one emoji containing the characteristics (or characteristics) of the metadata is reflected. More than one frame can be created. The electronic device may generate (render) image data including at least one frame and output the image data on a display (eg, the display device 160 of FIG. 1). An emoji is an image corresponding to the user of an electronic device (e.g., the first electronic device 310 and/or the second electronic device 320), and metadata is input into modeling data stored in the electronic devices 310 and 320. It may be an image created in this way. The metadata-based video call method can reduce the data used for video calls by exchanging metadata that is smaller in size than video data.

Below, an embodiment in which information related to a metadata-based video call is acquired before a call connection between the first electronic device 310 and the second electronic device 320 is described.

Figure 10A is a block diagram of an electronic device according to various embodiments of the present invention.

According to various embodiments of the present invention, the electronic device 1000 shown in FIG. 10A (e.g., the electronic device 101 in FIG. 1, the first electronic device 310 in FIGS. 3A and 3B, or the electronic device in FIG. 4 (400)) may include a communication module 1010 (e.g., the wireless communication module 192 of FIG. 1) and a processor 1020 (e.g., the processor 120 of FIG. 1).

According to various embodiments of the present invention, the communication module 1010 uses a communication channel (e.g., cellular communication channel) established through a call connection with an external electronic device (e.g., the second electronic device 320 of FIGS. 3A and 3B). You can transmit voice data or video data using .

According to various embodiments of the present invention, the processor 1020 uses a method defined in the session technology protocol to determine the characteristics of data transmitted and received when making a call between the external electronic device and the electronic device 1000. You can negotiate with.

Hereinafter, the electronic device 1000 corresponds to the transmitting terminal (e.g., the first electronic device 310 in FIG. 3A), and the external electronic device corresponds to the second electronic device (e.g., the second electronic device 320 in FIG. 3A). ), and a specific embodiment of obtaining information on whether metadata-based video calls are supported before call connection will be described.

According to various embodiments of the present invention, the processor 1020 executes a video call application that performs a video call with the second electronic device 320 (or sends a video call to the external electronic device 320 while executing the video call application). In response to detecting a user input requesting a call, a camera (e.g., the camera module 180 in FIG. 1) is activated, and a preview image collected using the camera 180 is displayed (e.g., the camera module 180 in FIG. 1). It can be displayed on the display device 160.

According to various embodiments of the present invention, the processor 1020 uses the communication module 1010 to transmit a call connection request message to the external electronic device 320 to establish a call channel between the external electronic device 320 and the electronic device 1000. ) can be controlled. For example, the call connection request message can be implemented in the SDP Invite message format defined in SDP.

According to various embodiments of the present invention, the call connection request message may include quality information of voice data or video data that the electronic device 1000 can transmit. For example, the quality information of voice data includes the sampling rate of voice data that can be generated using a microphone (e.g., microphone 311 in FIG. 3A) and transmitted through a network (e.g., network 320 in FIG. 3A). It may include a sampling rate of voice data that can be received or a sampling rate of voice data that an external electronic device can receive. The quality information of image data includes the resolution of image data that can be generated using a camera (e.g., camera 341 in FIG. 3B), the resolution of image data that can be transmitted through the network 320, or the resolution of image data that can be received by an external electronic device. The resolution of the video data may be included.

According to various embodiments of the present invention, before completing the call connection, the processor 1020 performs an electronic device related to whether the external electronic device 320 supports the metadata-based video call in order to check whether the metadata-based video call is supported. A call connection request message containing performance information of the device 1000 may be transmitted to the external electronic device 320.

According to various embodiments of the present invention, the performance information of the electronic device 1000 may include information indicating whether components of the electronic device 1000 required in a metadata-based video call method exist. According to one embodiment, the components of the electronic device 1000 may include at least one 3D model capable of rendering emojis, and the performance information of the electronic device 1000 may include the emoji model (or the emoji model's may include information indicating whether a group) exists in the electronic device 1000.

According to various embodiments of the present invention, the call connection request message may include identification information of a model reflecting metadata required in a metadata-based video call method. A model's identifying information includes the model's version information (e.g. information indicating the type of emoji (e.g. human, animal, or alien) the model can create using metadata), the model's manufacturer information, and the standards the model uses (e.g. : May include information indicating MPEG-4 face and body animation).

According to various embodiments of the present invention, the processor 1020 may receive a response message corresponding to the call connection request message from the external electronic device 320. For example, the response message may be implemented in the SDP 200 OK message format defined in the SDP protocol.

According to various embodiments of the present invention, the response message transmitted by the external electronic device 320 may include performance information of the external electronic device 320 related to the call connection.

According to various embodiments of the present invention, the external electronic device 320 selects a data transmission method (e.g., voice) that the electronic device 1000 can support based on the performance information of the electronic device 1000 included in the call connection request message. The performance information of the external electronic device 320 may be selected from among the data sampling method, sampling rate, video data compression format, and resolution. May contain information about the selected data transmission method.

According to various embodiments of the present invention, the performance information of the external electronic device 320 includes components (e.g., : May contain information about components implemented in software (e.g., a model to reflect metadata) or components implemented in hardware (e.g., graphics processing unit (GPU)). there is.

According to various embodiments of the present invention, the processor 1020 may determine whether to perform a metadata-based video call based on the performance information of the external electronic device 320 included in the response message.

According to various embodiments of the present invention, the processor 1020 checks the performance information of the external electronic device 320 included in the response message and determines whether the external electronic device 320 can support a metadata-based video call or If metadata-based video calls are supported, you can check information on the model to be used in metadata-based video calls or standard information to be used in creating metadata to be used in metadata-based video calls.

According to various embodiments of the present invention, the processor 1020 sends a call connection confirmation message including an indicator indicating whether to perform a metadata-based video call or an indicator indicating whether to perform a rendering operation using metadata to an external electronic device. It can be transmitted to device 320.

According to various embodiments of the present invention, the call connection confirmation message may be a message instructing to perform a call connection using a transmission method of voice data or video data included in the response message. The call connection confirmation message may be, for example, an SDP ACK message defined in the SDP session protocol.

According to various embodiments of the present invention, the electronic device 1000 performs various operations (for example, using the microphone 311 to generate voice data) for a call connection with the external electronic device 320 that has received the call connection confirmation message. An operation to activate, an operation to activate the camera 341 to generate image data, an operation to activate an image renderer (graphics library and/or graphics processing unit (GPU)) to render image data to be output on the screen, meta An operation of activating a model for a data-based video call or an operation of controlling the communication module 1010 to transmit or receive data for establishing a call channel may be performed. Various operations performed by the electronic device 1000 to connect a call with the external electronic device 320 may be implemented using functions provided by the running video call application. The operation of the electronic device 1000 activating a model for a metadata-based video call is, for example, performed by the electronic device (e.g., video call application 146, video renderer) using a 3D model stored in the non-volatile memory 134. modeling data), etc., to the volatile memory 132. According to various embodiments of the present invention, the processor 1020 may include electronic information related to a metadata-based video call within a call connection request message. By including performance information of the device 400, components required to perform a metadata-based video call (e.g., video renderer and/or 3D graphic model) are activated before the call connection with the external electronic device 320 is completed. You can do it. Therefore, after the call connection is completed, it is possible to prevent deterioration in the quality of the call channel that may occur as the components required to perform a metadata-based video call are activated, and by transmitting metadata that has a smaller size than the video data, the video call can be activated. Data consumption used for calls can be reduced.

10B is a block diagram of an electronic device according to various embodiments of the present invention.

Referring to FIG. 10B, the electronic device 1000 according to various embodiments of the present invention includes a call connection management unit 1030 that performs or manages a call connection, a processing operation for transmitting voice data, or processing received voice data. an audio data processing unit 1040 that performs an operation to transmit image data or an image data processing unit 1050 that performs an operation to process received image data, and a memory (e.g., the memory of FIG. 1 ( 130)) 1061, a camera (e.g., the camera module 180 of FIG. 1) 1062, a microphone (e.g., the input device 150 of FIG. 1) 1063, a display (e.g., the display device of FIG. 1) It may include (160)) (1064) and a speaker (e.g., the sound output device 155 of FIG. 1) (1065).

According to various embodiments of the present invention, the call connection management unit 1030 parses a message received from an external electronic device (e.g., the second electronic device 320 in FIGS. 3A and 3B) while performing a call connection ( An SDP parser 1031 that parses, an SDP writer 1032 that generates a message to be transmitted to the external electronic device 320 to perform a call connection, and a SDP writer 1032 that manages connection or termination of a call session. Metadata-based video call determination to determine whether to perform a metadata-based video call based on the performance information of the external electronic device 320 included in the response message received from the call session management unit 1033 and the external electronic device 320 It may include part 1034.

According to various embodiments of the present invention, the voice data processing unit 1040 from the viewpoint of transmitting voice data includes a voice acquisition unit 1041 that acquires voice data using a microphone 1063, and a filtering operation for voice data. A voice transmission filter (e.g., the transmission filter 313 in FIG. 3A) 1042, which is a component that performs a voice encoder ( Example; it may include an encoder 315 in FIG. 3A 1043 and a packetizer 1044 that converts the encoded voice data into voice packet data in packet form. You can.

According to various embodiments of the present invention, in terms of receiving voice data, the voice data processing unit 1040 includes a depacketizer (depacketizer) that converts voice packet data received from the external electronic device 320 into encoded voice data. Example: Depacketizer 321 in FIG. 3A) 1045, voice decoder for decoding encoded voice data (e.g., decoder 323 in FIG. 3A) 1046, changing the sampling rate of decoded voice data The signal may include a voice reception filter (e.g., reception filter 325 in FIG. 3A) 1047 and a voice output unit 1048 that outputs voice through a speaker 465.

According to various embodiments of the present invention, the image data processing unit 1050 from the viewpoint of transmitting image data includes an image acquisition unit 1051 that acquires an image captured by a camera 1062, and a filtering operation on the image data. Components that perform the audio transmission filter (e.g., the transmission filter 343 in FIG. 3B) 1052, and a video encoder that performs encoding on video data to compress the video data (e.g., the encoder 345 in FIG. 3B) ) 1053 and a packetizer (e.g., packetizer 347 in FIG. 3B) 1054 that converts the encoded video data into video packet data in the form of packets.

According to various embodiments of the present invention, the image data processing unit 1050 from the viewpoint of receiving image data includes a depacketizer (depacketizer) that converts the image data packet received from the external electronic device 320 into encoded image data. Example: Depacketizer 351) 1055 in FIG. 3B, video decoder (e.g. decoder 353 in FIG. 3B) 1056 that decodes the encoded video data converted by the depacketizer 1055. , an image reception filter that changes the resolution of the decoded image data (e.g., the reception filter 355 in FIG. 3B) 1057, and an image that outputs the image data generated by the reception filter 1057 through the display 464. It may include an output unit 1058.

According to various embodiments of the present invention, the SDP writer 1032 generates a call connection request message based on the control of the call session manager 1033 and connects the call to the external electronic device 320 through the communication module 1010. A request message can be sent. For example, the call connection request message can be implemented in the SDP Invite message format defined in SDP.

According to various embodiments of the present invention, the call connection message may include performance information of the electronic device 400 related to metadata-based video calls.

According to various embodiments of the present invention, the SDP parser 1031 may parse the response message received through the communication module 410. The response message may include performance information of the external electronic device 320 related to the metadata-based video call. For example, the response message may include an indicator indicating whether the external electronic device 320 can support a metadata-based video call, and a component for the external electronic device 320 to perform a metadata-based video call, e.g. It may include an indicator indicating whether a metadata depacketizer 1071 that decrypts metadata or an image renderer 1072 that renders metadata is provided.

According to various embodiments of the present invention, the performance information of the external electronic device 320 extracted by the SDP parser 1031 through parsing of the response message is transmitted to the metadata-based video call decision unit 1034 through the call session manager 1033. ) can be transmitted.

According to various embodiments of the present invention, the metadata-based video call decision unit 1034 may determine whether to perform a metadata-based video call based on performance information of the external electronic device 320.

According to various embodiments of the present invention, the metadata-based video call decision unit 1034 checks an indicator indicating whether metadata-based video call is supported included in the response message, and determines whether the external electronic device 320 supports the metadata. You can decide whether or not to support video calls.

According to various embodiments of the present invention, the metadata-based video call decision unit 1034 confirms that the external electronic device 320 supports metadata-based video call based on the performance information of the external electronic device 320, In response to the external electronic device 320 supporting metadata-based video calling, it may be determined to perform a metadata-based video call.

According to various embodiments of the present invention, in response to determining to perform a metadata-based video call, the metadata-based video call decision unit 1034 determines that the metadata-based video call will be performed, including components necessary for performing the metadata-based video call (e.g. A metadata generator 1073 detects feature points in the image acquired by the image acquisition unit 1051 and generates metadata based on the feature points, and metadata received from the metadata generator 1073 is stored in packet form. A metadata packetizer 1074 that converts into packet data, a metadata depacketizer 1071 that converts metadata in the form of packets received from the external electronic device 320, or a method of rendering metadata to create images. The generating image renderer (1072) can be activated.

According to various embodiments of the present invention, the call session manager 1033 may transmit a call connection confirmation message indicating whether to activate a metadata-based video call to the external electronic device 320 through the communication module 1010.

According to various embodiments of the present invention, components implemented on the processor 1020 of the electronic device 1000 may be components implemented in software, but some components may be implemented in hardware depending on the design method. It may be possible. For example, the image renderer 1072 may be implemented in software and may be a graphics library that can be stored in memory (e.g., memory 130 in FIG. 1), and the image renderer 1072 may be implemented in hardware. It may be a GPU (graphic processing unit).

FIG. 11A is a diagram illustrating an example of extracting metadata from an acquired image in an electronic device according to various embodiments of the present invention.

According to various embodiments of the present invention, an electronic device (e.g., electronic device 1000 in FIG. 10A) analyzes an image collected by a camera (e.g., camera 1062 in FIG. 10B) and detects at least one image included in the image. Feature points of the above objects can be extracted. For example, objects that make up the user's face may include eyebrows, eyes, nose, mouth, tongue, or facial features, and objects that make up part of the user's body may include hands, arms, torso, or legs. You can. A feature point of an image may mean a point that can express the characteristics of an object (e.g., the shape of the object, the size of the object, the location of the object). For example, a feature point may mean a point recognized by the electronic device 1000 as an object (eg, eyes or nose) constituting the user's face. According to one embodiment, the metadata generator 1073) implemented in software or hardware on the processor of the electronic device 1000 (e.g., the processor 1020 in FIG. 10A) is generated from the image acquisition unit 1051. Feature points of an object can be extracted from the acquired image.

According to various embodiments of the present invention, the electronic device 1000 groups at least some of the extracted feature points, obtains information on variables including the characteristics of the object based on the information on the feature points, and obtains information on variables including the information on the variables. Metadata can be created. The variables are elements included in the image to be rendered (e.g., elements for implementing the characteristics of the object included in the image to be rendered. If the object in the image is an animal, the elements may be eyes, nose, mouth, or body. ) may be a variable for determining the characteristics (e.g., facial expression of an object), and the information on the variables may be the characteristics of the objects (e.g., the position of the object, the size of the object, the direction of the object, or the rotation angle of the object) or between the objects. It may refer to specific values of variables used to determine a relationship (e.g., the distance between an object (eye) and an object (nose)). The metadata generator 1073 may generate metadata including characteristics of objects or relationships between objects created based on information on feature points (e.g., distances between feature points or positions of feature points).

According to various embodiments of the invention, variables and/or metadata containing the characteristics of the objects are feature points and/or facial animation parameters defined in MPEG-4 Face and Body Animation (MPEG-4 FBA). It may contain values corresponding to Face Animation Parameters (FAPs). For example, variables and/or metadata containing the characteristics of the objects include iris diameter (IRISD), eye separation (ES), eye-nose separation (ENS), mouth-nose separation (MN), and mouth width (MW). ), and may include data corresponding to at least some of the AU (angle units). According to various embodiments of the invention, metadata may be configured according to the syntax of FAP data.

Metadata may refer to data that groups information on variables. The metadata generator 1073 implemented in software or hardware on the processor 1020 of the electronic device 1000 acquires variable information based on information on the extracted feature points, and Based on this, metadata can be created.

Referring to FIG. 11A , the metadata generator 1073 may extract the object 1100 from an image acquired using the camera 1062 and extract feature points of the object 1100. Various feature point extraction algorithms can be used to extract feature points. For example, the metadata generator 1073 may generate data containing information (e.g., information about the shape of the feature point) related to the feature point stored in the memory 130 and the object 1100 of the image acquired using the camera 1062. ) can be compared, and the similarity between the shape of at least a part of the object 1100 and the feature points included in the data can be calculated. The metadata generator 1073 may extract feature points from at least a portion of the object 1100 in response to the fact that the calculated similarity is greater than or equal to a preset value.

The metadata generator 1073 may obtain information on variables based on information on extracted feature points. For example, the metadata generator 1073 determines the size of the object's pupils (1101), the distance between the object's eyes (1103), the distance between the object's eyes and nose (1105), and the distance between the object's nose and mouth. Information on various variables including (1107) and the width (1109) of the object's mouth can be obtained based on the information on the feature points. The meta data generator 1073 may generate meta data, which is a set of information on acquired variables.

FIG. 11B is a diagram illustrating an example of rendering an image based on a model and metadata in an electronic device according to various embodiments of the present invention.

According to various embodiments of the present invention, an electronic device (e.g., the electronic device 1000 in FIG. 10A) receives an external electronic device included in a response message received from an external electronic device (e.g., the external electronic device 320 in FIG. 3A). Based on the performance information of the device 320, a model to perform rendering can be selected among models implemented in the electronic device 1000, and an emoji 1110 to be included in the image to be rendered can be generated using the selected model. .

According to various embodiments of the present invention, the electronic device 1000 may store at least one model in memory (eg, memory 1061 in FIG. 10A). At least one model may be implemented differently depending on the type of object being rendered. For example, various models including models rendering dinosaurs, models rendering pets, and models rendering celebrities may be stored in the memory 1061. The electronic device 1000 can generate the emoji 1110 included in the image to be rendered by loading the selected model on the memory 1061 and inputting metadata received from the external electronic device 320 into the model. . According to one embodiment, the image renderer 1072 implemented on the processor 1020 receives metadata from the metadata depacketizer 1071 and inputs the metadata to the model loaded on the memory 130. By doing so, the emoji 1110 included in the image to be rendered can be generated. The image renderer 1072 may render an image including the emoji 1110 using resources provided by the processor 1020 or resources provided by a separate graphics processing unit. The rendered image may be displayed on a display (eg, display 1064 in FIG. 10B).

Referring to FIG. 11B, the electronic device 1000 selects a model for rendering a pet based on the response message from the external electronic device 320 and inputs metadata received from the external electronic device 320 into the selected model. can do. For example, the electronic device 1000 includes the size of the pupil of the object 1101, the distance between the two eyes of the object 1103, the distance between the eyes and the nose of the object 1105, and the distance between the nose and the object of the object included in the metadata. The emoji included in the image to be rendered can be created by inputting information about the mouth spacing (1107) and the object's mouth width (1109) into the selected model. The electronic device 1000 may render an image including the generated emoji 1110 and display the rendered image on the display 1064.

FIG. 12 is an operation flowchart illustrating an operation in which a first electronic device and a second electronic device perform a call connection according to various embodiments of the present invention.

In operation 1210, a first electronic device (e.g., the first electronic device 310 of FIGS. 3A and 3B or the electronic device 1000 of FIG. 10A) according to various embodiments of the present invention operates a second electronic device (e.g., A call connection request message (eg, an SDP Invite message defined in the SDP session protocol) may be transmitted to the second electronic device 320 of FIGS. 3A and 3B.

According to various embodiments of the present invention, before completing a call connection, the first electronic device 1000 determines whether the external electronic device 320 supports metadata-based video calls by checking whether metadata-based video calls are supported. A call connection request message containing performance information of the electronic device 1000 related to may be transmitted to the external electronic device 320. For example, the call connection request message may be implemented in the form shown in Table 3 below.

[-->] INVITE
m=audio 7010 RTP/AVP 110 100 98
b=AS:42
a=rtpmap:110EVS/16000
a=rtpmap:100 AMR-WB/16000/1
a=rtpmap:98 AMR/8000/1
......
m=video 7020 RTP/AVP 112 102 34
b=AS:1280
a=rtpmap:112 H265/90000
a=imageattr:112 send [x=720, y=1280] recv [x=720, y=1280]
a=emoji_metadata:112 modelinfo=samsung_emoji_model_set_v2 facial_feature=MPEG4_facial_animation_standard
a=rtpmap:102 H264/90000
a=imageattr:102 send [x=480, y=640] recv [x=480, y=640]
a=emoji_metadata:102 modelinfo=samsung_emoji_model_set_v2 facial_feature=MPEG4_facial_animation_standard
a=rtpmap:34 H263/90000
a=imageattr:34 send [x=176, y=144] recv [x=176, y=144]

Referring to Table 3, the call connection request message includes information indicating that the first electronic device 310 supports metadata-based video calls (a=emoji_metadata), which can be used when performing metadata-based emoji video calls. It may include model information (modelinfo=Samsung_emoji_model_set_v2) or metadata standard information (facial_feature=MPEG4_facial_animation_standard).

According to various embodiments of the present invention, in operation 1220, the second electronic device 320 may check the performance of the second electronic device 320 related to a metadata-based video call.

According to various embodiments of the present invention, the second electronic device 320 receives performance information of the first electronic device 310 related to a metadata-based video call of the first electronic device 310 included in the call connection request message. After confirming, the second electronic device 320 may select at least one video call method supported by the second electronic device 320 among the confirmed video call methods.

According to various embodiments of the present invention, in operation 1230, the second electronic device 320 may transmit a response message (e.g., an SDP 200 OK message defined in the SDP protocol) to the first electronic device 310.

According to various embodiments of the present invention, the response message may include the performance of the second electronic device 320 related to metadata-based video call. For example, the response message may include an indicator indicating whether metadata-based video calls are supported, an indicator indicating model information used for metadata-based video calls, or an indicator indicating standard information used to generate metadata. It can contain at least one. For example, the response message may be implemented in the form shown in Table 4 below.

[<--] SIP/2.0 200 OK
m=audio 12350 RTP/AVP 100
b=AS:41
a=rtpmap:100 AMR-WB/16000/1
......
m=video 15490 RTP/AVP 102
b=AS:640
a=rtpmap:102 H264/90000
a=imageattr:102 send [x=480, y=640] recv [x=480, y=640]
a=emoji_metadata:102 modelinfo=samsung_emoji_model_set_v2 facial_feature=MPEG4_facial_animation_standard

Referring to Table 4, the response message includes an indicator (a=emoji_metadata) indicating whether the second electronic device 320 supports video calls based on metadata, and a model used when performing video calls based on metadata. It may include information about (modelinfo+Samsung_emoji_model_set_v2) and standard information used when creating metadata (facial_feature=MPEG4_facial_animation_standard).

According to various embodiments of the present invention, in operation 1240, the first electronic device 1000 may determine whether to perform a video call based on metadata based on performance information of the second electronic device 320.

According to various embodiments of the present invention, the first electronic device 1000 may check an indicator indicating support for a metadata-based video call included in a response message and decide to perform a metadata-based video call. .

According to various embodiments of the present invention, the first electronic device 1000 checks the indicator indicating that metadata-based video call is not supported included in the response message and determines not to perform metadata-based video call. You can. Alternatively, the first electronic device 1000 may determine not to perform a metadata-based video call in response to confirming that there is no indicator indicating that metadata-based video call is supported in the response message. When the first electronic device 1000 does not perform a metadata-based video call, the first electronic device 1000 exchanges the video (or image stream) transmitted by the external electronic device 320 or the electronic device 1000. You may decide to conduct a call.

According to various embodiments of the present invention, in operation 1250, the first electronic device 1000 may transmit a call connection confirmation message (e.g., an SDP ACK message defined in the SDP session protocol) to the second electronic device 320. .

According to various embodiments of the present invention, the call connection confirmation message may include an indicator indicating whether a metadata-based video call is being performed.

According to various embodiments of the present invention, in operation 1261, the first electronic device 1000 may perform a setting operation for the components of the first electronic device 1000 for call connection.

According to various embodiments of the present invention, the electronic device 1000 performs various operations (for example, using the microphone 311 to generate voice data) for a call connection with the external electronic device 320 that has received the call connection confirmation message. An operation of activating, an operation of activating the camera 341 to generate video data, an operation of activating a model that performs a rendering operation included in a metadata-based video call method, or transmitting or receiving data for establishing a call channel. An operation of controlling the communication module 1010 may be performed.

According to various embodiments of the present invention, in operation 1263, the second electronic device 320 may perform a setting operation for the components of the second electronic device 320 for call connection.

According to various embodiments of the present invention, in response to determining that the second electronic device 320 will perform a metadata-based video call, the second electronic device 320 configures the components necessary to perform the metadata-based video call (e.g., in FIG. 10B). The metadata depacketizer 1071, the image renderer 1072 in FIG. 10B, the metadata generator 1073 in FIG. 10B, or the metadata packetizer 1074 in FIG. 10B can be activated. The operation of the second electronic device 320 activating a model for a metadata-based video call is, for example, an electronic device (e.g., video call application 146, video renderer) using a model stored in the non-volatile memory 134 ( An operation of loading modeling data) into the volatile memory 132 may be included.

According to various embodiments of the present invention, in operation 1270, the first electronic device 1000 and the second electronic device 320 complete the call connection and perform a call using the call channel.

According to various embodiments of the present invention, the first electronic device 1000 and the second electronic device 320 may transmit or receive metadata through a call channel. The first electronic device 1000 and the second electronic device 320 can generate the emoji 1110 included in the image to be rendered by inputting the received metadata into the model loaded in the memory 130. there is. The first electronic device 1000 and the second electronic device 320 may display an image generated by rendering an image including an emoji 1110 on the display 1061. The first electronic device 1000 and the second electronic device 320 can implement a video call with relatively low data consumption through a video call that exchanges metadata with a capacity smaller than an image stream.

An electronic device according to various embodiments of the present invention includes a communication module that transmits or receives data using a call channel established through a call connection with an external electronic device; and a processor, wherein the processor transmits a call connection request message for establishing a call channel between an external electronic device and the electronic device,

A response message is received from the external electronic device, and whether to perform a metadata-based video call using an image including an emoji generated based on metadata through the call connection is determined by the call connection included in the response message. Make a decision based on capability information of the relevant external electronic device, transmit a call connection confirmation message indicating whether to perform the metadata-based video call to the external electronic device, and complete the call channel establishment. The communication module is controlled, and the call connection request message may include performance information of the electronic device related to whether the metadata-based video call is supported.

In an electronic device according to various embodiments of the present invention, the performance information of the electronic device may include an indicator indicating whether a component of the electronic device for performing the metadata-based video call exists.

In an electronic device according to various embodiments of the present invention, components of the electronic device may include a model for generating an emoji based on the metadata.

In an electronic device according to various embodiments of the present invention, the call connection request message may include identification information of at least one model that the electronic device can support.

In an electronic device according to various embodiments of the present invention, the response message may include identification information of at least one model that the external electronic device can support.

In an electronic device according to various embodiments of the present invention, after completing the call channel establishment, the processor receives data for generating a video from an external electronic device that supports the metadata-based video call, and sends the data It can be set to input an emoji corresponding to the metadata by inputting it into the model, and output an image including the generated emoji on a display.

In an electronic device according to various embodiments of the present invention, the processor receives a user input requesting change of the model while connected to the external electronic device through the call channel, and displays an indicator indicating change of the model. may be set to transmit to the external electronic device.

In an electronic device according to various embodiments of the present invention, after completing the call channel establishment, the processor receives image data from an external electronic device that does not support a rendering operation based on the metadata, and displays the image data. It can be set to output on the screen.

FIG. 13 is an operation flowchart illustrating a method 1300 of operating an electronic device according to various embodiments of the present invention.

According to various embodiments of the present invention, in operation 1310, an electronic device (e.g., the electronic device 1000 of FIG. 10A) may transmit a call connection request message for establishing a call channel.

According to various embodiments of the present invention, before completing the call connection, the processor 1020 performs an electronic device related to whether the external electronic device 320 supports the metadata-based video call in order to check whether the metadata-based video call is supported. A call connection request message containing performance information of the device 1000 may be transmitted to the external electronic device 320.

According to various embodiments of the present invention, the performance information of the electronic device 1000 may include information indicating whether components of the electronic device 1000 required in a metadata-based video call method exist. According to one embodiment, the components of the electronic device 1000 may include a model for rendering metadata, and the performance information of the electronic device 1000 may be determined by the presence of a model for rendering metadata in the electronic device 1000. It may include information indicating whether or not to do so.

According to various embodiments of the present invention, the call connection request message may include identification information of a model that renders metadata required in a metadata-based video call method. The model's identification information may include model version information, model manufacturer information, and information indicating the standard (e.g., MPEG-4 face and body animation) used by the model.

According to various embodiments of the present invention, in operation 1320, the electronic device 1000 may receive a response message containing performance information of the external electronic device 320 related to the video call.

According to various embodiments of the present invention, the response message may be implemented in the SDP 200 OK message format defined in the SDP protocol.

According to various embodiments of the present invention, the response message transmitted by the external electronic device 320 may include performance information of the external electronic device 320 related to the call connection.

According to various embodiments of the present invention, the performance information of the external electronic device 320 includes components (e.g., : May include information about components implemented in software (model for rendering metadata) or components implemented in hardware (e.g., graphic processing unit (GPU)).

According to various embodiments of the present invention, in operation 1330, the electronic device 1000 may determine whether to perform a metadata-based video call based on performance information of the external electronic device 320.

According to various embodiments of the present invention, the processor 1020 checks the performance information of the external electronic device 320 included in the response message and determines whether the external electronic device 320 can support a metadata-based video call or If metadata-based video calls are supported, you can check information on the model to be used in metadata-based video calls or standard information to be used in creating metadata to be used in metadata-based video calls.

According to various embodiments of the present invention, in operation 1340, the electronic device 1000 may transmit a call connection confirmation message indicating whether to perform a metadata-based video call. For example, the call connection confirmation message may be implemented in the form of an SDP ACK message defined in the SDP session protocol.

According to various embodiments of the present invention, the electronic device 1000 performs various operations (for example, using the microphone 311 to generate voice data) for a call connection with the external electronic device 320 that has received the call connection confirmation message. An operation of activating, an operation of activating the camera 341 to generate video data, and an operation of generating an emoji (e.g., emoji 1110 in FIG. 11B) to be included in an image to be rendered included in a metadata-based video call method. An operation of activating a model or an operation of controlling the communication module 1010 to transmit or receive data for establishing a call channel may be performed.

FIG. 14 is an operation flowchart illustrating a method 1400 of operating an electronic device according to various embodiments of the present invention.

According to various embodiments of the present invention, in operation 1410, the electronic device (e.g., the second electronic device 320 of FIG. 3A) sends a call connection request message for establishing a call channel to an external electronic device (e.g., the electronic device of FIG. 10A). It can be received from the device 1000). For example, the call connection request message can be implemented in the SDP Invite message format defined in SDP.

According to various embodiments of the present invention, the call connection request message includes information indicating that the first electronic device 310 supports metadata-based video calls and a model that can be used in performing metadata-based emoji video calls. It may contain standard information or metadata.

According to various embodiments of the present invention, in operation 1420, the electronic device 320 may transmit a response message containing performance information of the electronic device 320 related to a metadata-based video call to the external electronic device 1000. . For example, the response message may be implemented in the SDP 200 OK message format defined in the SDP protocol.

According to various embodiments of the present invention, the response message may include the performance of the electronic device 320 related to the metadata-based video call. For example, the response message may include an indicator indicating whether metadata-based video calls are supported, an indicator indicating model information used for metadata-based video calls, or an indicator indicating standard information used to generate metadata. It can contain at least one.

According to various embodiments of the present invention, in operation 1430, the electronic device 320 may receive a call connection confirmation message from the external electronic device 1000 indicating whether to perform a metadata-based video call. For example, the call connection confirmation message may be implemented in the form of an SDP ACK message defined in the SDP session protocol.

According to various embodiments of the present invention, the call connection confirmation message may be implemented in the form of an SDP ACK message defined in the SDP session protocol. The call connection confirmation message may include an indicator indicating whether a metadata-based video call is being performed.

According to various embodiments of the present invention, in operation 1440, the electronic device 320 may set components for call connection.

According to various embodiments of the present invention, in response to determining that the electronic device 320 will perform a metadata-based video call, the electronic device 320 collects the components required to perform the metadata-based video call (e.g., the metadata of FIG. 10B The depacketizer 1071, the image renderer 1072 in FIG. 10B, the metadata generator 1073 in FIG. 10B, or the metadata packetizer 1074 in FIG. 10B can be activated.

A method of operating an electronic device according to various embodiments of the present invention includes transmitting a call connection request message to establish a call channel between an external electronic device and the electronic device; Receiving a response message from the external electronic device; Whether to perform a preprocessing operation of the content using a transmission filter that changes the quality of the content transmitted through the call connection based on the capability information of the external electronic device related to the call connection included in the response message. The action of determining whether; Transmitting a call connection confirmation message indicating whether to perform a preprocessing operation of the content to the external electronic device; and completing the call channel establishment, wherein the call connection request message may include performance information of the electronic device related to the transmission filter.

In a method of operating an electronic device according to various embodiments of the present invention, the performance information of the electronic device may include an indicator indicating whether a component of the electronic device for processing the transmission filter exists.

In a method of operating an electronic device according to various embodiments of the present invention, components of the electronic device may include a neural processing unit (NPU).

In a method of operating an electronic device according to various embodiments of the present invention, the call connection request message may be set to include identification information of the NPU.

In a method of operating an electronic device according to various embodiments of the present invention, the call connection request message may be set to include identification information of the transmission filter.

In a method of operating an electronic device according to various embodiments of the present invention, the connection request message may include an indicator indicating whether processing of the content based on the transmission filter is supported.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. In this document, “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “A. Each of the phrases such as “at least one of , B, or C” may include all possible combinations of the items listed together in the corresponding phrase. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to those components in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” Where mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). It may be implemented as software (e.g., program 140) including these. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. Device-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is stored semi-permanently in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smartphones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store server, or a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single entity or a plurality of entities. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

Claims (20)

In electronic devices,
A communication module that transmits or receives data using a call channel established through a call connection with an external electronic device; and
Includes a processor,
The processor is
Transmitting a call connection request message to establish a call channel between an external electronic device and the electronic device,
Receive a response message from the external electronic device,
A preprocessing operation of the content using a transmission filter that changes the quality of the content transmitted through the call connection based on capability information of the external electronic device related to the call connection included in the response message, Before the content is encoded, decide whether to perform
After decoding the content, transmitting a call connection confirmation message indicating whether to perform a post-processing operation by the external electronic device to the external electronic device,
Control the communication module to complete the call channel establishment,
The connection request message includes performance information of the electronic device related to the transmission filter.
According to clause 1,
Performance information of the electronic device is
An electronic device comprising an indicator indicating whether a component of the electronic device for processing the transmission filter exists.
According to clause 2,
An electronic device wherein the components of the electronic device include a neural processing unit (NPU).
According to clause 3,
The call connection request message is
An electronic device configured to include identification information of the NPU.
According to claim 1,
The call connection request message is
An electronic device configured to include identification information of the transmission filter.
According to clause 5,
Performance information of the electronic device is
An electronic device including a resolution or sampling rate of content that can be processed using the transmission filter.
According to paragraph 1,
The connection request message is
An electronic device including an indicator indicating whether a processing operation of the content based on the transmission filter is supported.
According to clause 1,
The processor is
Based on the response message, determine whether a reception filter corresponding to the transmission filter exists in the external electronic device,
An electronic device configured to determine whether to use the transmission filter based on the confirmation result.
According to clause 7,
The processor is
An electronic device configured to activate the transmission filter and components for controlling the transmission filter before the call connection is completed, in response to determining to use the transmission filter.
In electronic devices,
A communication module that transmits or receives data using a call channel established through a call connection with an external electronic device; and
Includes a processor,
The processor is
Receiving a call connection request message to establish a call channel between an external electronic device and the electronic device,
Transmitting a response message containing performance information of the electronic device related to a reception filter that processes content received through the call connection after decoding the content to the external electronic device,
Receiving a call connection confirmation message from the external electronic device that includes an indicator indicating whether a preprocessing operation of the content using a transmission filter corresponding to the reception filter is performed after the encoding operation of the content,
An electronic device configured to perform an operation of activating the reception filter based on the indicator.
In electronic devices,
A communication module that transmits or receives data using a call channel established through a call connection with an external electronic device; and
Includes a processor,
The processor is
Receiving a call connection request message to establish a call channel between an external electronic device and the electronic device,
Transmitting a response message containing performance information of the electronic device related to a reception filter that preprocesses content received through the call connection to the external electronic device,
Receiving a call connection confirmation message from the external electronic device including an indicator indicating whether to perform a pre-processing operation of the content using a transmission filter corresponding to the reception filter,
An electronic device configured to perform an operation of activating the reception filter based on the indicator.
According to claim 10,
The response message is
An electronic device including identification information of the reception filter or an indicator indicating whether a processing operation of the content based on the reception filter is supported.
In electronic devices,
A communication module that transmits or receives data using a call channel established through a call connection with an external electronic device; and
Includes a processor,
The processor is
Transmitting a call connection request message to establish a call channel between an external electronic device and the electronic device,
Receive a response message from the external electronic device,
Capability information of the external electronic device related to the call connection included in the response message whether to perform a metadata-based video call using an image including an emoji generated based on metadata through the call connection. ) and decide based on
Transmitting a call connection confirmation message indicating whether to perform the metadata-based video call to the external electronic device,
Control the communication module to complete the call channel establishment,
The call connection request message is
An electronic device including performance information of the electronic device indicating whether it supports the metadata-based video call.
According to clause 13,
Performance information of the electronic device is
An electronic device including an indicator indicating whether a component of the electronic device for performing the metadata-based video call exists.
According to clause 14,
The components of the electronic device include a model for generating an emoji based on the metadata.
According to clause 13,
The call connection request message is
An electronic device including identification information of at least one model that the electronic device can support.
According to clause 13,
The response message is
An electronic device including identification information of at least one model that the external electronic device can support.
According to clause 13,
The processor is
After completing the call channel establishment, receive data for generating a video from an external electronic device that supports the metadata-based video call,
Input the data into a model that generates an emoji based on the metadata to generate an emoji corresponding to the metadata,
An electronic device configured to output an image including the generated emoji on a display.
According to clause 18,
The processor is
While connected to the external electronic device through the call channel, receive a user input requesting a change in a model for generating emojis based on the metadata,
An electronic device configured to transmit an indicator indicating a change in a model for generating emojis based on the metadata to the external electronic device.
According to clause 13,
The processor is
After completing the call channel establishment, receive image data from an external electronic device that does not support a rendering operation based on the metadata,
An electronic device configured to output the image data on a display.

Images