KR20250066353A - Method for providing user interface in extended reality environment and an electronic device thereof - Google Patents

Abstract

A head mounted electronic device according to an embodiment disclosed in the present document may include at least one camera, at least one display, a memory storing instructions, and a processor. The processor may be configured to acquire a first background image through the at least one camera when the instructions are executed. The processor may be configured to generate the first background image as a background blur image when the instructions are executed. The processor may be configured to check display information for a user interface to be displayed on the first background image when the instructions are executed. The processor may be configured to acquire a first portion of the background blur image to be synthesized with the user interface based on the display information when the instructions are executed. The processor may be configured to synthesize the first background image, the first portion of the background blur image, and the user interface when the instructions are executed, and display the synthesized image on the at least one display. In addition to these, various embodiments are possible as identified through the present document.

Description

METHOD FOR PROVIDING USER INTERFACE IN EXTENDED REALITY ENVIRONMENT AND AN ELECTRONIC DEVICE THEREOF

Various embodiments disclosed in this document relate to a method and an electronic device for providing a user interface in an augmented reality environment.

Recently, services that provide extended reality (XR), including augmented reality (AR), virtual reality (VR), and mixed reality (MR), are being developed. In particular, among these XR services, various XR applications related to cameras and games are being widely used by users.

The above information may be provided as related art for the purpose of assisting in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

An electronic device according to an embodiment disclosed in the present document may include at least one camera, at least one display, a memory storing instructions, and a processor. The processor may be configured to acquire a first background image through the at least one camera when the instructions are executed. The processor may be configured to generate the first background image as a background blur image when the instructions are executed. The processor may be configured to check display information for a user interface to be displayed on the first background image when the instructions are executed. The processor may be configured to acquire a first portion of the background blur image to be synthesized with the user interface based on the display information when the instructions are executed. The processor may be configured to synthesize the first background image, the first portion of the background blur image, and the user interface and display the synthesized image on the at least one display when the instructions are executed.

FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.
Figure 2 is a drawing illustrating a method of representing a user interface in which multiple applications are executed, according to a comparative example.
FIG. 3 is a drawing illustrating a configuration of an electronic device according to one embodiment.
FIG. 4 is a drawing illustrating a method of representing a user interface in an electronic device according to one embodiment.
FIG. 5 is a diagram illustrating a method of displaying a user interface by synthesizing it with a background blur image in an electronic device according to one embodiment.
FIG. 6 is a diagram illustrating a method of displaying a blurred user interface on a background image in an electronic device, according to one embodiment.
FIGS. 7A and 7B are diagrams illustrating a blur processing method when the position of a window included in a user interface in an electronic device is changed according to one embodiment.
FIGS. 8A and 8B are diagrams illustrating a blur processing method when the rotation angle of a window included in a user interface in an electronic device changes according to one embodiment.
FIGS. 9A and 9B are diagrams illustrating a blur processing method when the size of a window included in a user interface in an electronic device is changed according to one embodiment.
FIGS. 10A, 10B, 10C, and 10D are diagrams illustrating a blur processing method when performing stereo rendering in an electronic device according to one embodiment.
FIGS. 11A, 11B, and 11C are diagrams illustrating a method of applying a blur intensity based on depth data of a background image in an electronic device, according to one embodiment.
FIGS. 12a, 12b and 12c are diagrams illustrating a method of providing various types of user interfaces by blurring them, according to one embodiment.
FIG. 13 is a flowchart illustrating a method of operating an electronic device according to one embodiment.
FIG. 14 is a flowchart illustrating a method of displaying a user interface by compositing it with a background blur image in an electronic device according to one embodiment.
In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

Hereinafter, various embodiments disclosed in this document will be described with reference to the attached drawings. It should be understood that this is not intended to limit the various embodiments of the present invention to a specific form, but includes various modifications, equivalents, and/or alternatives of the present invention.

In order to secure the visibility of content displayed through a display in an XR environment, an electronic device may apply a graphic effect, such as blur, to an image displayed through the display. For example, when content including text is overlapped and displayed on a portion of a background image, if the content is displayed in an opaque form, the visibility of the background image may be reduced, and if the content is displayed in a transparent form, the visibility of the content may be reduced. To prevent this phenomenon, an electronic device may apply a blur effect to the background area where the content is displayed.

Electronic devices can blur certain areas of an image displayed on a display by preprocessing the rendering of the entire image area and synthesizing the blurred image with the original image to perform rendering processing. However, the performance (e.g., processing speed) of the electronic device may be degraded during this process. When using multiple applications in an XR environment, the electronic device must repeatedly perform these graphic processing tasks for multiple application UI (user interface) objects (e.g., windows), so the performance issue of the electronic device may be further highlighted.

In various embodiments of the present document, when providing a graphic effect such as blur to a user interface in a VST (video see through) environment, various embodiments can be provided for quickly processing complex graphic tasks and preventing performance degradation of an electronic device by obtaining a partial blur image corresponding to a background area of the user interface from an entire blur image of a background image corresponding to the entire view area, and synthesizing the partial blur image with the user interface and displaying it.

FIG. 1 is a diagram illustrating an electronic device within a network environment (100) according to one embodiment.

Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may include one or more other components. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphic processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a given function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

The auxiliary processor (123) may control at least a part of functions or states associated with at least one of the components of the electronic device (101) (e.g., the display module (160), the sensor module (176), or the communication module (190)), for example, while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, in the electronic device (101) itself on which the artificial intelligence model is executed, or may be performed through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may additionally or alternatively include a software structure.

The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or nonvolatile memory (134).

The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The audio output module (155) can output an audio signal to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive an incoming call. According to one embodiment, the receiver can be implemented separately from the speaker or as a part thereof.

The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) can include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can obtain sound through an input module (150), or output sound through an audio output module (155), or an external electronic device (e.g., an electronic device (102)) (e.g., a speaker or a headphone) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect an operating state (e.g., power or temperature) of the electronic device (101) or an external environmental state (e.g., user state) and generate an electric signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., the electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that a user can perceive through a tactile or kinesthetic sense. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) can include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery (189) can power at least one component of the electronic device (101). In one embodiment, the battery (189) can include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., the electronic device (102), the electronic device (104), or the server (108)), and performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., the application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, a corresponding communication module may communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, Wi-Fi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

The wireless communication module (192) can support a 5G network and next-generation communication technology after a 4G network, for example, NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), terminal power minimization and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) may support various requirements specified in an electronic device (101), an external electronic device (e.g., electronic device (104)), or a network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL) each, or 1 ms or less for round trip) for URLLC realization.

The antenna module (197) can transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and the external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC positioned on or adjacent a first surface (e.g., a bottom surface) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) positioned on or adjacent a second surface (e.g., a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

At least some of the above components may be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)).

According to one embodiment, a command or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform at least a part of the function or service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result, as is or additionally processed, as at least a part of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an IoT (Internet of Things) device. The server (108) may be an intelligent server using machine learning and/or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199). The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

FIG. 2 is a drawing illustrating a method of representing a user interface in which multiple applications are executed according to a comparative example. In the comparative example, an electronic device (e.g., the electronic device (101) of FIG. 1) may perform a rendering pipeline for providing a user interface including execution screens of multiple applications in a VST (video see through) environment.

Referring to FIG. 2, the electronic device (101) can perform an application rendering (210) process and a composition (220) process to provide a user interface including execution screens of multiple applications.

In a comparative example, the application rendering (210) process renders the execution screen of each application to a buffer corresponding to each application to display the execution screen of the application executed on the electronic device (101), and may include a background capture (211) operation, a blur processing (212) operation, or a rendering performance (213) operation.

In the comparative example, the background capture (211) operation is for obtaining a background image displayed in the entire view area shown to the user, and the entire view area may correspond to a field of view area that the user can see at one time. The electronic device (101) may check the user's head posture and view direction while the user wears the electronic device (101), and determine the entire view area in consideration of the checked head posture and/or view direction. The electronic device (101) may obtain a background image for the actual space and/or virtual space based on the determined entire view area.

In the comparative example, the blur processing (212) operation is to secure the visibility of the execution screen of the application displayed on the background image, and can be performed by lowering the sharpness (or increasing the transparency) of a portion of the background image corresponding to the position where the execution screen is placed. The electronic device (101) can check the coordinates of the position where the execution screen of the application is displayed on the background image, and identify the background area of the execution screen within the background image based on the checked coordinates. The electronic device (110) can capture the identified background area from the background image or crop the identified background area to obtain an image of the background area, and apply a blur effect to the image of the background area.

In the comparative example, the rendering performance (213) operation may be rendering the execution screen of the application on the image of the blurred background area. The electronic device (101) may perform rendering on the execution screen in a manner of synthesizing the execution screen with the image of the blurred background area. In the comparative example, the electronic device (101) may repeatedly perform the background capture (211) operation, the blur processing (212) operation, and the rendering performance (213) operation as many times as the number of applications being executed.

In a comparative example, the composition (220) process is a process for expressing execution screens of multiple applications blurred through the application rendering (210) process on the three-dimensional space of the background image, and may include a left-eye rendering (221) operation, a right-eye rendering (222) operation, a distortion processing (223, 224) operation, a color correction (225, 226) operation, a composition (227) operation, or a display (228) operation.

In a comparative example, the left-eye rendering (221) operation may be for rendering the execution screens in a three-dimensional space of the background image based on the user's left field of vision, and the right-eye rendering (222) operation may be for rendering the execution screens in a two-dimensional form in the three-dimensional space of the background image based on the user's right field of vision. The electronic device (101) may synthesize buffers corresponding to each application for the three-dimensional space of the background image displayed on each of the user's left-eye display and the right-eye display. At this time, the electronic device (101) may sequentially perform synthesis by considering the position and depth data of the background image in which the execution screen of each application is arranged within the three-dimensional space.

In a comparative example, the distortion processing (223) operation may be for processing distortion of execution screens rendered based on the user's left field of view, and the distortion processing (224) operation may be for processing distortion of execution screens rendered based on the user's right field of view. The electronic device (101) may process distortion for left and right buffers corresponding to each application in order to correct distortions shown to the user's left and right fields of view by convex lenses arranged to widen the field of view of the curved left and right displays.

In a comparative example, the color correction (225) operation may be for correcting the colors of execution screens based on the user's left field of vision, and the color correction (226) operation may be for correcting the colors of execution screens rendered based on the user's right field of vision. The electronic device (101) may perform color correction for the left/right buffers corresponding to each application in order to remove chromatic aberration due to the difference in refractive index for each RGB wavelength.

In a comparative example, the electronic device (101) can synthesize left and right buffers corresponding to each application into one image through a synthesis (227) operation.

In a comparative example, the electronic device (101) can display the synthesized image by transmitting it to the left/right displays through the operation of the display (228).

In a comparative example, the electronic device (101) can perform the application rendering (210) process as many times as the number of running applications and then display them on the left and right displays through the composition (220) process. However, since these processes require a lot of calculations, the performance (e.g., processing speed) of the electronic device (101) may be reduced.

FIG. 3 is a drawing illustrating the configuration of an electronic device (300) according to one embodiment.

Referring to FIG. 3, an electronic device (300) is a device that obtains a partial blur image corresponding to a background area at a location where a user interface is displayed from a full background blur image of a background image corresponding to an entire view area to be displayed, and displays the obtained partial blur image by synthesizing it with a user interface and the background image, and may include one or more sensors (310), at least one camera (320), at least one display (330), a memory (340), or a processor (350). In FIG. 3, the electronic device (300) may correspond to the electronic device (101) illustrated in FIG. 1. According to various embodiments, the electronic device (300) may be a wearable type device that can be worn on a part of a user's body (e.g., an HMD (head mounted display) device or AR glasses) or a portable multimedia device that a user can carry (e.g., a smartphone or a tablet PC).

In one embodiment, one or more sensors (310) (e.g., sensor module (176) of FIG. 1) can measure posture information of the electronic device (300) or a user wearing the electronic device (300). For example, the one or more sensors (310) can include a gyro sensor, an acceleration sensor, or various other types of sensors capable of measuring posture information of the user and/or the electronic device.

In one embodiment, at least one camera (320) (e.g., camera module (180) of FIG. 1) can recognize at least one eye of the user, either the left or right eye, and track the direction in which the recognized user eye is looking. According to various embodiments, at least one camera (320) can also obtain a real space image and provide it to the user.

In one embodiment, at least one display (330) (e.g., the display module (160) of FIG. 1) may display a background image corresponding to the entire view area and a user interface arranged on the background image according to a user's operation. For example, at least one display (330) may display a real space image captured using at least one camera (320) or a virtual space image acquired from an external device via a communication circuit (e.g., the communication module (190) of FIG. 1) as the background image. As another example, at least one display (330) may display a user interface including one or more windows by rendering it on the background image. The one or more windows may include at least one of an execution screen of one or more applications executed on the electronic device (300), multimedia data, text, or icons.

In one embodiment, at least one display (330) may be configured as at least one of a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), an organic light emitting diodes (OLED), a light emitting diode (LED), an active matrix organic LED (AMOLED), a flexible display, and a 3 dimension display. Additionally, some of these displays may be configured as transparent or light-transmitting so that the outside can be viewed therethrough. This may be configured as a transparent display including a TOLED (transparent OLED).

According to various embodiments, at least one display (330) may be a separate display for the user's left eye and right eye, or may be an integrated display in which the left and right displays are not separated.

In one embodiment, the memory (340) (e.g., the memory (130) of FIG. 1) may store instructions that, when executed, cause the electronic device (300) to perform various operations by the processor (350) (e.g., the processor (120) of FIG. 1). For example, the processor (350) may control operations to perform blur processing for a user interface expressed on a background image more quickly.

In one embodiment, the processor (350) may obtain a first background image corresponding to the entire view area, and obtain a background blur image based on the first background image. The entire view area may be determined based on posture information of the electronic device (300) measured by one or more sensors (310) (or posture information of a user wearing the electronic device (300)) or eye gaze direction information of the user detected by at least one camera (320), and may correspond to a field of view area that the user can see at one time. The first background image may include at least one of a real space image obtained from at least one camera (320) or a virtual space image obtained from an external device using a communication circuit (e.g., the communication module (190) of FIG. 1). The processor (350) may obtain a first background image corresponding to the entire view area from at least one of the real space image and the virtual space image, and may apply a blur effect to the first background image to generate a background blur image. For example, the processor (350) may generate the background blur image by mixing surrounding pixels based on a specific pixel in the first background image to create a pixel with a new color.

In one embodiment, the processor (350) may determine whether one or more windows included in a user interface to be displayed on the first background image are targets for blur processing. For example, the processor (350) may determine whether an application or content being executed has been designated as a target for blur effect application by a user, and if a window associated with an application or content designated as a target for blur effect application exists as a result of the determination, the processor may determine the user interface as a target for blur processing.

In one embodiment, the processor (350) may determine blur option data based on rendering environment information if there is at least one window identified as the blur processing target. The rendering environment information may include at least one of the brightness of the user interface, depth data of the background image, the size of the graphic buffer, the performance of hardware or the processor (350) included in the electronic device (300), or the physical size of at least one display (330). In addition, the processor (350) may additionally consider usage environment information such as CPU usage or runtime status to determine the blur option data. In one embodiment, the blur option data may include at least one of a blur type or blur intensity applicable for blur processing for the at least one window. The blur type may include at least one of a Gaussian blur or a Kawase blur. The blur intensity indicates the degree to which the image is blurred, and different blurred images may be generated depending on the blur intensity applied to the image. For example, as the blur intensity is set to a large value, a blurry blur image can be generated, and as the blur intensity is set to a small value, a clear blur image can be generated. According to various embodiments, the blur option data may be determined according to a pre-specified configuration, or may be determined based on a value estimated by learning accumulated user data. The processor (350) may obtain the background blur image by applying a blur effect to the first background image based on the determined blur option data.

In one embodiment, the processor (350) may check display information for a user interface to be displayed on the first background image. The display information may include display position information or size information of the user interface. The display position information may include position information on at least one display (330) or background image where one or more windows included in the user interface are displayed, and may include at least one of position coordinates or rotation angles. The size information may be related to a size at which one or more windows included in the user interface are displayed on at least one display (330) or background image. For example, the processor (350) may project three-dimensional coordinates of each user interface composed of three-dimensional information onto the background image to obtain two-dimensional coordinates, and may check the display information based on the obtained two-dimensional coordinates.

In one embodiment, the processor (350) may obtain a first portion of the background blur image to be composited with the user interface based on the display information. The first portion of the background blur image may be a partial blur image corresponding to a background area of the user interface displayed on the first background image. For example, the processor (350) may calculate a background area on which the user interface is overlapped and displayed among the entire area of the first background image based on the display information. The processor (350) may crop a partial image corresponding to the calculated background area from the background blur image to obtain the first portion of the background blur image.

In one embodiment, the processor (350) may synthesize a first portion of the acquired background blur image with the user interface, and display the blurred user interface through the synthesis with the first background image on at least one display (330).

In one embodiment, when the processor (350) determines that the display information of the user interface is changed, the processor (350) may obtain a second portion of the background blur image to be synthesized with the user interface based on the changed display information. For example, when the processor (350) determines that the position of a first window among the windows included in the user interface is moved, the processor (350) may calculate a background area changed according to the position movement of the first window. For another example, when the processor (350) determines that the first window is rotated by a predetermined angle, the processor (350) may calculate a background area changed according to the rotation of the first window. For another example, when the processor (350) determines that the size of the first window is enlarged (or reduced), the processor (350) may calculate a background area changed according to the enlargement (or reduction) of the size of the first window. The processor (350) may crop a partial image corresponding to the changed background area from the background blur image to obtain a second portion of the background blur image, and may synthesize the second portion of the background blur image with the user interface. The processor (350) may display the blurred user interface through the synthesis on at least one display (330) by synthesizing it with a second background image according to the changed display information. According to various embodiments, the second background image may be the same image as the first background image. For example, if the processor (350) determines that even if the display information of the user interface is changed, the posture information of the electronic device (300) or the movement or gaze direction change of the user wearing the electronic device (300) is not detected, and only the position or size of at least one window included in the user interface is changed, the processor (350) may display the second background image, which is the same as the first background image, by synthesizing it with the blurred user interface.

According to various embodiments, the second background image may be a different image from the first background image. In one embodiment, if the processor (350) determines that the background image is changed while displaying the user interface, the processor (350) may synthesize the second background image, which is different from the first background image, with the blurred user interface and display it. For example, if the processor (350) detects a change in the posture of the electronic device (300), a movement of a user wearing the electronic device (300), or a change in the user's gaze direction, the processor (350) may determine that the background image is changed according to a change in the entire view area. For another example, if the background image is of a type that is automatically updated over time, the processor (350) may determine that the background image is changed when a criterion (e.g., time, date) for updating the background image is satisfied. As another example, when a new object (e.g., a person) appears in a real space while the processor (350) is displaying a background image including a virtual space, the processor (350) may determine that the background image has changed by displaying a real-time image acquired from at least one camera (320) on at least a portion of the background image. When the second background image is different from the first background image, the processor (350) may determine blur option data for the changed second background image, and apply a blur effect to the second background image based on the blur option data to update the background blur image.

In one embodiment, the processor (350) may apply a blur intensity differently based on depth data for at least one object included in the first background image (or the second background image). For example, if a first object is located closer than a second object based on a user's viewpoint within the first background image (or the second background image), the processor (350) may apply a blur intensity differently to generate the background blur image so that an area corresponding to the first object is expressed more clearly than an area of the second object.

According to various embodiments, when at least one display (330) is divided into a left-eye display and a right-eye display, rendering for the two displays may be required. In this case, the processor (350) may generate a single image by connecting non-overlapping portions (e.g., portions included only in the left-eye background image and/or portions included only in the right-eye background image) based on overlapping portions of the left-eye background image and the right-eye background image, or may generate a single image by overlapping the left-eye background image and the right-eye background image, and applying a blur effect to the single image at once, thereby increasing blur processing performance.

FIG. 4 is a drawing illustrating a method of expressing a user interface in an electronic device (300) according to one embodiment.

Referring to FIG. 4, the electronic device (300) can express a structure in which a two-dimensional user interface is arranged in a three-dimensional space by utilizing a method in which two or more layers are overlapped and displayed. For example, a first layer (410) can be arranged on the background of the entire view area visible to the user's field of vision, and a second layer (420), a third layer (430), or a fourth layer (440) can be arranged on the first layer (410). The number of layers that are overlapped and arranged on the first layer (410) can correspond to the number of windows to be displayed through the user interface.

In one embodiment, the electronic device (300) may render a background image on the first layer (410) and render one or more windows to be displayed on the second layer (420), the third layer (430), or the fourth layer (440) that are overlappingly arranged on the first layer (410). At this time, the second layer (420), the third layer (430), or the fourth layer (440) may be generated on the first layer (410) so as to correspond to the position/size/rotation angle at which one or more windows included in the user interface are displayed on the background image. The background image may include at least one of an image of a real space acquired in real time through a camera (e.g., the camera module (180) of FIG. 1 or at least one camera (320) of FIG. 3) or an image of a virtual space acquired from an external device through a communication circuit (e.g., the communication module (190) of FIG. 1). The one or more windows may include at least one of an execution screen, multimedia data, text, or icon of an application running on the electronic device (300).

FIG. 5 is a drawing illustrating a method of displaying a user interface by synthesizing it with a background blur image in an electronic device (300) according to one embodiment.

Referring to FIG. 5, the electronic device (300) may blur at least a portion of an area on which the user interface (510) is overlapped on the background image to increase the visibility of the background image and/or the user interface (510) during the process of displaying the user interface (510). In one embodiment, the electronic device (300) may obtain a background image to be rendered on a first layer (e.g., the first layer (410) of FIG. 4) based on an entire view area determined based on at least one of the posture information of the electronic device (or the posture information of a user wearing the electronic device (300)) or the detected eye gaze direction information of the user. The background image may include at least one of a real space image obtained in real time from at least one camera (e.g., the camera module (180) of FIG. 1 or at least one camera (320) of FIG. 3) or a virtual space image obtained from an external device.

In one embodiment, the electronic device (300) may apply a blur effect to the background image to generate a background blur image (520). The electronic device (300) may determine blur option data based on rendering environment information confirmed at the time of generating the background blur image (520). The rendering environment information may include at least one of the brightness of the user interface (510), depth data of the background image, the size of the graphic buffer, the performance of the hardware included in the electronic device (300), or the physical size of at least one display (330). The blur option data may include at least one of a blur type or blur intensity applicable for blur processing for the at least one interface. According to various embodiments, the blur option data may be determined according to a pre-specified configuration, or may be determined based on a value estimated by learning accumulated user data. The electronic device (300) may apply a blur effect to the background image based on the determined blur option data to generate the background blur image (520).

In one embodiment, the electronic device (300) can check display information including at least one of a size, a position, or a rotation angle of a user interface (510) displayed on the background image, and calculate a background area where the user interface (510) is overlapped and displayed among the entire area of the background image based on the display information. The electronic device (300) can crop a partial image corresponding to the background area from a background blur image (520) to obtain a partial blur image (521).

In one embodiment, the electronic device (300) may perform blur processing on the user interface (510) by synthesizing the acquired partial blur image (521) with the user interface (510). The electronic device (300) may display the blurred user interface (530) on the background image, which may be expressed as in FIG. 6.

FIG. 6 is a diagram illustrating a method of displaying a blurred user interface on a background image in an electronic device (300), according to one embodiment.

Referring to FIG. 6, the electronic device (300) can render a background image corresponding to the entire view area on the first layer (610) and render a blurred user interface on the second layer (620) and the third layer (630) that are overlaid on the first layer (610).

According to various embodiments, the number, size, and/or position of layers overlappingly arranged on the first layer (610) is not limited to any one embodiment, and may be arranged to correspond to the number/size/position of windows included in the user interface displayed on the background image.

FIGS. 7A and 7B are drawings explaining a blur processing method when the position of a window included in a user interface in an electronic device (300) is changed according to one embodiment.

Referring to FIG. 7A, while displaying a user interface on a background image (700), the position of the first window (701) may be moved by a user input. For example, the electronic device (300) may confirm that the first window (701) is moved from a first position (710) to a second position (720) by a user input corresponding to movement 1, and then moved from the second position (720) to a third position (730) by a user input corresponding to movement 2.

In one embodiment, the electronic device (300) can update the blur processing for the first window (701) whenever the position of the first window (701) is moved. When the electronic device (300) determines that the position of the first window (701) is moved to the second position (720) by the movement 1, the electronic device (300) can calculate a changed background area corresponding to the second position (720) among the entire area of the background image (700) and perform blur processing for the first window (701) using the partial blur image of the changed background area. In FIG. 7b, the electronic device (300) can obtain the partial blur image of the changed background area from the background blur image (740) generated by applying a blur effect to the background image (700). For example, the electronic device (300) can generate a blurred first window (741) corresponding to the second location (720) by cropping a partial image corresponding to the changed background area from the background blur image (740) to obtain the partial blur image, and synthesizing the partial blur image with the first window (701).

In one embodiment, when the electronic device (300) determines that the position of the first window (701) is further moved to the third position (730) by the movement 2, the electronic device (300) may calculate a changed background area corresponding to the third position (730) among the entire area of the background image (700), and may re-perform blur processing for the first window (701) using the partial blur image of the changed background area. In FIG. 7b, the electronic device (300) may crop a partial image corresponding to the changed background area from the background blur image (740) to obtain the partial blur image, and may generate a blurred first window (742) corresponding to the third position (730) by synthesizing the partial blur image with the first window (701).

FIGS. 8A and 8B are drawings explaining a blur processing method when the rotation angle of a window included in a user interface in an electronic device (300) changes according to one embodiment.

Referring to FIG. 8A, while displaying a user interface on a background image (800), the first window (801) may be rotated to a predetermined angle by a user input. For example, the electronic device (300) may confirm that while the first window (801) is displayed at a first angle (810) based on the user's viewpoint, a first rotation occurs due to a user's movement, a change in the gaze direction, or a user input, thereby changing the display angle of the first window (801) to a second angle (820), and then a second rotation additionally occurs due to the user's movement, a change in the gaze direction, or a user input, thereby changing the display angle of the first window (801) to a third angle (830).

In one embodiment, the electronic device (300) may update blur processing for the first window (801) whenever the first window (801) is rotated and the display angle changes. When the electronic device (300) determines that the display angle of the first window (801) is changed to the second angle (820) by rotation 1, the electronic device (300) may calculate a background area that has been changed in response to rotation 1 among the entire area of the background image (800), and perform blur processing for the first window (801) using the partial blur image of the changed background area. In FIG. 8b, the electronic device (300) may obtain a partial blur image of the changed background area from a background blur image (840) generated by applying a blur effect to the background image (800). For example, the electronic device (300) can generate a first window (841) that is blurred in response to a second angle (820) rotation by cropping a partial image corresponding to the changed background area from the background blur image (840) to obtain the partial blur image, and synthesizing the partial blur image with the first window (801).

In one embodiment, when the electronic device (300) determines that the display angle of the first window (801) is additionally changed to a third angle (830) by rotation 2, the electronic device (300) may calculate a background area changed in response to rotation 2 among the entire area of the background image (800), and re-perform blur processing for the first window (801) using a partial blur image of the changed background area. In FIG. 8b, the electronic device (300) may crop a partial image corresponding to the changed background area from the background blur image (840) to obtain the partial blur image, and may generate a first window (842) blurred in response to the rotation at the third angle (830) by synthesizing the partial blur image with the first window (801).

FIGS. 9A and 9B are drawings explaining a blur processing method when the size of a window included in a user interface in an electronic device (300) is changed according to one embodiment.

Referring to FIG. 9A, the size of the first window (901) may be changed by a user input while displaying a user interface on a background image (900). For example, the electronic device (300) may confirm that the first window (901) is enlarged from a first size (910) to a second size (920) by a user input corresponding to expansion 1, and then reduced from the second size (920) to a third size (930) by a user input corresponding to reduction 2.

In one embodiment, the electronic device (300) may update the blur processing for the first window (901) whenever the size of the first window (901) changes. When the electronic device (300) determines that the size of the first window (901) is changed to the second size (920) by the expansion 1, the electronic device (300) may calculate a changed background area corresponding to the expansion 1 among the entire area of the background image (900) and perform blur processing for the first window (901) using the partial blur image of the changed background area. In FIG. 9b, the electronic device (300) may obtain the partial blur image of the changed background area from the background blur image (940) generated by applying a blur effect to the background image (900). For example, the electronic device (300) can crop a partial image corresponding to the changed background area from the background blur image (940) to obtain the partial blur image, and generate a blurred first window (941) corresponding to the second size (920) by synthesizing the partial blur image with the first window (901).

In one embodiment, when the electronic device (300) determines that the size of the first window (901) is further changed to the third size (930) by reduction 2, the electronic device (300) may calculate a background area changed in response to reduction 2 among the entire area of the background image (900), and re-perform blur processing for the first window (901) using a partial blur image of the changed background area. In FIG. 9b, the electronic device (300) may crop a partial image corresponding to the changed background area from the background blur image (940) to obtain the partial blur image, and may generate a first window (942) blurred in response to the third size (930) by synthesizing the partial blur image with the first window (901).

FIGS. 10A, 10B, 10C, and 10D are diagrams illustrating a blur processing method when performing stereo rendering in an electronic device (300) according to one embodiment. According to various embodiments, the electronic device (300) may perform rendering for each of the left and right eyes for stereo rendering.

Referring to FIG. 10A, the left-eye background image (1010) displayed for the left eye and the right-eye background image (1020) displayed for the right eye based on the two eyes of the user (1000) do not match, but may have overlapping portions. The electronic device (300) may improve blur processing performance by applying the blur effect to the left-eye background image (1010) and the right-eye background image (1020) at once instead of individually so that the blur effect is not applied repeatedly to the overlapping portions.

In one embodiment, the electronic device (300) may apply a blur effect by attaching a portion (1031) not included in the left-eye background image (1010) of the right-eye background image (1020) to a portion (1030) corresponding to the left-eye background image (1010), as shown in FIG. 10b.

In one embodiment, the electronic device (300) may also apply a blur effect in batches while the left-eye background image (1041) and the right-eye background image (1042) overlap each other, as shown in FIG. 10c.

In one embodiment, there may be areas within a background image that can be viewed from both the left and right eyes of the user. However, even when the user views the same object within the background image from the left and right eyes, the angles at which the left and right eyes view the object may be different. As a result, even if there are overlapping areas in the left-eye background image (1010) and the right-eye background image (1020), they may be displayed differently. However, since the images become blurred when a blur effect is applied, the difference between the left-eye blur image (1051) and the right-eye blur image (1052) may not be perceived with the naked eye. Therefore, the electronic device (300) may reduce unnecessary operations by utilizing a partial image cropped from the left-eye blur image (1051) for the right eye.

FIGS. 11A, 11B, and 11C are diagrams illustrating a method of applying a blur intensity based on depth data of a background image in an electronic device (300), according to one embodiment.

Referring to FIG. 11A, the electronic device (300) may apply a blur intensity differently based on depth data for a space within a background image (1110). For example, the electronic device (300) may apply a blur intensity differently according to a distance range of a space within a background image (1110) based on a user viewpoint (1100). In FIG. 11A, the electronic device (300) may determine that an area/object belonging to a first distance range has a first depth value (1111), that an area/object belonging to a second distance range has a second depth value (1112), and that an area/object belonging to a third distance range has a third depth value (1113). The electronic device (300) may apply a first blur intensity to the first depth value (1111), and may apply a second blur intensity greater than the first blur intensity to the second depth value (1112) and the third depth value (1113). In this case, a first object area (1121) having the first depth value (1111) in a background image (1120) to which a blur effect is applied may be expressed more clearly than a second object area (1122) having the second depth value (1112) and the third depth value (1113).

In one embodiment, the electronic device (300) may synthesize at least a portion of a blurred image to which a blur intensity is differentially applied with a user interface and display the result on a background image. For example, as shown in FIG. 11b, the blur intensity may be applied differently according to a depth value of each background area in the user interface (1140), so that a background corresponding to a first area (1141) in the user interface (1140) may be expressed more clearly than a background corresponding to a second area (1142). As another example, as shown in FIG. 11c, when a user interface including two different windows (1151, 1152) is displayed, the blur intensity may be applied differently according to a depth value of a location in the background image (1150), so that a background corresponding to the second window (1152) may be expressed more blurry than a background corresponding to the first window (1151).

FIGS. 12A, 12B, and 12C are diagrams illustrating a method of providing various types of user interfaces by blurring them according to one embodiment. According to various embodiments, the electronic device (300) may perform blurring on various types of user interfaces in addition to the execution screen of a running application.

Referring to FIG. 12A, the electronic device (300) may perform blur processing to display a graph (1210) representing specific information on a background image. For example, the graph (1210) may be related to a progress status or a battery level for a specific task being performed by the electronic device (300). When the progress status or the battery level for the specific task is 64%, the electronic device (300) may perform blur processing on a graph corresponding to the remaining 36%.

Referring to FIG. 12b, the electronic device (300) can perform blur processing on an icon (1220) displayed on a background image. The icon (1220) can represent a specified type of information, such as real-time weather information, date, time, or surrounding environment data.

Referring to FIG. 12c, the electronic device (300) can perform blur processing on text (1230) displayed on a background image to increase the visibility of the text (1230).

FIG. 13 is a flowchart illustrating an operating method of an electronic device (300) according to an embodiment. According to an embodiment, the electronic device (300) is a device that obtains a partial blur image corresponding to a background area at a location where a user interface is displayed from a full background blur image of a background image corresponding to an entire view area, and displays the obtained partial blur image by synthesizing it with a user interface and the background image, and may correspond to the electronic device (101) illustrated in FIG. 1. The operations of FIG. 13 may be performed by a processor included in the electronic device (300) (e.g., the processor (120) of FIG. 1 or the processor (350) of FIG. 3).

Referring to FIG. 13, in operation 1310, the electronic device (300) may generate a first background image corresponding to the entire view area as a background blur image. The electronic device (300) may determine the entire view area based on posture information of the electronic device (300) (or posture information of a user wearing the electronic device (300)) measured using one or more sensors (e.g., the sensor module (176) of FIG. 1 or one or more sensors (310) of FIG. 3) or eye gaze direction information of the user detected using at least one camera (e.g., the camera module (180) of FIG. 1 or at least one camera (320) of FIG. 3), and may correspond to a field of view area that the user can see at one time. The background image may include at least one of a real space image acquired from at least one camera (320) or a virtual space image acquired from an external device using a communication circuit (e.g., the communication module (190) of FIG. 1). In operation 1310, the electronic device (300) may obtain a first background image corresponding to the entire view area from at least one of the real space image or the virtual space image, and may apply a blur effect to the first background image to generate the background blur image. For example, the electronic device (300) may generate the background blur image by mixing pixels surrounding a specific pixel in the first background image to create a pixel having a new color.

According to various embodiments, the electronic device (300) may obtain the background blur image by determining blur option data based on rendering environment information in operation 1310 and applying a blur effect to the first background image based on the determined blur option data. The rendering environment information may include at least one of brightness of a user interface to be displayed on the background image, depth data of the background image, a size of a graphic buffer, performance of hardware or a processor (350) included in the electronic device (300), or a physical size of at least one display (e.g., the display module (160) of FIG. 1 or at least one display (330) of FIG. 3). In addition, the electronic device (300) may determine the blur option data by additionally considering usage environment information such as CPU usage or a runtime status. The blur option data may include at least one of a blur type or a blur intensity applicable for blur processing for the interface. The blur type may include at least one of a Gaussian blur or a Kawase blur. The above blur intensity indicates the degree to which the image is blurred, and different blur images can be generated depending on the blur intensity applied to the image. For example, the greater the blur intensity is set, the more blurry the blur image is generated, and the smaller the blur intensity is, the more clear the blur image is generated. According to various embodiments, the blur option data may be determined according to a pre-specified configuration, or may be determined based on a value estimated by learning accumulated user data.

According to various embodiments, the electronic device (300) may, in operation 1310, apply a blur intensity differently based on depth data for at least one object included in the first background image. For example, if a first object is located closer than a second object based on a user's viewpoint within the first background image, the electronic device (300) may generate the background blur image by differentially applying a blur intensity so that an area corresponding to the first object is expressed more clearly than an area of the second object.

According to one embodiment, in operation 1320, the electronic device (300) may check display information for a user interface to be displayed on the first background image. The display information may include display position information or size information of the user interface. The display position information may include position information on at least one display (330) or the first background image where one or more windows included in the user interface are displayed, and may include at least one of position coordinates or a rotation angle. The size information may indicate at what size one or more windows included in the user interface are displayed on at least one display (330) or the first background image. For example, the electronic device (300) may project three-dimensional coordinates of each user interface composed of three-dimensional information onto the background image to obtain two-dimensional coordinates, and may check the display information based on the obtained two-dimensional coordinates.

According to one embodiment, in operation 1330, the electronic device (300) may obtain a first portion of the background blur image to be synthesized with the user interface based on the display information. The first portion of the background blur image may be a partial blur image corresponding to a background area of the user interface displayed on the first background image. For example, the electronic device (300) may calculate a background area on which the user interface is overlapped and displayed among the entire area of the first background image based on the display information. In operation 1330, the electronic device (300) may crop a partial image corresponding to the calculated background area from the background blur image to obtain the first portion of the background blur image.

According to one embodiment, in operation 1340, the electronic device (300) may synthesize a first part of the acquired background blur image with the user interface, and display the blurred user interface through the synthesis with the first background image on at least one display (330).

According to one embodiment, in operation 1350, the electronic device (300) may determine that display information of the user interface is changed while displaying the user interface, and may obtain a second part of the background blur image to be synthesized with the user interface based on the changed display information. For example, if the electronic device (300) determines that at least one of a position, a size, and a rotation angle of a first window among windows included in the user interface is changed, the electronic device (300) may calculate a changed background area for the first window. The electronic device (300) may crop a partial image corresponding to the changed background area from the background blur image to obtain a second part of the background blur image.

According to one embodiment, in operation 1360, the electronic device (300) may synthesize a second background image, a second part of the background blur image, and the user interface. The electronic device (300) may synthesize the second part of the acquired background blur image with the user interface, and may synthesize the blurred user interface through the synthesis with the second background image according to the changed display information and display the synthesized user interface on at least one display (330). According to various embodiments, the second background image may be the same image as the first background image. For example, if the electronic device (350) determines that even if the display information of the user interface is changed, the posture information of the electronic device (300) or the movement or gaze direction change of the user wearing the electronic device (300) is not detected, and only the position or size of at least one window included in the user interface is changed, the electronic device (350) may synthesize the second background image, which is the same as the first background image, with the blurred user interface and display the synthesized user interface.

According to various embodiments, the second background image may be a different image from the first background image. When the electronic device (300) determines that the background image is changed while displaying the user interface, the electronic device (300) may synthesize the second background image, which is different from the first background image, with the blurred user interface and display it. For example, when the electronic device (300) detects a change in the posture of the electronic device (300), a movement of a user wearing the electronic device (300), or a change in the user's gaze direction, the electronic device (300) may determine that the background image is changed according to a change in the entire view area. For another example, when the background image is of a type that is automatically updated over time, the electronic device (300) may determine that the background image is changed when a criterion (e.g., time, date) for updating the background image is satisfied. As another example, when a new object (e.g., a person) appears in a real space while the electronic device (300) is displaying a background image including a virtual space, the electronic device (300) may determine that the background image has changed by displaying a real-time image acquired from at least one camera (320) on at least a portion of the background image. When the second background image is different from the first background image, the electronic device (300) may determine blur option data for the changed second background image, and apply a blur effect to the second background image based on the blur option data to update the background blur image.

FIG. 14 is a flowchart illustrating a method of displaying a user interface by synthesizing it with a background blur image in an electronic device (300) according to one embodiment. The operations of FIG. 14 may be understood as functions performed by at least one processor (e.g., the processor (120) of FIG. 1 or the processor (350) of FIG. 3) included in the electronic device (300).

Referring to FIG. 14, in operation 1410, the electronic device (300) may obtain a background image and a user interface displayed on a display (e.g., the display module (160) of FIG. 1 or at least one display (330) of FIG. 3). The background image is an image corresponding to an entire view area determined based on posture information of the electronic device (300) measured by one or more sensors (310) (or posture information of a user wearing the electronic device (300)) or eye gaze direction information of the user detected by at least one camera (320), and may include at least one of a real space image obtained from at least one camera (e.g., the camera module (180) of FIG. 1 or at least one camera (320) of FIG. 3)) or a virtual space image obtained from an external device using a communication circuit (e.g., the communication module (190) of FIG. 1). The above user interface includes one or more windows, and the one or more windows may include at least one of an execution screen, multimedia data, text, or icon of one or more applications running on the electronic device (300).

According to one embodiment, in operation 1420, the electronic device (300) may determine whether one or more windows included in the user interface are targets of blur processing. For example, the electronic device (300) may determine whether an application or content being executed has been designated as a target of blur effect by a user, and if a window associated with the application or content designated as a target of blur effect exists as a result of the determination, the electronic device may determine the user interface as a target of blur processing.

If the user interface is a target of blur processing as a result of the determination in operation 1420 (operation 1420-Yes), the electronic device (300) can determine whether a change has occurred in the background image in operation 1430. For example, if the electronic device (300) detects a change in the posture of the electronic device (300), a movement of a user wearing the electronic device (300), or a change in the direction of the user's gaze, the electronic device (300) can determine that the background image has changed according to a change in the entire view area. For another example, if the background image is of a type that is automatically updated over time, the electronic device (300) can determine that the background image has changed when a criterion (e.g., time, date) for updating the background image is satisfied. For another example, if the electronic device (300) is displaying a background image including a virtual space and a new object (e.g., a person) appears in an actual space, the electronic device (300) can display a real-time image acquired from at least one camera (320) in at least a portion of the background image to determine that the background image has changed.

If the background image is changed as a result of the determination in operation 1430 (operation 1430-Yes), the electronic device (300) can determine blur option data for the changed background image in operation 1435. The blur option data includes at least one of a blur type or a blur intensity applicable for blur processing for the user interface, wherein the blur type includes at least one of a Gaussian blur or a Kawase blur, and the blur intensity can indicate the degree to which the image is blurred. The electronic device (300) applies a blur effect to the background image in operation 1440 to generate a background blur image, and in operation 1450, the electronic device (300) can calculate a background area on which the user interface is overlapped and displayed among the entire area of the background image based on display information for the user interface.

If the judgment result of operation 1430 indicates that the background image has not been changed (operation 1430-No), the electronic device (300) may skip operations 1435 to 1440, and calculate a background area where one or more windows included in the user interface are overlapped and displayed among the entire area of the background image in operation 1450.

According to one embodiment, in operation 1460, the electronic device (300) may obtain a partial blur image of a background blur image for the user interface based on the calculated background area. The electronic device (300) may crop a portion corresponding to the calculated background area from the background blur image to obtain the partial blur image for the user interface.

According to one embodiment, in operation 1470, the electronic device (300) may synthesize the partially blurred image with the user interface, and display the blurred user interface through the synthesis with the background image on at least one display (330).

If the user interface is not a target for blur processing as a result of the judgment of operation 1420 (operation 1420-No), the electronic device (300) may display the non-blurred user interface on the background image without performing operations 1430 to 1470.

According to an embodiment, a head-mounted electronic device (e.g., electronic device (300)) includes at least one camera (e.g., camera (320)), at least one display (e.g., display (330)), a memory (e.g., memory (340)) configured to store instructions, and a processor (e.g., processor (350)), wherein, when the instructions are executed, the processor (350) causes the electronic device (300) to acquire a first background image through the at least one camera, generate the first background image as a background blur image, check display information for a user interface to be displayed on the first background image, acquire a first part of the background blur image to be synthesized with the user interface based on the display information, synthesize the first background image, the first part of the acquired background blur image, and the user interface and display them on the at least one display, and, as the display information of the user interface is changed, acquire a second part of the background blur image to be synthesized with the user interface based on the changed display information, and generate the second part of the background blur image to be synthesized with the user interface based on the changed display information. Instructions may be stored for synthesizing a background image, a second portion of the acquired background blur image, and the user interface and displaying the result on at least one display.

In one embodiment, the electronic device (300) further includes one or more sensors (e.g., sensor (310)) configured to obtain attitude information of the electronic device, and the processor (350) may be configured to obtain the second background image corresponding to the changed attitude information through the at least one camera when the instructions are executed and the electronic device (300) determines that the attitude information of the electronic device has been changed.

In one embodiment, the processor (350) may be configured to, when the instructions are executed, determine whether one or more windows included in the user interface are targets of blur processing, determine blur option data based on rendering environment information if there is a window identified as a target of blur processing, and perform blur processing on the first background image based on the determined blur option data to obtain the background blur image.

In one embodiment, the rendering environment information may include at least one of brightness of the user interface, depth data of the first background image, a size of a graphics buffer, performance of the processor, or a size of the at least one display.

In one embodiment, the processor (350) may be configured such that, when the instructions are executed, the electronic device (300) calculates a background area for one or more windows included in the user interface based on the display information, and crops a partial image corresponding to the calculated background area from the background blur image to obtain a first portion of the background blur image.

In one embodiment, the processor (350) may be configured to cause the electronic device (300) to calculate the background area based on at least one of the size, position, or rotation angle of each of the one or more windows when the instructions are executed.

In one embodiment, the processor (350), when the instructions are executed, if the electronic device (300) determines that display information of a first window among one or more windows included in the user interface has been changed, calculates a changed background area for the first window based on the changed display information, crops a partial image corresponding to the changed background area from the background blur image to obtain a third part of the background blur image, and synthesizes and displays the first background image, the third part of the background blur image, and the user interface.

In one embodiment, the processor (350) may cause the electronic device (300) to, when the instructions are executed, check blur option data for the second background image when the second background image is obtained, and generate a background blur image corresponding to the changed second background image based on the checked blur option data.

In one embodiment, the processor (350) may cause the electronic device (300) to, when the instructions are executed, identify depth data for at least one object included in the first background image, and differentially apply a blur intensity for each area corresponding to the at least one object in the first background image based on the identified depth data to generate the background blur image.

In one embodiment, the one or more windows may include at least one of an execution screen of an application running on the electronic device, multimedia data, text, or an icon.

According to an embodiment, a method for operating a head mounted electronic device (e.g., electronic device (300)) may include an operation of acquiring a first background image, an operation of generating the first background image as a background blur image, an operation of checking display information for a user interface to be displayed on the first background image, an operation of acquiring a first part of the background blur image to be synthesized with the user interface based on the display information, an operation of synthesizing and displaying the first background image, the first part of the acquired background blur image, and the user interface, an operation of acquiring a second part of the background blur image to be synthesized with the user interface based on the changed display information as the display information of the user interface is changed, and an operation of synthesizing and displaying the second background image, the second part of the acquired background blur image, and the user interface.

In one embodiment, the method may further include an operation of obtaining the second background image corresponding to the changed posture information when it is determined that the posture information of the electronic device has changed.

In one embodiment, the operation of generating the first background image as a background blur image may include the operation of checking whether one or more windows included in the user interface are targets of blur processing, if there is a window identified as a target of blur processing, the operation of determining blur option data based on rendering environment information, and the operation of performing blur processing on the first background image based on the determined blur option data to obtain the background blur image.

In one embodiment, the rendering environment information may include at least one of brightness of the user interface, depth data of the first background image, a size of a graphics buffer, performance of the processor, or a size of the at least one display.

In one embodiment, the operation of obtaining the first portion of the background blur image may further include the operation of calculating a background area for one or more windows included in the user interface based on the display information, and the operation of cropping a partial image corresponding to the calculated background area from the background blur image to obtain the first portion of the background blur image.

In one embodiment, the operation of calculating the background area may include calculating the background area based on at least one of a size, a position, or a rotation angle of each of the one or more windows.

In one embodiment, the method may further include, when it is determined that display information of a first window among one or more windows included in the user interface has been changed, calculating a changed background area for the first window based on the changed display information, cropping a partial image corresponding to the changed background area from the background blur image to obtain a third portion of the background blur image, and synthesizing and displaying the first background image, the third portion of the background blur image, and the user interface.

In one embodiment, the method may further include, when obtaining the second background image, an operation of checking blur option data for the second background image, and an operation of generating a background blur image corresponding to the changed second background image based on the checked blur option data.

In one embodiment, the method may further include an operation of identifying depth data for at least one object included in the first background image, and an operation of differentially applying a blur intensity for each area corresponding to the at least one object in the first background image based on the identified depth data to generate the background blur image.

In one embodiment, the one or more windows may include at least one of an execution screen of an application running on the electronic device, multimedia data, text, or an icon.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs.

Electronic devices according to various embodiments disclosed in this document may be devices of various forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliance devices. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" as used in this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more commands stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the called at least one command. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means only that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store??) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

According to various embodiments, each of the components (e.g., modules or programs) described above may include a single or multiple entities. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., modules or programs) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the components of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. According to various embodiments, the operations performed by the modules, programs or other components may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims (10)

In a head mounted electronic device (300),
At least one camera (320);
At least one display (330);
A memory (340) configured to store instructions; and
Contains a processor (350),
The above processor, when the above instructions are executed, the electronic device (300),
Acquiring a first background image through at least one camera,
Generate the above first background image as a background blur image,
Check display information for a user interface to be displayed on the first background image, and the display information includes display position information and size information of the user interface,
Based on the above display information, a first part of the background blur image to be composited with the user interface is obtained,
Compositely displaying the first background image, the first part of the acquired background blur image, and the user interface on at least one display,
As the display information of the above user interface is changed, a second part of the background blur image to be synthesized with the user interface is obtained based on the changed display information,
A second background image, a second part of the acquired background blur image, and the user interface are set to be synthesized and displayed on at least one display,
An electronic device wherein the first background image and the second background image are the same or different. In claim 1,
One or more sensors (310) configured to obtain detailed information of the electronic device (300) are further included.
The above processor (350) causes the electronic device (300) to, when the above instructions are executed,
If you confirm that the detailed information of the above electronic device has changed,
An electronic device set to acquire the second background image corresponding to the changed posture information through at least one camera (320). In claim 1,
The above processor (350) causes the electronic device (300) to, when the above instructions are executed,
Check whether one or more windows included in the above user interface are subject to blur processing,
If there is a window identified as a target for the above blur processing, the blur option data is determined based on the rendering environment information,
An electronic device configured to perform blur processing on the first background image based on the determined blur option data to obtain the background blurred image. In claim 3,
The above rendering environment information is:
An electronic device comprising at least one of the brightness of the user interface, depth data of the first background image, a size of a graphics buffer, performance of the processor, or a size of the at least one display. In claim 1,
The above processor (350) causes the electronic device (300) to, when the above instructions are executed,
Calculating a background area for one or more windows included in the user interface based on the above display information,
An electronic device configured to obtain a first portion of the background blur image by cropping a partial image corresponding to the calculated background area from the background blur image. In claim 5,
The above processor (350) causes the electronic device (300) to, when the above instructions are executed,
An electronic device configured to calculate the background area based on at least one of the size, position, or rotation angle of each of the one or more windows. In claim 1,
The above processor (350) causes the electronic device (300) to, when the above instructions are executed,
When it is confirmed that the display information of a first window among one or more windows included in the user interface has changed, a changed background area for the first window is calculated based on the changed display information,
A third part of the background blur image is obtained by cropping a partial image corresponding to the changed background area from the background blur image,
An electronic device set to display a composite image of the first background image, a third portion of the background blur image, and the user interface. In claim 2,
The above processor (350) causes the electronic device (300) to, when the above instructions are executed,
When the second background image is obtained, the blur option data for the second background image is checked,
An electronic device that generates a background blur image corresponding to the changed second background image based on the above-mentioned confirmed blur option data. In claim 1,
The above processor (350) causes the electronic device (300) to, when the above instructions are executed,
Verifying depth data for at least one object included in the first background image,
An electronic device configured to generate the background blur image by differentially applying a blur intensity to each area corresponding to at least one object in the first background image based on the verified depth data. In claim 1,
One or more of the above windows,
An electronic device comprising at least one of an execution screen, multimedia data, text, or icon of an application running on the electronic device.

Images