KR20240100318A - Audio Signal - Based Device And Control Method Thereof - Google Patents

Abstract

The present invention relates to a control method of an electronic device including a microphone, a memory, a control unit, and a driving unit. the control method of an electronic device includes the steps of: receiving an audio signal from a user; determining whether the input audio signal is a whistle sound; analyzing characteristics of the whistle sound when it is determined that the audio signal is the whistle sound; and controlling the operation of the electronic device using the analysis result.

Description

Audio signal based device and control method thereof {Audio Signal - Based Device And Control Method Thereof}

The present invention relates to a device using an audio signal such as a whistle and a method of controlling the same.

Recently, artificial intelligence technologies such as deep learning are being applied to process audio. Audio identification technology, one of the processing technologies related to audio, is developed for the purpose of detecting from which subject the audio input is generated and under what circumstances the audio input is generated.

As audio identification technology continues to develop, research on application technologies using it is also actively underway. Specifically, various methods for controlling electronic devices based on user utterances are emerging. Examples of commercialized voice recognition technology include Siri and S Voice.

On the other hand, methods of controlling electronic devices using non-verbal audio that can be generated by humans have limited application. For example, simple methods such as turning lights on and off in response to applause are used.

Unlike verbal audio, nonverbal audio such as applause has some limitations in containing meaning, making it difficult to control electronic devices for various purposes.

Meanwhile, a whistling sound that can be easily generated by multiple users may have various features such as pitch, sound change pattern, and melody. Therefore, if audio identification technology is applied to the whistling sound generated by the user, it becomes possible to control electronic devices with a simple whistling sound without the user speaking.

In the present invention, we would like to propose a method of extracting operation commands of an electronic device from a whistle sound generated by a user using audio identification technology and controlling the electronic device based on the extracted operation command.

The technical object of the present invention is to provide an electronic device that operates based on non-verbal audio rather than voice.

The technical problem of the present invention is to provide a control method that can operate electronic devices based on the sound of a whistle.

The technical object of the present invention is to provide a control method that analyzes the pitch, melody, and change pattern of a whistle sound and operates an electronic device based on the analysis results.

The present invention relates to a control method of an electronic device including a microphone, a memory, a control unit, and a driver, comprising: receiving an audio signal from a user; determining whether the input audio signal is a whistle sound; and When it is determined that the signal is a whistle sound, the method includes analyzing the characteristics of the whistle sound and controlling the operation of the electronic device using the analysis result.

According to the present invention, the user can operate the electronic device simply by making a whistling sound, thereby improving the user's convenience.

According to the present invention, there is an advantage in that the limitation of the trigger of audio-based electronic device control technology being limited to the user's speech can be solved by using non-verbal signals.

Figure 1 is a conceptual diagram showing the components of an electronic device that is the target of the control method proposed in the present invention.
Figure 2 is a flowchart showing a method for controlling an electronic device according to an embodiment of the present invention.
3A and 3B are block diagrams showing components of a refrigerator to which the electronic device control method according to the present invention is applied.
Figure 4 is a conceptual diagram showing a method of controlling the operation of a refrigerator using a whistle sound.
Figure 5 is a block diagram showing the components of an AI speaker to which the electronic device control method according to the present invention is applied.
Figure 6 is a conceptual diagram showing a method of controlling the operation of an AI speaker using a whistle sound.

Hereinafter, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the drawings and examples disclosed below. In addition, in order to clearly explain the present invention in the drawings, parts not related to the present invention are omitted, and identical or similar symbols in the drawings indicate identical or similar components.

The purpose and effect of the present invention can be naturally understood or become clearer through the following description, and if it is judged that it may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Therefore, the purpose of the present invention can be understood only by the following description. And the effect is not limited.

In Figure 1, the components of the electronic device 100 are explained.

The electronic device 100 described in this specification is defined as various devices that perform data communication, data processing, and perform certain operations, such as refrigerators, artificial intelligence speakers, vacuum cleaners, mobile terminals, washing machines, and dryers. It can be.

For example, when the electronic device 100 is a mobile terminal, the electronic device 100 may be a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, personal digital assistants (PDAs), or a portable PMP (Personal Digital Assistant). multimedia player, navigation, slate PC, tablet PC, ultrabook, wearable device (e.g., smartwatch), glass-type terminal (smart glass) , HMD (head mounted display), etc. may be included.

However, those skilled in the art will easily understand that, except for the case where the configuration according to the embodiment described in this specification is applicable only to mobile terminals, it can also be applied to fixed terminals such as digital TVs, desktop computers, digital signage, etc. will be.

The electronic device 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, and a power supply unit 190. ), etc. may be included.

More specifically, among the above components, the wireless communication unit 110 is used between the electronic device 100 and the wireless communication system, between the electronic device 100 and another electronic device 100, or between the electronic device 100 and an external server. It may include one or more modules that enable wireless communication between the devices. Additionally, the wireless communication unit 110 may include one or more modules that connect the electronic device 100 to one or more networks.

This wireless communication unit 110 may include at least one of a broadcast reception module 111, a mobile communication module 112, a wireless Internet module 113, a short-range communication module 114, and a location information module 115. .

The input unit 120 includes a camera 121 or an image input unit for inputting an image signal, a microphone 122 or an audio input unit for inputting an audio signal, and a user input unit 123 for receiving information from a user, for example. , touch keys, push keys (mechanical keys, etc.). Voice data or image data collected by the input unit 120 may be analyzed and processed as a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of information within the collection device, information on the surrounding environment surrounding the collection device, and user information. For example, the sensing unit 140 includes a proximity sensor (141), an illumination sensor (142), a touch sensor, an acceleration sensor, a magnetic sensor, and a gravity sensor. Sensor (G-sensor), gyroscope sensor, motion sensor, RGB sensor, IR sensor (infrared sensor), fingerprint scan sensor, ultrasonic sensor , optical sensors (e.g., cameras (see 121)), microphones (see 122), battery gauges, environmental sensors (e.g., barometers, hygrometers, thermometers, radiation detection sensors, It may include at least one of a heat detection sensor, a gas detection sensor, etc.) and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.).

The output unit 150 is for generating output related to vision, hearing, or tactile sense, and includes at least one of a display unit 151, an audio output unit 152, a haptip module 153, and an optical output unit 154. can do. The display unit 151 can implement a touch screen by forming a layered structure or being integrated with the touch sensor. This touch screen functions as a user input unit 123 that provides an input interface between the electronic device 100 and the user, and can simultaneously provide an output interface between the electronic device 100 and the user.

The interface unit 160 serves as a passageway for various types of external devices connected to the electronic device 100. This interface unit 160 connects devices equipped with a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and an identification module. It may include at least one of a port, an audio input/output (I/O) port, a video input/output (I/O) port, and an earphone port. In response to the external device being connected to the interface unit 160, the electronic device 100 may perform appropriate control related to the connected external device.

Additionally, the memory 170 stores data supporting various functions of the electronic device 100. The memory 170 may store a plurality of application programs (application programs or applications) running on the electronic device 100, data for operating the electronic device 100, and commands. At least some of these applications may be downloaded from an external server via wireless communication. Additionally, at least some of these applications may be present on the electronic device 100 from the time of shipment for basic functions of the electronic device 100 (e.g., incoming and outgoing calls, receiving and sending functions). Meanwhile, the application program may be stored in the memory 170, installed on the electronic device 100, and driven by the control unit 180 to perform the operation (or function) of the audio analysis information collection device.

In addition to operations related to the application program, the control unit 180 typically controls the overall operation of the electronic device 100. The control unit 180 can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above, or by running an application program stored in the memory 170.

Additionally, the control unit 180 may control at least some of the components examined with FIG. 1 in order to run an application program stored in the memory 170. Furthermore, the control unit 180 may operate at least two of the components included in the electronic device 100 in combination with each other in order to run the application program.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 and supplies power to each component included in the electronic device 100. This power supply unit 190 includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the above components may cooperate with each other to implement the operation, control, or control method of the audio analysis information collection device according to various embodiments described below. Additionally, the operation, control, or control method of the audio analysis information collection device may be implemented on the audio analysis information collection device by running at least one application program stored in the memory 170.

Below, before looking at various embodiments implemented through the electronic device 100 discussed above, the components listed above will be looked at in more detail with reference to FIG. 1.

First, looking at the wireless communication unit 110, the broadcast reception module 111 of the wireless communication unit 110 receives broadcast signals and/or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channels may include satellite channels and terrestrial channels. Two or more broadcast reception modules may be provided in the electronic device 100 for simultaneous broadcast reception of at least two broadcast channels or broadcast channel switching.

The mobile communication module 112 supports technical standards or communication methods for mobile communication (e.g., Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (CDMA2000), EV -DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced, etc.), wireless signals are transmitted and received with at least one of a base station, an external terminal, and a server on a mobile communication network.

The wireless signal may include various types of data based on voice call signals, video call signals, or text/multimedia message transmission and reception.

The wireless Internet module 113 refers to a module for wireless Internet access and may be built into or external to the electronic device 100. The wireless Internet module 113 is configured to transmit and receive wireless signals in a communication network based on wireless Internet technologies.

Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), and WiMAX (Worldwide). Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc., and the wireless Internet module ( 113) transmits and receives data according to at least one wireless Internet technology, including Internet technologies not listed above.

From the perspective that wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A, etc. is achieved through a mobile communication network, the wireless Internet module 113 performs wireless Internet access through the mobile communication network. ) may be understood as a type of the mobile communication module 112.

The short-range communication module 114 is for short-range communication, including Bluetooth®, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Short-distance communication can be supported using at least one of NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology. This short-range communication module 114 is between the electronic device 100 and a wireless communication system, between the electronic device 100 and another electronic device 100, or between the electronic device 100 through wireless area networks. ) can support wireless communication between a network where another terminal (or external server) is located. The short-range wireless communication networks may be wireless personal area networks.

Here, the other terminal is a wearable device capable of exchanging data with (or interoperating with) the electronic device 100 according to the present invention, for example, a smartwatch, smart glass. , it can be a head mounted display (HMD). The short-range communication module 114 may detect (or recognize) a wearable device capable of communicating with the electronic device 100 around the electronic device 100 . Furthermore, if the detected wearable device is a device authenticated to communicate with the electronic device 100 according to the present invention, the control unit 180 sends at least a portion of the data processed by the electronic device 100 to the short-range communication module ( 114) can be transmitted to a wearable device. Accordingly, a user of a wearable device can use data processed in the electronic device 100 through the wearable device. For example, according to this, when a call is received on the electronic device 100, the user may make a phone call through a wearable device, or when a message is received on the electronic device 100, the user may make a phone call through the wearable device. It is possible to check the message.

The location information module 115 is a module for acquiring the location (or current location) of a mobile terminal, and representative examples thereof include a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, if the audio analysis information collection device utilizes a GPS module, the location of the audio analysis information collection device can be acquired using signals sent from GPS satellites. As another example, when the audio analysis information collection device utilizes a Wi-Fi module, the location of the audio analysis information collection device is based on information from the Wi-Fi module and the wireless AP (Wireless Access Point) that transmits or receives wireless signals. can be obtained. If necessary, the location information module 115 may replace or additionally perform any of the functions of other modules of the wireless communication unit 110 to obtain data regarding the location of the audio analysis information collection device. The location information module 115 is a module used to obtain the location (or current location) of the audio analysis information collection device, and is not limited to a module that directly calculates or obtains the location of the audio analysis information collection device.

Next, the input unit 120 is for inputting video information (or signal), audio information (or signal), data, or information input from the user. For input of video information, the electronic device 100 is one. Alternatively, a plurality of cameras 121 may be provided. The camera 121 processes image frames such as still images or moving images obtained by an image sensor in video call mode or shooting mode. The processed image frame may be displayed on the display unit 151 or stored in the memory 170. Meanwhile, the plurality of cameras 121 provided in the electronic device 100 may be arranged to form a matrix structure, and through the cameras 121 forming the matrix structure, the electronic device 100 can be viewed at various angles or focuses. A plurality of image information may be input. Additionally, the plurality of cameras 121 may be arranged in a stereo structure to acquire left and right images for implementing a three-dimensional image.

The microphone 122 processes external acoustic signals into electrical voice data. The processed voice data can be used in various ways depending on the function (or application program being executed) being performed by the electronic device 100. Meanwhile, various noise removal algorithms may be implemented in the microphone 122 to remove noise generated in the process of receiving an external acoustic signal.

The user input unit 123 is for receiving information from the user. When information is input through the user input unit 123, the control unit 180 can control the operation of the electronic device 100 to correspond to the input information. . This user input unit 123 is a mechanical input means (or mechanical key, for example, a button located on the front, rear, or side of the electronic device 100, a dome switch, a jog wheel, a jog switches, etc.) and touch-type input means. As an example, the touch input means consists of a virtual key, soft key, or visual key displayed on the touch screen through software processing, or a part other than the touch screen. It can be done with a touch key placed in . Meanwhile, the virtual key or visual key can be displayed on the touch screen in various forms, for example, graphic, text, icon, video or these. It can be made up of a combination of .

Meanwhile, the sensing unit 140 senses at least one of information within the collection device, information on the surrounding environment surrounding the collection device, and user information, and generates a sensing signal corresponding thereto. Based on these sensing signals, the control unit 180 may control the driving or operation of the electronic device 100 or perform data processing, functions, or operations related to an application installed on the electronic device 100. Representative sensors among the various sensors that may be included in the sensing unit 140 will be looked at in more detail.

First, the proximity sensor 141 refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing nearby without mechanical contact using the power of an electromagnetic field or infrared rays. This proximity sensor 141 may be placed in the inner area of the collection device surrounded by the touch screen as seen above or near the touch screen.

Examples of the proximity sensor 141 include a transmissive photoelectric sensor, a direct reflection photoelectric sensor, a mirror reflection photoelectric sensor, a high-frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, the proximity sensor 141 may be configured to detect the proximity of a conductive object by changing the electric field according to the proximity of the object. In this case, the touch screen (or touch sensor) itself can be classified as a proximity sensor.

Meanwhile, for convenience of explanation, the act of approaching an object on the touch screen without touching it so that the object is recognized as being located on the touch screen is called "proximity touch," and the touch The act of actually touching an object on the screen is called “contact touch.” The position at which an object is close-touched on the touch screen means the position at which the object corresponds perpendicularly to the touch screen when the object is close-touched. The proximity sensor 141 can detect proximity touch and proximity touch patterns (e.g., proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch position, proximity touch movement state, etc.). there is. Meanwhile, the control unit 180 processes data (or information) corresponding to the proximity touch operation and proximity touch pattern detected through the proximity sensor 141, as described above, and further provides visual information corresponding to the processed data. It can be output on the touch screen. Furthermore, the control unit 180 may control the electronic device 100 to process different operations or data (or information) depending on whether a touch to the same point on the touch screen is a proximity touch or a contact touch. .

The touch sensor detects a touch (or touch input) applied to the touch screen (or display unit 151) using at least one of several touch methods such as resistive method, capacitive method, infrared method, ultrasonic method, and magnetic field method. sense

As an example, a touch sensor may be configured to convert changes in pressure applied to a specific part of the touch screen or capacitance occurring in a specific part into an electrical input signal. The touch sensor may be configured to detect the position, area, pressure at the time of touch, capacitance at the time of the touch, etc., at which the touch object that touches the touch screen is touched on the touch sensor. Here, the touch object is an object that applies a touch to the touch sensor, and may be, for example, a finger, a touch pen or stylus pen, or a pointer.

In this way, when there is a touch input to the touch sensor, the corresponding signal(s) are sent to the touch controller. The touch controller processes the signal(s) and then transmits the corresponding data to the control unit 180. As a result, the control unit 180 can determine which area of the display unit 151 has been touched. Here, the touch controller may be a separate component from the control unit 180 or may be the control unit 180 itself.

Meanwhile, the control unit 180 may perform different controls or the same control depending on the type of touch object that touches the touch screen (or touch keys provided other than the touch screen). Whether to perform different controls or the same controls depending on the type of touch object may be determined depending on the current operating state of the electronic device 100 or the application program being executed.

Meanwhile, the touch sensor and proximity sensor discussed above are used independently or in combination to provide short (or tap) touch, long touch, multi touch, and drag touch on the touch screen. ), flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, etc. Touch can be sensed.

An ultrasonic sensor can recognize the location information of a sensing object using ultrasonic waves. Meanwhile, the control unit 180 is capable of calculating the location of the wave source through information sensed from an optical sensor and a plurality of ultrasonic sensors. The location of the wave source can be calculated using the property that light is much faster than ultrasonic waves, that is, the time for light to reach the optical sensor is much faster than the time for ultrasonic waves to reach the ultrasonic sensor. More specifically, the location of the wave source can be calculated using the time difference between the arrival time of ultrasonic waves using light as a reference signal.

Meanwhile, looking at the configuration of the input unit 120, the camera 121 includes at least one of a camera sensor (eg, CCD, CMOS, etc.), a photo sensor (or image sensor), and a laser sensor.

The camera 121 and the laser sensor can be combined with each other to detect the touch of the sensing object for a 3D stereoscopic image. A photo sensor can be laminated on the display element, and this photo sensor is configured to scan the movement of a detection object close to the touch screen. More specifically, the photo sensor mounts photo diodes and TR (transistors) in rows/columns and scans the contents placed on the photo sensor using electrical signals that change depending on the amount of light applied to the photo diode. In other words, the photo sensor calculates the coordinates of the sensing object according to the amount of change in light, and through this, location information of the sensing object can be obtained.

The display unit 151 displays (outputs) information processed by the electronic device 100. For example, the display unit 151 may display execution screen information of an application running on the electronic device 100, or UI (User Interface) and GUI (Graphic User Interface) information according to such execution screen information. .

Additionally, the display unit 151 may be configured as a three-dimensional display unit that displays a three-dimensional image.

A three-dimensional display method such as a stereoscopic method (glasses method), an autostereoscopic method (glasses-free method), or a projection method (holographic method) may be applied to the three-dimensional display unit.

The audio output unit 152 may output audio data received from the wireless communication unit 110 or stored in the memory 170 in call signal reception, call mode or recording mode, voice recognition mode, broadcast reception mode, etc. The sound output unit 152 also outputs sound signals related to functions performed in the electronic device 100 (eg, call signal reception sound, message reception sound, etc.). This sound output unit 152 may include a receiver, speaker, buzzer, etc.

The haptic module 153 generates various tactile effects that the user can feel. A representative example of a tactile effect generated by the haptic module 153 may be vibration. The intensity and pattern of vibration generated by the haptic module 153 can be controlled by the user's selection or settings of the control unit. For example, the haptic module 153 may synthesize and output different vibrations or output them sequentially.

In addition to vibration, the haptic module 153 responds to stimulation such as pin arrays moving perpendicular to the contact skin surface, blowing force or suction force of air through a nozzle or inlet, grazing the skin surface, contact with electrodes, and electrostatic force. A variety of tactile effects can be generated, such as effects by reproducing hot and cold sensations using elements that can absorb heat or heat.

The haptic module 153 can not only deliver a tactile effect through direct contact, but can also be implemented so that the user can feel the tactile effect through muscle senses such as fingers or arms. Two or more haptic modules 153 may be provided depending on the configuration of the electronic device 100.

The optical output unit 154 uses light from the light source of the electronic device 100 to output a signal to notify that an event has occurred. Examples of events that occur in the electronic device 100 may include receiving a message, receiving a call signal, a missed call, an alarm, a schedule notification, receiving an email, receiving information through an application, etc.

The signal output by the optical output unit 154 is realized as the collection device emits light of a single color or multiple colors to the front or back. The signal output may be terminated when the collection device detects the user's confirmation of the event.

The interface unit 160 serves as a passageway for all external devices connected to the electronic device 100. The interface unit 160 receives data from an external device, receives power and transmits it to each component inside the electronic device 100, or transmits data inside the electronic device 100 to an external device. For example, wired/wireless headset port, external charger port, wired/wireless data port, memory card port, port for connecting a device equipped with an identification module. A port, an audio input/output (I/O) port, a video input/output (I/O) port, an earphone port, etc. may be included in the interface unit 160.

Meanwhile, the identification module is a chip that stores various information for authenticating the right to use the electronic device 100, including a user identification module (UIM), subscriber identity module (SIM), and general user authentication. It may include a module (universal subscriber identity module; USIM), etc. A device equipped with an identification module (hereinafter referred to as an 'identification device') may be manufactured in a smart card format. Therefore, the identification device can be connected to the terminal 100 through the interface unit 160.

In addition, the interface unit 160 serves as a path through which power from the cradle is supplied to the electronic device 100 when the electronic device 100 is connected to an external cradle, or the interface unit 160 serves as a path for supplying power from the cradle to the electronic device 100 by the user. It can be a passage through which various command signals are transmitted to the electronic device 100. Various command signals or the power input from the cradle may be operated as signals for recognizing that the electronic device 100 is correctly mounted on the cradle.

The memory 170 can store programs for operating the control unit 180, and can also temporarily store input/output data (eg, phone book, messages, still images, videos, etc.). The memory 170 may store data on various patterns of vibration and sound output when a touch is input on the touch screen.

The memory 170 is a flash memory type, hard disk type, solid state disk type, SDD type (Silicon Disk Drive type), and multimedia card micro type. ), card-type memory (e.g. SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), EEPROM (electrically erasable programmable read) -only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk may include at least one type of storage medium. The electronic device 100 may be operated in connection with web storage that performs a storage function of the memory 170 on the Internet.

Meanwhile, as discussed above, the control unit 180 controls operations related to application programs and generally controls the overall operation of the electronic device 100. For example, if the state of the collection device satisfies a set condition, the control unit 180 may execute or release a lock state that restricts the user's input of control commands to applications.

In addition, the control unit 180 performs control and processing related to voice calls, data communications, video calls, etc., or performs pattern recognition processing to recognize handwriting input or drawing input made on the touch screen as text and images, respectively. You can. Furthermore, the control unit 180 may control any one or a combination of the components described above in order to implement various embodiments described below on the electronic device 100 according to the present invention.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 and supplies power necessary for the operation of each component. The power supply unit 190 includes a battery, and the battery can be a built-in battery that can be recharged, and can be detachably coupled to the terminal body for charging, etc.

Additionally, the power supply unit 190 may be provided with a connection port, and the connection port may be configured as an example of the interface 160 to which an external charger that supplies power for charging the battery is electrically connected.

As another example, the power supply unit 190 may be configured to charge the battery wirelessly without using the connection port. In this case, the power supply unit 190 uses one or more of the inductive coupling method based on magnetic induction phenomenon or the magnetic resonance coupling method based on electromagnetic resonance phenomenon from an external wireless power transmitter. Power can be transmitted.

Meanwhile, various embodiments below may be implemented in a recording medium readable by a computer or similar device, for example, using software, hardware, or a combination thereof.

The location information module 115 provided in the collection device is used to detect, calculate, or identify the location of the collection device, and representative examples may include a Global Position System (GPS) module and a Wireless Fidelity (WiFi) module. If necessary, the location information module 115 may replace or additionally perform any of the functions of other modules of the wireless communication unit 110 to obtain data regarding the location of the collection device.

The GPS module 115 calculates distance information and accurate time information from three or more satellites and then applies trigonometry to the calculated information to accurately calculate three-dimensional current location information according to latitude, longitude, and altitude. can do. Currently, a method of calculating location and time information using three satellites and correcting errors in the calculated location and time information using another satellite is widely used. Additionally, the GPS module 115 can calculate speed information by continuously calculating the current location in real time. However, it is difficult to accurately measure the location of the collection device using a GPS module in shaded areas of satellite signals, such as indoors. Accordingly, WPS (WiFi Positioning System) can be used to compensate for GPS-based positioning.

The WiFi Positioning System (WPS) uses a WiFi module provided in the electronic device 100 and a wireless AP (Wireless Access Point) that transmits or receives wireless signals from the WiFi module. As a technology for tracking the location of a device, it refers to a location determination technology based on WLAN (Wireless Local Area Network) using WiFi.

The Wi-Fi location tracking system may include a Wi-Fi location determination server, an electronic device 100, a wireless AP connected to the electronic device 100, and a database storing arbitrary wireless AP information.

The electronic device 100 connected to the wireless AP may transmit a location information request message to the Wi-Fi location determination server.

The Wi-Fi location determination server extracts information on the wireless AP connected to the electronic device 100 based on the location information request message (or signal) of the electronic device 100. Information on the wireless AP connected to the electronic device 100 may be transmitted to the Wi-Fi location determination server through the electronic device 100, or may be transmitted from the wireless AP to the Wi-Fi location determination server.

Based on the location information request message of the electronic device 100, the extracted wireless AP information includes MAC Address, Service Set IDentification (SSID), Received Signal Strength Indicator (RSSI), Reference Signal Received Power (RSRP), and RSRQ ( Reference Signal Received Quality), channel information, privacy, network type, signal strength, and noise strength.

As described above, the Wi-Fi location determination server can receive information on the wireless AP connected to the electronic device 100 and extract wireless AP information corresponding to the wireless AP to which the collection device is connected from a pre-built database. At this time, information on any wireless APs stored in the database includes MAC Address, SSID, channel information, Privacy, Network Type, latitude and longitude coordinates of the wireless AP, name of the building where the wireless AP is located, number of floors, and detailed indoor location information (GPS coordinates). available), and may be information such as the AP owner’s address and phone number. At this time, in order to remove mobile APs or wireless APs provided using illegal MAC addresses from the positioning process, the Wi-Fi location determination server may extract only a certain number of wireless AP information in order of high RSSI.

Thereafter, the Wi-Fi location determination server may extract (or analyze) location information of the electronic device 100 using at least one wireless AP information extracted from the database. By comparing the included information with the received wireless AP information, the location information of the electronic device 100 is extracted (or analyzed).

As a method for extracting (or analyzing) the location information of the electronic device 100, the Cell-ID method, the fingerprint method, the triangulation method, and the landmark method can be used.

The Cell-ID method is a method of determining the location of the wireless AP with the strongest signal strength among the surrounding wireless AP information collected by the collection device as the location of the collection device. It has the advantage of being simple to implement, requiring no additional costs, and being able to obtain location information quickly, but has the disadvantage of lowering the positioning accuracy if the installation density of wireless APs is low.

The fingerprint method is a method of collecting signal strength information by selecting a reference location in the service area and estimating the location through signal strength information transmitted from a collection device based on the collected information. In order to use the fingerprint method, it is necessary to create a database of radio wave characteristics in advance.

The triangulation method is a method of calculating the location of a collection device based on the coordinates of at least three wireless APs and the distance between the collection devices. To measure the distance between a collection device and a wireless AP, the signal strength is converted into distance information, or the time when the wireless signal is transmitted (Time of Arrival, ToA), or the time difference between the signal transmission (Time Difference of Arrival, TDoA) , the angle at which the signal is transmitted (Angle of Arrival, AoA), etc. can be used.

The landmark method is a method of measuring the location of a collection device using a landmark transmitter whose location is known.

In addition to the methods listed, various algorithms can be used as a method to extract (or analyze) the location information of the collection device.

The location information of the electronic device 100 extracted in this way is transmitted to the electronic device 100 through the Wi-Fi location determination server, so that the electronic device 100 can obtain location information.

The electronic device 100 can obtain location information by connecting to at least one wireless AP. At this time, the number of wireless APs required to obtain location information of the electronic device 100 may vary depending on the wireless communication environment in which the electronic device 100 is located.

As previously seen in Figure 1, the collection device according to the present invention includes Bluetooth (Bluetooth??), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, and NFC ( Near field communication (Near Field Communication) and Wireless USB (Wireless Universal Serial Bus) may be applied.

Among these, the NFC module provided in the collection device supports non-contact, short-range wireless communication between devices at a distance of about 10 cm. The NFC module can operate in any of card mode, reader mode, and P2P mode. In order for the NFC module to operate in card mode, the electronic device 100 may further include a security module that stores card information. Here, the security module may be a physical medium such as a Universal Integrated Circuit Card (UICC) (e.g., Subscriber Identification Module (SIM) or Universal SIM (USIM)), Secure micro SD, and a sticker, or it may be a logical medium embedded in the collection device ( For example, it may be an embedded SE (Secure element). Data exchange based on SWP (Single Wire Protocol) can be performed between the NFC module and the security module.

When the NFC module is operated in card mode, the collection device can transmit the stored card information to the outside like a traditional IC card. Specifically, by bringing the collection device that stores the card information of payment cards such as credit cards or bus cards close to the fare payment machine, mobile short-distance payments can be processed, and the collection device that stores the card information of the access card can be used to authorize access. Upon approaching the flag, the access approval process can begin. Cards such as credit cards, transportation cards, and access cards are mounted on the security module in the form of an applet, and the security module can store card information for the mounted cards. Here, the card information of the payment card may be at least one of the card number, balance, and usage history, and the card information of the access card may be at least one of the user's name, number (e.g., user's student number or employee number), and access history. It could be one.

When the NFC module is operated in reader mode, the collection device can read data from an external tag. At this time, the data that the collection device receives from the tag may be coded in the NFC Data Exchange Format defined by the NFC Forum. In addition, the NFC Forum defines four record types. Specifically, the NFC Forum defines four Record Type Definitions (RTDs): Smart Poster, Text, Uniform Resource Identifier (URI), and General Control. If the data received from the tag is a smart poster type, the control unit can run a browser (eg, an Internet browser), and if the data received from the tag is a text type, the control unit can run a text viewer. If the data received from the tag is a URI type, the control unit can run a browser or make a phone call, and if the data received from the tag is a general control type, the control unit can execute an appropriate operation according to the control contents.

When the NFC module is operated in P2P (Peer-to-Peer) mode, the collection device can perform P2P communication with other collection devices. At this time, LLCP (Logical Link Control Protocol) may be applied to P2P communication. For P2P communication, a connection can be created between a collection device and another collection device. At this time, the created connection can be divided into a connectionless mode, which terminates after exchanging one packet, and a connection-oriented mode, which continuously exchanges packets. Through P2P communication, data such as electronic business cards, contact information, digital photos, URLs, and setup parameters for Bluetooth and Wi-Fi connections can be exchanged. However, since the available distance of NFC communication is short, P2P mode can be effectively used to exchange small data.

Hereinafter, embodiments related to the control method that can be implemented in the collection device configured as described above will be discussed with reference to the attached drawings. It is obvious to those skilled in the art that the present invention can be embodied in other specific forms without departing from the spirit and essential features of the present invention.

Meanwhile, the control unit 180 of the electronic device 100 according to the present invention may be equipped with an artificial intelligence engine that performs audio recognition.

Specifically, the artificial intelligence engine mounted on the control unit 180 may be configured to identify which audio event the nonverbal audio input corresponds to.

In one example, the artificial intelligence engine may perform self-learning using training data labeled as a whistle sound and training data labeled as a non-whistle sound. In other words, the artificial intelligence engine builds an artificial neural network, receives training data labeled as to whether or not it is a whistle sound, and uses the input training data to use machine learning or deep learning to identify whether the unknown input audio signal is a whistle sound. Running is possible.

In another example, the artificial intelligence engine can identify the label of the audio signal when an audio signal is input, and determine at least one of the height, intensity, and duration of at least a portion of the audio signal identified by the specific label. It can be determined.

Hereinafter, in FIG. 2, a method of controlling the above-described electronic device 100 with an audio signal will be described.

First, as shown in FIG. 2, the sensing unit 140 or microphone of the electronic device 100 can receive an audio signal from the user (S201).

The control unit 180 may determine whether the input audio signal is a whistle sound (S202).

Specifically, the control unit 180 can determine whether the input audio signal is a whistle sound using a pre-trained artificial intelligence model. At this time, the artificial intelligence model may be configured to identify information related to the subject that generated the input audio signal.

For example, an artificial intelligence model can be learned using training data labeled for each of a plurality of classes, and calculate the probability that input audio corresponds to each of the plurality of classes. In this case, the plurality of classes may include whistle sounds.

In another example, an artificial intelligence model may be trained by training data labeled as a whistle sound and training data labeled as a non-whistle sound to determine whether the input audio is a whistle sound.

In addition, if the control unit 180 determines that the audio signal is a whistle sound, it can analyze the characteristics of the whistle sound (S203).

Here, the characteristics of the whistle sound may be related to at least one of the height, size, and duration of the whistle sound.

Specifically, the control unit 180 may detect information related to changes in the whistle sound. That is, the control unit 180 may detect whether the pitch of the whistling sound changes from a low tone to a high tone, and conversely, it may detect whether the pitch of the whistling sound changes from a high tone to a low tone.

Additionally, the control unit 180 can detect whether the whistle sound is of a specific height. Furthermore, the control unit 180 can also detect whether the whistle sound corresponds to a specific melody.

In one embodiment, when performing the step (S203) of analyzing the whistle sound, the control unit 180 determines at least one of the height, intensity, and duration of the whistle sound, and based on the determination result, the electronic device ( The process of generating an operation command corresponding to 100) can be performed.

More specifically, when performing the step of analyzing the whistle sound (S203), the control unit 180 detects a change in the height of the whistle sound and generates an operation command corresponding to the electronic device based on the detected height change. The process can be performed.

Afterwards, the control unit 180 can control the operation of the electronic device 100 using the analysis results (S204).

Specifically, when performing the step (S204) of controlling the operation of the electronic device 100, when it is detected that the height of the whistle sound increases, the control unit 180 operates a driving unit to perform the first operation of the electronic device. A process of controlling, and when it is detected that the height of the whistle sound decreases, a process of controlling the driving unit to perform a second operation of the electronic device, and when it is detected that the height of the whistle sound is maintained, a second operation of the electronic device. 3 The process of controlling the driving part to perform the operation can be performed.

For example, if the electronic device 100 is a refrigerator, the control unit 180 may control the door driver to open the door of the refrigerator when it is analyzed that the whistle sound has changed from a low sound to a high sound.

In one embodiment, the audio-based electronic device control method proposed in the present invention can be applied to a refrigerator.

At this time, the refrigerator may include a microphone for receiving an audio signal, a control unit that determines whether the input audio signal is a whistle, and a driving unit that operates according to the whistle sound. Hereinafter, for convenience of explanation, the driving unit is defined as including a door opening and closing device for opening and closing the door of the refrigerator, and a display that outputs information related to the operation of the refrigerator. In order to open or close a refrigerator door, a door opening and closing device may include a motor that provides driving force to a hinge portion of the door, and a guide that defines the direction of the driving force.

For example, when the controller determines that the whistle sound changes from a low sound to a high sound, it can control the door opening and closing device to open the door of the refrigerator. Conversely, the controller may control the door opening and closing device to close the door when the whistle sound changes from a high sound to a low sound.

In another example, the controller may change the output screen of the refrigerator display when the pitch of the whistle is maintained.

In another embodiment, the audio-based electronic device control method proposed in the present invention can be applied to Internet of Things devices such as artificial intelligence speakers.

In general, an artificial intelligence speaker communicates with a plurality of devices present in an installed location and plays a role in controlling the operation of the plurality of devices. The artificial intelligence speaker may include a microphone that receives an audio signal, a control unit that processes the audio signal, a memory that stores information related to the audio signal, and a communication unit that transmits an operation command to the plurality of devices.

For example, when the whistle sound changes from a low tone to a high tone, the control unit of the artificial intelligence speaker may generate an operation command corresponding to the first device paired with the artificial intelligence speaker. Additionally, the control unit of the artificial intelligence speaker may generate an operation command corresponding to the second device paired with the artificial intelligence speaker when the whistle sound changes from a high tone to a low tone. In this way, the control unit of the artificial intelligence speaker can generate operation commands for different devices according to changes in the whistle sound, and can control the operation of each device based on the generated operation commands.

In another example, the control unit of an artificial intelligence speaker may generate different operation commands for one device according to changes in the whistle sound.

Below, in Figure 3, the components of a refrigerator to which the electronic device control method according to the present invention is applied are explained.

Figure 3A is a control block diagram of one embodiment according to the present invention.

The present invention may include a control unit 300 that divides and manages one photo taken by the camera 70 into a plurality of images. At this time, the image may mean a portion of a photo that has been processed or corrected by the control unit 300.

The control unit 300 can command the camera 70 to take a picture and receive a picture taken by the camera 70.

Additionally, the control unit 300 may provide a portion of the divided image to the display 14 to provide the user with up-to-date information about food stored in the storage compartment 22. At this time, the display 14 can be installed on the front of the refrigerator, or it can be a separate device separated from the refrigerator. That is, it is possible for a user to receive an image about the storage room 22 and obtain information through an external communication terminal such as a mobile phone.

At this time, the control unit 300 can divide the photos taken by the camera 70 into each area, divide them into a plurality of independent images, and provide them to the display 14. At this time, the photo selected by the control unit 100 may include information about the most recent state of the storage room 22 when the user last accessed the storage room 22 and retrieved or received food from the storage room.

An embodiment of the present invention may include a door switch 310 that can detect whether the door 20 opens or closes the storage compartment 22. At this time, the door switch 310 is provided in the outer case 10, and when the door 20 contacts the outer case 10, it detects that the door 20 seals the storage compartment 22. can do. Additionally, if the door 20 does not contact the outer case 10, it can be detected that the door 20 is opening the storage compartment 22.

Additionally, an embodiment of the present invention may include a door sensor 320 that detects the rotation angle of the door 20. At this time, the door sensor 320 can detect the rotation direction of the door 20 and the rotation angle of the door 20. For example, when the door 20 is rotated more than a certain angle, a change occurs in the door sensor 320, so that the door sensor 320 can detect that the door 20 has been rotated more than a certain angle. there is. Additionally, when the door 20 is rotated in a specific direction, it is possible to detect the direction of rotation of the door 20 by detecting changes in pulses generated depending on the direction. Of course, the door sensor 320 may be modified into various forms other than those described above.

In addition, the door sensor 320 has a light emitting unit and a light receiving unit, so that it is possible to detect the rotation angle of the door 20 by determining whether the light emitted from the light emitting unit is transmitted to the light receiving unit.

In particular, the door sensor 320 can be driven only when the door switch 310 determines that the door 20 has opened the storage compartment 22.

Meanwhile, if the door sensor 320 determines that the door 20 has been rotated at a specific angle, it is also possible to take a picture with the camera 70. In this case, even if the door switch 310 detects the opening of the door 20, the camera 70 does not immediately start taking pictures, but takes pictures when the door sensor 320 reaches a certain angle. will be filmed.

Once the camera 70 starts taking pictures, it is also possible to continue taking pictures at predetermined time intervals. Of course, if the door sensor 320 determines that the door 20 is open by a certain angle, the camera 70 can continue to take pictures, not just one picture, until a command to stop taking pictures is provided. do.

The control unit 300 may include a drawer detection unit 330 that detects whether the drawer 50 is withdrawn or retracted.

At this time, the drawer detection unit 330 may mean a part that reads the image of a photo taken by the camera 70 within the control unit 300. The drawer detection unit 330 is capable of detecting not only the movement of the drawer 50 but also the direction of movement of the drawer 50.

In particular, the drawer detection unit 330 may be implemented by software. The drawer detection unit 330 can detect the location of the drawer 50 using information captured in a photo taken by the camera 70.

The drawer 50 is equipped with an identification mark called a marker, and the marker can be used to determine whether the drawer 50 has entered a specific section, stopped if entered, and changed the direction of movement after stopping through a photo taken with the marker. You can find information about it. The marker will be described in detail later.

Since the marker is displayed on the drawer, it has the same movement as the drawer 50. Therefore, it is possible to obtain various information about the movement of the drawer 50 using information from the photo taken by the camera 70, without using a separate sensor to detect the movement of the drawer.

Photos taken by the camera 70 may be stored in the storage unit 18. At this time, the storage unit 18 may be provided in a refrigerator or in a device separate from the refrigerator. At this time, the storage unit 18 may be placed in a refrigerator together with the display 14, or may be installed together with the display 14 in a device other than the refrigerator, a server connected to the refrigerator and a network, or a terminal connected to the refrigerator and the network. there is.

At this time, the storage unit 18 may not store all photos taken by the camera 70. For example, if a command to select a specific photo is not issued by the control unit 300, the storage unit 18 can delete the previously taken and stored photo and store the currently captured photo (First In , First Out). The storage unit 18 stores only some of the photos taken by the camera 70, thereby reducing storage capacity.

Figure 3(b) is a control block diagram of a modified form of one embodiment according to the present invention.

In FIG. 3(b), unlike FIG. 3(a), the drawer detection unit 330 does not use information obtained from a photo taken by the camera 70, but separately moves the drawer 50. It can be sensed.

For example, the drawer detection unit 330 may be formed in the form of a plurality of hall sensors. A plurality of hall sensors may be installed in the movement path of the drawer 50. When the drawer 50 is moved, a plurality of hall sensors can detect each change, thereby detecting the position and direction of movement of the drawer 50.

The drawer detection unit 330 detects the movement of the drawer, and when taking a photo with the camera 70, it is possible to determine the point in time when the interior of the drawer is included in the photo taken by the camera 70. . That is, the drawer detection unit 330 can detect a point in time when information about an item stored inside the drawer can be expressed in a photo taken by the camera.

The drawer detection unit 330 can detect the movement of the drawer and detect the point in time when the drawer is determined to be up to date after the user has used it. That is, the drawer detection unit 330 can detect the point in time when the photo taken by the camera includes the latest information that the user has finished using the drawer. In other words, it is possible to indirectly determine the completion point of product arrival and departure or the end point of product update.

Figure 4 is a conceptual diagram showing a method of controlling the operation of a refrigerator using a whistle sound. Referring to Figure 4, the user can control the operation of the refrigerator by blowing a whistle. At this time, the user can blow the whistle in various ways to control the operation of the refrigerator, and the control unit analyzes the way the user whistles, and the refrigerator performs a preset pattern of operation corresponding to the analysis result.

Various methods of blowing the whistle include changing the pitch of the whistle, varying the length of the whistle, varying the intensity of the whistle, and blowing to a specific melody. In detail, the method of changing the pitch may include increasing the pitch of the sound, decreasing the pitch of the sound, or keeping the pitch of the sound constant, and the method of varying the length of the whistle sound may be short. This may include making a long whistle sound repeatedly, or making a short or long whistle sound a set number of times. The method of varying the intensity of the whistle sound is to vary the sound volume (dB) depending on the intensity of the whistle. It may include a method of blowing with a specific melody, and the method of blowing with a specific melody may include a method of whistling according to the pitch of the note corresponding to the set melody. Additionally, users can whistle in more ways by combining the above methods.

The operation of the refrigerator corresponding to the way the user whistles may include opening or closing the door of the refrigerator, changing the external display output screen, controlling the temperature and humidity inside the refrigerator or freezer, freezing ice, etc. . In detail, the number of doors may vary depending on the structure of the refrigerator, which may include actions to open or close all doors or actions to open or close each door. If the refrigerator has an external display, the external display It may include an action to change the displayed information to other information you want to see or to retrieve specific information, and it may include an action to raise or lower the temperature or humidity of the freezer or refrigerator compartment of the refrigerator in stages or to change it to a specific level. In the case of a refrigerator with a built-in ice maker, the operation of freezing ice may be included. Additionally, in addition to the above operations, the refrigerator may be equipped to operate additional functions provided by the refrigerator.

As above, there may be various ways for the user to whistle, and there may be various operations of the refrigerator that correspond to them. These various methods of blowing the whistle and the operation of the refrigerator can be matched to perform a preset operation. In addition, even if it is not a preset operation, a method of matching the operation of the refrigerator desired by the user with a specific whistle method and additionally setting the operation of the refrigerator corresponding to the whistle method may be included, and only one operation can be used for the whistle method. A method in which a plurality of operations correspond to each other and proceed sequentially may also be included.

Figure 5 is a block diagram showing the components of an AI speaker to which the electronic device control method according to the present invention is applied.

The AI speaker 500 is widely used in TVs, projectors, mobile phones, smartphones, desktop computers, laptops, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation, tablet PCs, wearable devices, and set-top boxes (STBs). ), DMB receivers, radios, washing machines, refrigerators, desktop computers, digital signage, robots, vehicles, etc. can be implemented as fixed or movable devices.

Referring to FIG. 5, the AI speaker 500 includes a communication unit 510, an input unit 520, a learning processor 530, a sensing unit 540, an output unit 550, a memory 570, and a processor 580. may include.

The communication unit 510 may transmit and receive data with external devices such as other AI devices 500a to 500e or an AI server (not shown) using wired or wireless communication technology. For example, the communication unit 510 may transmit and receive sensor information, user input, learning models, and control signals with external devices.

At this time, the communication technologies used by the communication unit 510 include Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Long Term Evolution (LTE), 5G, Wireless LAN (WLAN), and Wireless-Fidelity (Wi-Fi). ), Bluetooth??, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

The input unit 520 can acquire various types of data.

At this time, the input unit 520 may include a camera for inputting video signals, a microphone for receiving audio signals, and a user input unit for receiving information from the user. Here, the camera or microphone may be treated as a sensor, and the signal obtained from the camera or microphone may be referred to as sensing data or sensor information.

The input unit 520 may acquire training data for model learning and input data to be used when obtaining an output using the learning model. The input unit 520 may acquire unprocessed input data, and in this case, the processor 580 or the learning processor 530 may extract input features by preprocessing the input data.

The learning processor 530 can train a model composed of an artificial neural network using training data. Here, the learned artificial neural network may be referred to as a learning model. A learning model can be used to infer a result value for new input data other than learning data, and the inferred value can be used as the basis for a decision to perform an operation.

At this time, the learning processor 530 may perform AI processing together with the learning processor (not shown) of the AI server.

At this time, the learning processor 530 may include memory integrated or implemented in the AI speaker 500. Alternatively, the learning processor 530 may be implemented using the memory 570, an external memory directly coupled to the AI speaker 500, or a memory maintained in an external device.

The sensing unit 540 may use various sensors to obtain at least one of internal information of the AI speaker 500, information about the surrounding environment of the AI speaker 500, and user information.

At this time, the sensors included in the sensing unit 540 include a proximity sensor, illuminance sensor, acceleration sensor, magnetic sensor, gyro sensor, inertial sensor, RGB sensor, IR sensor, fingerprint recognition sensor, ultrasonic sensor, light sensor, microphone, and lidar. , radar, etc.

The output unit 550 may generate output related to vision, hearing, or tactile sensation.

At this time, the output unit 550 may include a display unit that outputs visual information, a speaker that outputs auditory information, and a haptic module that outputs tactile information.

The memory 570 can store data supporting various functions of the AI speaker 500. For example, the memory 570 may store input data, learning data, learning models, learning history, etc. obtained from the input unit 520.

The processor 580 may determine at least one executable operation of the AI speaker 500 based on information determined or generated using a data analysis algorithm or a machine learning algorithm. Additionally, the processor 580 can control the components of the AI speaker 500 to perform the determined operation.

To this end, the processor 580 may request, retrieve, receive, or utilize data from the learning processor 530 or the memory 570, and may perform a predicted operation or an operation determined to be desirable among the at least one executable operation. Components of the AI speaker 500 can be controlled to execute.

At this time, if linkage with an external device is necessary to perform the determined operation, the processor 580 may generate a control signal to control the external device and transmit the generated control signal to the external device.

The processor 580 may obtain intent information for user input and determine the user's request based on the obtained intent information.

At this time, the processor 580 uses at least one of a STT (Speech To Text) engine for converting voice input into a character string or a Natural Language Processing (NLP) engine for acquiring intent information of natural language, so that the user Intent information corresponding to the input can be obtained.

At this time, at least one of the STT engine or the NLP engine may be composed of at least a portion of an artificial neural network learned according to a machine learning algorithm. In addition, at least one of the STT engine or NLP engine may be learned by the learning processor 530, by a learning processor (not shown) of the AI server, or by distributed processing thereof. there is.

The processor 580 collects history information including the operation content of the AI speaker 500 or user feedback on the operation, and stores it in the memory 570 or the learning processor 530, or in an external device such as an AI server. Can be transmitted. The collected historical information can be used to update the learning model.

The processor 580 may control at least some of the components of the AI speaker 500 to run the application program stored in the memory 570. Furthermore, the processor 580 can operate two or more of the components included in the AI speaker 500 in combination with each other in order to run the application program.

Figure 6 is a conceptual diagram showing a method of controlling the operation of an AI speaker using a whistle sound. Referring to Figure 6, the user can control the operation of the AI speaker by blowing a whistle. At this time, the user can whistle in various ways to control the operation of the AI speaker, and the control unit analyzes the way the user whistles, and the AI speaker performs the operation of a preset pattern corresponding to the analysis result.

Various methods of blowing the whistle include changing the pitch of the whistle, varying the length of the whistle, varying the intensity of the whistle, and blowing to a specific melody. In detail, the method of changing the pitch may include increasing the pitch of the sound, decreasing the pitch of the sound, or keeping the pitch of the sound constant, and the method of varying the length of the whistle sound may be short. This may include making a long whistle sound repeatedly, or making a short or long whistle sound a certain number of times. The method of varying the intensity of the whistle sound is to vary the sound volume (dB) depending on the intensity of the whistle. It may include a method of blowing with a specific melody, and the method of blowing with a specific melody may include a method of whistling according to the pitch of the note corresponding to the set melody. Additionally, users can whistle in more ways by combining the above methods.

The operations of the AI speaker corresponding to the method of the user whistling include controlling the power of the AI speaker, controlling the search function, controlling the AI speaker when playing music, and controlling other devices linked to the AI speaker. It may include actions such as searching for a melody, etc. In detail, it may include turning on or off the power of the AI speaker, turning on or off the search function of the corresponding category to search for weather, fine dust, news, etc., or searching for weather or fine dust in a specific time or region. This may include operating to search, and controlling the AI speaker when playing music to play music, pause, play next song, play previous song, repeat one song, random play, set and release start and end points of section repeat, etc. In order to control other devices linked to the AI speaker, it may include turning on and off the power of the other device and performing functions provided by the other device. If there is a function to express the melody and search for music similar to the melody, an action to turn the function on and off can be included. Additionally, in addition to the above operations, additional functions provided by the AI speaker may also be provided to operate.

As shown above, there may be various ways for the user to whistle, and there may be various actions of the AI speaker corresponding to them. These various methods of blowing the whistle and the operation of the AI speaker can be matched to perform a preset operation. In addition, even if it is not a preset operation, a method of matching the operation of the AI speaker desired by the user with a specific whistle method and setting the operation of the AI speaker corresponding to the whistle method may be included, and only one action can be used for the whistle method. A method in which a plurality of operations correspond to each other and proceed sequentially may also be included.

The present invention described above relates to one embodiment, which is only an embodiment, and various modifications and equivalent other embodiments may be possible by those skilled in the art. Accordingly, the scope of the present invention is not limited by the above-described embodiments and the attached drawings.

Claims (5)

A microphone that receives an audio signal from the user;
a control unit that determines whether the audio signal input by the microphone is a whistle sound; and
It includes a driver that operates according to the characteristics of the audio signal,
The control unit,
An artificial neural network receives training data labeled as a whistle sound and training data labeled as a non-whistle sound, and uses the input training data to perform machine learning or deep learning to identify whether the audio signal is a whistle sound. Equipped with an artificial intelligence engine that builds
Calculate the probability that the audio signal corresponds to a whistle sound based on the learning results of the artificial intelligence, and determine whether the audio signal is a whistle sound based on the calculated probability,
If it is determined that the audio signal is a whistle sound, the characteristics of the whistle sound are analyzed,
According to the analysis results, the driving unit is operated in a preset pattern,
The driving unit,
An audio signal-based electronic device comprising a door opening and closing device for opening and closing the door of a refrigerator, and a display that outputs information related to the operation of the refrigerator.
According to paragraph 1,
The control unit,
Analyzing the characteristics of the whistle sound by determining at least one of the height, intensity, and duration of the whistle sound,
An audio signal-based electronic device, characterized in that it generates an operation command corresponding to the driving unit based on the analysis results.
According to paragraph 2,
The control unit,
An audio signal-based electronic device, characterized in that detecting a change in height of the whistle sound and generating a corresponding operation command for the electronic device based on the detected change in height.
According to paragraph 3,
The control unit,
When it is detected that the height of the whistle sound increases, controlling the driving unit to perform a first operation of the electronic device,
When it is detected that the height of the whistle sound decreases, controlling the driving unit to perform a second operation of the electronic device,
An audio signal-based electronic device, characterized in that when it is detected that the height of the whistle sound is maintained, the driving unit is controlled to perform a third operation of the electronic device.
According to paragraph 3,
The control unit,
An audio signal-based electronic device, characterized in that it determines whether the whistle sound corresponds to a preset melody and, based on the determination result, generates an operation command corresponding to the electronic device.