JP2012191445A - Mobile terminal device and control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve operability and convenience when a user operates a mobile terminal device in one hand.SOLUTION: The mobile terminal device having a touch panel display fulfills a function using a user's input operation on the touch panel display as a trigger. The mobile terminal device further comprises: a sensor which detects a user's holding state; a GUI arrangement change part which changes a graphical element of an operation button 3-2 to be displayed on the touch panel display based on a detection result of the sensor; and a display part which displays the changed operation button 3-2 on the touch panel display.

Description

  The present invention relates to a mobile terminal device and a control program that exhibit a function triggered by a user input operation on a touch panel display.

  Conventionally, a mobile phone provided with a touch panel display is known. In general, when a user uses a mobile phone, the mobile phone can be used with the other hand even when one hand cannot be used, such as when holding a mobile phone in one hand and holding an object in the other hand. There are many situations where you want to perform the operation. However, for many users, the mobile phone is larger than the palm, so when operating with one hand, it is difficult for the finger to reach a button far from the hand it holds, and the button different from the target button is pressed. There is a problem. Therefore, a graphical user interface (GUI) that allows easy input with one hand is required.

  For example, Patent Document 1 discloses a control method and apparatus for moving or enlarging / reducing a software keyboard when a user moves a mobile phone in the up / down, left / right, or rearward direction. In this mobile phone, for example, when the user holds the mobile phone in the right hand and moves the mobile phone to the left, the software keyboard moves to the right. Thus, the user can operate with the right hand while holding the mobile phone with the right hand.

  Further, in Patent Document 2, it is determined by using an infrared sensor which direction of the touch panel the hand performing the input operation has touched. Then, the portion shielded by the input operation hand is displayed in a non-shielded direction. This makes it easy for the user to hold the mobile phone with one hand and operate the software keyboard with the other hand.

JP 2010-282458 A JP 2010-287032 A

  However, the technique described in Patent Document 1 has a problem that when the mobile phone is moved, the software keyboard displayed on the touch panel display moves freely, and thus the keyboard moves against the user's intention. Further, when moving the software keyboard, the user must determine the amount of movement and the direction of movement with his / her own judgment and operation. For an unfamiliar user, it is not easy to move the software keyboard to a position where it can be easily operated.

  In the technique described in Patent Document 2, it is assumed that the mobile phone is held with one hand and the buttons on the software keyboard are operated with the other hand. However, when this cellular phone is operated with one hand, there is a problem that the operation button moves in a direction far from the gripping hand, which makes it difficult to operate.

  The present invention has been made in view of such circumstances, and provides a portable terminal device and a control program capable of improving operability and convenience when a user operates the portable terminal device with one hand. The purpose is to provide.

  (1) In order to achieve the above object, the present invention takes the following measures. That is, the mobile terminal device according to the present invention is a mobile terminal device that includes a touch panel display and exhibits a function triggered by a user input operation on the touch panel display, and includes a sensor that detects a gripping state of the user, A GUI arrangement changing unit that changes a graphic element of an operation button displayed on the touch panel display based on a detection result, and a display unit that displays the changed operation button on the touch panel display. And

  As described above, since the graphical element of the operation button displayed on the touch panel display is changed based on the detection result of the gripping state of the user, the operation button that is easier to operate is displayed when the user operates with one hand. Is possible. As a result, the convenience of the mobile terminal device can be improved.

  (2) In the mobile terminal device according to the present invention, the sensor may be one of a left hand grip left hand operation, a right hand grip right hand operation, a right hand grip left hand operation, a left hand grip right hand operation, and a two-hand grip two-hand operation as a grip state of the user. Is detected.

  With this configuration, it is possible to change the graphical elements of the operation buttons displayed on the touch panel display according to the gripping state of the user. As a result, the convenience of the mobile terminal device can be improved.

  (3) Further, in the mobile terminal device of the present invention, the GUI arrangement changing unit changes a size or a position of an operation button displayed on the touch panel display according to a gripping state of the user. .

  Thus, since the size or position of the operation button displayed on the touch panel display is changed according to the gripping state of the user, it is possible to display an operation button that is easier to operate when the user operates with one hand. It becomes possible. As a result, the convenience of the mobile terminal device can be improved.

  (4) Further, in the mobile terminal device of the present invention, the touch panel display accepts an input of a gripping state from a user, and the GUI arrangement detection unit is applied to the touch panel display according to the gripping state input from the user. The size or position of the operation button to be displayed is changed.

  Thus, since the size or position of the operation button displayed on the touch panel display is changed according to the gripping state input from the user, the size of the operation button is based on the gripping state according to the user's preference. Alternatively, the position can be changed. As a result, the convenience of the mobile terminal device can be improved.

  (5) In the mobile terminal device of the present invention, the GUI arrangement changing unit displays the information on the touch panel display based on information indicating a user's gripping tendency obtained from the gripping state of the user detected within a certain period. The size or position of the operation button is changed.

  As described above, since the size or position of the operation button displayed on the touch panel display is changed based on the information indicating the user's grip tendency obtained from the grip state of the user detected within the predetermined period, It is possible to display operation buttons in accordance with user operation trends. As a result, the convenience of the mobile terminal device can be improved.

  (6) Moreover, in the portable terminal device of the present invention, the GUI arrangement changing unit changes the size or position of an operation button displayed on the touch panel display based on a parameter set by a user. To do.

  As described above, since the size or position of the operation button displayed on the touch panel display is changed based on the parameter set by the user, it is possible to display the operation button according to the user's preference. As a result, the convenience of the mobile terminal device can be improved.

  (7) Moreover, in the portable terminal device of the present invention, when the orientation of the touch panel display is changed according to the use state of the user, the GUI arrangement changing unit uses a parameter corresponding to the detected gripping state of the user. Then, the size or position of the operation button displayed on the touch panel display is changed.

  As described above, when the orientation of the touch panel display is changed according to the use state of the user, the size or position of the operation button to be displayed on the touch panel display is determined using the parameter corresponding to the already detected gripping state of the user. Since the change is made, it becomes possible to display the operation buttons according to the screen orientation preferred by the user. As a result, the convenience of the mobile terminal device can be improved.

  (8) A control program according to the present invention is a control program for a portable terminal device that includes a touch panel display and exhibits a function triggered by a user input operation on the touch panel display. A process of detecting, a process of changing a graphical element of an operation button displayed on the touch panel display based on a detection result of the sensor, and a process of displaying the changed operation button on the touch panel display. A series of processing is executed by a computer.

  As described above, since the graphical element of the operation button displayed on the touch panel display is changed based on the detection result of the gripping state of the user, the operation button that is easier to operate is displayed when the user operates with one hand. Is possible. As a result, the convenience of the mobile terminal device can be improved.

  (9) Further, in the control program of the present invention, in the sensor, any one of left hand grip left hand operation, right hand grip right hand operation, right hand grip left hand operation, left hand grip right hand operation or both hand grip two-hand operation as the grip state of the user in the sensor The method further includes a process of detecting.

  With this configuration, it is possible to change the graphical elements of the operation buttons displayed on the touch panel display according to the gripping state of the user. As a result, the convenience of the mobile terminal device can be improved.

  (10) The control program according to the present invention further includes a process of changing a size or a position of an operation button displayed on the touch panel display according to a gripping state of the user.

  Thus, since the size or position of the operation button displayed on the touch panel display is changed according to the gripping state of the user, it is possible to display an operation button that is easier to operate when the user operates with one hand. It becomes possible. As a result, the convenience of the mobile terminal device can be improved.

  (11) The control program according to the present invention includes a process for receiving an input of a gripping state from a user via the touch panel display, and an operation button displayed on the touch panel display according to the gripping state input from the user. And a process of changing the size or the position of.

  Thus, since the size or position of the operation button displayed on the touch panel display is changed according to the gripping state input from the user, the size of the operation button is based on the gripping state according to the user's preference. Alternatively, the position can be changed. As a result, the convenience of the mobile terminal device can be improved.

  (12) Further, the control program of the present invention sets the size or position of the operation button displayed on the touch panel display based on information indicating a user's gripping tendency obtained from the gripping state of the user detected within a certain period. It further includes a process for changing.

  As described above, since the size or position of the operation button displayed on the touch panel display is changed based on the information indicating the user's grip tendency obtained from the grip state of the user detected within the predetermined period, It is possible to display operation buttons in accordance with user operation trends. As a result, the convenience of the mobile terminal device can be improved.

  (13) Further, the control program of the present invention further includes a process of changing the size or position of the operation button displayed on the touch panel display based on a parameter set by a user.

  As described above, since the size or position of the operation button displayed on the touch panel display is changed based on the parameter set by the user, it is possible to display the operation button according to the user's preference. As a result, the convenience of the mobile terminal device can be improved.

  (14) Further, when the orientation of the touch panel display is changed according to the use state of the user, the control program of the present invention displays the parameter on the touch panel display using a parameter corresponding to the detected gripping state of the user. The method further includes a process of changing the size or position of the operation button.

  As described above, when the orientation of the touch panel display is changed according to the use state of the user, the size or position of the operation button to be displayed on the touch panel display is determined using the parameter corresponding to the already detected gripping state of the user. Since the change is made, it becomes possible to display the operation buttons according to the screen orientation preferred by the user. As a result, the convenience of the mobile terminal device can be improved.

  According to the present invention, since the graphical element of the operation button displayed on the touch panel display is changed based on the detection result of the gripping state of the user, the operation button that is easier to operate is displayed when the user operates with one hand. It becomes possible to do. As a result, the convenience of the mobile terminal device can be improved.

It is a figure which shows a mode that a user holds the mobile telephone 1 with a left hand and operates with a left hand. It is a figure which shows a mode that a user holds the mobile telephone 1 with a right hand and operates with a right hand. It is a figure which shows a mode that a user holds the mobile telephone 1 with a right hand and operates with a left hand. It is a figure which shows a mode that a user holds the mobile telephone 1 with a left hand and operates with a right hand. It is a figure which shows a mode that a user holds the mobile telephone 1 with both hands and operates with both hands. 6 is a diagram showing an example of a GUI character input screen in the mobile phone 1; FIG. It is a figure which shows the example of the ten keyboard by software. It is a figure which shows the image of the GUI arrangement | positioning change of the mobile telephone. It is a figure which shows the image of the GUI arrangement | positioning change of the mobile telephone 1. FIG. It is a block diagram which shows schematic structure of the mobile telephone which concerns on this embodiment. It is a figure which shows a mode that data are acquired when a user's touch operation is performed. It is a figure which shows an example of a learning data storage table. It is a flowchart which shows operation | movement of the initial learning mode. FIG. 6 is a diagram showing a region assumed to be easy to press a button in an operation pattern a by a hatched portion. It is a figure which shows the GUI arrangement | positioning change figure in the operation pattern a. It is a figure which shows the GUI arrangement | positioning change figure in the operation pattern a. It is a figure which shows the GUI arrangement | positioning change figure in the operation pattern a. It is a figure which shows the GUI arrangement | positioning change conceptual diagram by both holding | grip both inputs. It is a figure which shows the GUI arrangement | positioning change conceptual diagram by both holding | grip both inputs. It is a flowchart which shows operation | movement of normal mode. It is a figure which shows the mobile telephone 1 of the 1st state in the application example of this embodiment. It is a figure which shows the mobile telephone 1 of the 2nd state in the application example of this embodiment. It is a figure which shows the mobile telephone 1 of the 3rd state in the application example of this embodiment. It is a figure which shows the mobile telephone 1 of the 4th state in the application example of this embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. In the present invention, the portable terminal device means a device that can communicate while moving, and is a concept including a mobile phone, a PDA (Personal Digital Assistant), a smartphone, and the like. In the present embodiment, a mobile phone will be described as an example, but the present invention is not limited to the mobile phone.

  1A to 1E are diagrams illustrating a concept of a state in which a user holds or operates a mobile phone. FIG. 1A shows a state in which the user holds the mobile phone 1 with the left hand and operates with the left hand (left grip left operation). FIG. 1B shows a state in which the user holds the mobile phone 1 with the right hand and operates with the right hand (right grip right operation). FIG. 1C shows a state in which the user holds the mobile phone 1 with the right hand and operates with the left hand (right grip left operation). FIG. 1D shows a state in which the user holds the mobile phone 1 with the left hand and operates with the right hand (left grip right operation). FIG. 1E shows a state in which the user holds the mobile phone 1 with both hands and operates with both hands (both gripping and both operations). As described above, in this embodiment, the user's hand holding the mobile phone 1 and the operating hand are combined and expressed in five patterns. The mobile phone 1 according to the present embodiment has a function of changing the arrangement of a GUI (Graphical User Interface), and the GUI displays a software keyboard and operation buttons.

  FIG. 2A is a diagram illustrating an example of a GUI character input screen in the mobile phone 1. In this character input screen, a software keyboard 2-1 and operation buttons 2-2 as input buttons are displayed on the screen 2. Moreover, FIG. 2B is a figure which shows the example of the ten keyboard by software. In this numeric keyboard, operation buttons 2-3 and 2-4 are displayed on the screen 2.

  3A and 3B are diagrams illustrating an image of changing the GUI arrangement of the mobile phone 1. FIG. 3A shows a screen display example before the change. As shown in FIG. 3A, the software keyboard 3-1 is displayed on the screen 3 at a normal size and a normal position. On the other hand, FIG. 3B shows a screen display example after the change. The software keyboard 3-2 is reduced and displayed at the lower left of the screen 2. In this example, since it is detected that the user is performing a left grip left operation, the software keyboard 3-2 is reduced and moved to the left in order to facilitate the operation. As a result, the user can easily perform a one-handed operation.

  In the present embodiment, the initial learning mode for learning the identification parameter of the identification unit for identifying the user's operation pattern and the GUI parameter for changing the GUI arrangement, the result of learning in the initial learning mode, and the user's operation The normal mode for identifying the operation pattern from the information at the time and changing the GUI layout is provided.

  As shown in FIGS. 1A to 1E, the operation patterns include five patterns of left grip left input, right grip right input, left grip right input, right grip left input, and two-hand grip two-hand input. Here, the operation patterns are a, b, c, d, and e, respectively. The horizontal direction of the display unit is referred to as the X direction, the vertical direction as the Y direction, and the vertical direction as the Z direction. Among these, the operation patterns a and b correspond to one-hand operation as described above. For each of the above-described operation patterns, physical operations such as shaking and tilting of the mobile phone 1 due to user input operations, positions touched by the user, and the like are different. Therefore, it is possible to identify the operation pattern by using these pieces of information.

  As shown in FIGS. 1A to 1E, the horizontal direction of the display unit is referred to as the X direction, the vertical direction as the Y direction, and the vertical direction as the Z direction.

FIG. 4 is a block diagram showing a schematic configuration of the mobile phone according to the present embodiment. The main control unit 41 controls the operation of each component of the mobile phone 1. The display unit 42 displays various operation screens, a software keyboard, and operation buttons. The touch panel input unit 43 receives an input from the user via the touch panel display. The touch panel input unit 43 acquires the following data (touch information) when the user touches. FIG. 5 is a diagram illustrating a state in which data is acquired when a touch operation is performed by the user.
・ Coordinates touched on touch panel (x, y)
・ Center coordinates (cx, cy) of the button that responded to touch
・ Size of touched button (w, h)

  The acceleration sensor 44 detects acceleration information of the mobile phone 1. The detected acceleration is three axes of Ax, Ay, and Az. The learning data storage unit 45 operates in the initial learning mode. For each user touch operation, the operation pattern, acceleration information, and touch information of the mobile phone are stored. In this specification, these pieces of information are referred to as learning data and are stored as a storage table shown in FIG. The identification parameter calculation unit 46 operates in the initial learning mode. An identification parameter is calculated from the acceleration information of the mobile phone 1. This identification parameter is used to identify the operation pattern in the normal mode. The identification parameter storage unit 47 stores the identification parameter calculated by the identification parameter calculation unit 46.

  The operation pattern identification unit 48 operates in the normal mode. The operation pattern is identified from the identification parameter read from the identification parameter storage unit 47 and the acceleration information measured in the normal mode. The GUI layout change parameter calculation unit 49 operates in the initial learning mode. A GUI layout change parameter is calculated from the touch information. The GUI placement change parameter storage unit 50 stores a GUI placement change parameter. The GUI arrangement control unit 51 operates in the normal mode. The GUI arrangement is changed from the operation pattern identified by the operation pattern identification unit 48 and the GUI arrangement change parameter.

  Next, the operation of the mobile phone 1 according to this embodiment configured as described above will be described. The timing for identifying the operation pattern in the normal mode is preferably before the user holds the mobile phone 1 and performs a desired operation. This is because a large change in the GUI layout while the user is performing a desired operation leads to a decrease in operability. In the present embodiment, as a timing for identifying an operation pattern in the normal mode, a case where the present invention is applied to an input operation using a numeric keyboard when inputting the personal identification number of the mobile phone 1 and releasing the lock of sleep will be described. To do. Here, the touch information and acceleration information learned in the initial learning mode are data when input using a numeric keyboard.

  Hereinafter, the initial learning mode will be described. FIG. 7 is a flowchart showing the operation in the initial learning mode. In the initial learning mode, an identification parameter for identifying the operation pattern of the user's cellular phone 1 and a GUI arrangement change parameter for changing the GUI arrangement are learned. First, the cellular phone 1 is activated (step S1), and a learning data acquisition numeric keyboard is displayed. The numeric keyboard shown in FIG. 2B and operation pattern options are displayed on the display unit. Next, the user selects operation patterns a to e shown in FIGS. 1A to 1E (step S2), and the result is stored in the learning data storage unit 45.

  Next, learning data is acquired. Displays a string of numbers that the user can enter. The touch information and the acceleration information are measured from when the user presses the start button until the end button is pressed, or until all the inputs are completed after the input of the numeric string is started (step S3, step S3). S4). The acceleration information is only data when a touch is detected. The measured result is stored in the learning data storage unit 45 (step S5). Then, as shown in FIG. 6, data is stored in the learning data storage unit 45 according to the data format.

  Next, another operation pattern is input and learning data is acquired (step S6). In the same manner as described above, the user selects another operation pattern and inputs a numeric string, and acquires touch information and acceleration information. It is determined whether learning has ended (step S7). If it has not ended, the process proceeds to step S3. If learning has ended, the process proceeds to step S8. In step S8, the identification parameter is calculated and held. After the acquisition of the learning data, the identification parameter of the operation pattern identification unit 48 is calculated (step S8). In the normal mode, the operation pattern identification unit 48 uses the acceleration as an input value and the operation patterns a to e as output values. That is, based on the acceleration data during the input operation by the user, the user's operation pattern includes five patterns of left grip left input, right grip right input, left grip right input, right grip left input, and both grip input. Identify which one is.

  Instead of acceleration, touch information, both information of acceleration and touch information, or both information and their time differential values may be used as input values. The identification parameter is a parameter used for the identification.

  As the operation pattern identification unit 48, a widely used classifier such as SVM, k-nearest neighbor method, Bayes classification, neural network, or the like is used. Since these discriminators identify an operation pattern from an input value at a certain time, the discrimination results can show different results as time changes. For this reason, the operation pattern identification unit 48 identifies the operation pattern having a large ratio among the operation patterns identified by the classifier within a certain period of time or within a certain number of keystrokes as the final operation pattern. However, in order to operate the identification unit, a terminal having high specifications to some extent is required. Therefore, these identification units may be arranged not only inside the mobile phone but also on a server terminal that can communicate with the mobile phone. At this time, the transmission unit of the mobile phone transmits the acceleration and touch information acquired during the user's operation to the server, and based on these, the server identifies the operation pattern, and the reception unit of the mobile phone receives the result. The configuration is as follows. Alternatively, it may be an identification unit that presents a GUI for selecting an operation pattern to the user and allows the user to select one.

  In the present invention, the operation pattern identifying unit 48 has five patterns of the user's operation pattern during the input operation: left grip left input, right grip right input, left grip right input, right grip left input, and both grip both inputs. As long as one of them is identified, its configuration and detailed implementation method are not limited. The identification parameter calculation result is stored in the identification unit parameter storage unit.

Next, learning for GUI layout change is performed (step S9). That is, learning for GUI layout change for the operation pattern i will be described. Change the GUI layout in the X and Y directions
-The button position consists of one or any combination of distance pi, qi parallel movement,-button size ri, si magnification. i corresponds to the operation patterns a, b, c, d, and e in FIGS. These pi to si are used as GUI layout change parameters.

These GUI layout changes are defined based on the following values that can be obtained from the learning data of the operation pattern i or the learning data, and the values on the display.
・ Acceleration (Axi, Ayi, Azi)
・ Touch coordinates (xi, yi)
・ Center coordinates (cx, cy) of the responded button
・ Touch error (Δxi, Δyi) = (cx-xi, cy-yi) between the touch coordinates and the center coordinates of the reacted button
・ Size of touched button (w, h)
-Display size (W, H)
Software keyboard size (Lx, Ly)
・ Distance between the touched button and the previous touched button ・ Distance between adjacent buttons ・ The above time differential value

  An example of changing the GUI layout of the software keyboard is given below. FIG. 8 is a diagram showing a region assumed to be easy to press the button in the operation pattern a by a hatched portion. The shaded area in the figure is an area sandwiched between two arcs centered on the lower left of the display, but is not limited to an arc. The area other than the shaded area in FIG. 8 is an area where the user hardly presses the button because it is too close to the hand or too far away. Therefore, when changing the GUI layout, it is desirable to move within the shaded area.

  On the other hand, in the case of two-handed operation (the operation pattern is c or d), it is not necessary to consider an area that is difficult to press, and it is not necessary to move the button greatly.

  In the operation pattern e, it is considered that the operability is improved by dividing the software keyboard into left and right. As described above, the policies of the GUI arrangement changing method are different between the operation patterns a and b, the operation patterns c and d, and the operation pattern e.

  Furthermore, in each operation pattern, the distribution of touch coordinates differs depending on the dominant hand, how to hold the terminal (how to attach the hand to the terminal), and the finger pressing the button, even if the buttons are the same. It is considered that a detailed change can be made by changing the GUI arrangement for each operation pattern based on the touch error.

Therefore, the GUI layout is changed by the following two-step procedure.
(Procedure 1) For one-handed operation (operation patterns a and b), move the software keyboard roughly to an assumed area where one-handed operation is easy. No change if operation pattern c, d. If operation pattern e, it is divided into left and right. No change if using both hands.
(Procedure 2) For each operation pattern, use the touch error to translate the software keyboard in the X or Y direction.

  First, the one-handed operation of procedure 1 will be described. 9A to 9C are diagrams showing GUI layout change diagrams in the operation pattern a. The point O1 (xmin, ymin) is one of the end points of the region assumed to be difficult to operate because it is close to the hand. The point O2 (xmax, ymax) is one of the end points of the region that is assumed to be difficult to operate because it is far from the hand. It is assumed that the points O1 and O2 are obtained from the center coordinates of the touched button, the acceleration and the touch error, and the average / variance thereof in the learning data. Alternatively, it may be given in advance as a constant. Point C1 (cx1, cy1) is the center coordinate of the button closest to the lower left end of the software keyboard, and point C2 (cx2, cy2) is the center coordinate of the button closest to the upper right end.

  If the point C1 and the point C2 are moved to an area where it is assumed that the points C2 and C2 are easy to push, it is considered that the entire GUI is easy to push. To do so, first move C1 to O1, C2 to O2, and then shrink the entire software keyboard accordingly. Due to this GUI layout change, a part and all of all buttons are included in an area that is assumed to be easy to press.

This GUI layout change can be explained as follows using the GUI layout change parameters pa, qa, ra, and sa. First, the size of the software keyboard is multiplied by ra times in the X direction and sa times in the Y direction, with the coordinates at the lower left corner of the keyboard (x0, y0) as the center, and then translated by pa in the X direction and qa in the Y direction. Each parameter is shown below.
pa = xmin-x0
qa = ymin-y0
ra = (xmax -xmin) / (Lx-w)
sa = (ymax-ymin) / (Ly-h)

  In the operation pattern b, the above-described example may be changed in the same GUI layout by inverting the X direction with respect to the display center.

  Next, the operation pattern e will be described. FIG. 10A and FIG. 10B are diagrams showing a GUI layout change conceptual diagram with both gripping inputs. Using the touch position and touch error, it is discriminated whether the button is pressed more frequently on the left or right for each button, and the software keyboard is divided with a boundary between buttons. For example, the positive / negative of ΣΔxi / Mi for each button is compared. If the button is positive, the left side of the button is pressed a lot.

  In the operation pattern b, the same GUI layout change may be performed by inverting the X direction with respect to the center of the display.

Next, procedure 2 will be described. For each operation pattern i, the software keyboard is moved pi to the X coordinate and qi to the Y coordinate from the average touch error.
pi and qi are defined by the following equations.
pi = ΣΔxi / Mi
qi ＝ ΣΔyi / Mi
It is defined as Here, Mi is the number of learning data of the operation pattern i shown in FIG.

  As described above, it is considered that the GUI layout change suitable for the user's operation pattern can be performed by repeatedly learning the procedures 1 and 2. This procedure may be applied not only to the software keyboard but also to other input scenes.

  Next, the normal mode will be described. FIG. 11 is a flowchart showing the operation in the normal mode. In the normal mode, the operation pattern is identified based on the acceleration information, and the GUI layout is changed. First, a password is entered (step T1). A numeric keyboard as shown in FIG. 2B is displayed, and the user is prompted to enter a password. Next, acceleration information and touch information are acquired (step T2). As in the initial learning mode, acceleration and touch information during input of the password are acquired. Next, initial identification of the operation pattern is performed (step T3). When the input is completed, the operation pattern is identified by the identification unit based on the acceleration information. Next, the GUI layout is changed (step T4). Based on the identified operation pattern, the GUI parameter learned in the initial learning mode is read from the GUI parameter storage unit, and the GUI arrangement is changed. The change method is as described above (learning for GUI layout change).

Next, normal operation is performed (step T5). Thereafter, the user uses the mobile phone 1 as usual. Thereafter, acceleration and touch information when the software keyboard and buttons are pressed are stored (step T6). Next, it is determined whether or not an end operation has been performed (step T7). If the user has not performed an operation to end the portable operation, the operation pattern and GUI change parameter are re-identified and re-changed (step T8, T9). It is conceivable that a pattern different from the actual operation pattern is erroneously identified, or the user changes the operation pattern of the mobile phone 1 during the normal operation. Therefore, the operation pattern is re-identified and re-changed based on a certain determination. The timing for determining whether re-identification / re-change is possible is one of the following.
・ At certain time or keystroke interval ・ When the character input cancel button is pressed K times within a certain time or keystroke ・ When the average value of touch error absolute value within a certain time or keystroke is greater than a certain value ・When the variance of acceleration within a certain time or number of keystrokes is greater than a certain value ・ When the mobile phone goes to sleep ・ When the user directly points out an operation pattern discrimination error ・ When any of the above combinations occur

  Further, the acceleration and touch panel information measured with the misidentified operation pattern from the start of the operation to the change are stored as learning data in association with the correct operation pattern. Further, after the operation pattern is re-identified and re-changed, the GUI change parameter is also learned again.

  On the other hand, when the user performs an operation to end the portable operation in step T7, the recorded acceleration and touch information are added to the learning data, the identification parameter and the GUI change parameter are re-learned (step T10), and the process ends ( Step T11).

  In this embodiment, a method of inputting a personal identification number using a numeric keyboard is used for initial identification in the initial learning mode and the normal mode. However, another character string / numeric string is input using another software keyboard. A method may be used. In this embodiment, the parameter calculation in the initial learning mode is performed after the user performs the cellular phone termination process, but may be performed when the cellular phone goes to sleep. In the normal mode, the identification parameter and the GUI change parameter obtained in the initial learning mode are used. However, when the initial learning mode is not performed, a preset parameter may be used.

  In this embodiment, the identification parameter and the GUI change parameter are stored in the learning data storage unit, but may have a function of transmitting / receiving to / from an external server that can store both parameters. As a result, even if the mobile phone owned by the user is changed, the operation pattern can be identified and the GUI layout can be changed without the initial learning mode.

  In the present embodiment, the movement for changing the GUI arrangement is performed. Alternatively, the keystroke detection area of the button on the touch panel may be moved. 12A to 12D are diagrams illustrating application examples of the present embodiment. When the orientation of the mobile phone 1 is 90 degrees, 180 degrees, and 270 degrees rotated about the vertical line of the display unit from the first state in the first state as the second state, the third state, and the fourth state, respectively. The identification unit parameter and the GUI parameter obtained from any one of the first to fourth states may be adapted to the identification of the operation modes of the other three states by acceleration coordinate conversion.

Assuming that the acceleration measured in the i-th state is (Aix, Aiy, Aiz) (i = 1, 2, 3, 4)
(A2x, A2y, A2z) = (− A1y, A1x, A1z)
(A3x, A3y, A3z) = (− A1x, −A1y, A1z)
(A4x, A4y, A4z) = (A1y, −A1x, A1z)
By doing so, the user's operation pattern can be identified. The input GUI can be changed by changing the direction in which the input GUI is moved according to the direction of rotation of the mobile phone and scaling the GUI parameters using the mobile phone height and width information. Assuming that the height of the mobile phone in the first state is H and the width is W, for example, in the second and fourth states, scaling is performed by multiplying the amount of movement in the first state GUI layout change by H / W. Can be mentioned. The state determinations 1 to 4 of the mobile phone 1 are performed by an inclination detection sensor or the like.

  In this embodiment, the motion acceleration is used for identifying the operation pattern and changing the input GUI, but information obtained from the geomagnetic sensor and the gyro sensor may be used for each.

  The characteristic operation of the present invention can also be realized by causing a computer to execute a control program. That is, the control program according to the present embodiment is a control program for a mobile terminal device that includes a touch panel display and that functions when triggered by a user input operation on the touch panel display, and detects a user's gripping state by a sensor. A process of changing a graphical element of an operation button displayed on the touch panel display based on a detection result of the sensor, and a process of displaying the changed operation button on the touch panel display. The above processing is executed by a computer.

  As described above, since the graphical element of the operation button displayed on the touch panel display is changed based on the detection result of the gripping state of the user, the operation button that is easier to operate is displayed when the user operates with one hand. Is possible. As a result, the convenience of the mobile terminal device can be improved.

  As described above, according to the present embodiment, since the graphic element of the operation button displayed on the touch panel display is changed based on the detection result of the user's gripping state, when the user operates with one hand, It is possible to display operation buttons that are easy to operate. As a result, the convenience of the mobile terminal device can be improved.

DESCRIPTION OF SYMBOLS 1 Cellular phone 2 Screen 2-1 Software keyboard 2-2 Operation button 2-3 Screen 3-1 Software keyboard 41 Main control part 42 Display part 43 Touch panel input part 44 Acceleration sensor 45 Learning data storage part 46 Identification parameter calculation part 47 Identification Parameter storage unit 48 Operation pattern identification unit 49 GUI layout change parameter calculation unit 50 GUI layout change parameter storage unit 51 GUI layout control unit

Claims (14)

A mobile terminal device that includes a touch panel display and that exhibits a function triggered by a user input operation on the touch panel display,
A sensor for detecting a user's gripping state;
A GUI arrangement changing unit that changes a graphical element of an operation button displayed on the touch panel display based on a detection result of the sensor;
A portable terminal device comprising: a display unit configured to display the changed operation button on the touch panel display.   2. The sensor according to claim 1, wherein the sensor detects one of a left hand operation, a right hand operation, a right hand operation, a right hand operation, a left hand operation, a left hand operation, a right hand operation, and a two-hand operation. Mobile terminal device.   The mobile terminal device according to claim 1, wherein the GUI arrangement changing unit changes a size or a position of an operation button displayed on the touch panel display according to a gripping state of the user. The touch panel display accepts an input of a gripping state from a user,
The mobile terminal device according to claim 1, wherein the GUI arrangement detection unit changes a size or a position of an operation button displayed on the touch panel display according to a gripping state input from the user.   The GUI arrangement changing unit changes a size or a position of an operation button displayed on the touch panel display based on information indicating a user's grip tendency obtained from a user's grip state detected within a predetermined period. The mobile terminal device according to claim 1.   The mobile terminal device according to claim 1, wherein the GUI arrangement changing unit changes the size or position of an operation button displayed on the touch panel display based on a parameter set by a user.   When the orientation of the touch panel display is changed according to the usage state of the user, the GUI arrangement changing unit uses the parameter corresponding to the detected gripping state of the user, and the size of the operation button displayed on the touch panel display. The mobile terminal device according to claim 1, wherein the position is changed. A control program for a mobile terminal device that includes a touch panel display and that exhibits a function triggered by a user input operation on the touch panel display,
A process of detecting a user's gripping state by a sensor;
Based on the detection result of the sensor, processing to change the graphical element of the operation button displayed on the touch panel display;
A control program that causes a computer to execute a series of processes of displaying the changed operation button on the touch panel display.   The sensor further includes a process of detecting any one of a left-hand operation, a left-hand operation, a right-hand operation, a right-hand operation, a left-hand operation, a left-hand operation, a left-hand operation, a two-hand operation as a user's holding state. Item 9. The control program according to Item 8.   The control program according to claim 8, further comprising a process of changing a size or a position of an operation button displayed on the touch panel display according to a gripping state of the user. A process of receiving an input of a gripping state from a user via the touch panel display;
The control program according to claim 8, further comprising a process of changing a size or a position of an operation button displayed on the touch panel display according to a gripping state input from the user.   The information processing apparatus further includes a process of changing a size or a position of an operation button displayed on the touch panel display based on information indicating a user's grip tendency obtained from the grip state of the user detected within a certain period. Item 9. The control program according to Item 8.   The control program according to claim 8, further comprising a process of changing a size or a position of an operation button displayed on the touch panel display based on a parameter set by a user.   When the orientation of the touch panel display is changed according to the use state of the user, a process of changing the size or position of the operation button displayed on the touch panel display using a parameter corresponding to the detected grip state of the user. The control program according to claim 8, further comprising:

Images