IT202300015498A1 - An improvement of a method for estimating a subject's body fat mass using digital images. - Google Patents

Description

TITLE

Description accompanying the patent application for an industrial invention entitled:

AN IMPROVEMENT OF A METHOD FOR ESTIMATING MASS

IMAGE OF A SUBJECT THROUGH DIGITAL IMAGES

Scope of the invention

[001] The present invention relates to the technical sector relating to devices for determining one or more physiological parameters of a subject, whether human or animal in general.

[002] In particular, the invention refers to an improvement of a method that allows for the estimation of body composition, such as the quantity of fat mass and/or lean mass contained in a body.

Brief notes on the prior art

[003] The determination of body fat mass is of significant importance in many sectors.

[004] In the medical field, for example, determining body fat mass allows for diagnosis, prevention and helps plan any interventions aimed at both weight loss and weight gain.

[005] ? The European publication EP3304490 in the name of Talluri Antonio, dated 01-03-2016 is already known and is here considered to be contained entirely within this text for reference.

[006] This patent application discusses a method that allows you to determine your body fat mass by acquiring an image with your mobile phone device, for example by downloading a specific application.

[007] This scientific method has also been validated and the following Internet links are provided to the validation articles published by the prestigious magazine Sensors:

[008] http://www.mdpi.com/1424-8220/16/12/2163/pdf;

[009] Sensors 2022, 22, 8365:

https://doi.org/10.3390/s22218365

[0010] What is described in the cited articles and in the aforementioned European publication EP3304490 is also considered included herein by reference.

[0011] The acquired image obviously refers to the subject under examination who must be positioned in lateral profile, "at attention". Through methods, such as those of AI with known libraries that are able to remove the background, only the photograph of the subject, without the background (silhouette), is automatically obtained. For example, an online method called:

Remove (https://www.remove.bg/).

[0012] The acquired image obviously refers to the subject who must position himself with respect to a uniform background without wearing clothes or at least minimally dressed. In this way the subject presents a sufficiently uniform coloration to be able to distinguish his profile from the background.

[0013] At this point the software, with an analysis of the image, determines which pixels make up the profile of the subject and which constitute the background and this information, with a specific algorithm, is then used to obtain an area occupied by the subject with respect to an area of a frame that surrounds the image of the subject and therefore determine with a further specific algorithm the value of the fat mass.

[0014] Further improved versions of this well-known application have been filed.

[0015] For example, the Italian publication 102017000091318 in the name of Talluri Antonio dated 07-08-2017 is also known and is considered to be entirely incorporated here for reference.

[0016] In all these solutions, a mobile phone device is used to acquire an image and, once the image has been acquired, a specific algorithm is used to determine body composition, in particular fat and/or lean mass.

[0017] Now, with all of this in mind, one technical issue that has emerged over time stems from the quality of image acquisition.

[0018] In order for an image to be suitable for precise calculation, it must be acquired according to the correct angles and framing.

[0019] An image that frames a 'silhouette' incorrectly obviously affects the calculation or in any case decreases the predictive precision and the result that is then processed by the algorithm that has to determine the body composition is not correct.

[0020] For example, if the image frames the subject from the bottom upwards, the subject tends to appear with longer legs and this, obviously, can affect the calculation.

[0021] If the image frames the subject from top to bottom, then the subject in the frame appears stockier than normal and this also affects the quality of the calculation relating to body composition (for example fat and/or lean mass).

[0022] However, there is currently no guidance system that optimizes the image acquisition quality, which can lead to situations where the images are not perfectly centered or excessively tilted.

[0023] Furthermore, the body position must also be correct. If the subject maintains incorrect postures, for example too closed in on himself, in this case too the processing result could be distorted.

Summary of the invention

[0024] ? It is therefore the aim of the present invention to provide an innovative method that solves the aforementioned technical drawbacks.

[0025] In particular, the purpose of the present invention is to provide an improvement that allows the user to be guided in acquiring an image that is as correct and usable as possible by the program in use.

[0026] These and other objects are therefore obtained with a method according to claim 1.

[0027] This method is designed to allow the correct acquisition of an image portraying a subject, for example preferably a person, in such a way that the outline of the portrayed subject (for example the person) is electronically processed with an algorithm for determining a body composition, for example fat and/or lean mass.

[0028] As mentioned, algorithms are already known that are capable of processing such body composition information, including the subject's fat and/or lean mass, based on the image portraying the subject.

[0029] In accordance with the invention, the method now comprises the following steps:

- The arrangement of a mobile device equipped with a screen and camera to frame a subject, for example a person, in such a way that, once the camera has framed that subject, the image is visible on the screen of said mobile device.

- The mobile device screen now features a frame (11, 12, 13) whose function is that of a diopter which helps the user to correctly frame the subject (for example the person) in the image so that it is intercepted by the said frame and/or positioned within the limits of the said frame;

- The method providing for the sending to the user of one or more corrective information relating to the position that the mobile device must assume with respect to a reference plane, said corrective information being obtained through one or more sensors of the mobile device capable of determining the position assumed by the mobile device;

- And in which, once the image is acquired, it is processed by positioning a plurality of landmarks along at least part of the subject's outline; - This is then followed by a joining phase with segments that join the succession of landmarks in such a way as to define an angular positioning of one or more parts of the subject's body, for example the limbs. - In this way, the angles of the segments joining the said landmarks represent an angular positioning of one or more parts of the body, for example the limbs, and these angles are now compared with reference angles.

[0030] In this way, it is advantageously possible to verify whether the position assumed by the subject (for example a person) is suitable or not for the subsequent processing of body composition (for example fat mass and/or lean mass).

[0031] Advantageously, in this regard, a warning message can be sent indicating whether the detected angular positioning is correct or incorrect.

[0032] In particular, now, the subject portrayed in the image is framed correctly and also his posture, thanks to the landmarks, is correct.

[0033] This allows, following image processing with well-known algorithms, to precisely determine subject parameters such as fat or lean mass.

[0034] Advantageously, said one or more sensors may comprise at least one gyroscope and/or one or more accelerometers.

[0035] Advantageously, said diopter can comprise two horizontal lines (11, 12) parallel to each other joined by a transverse line (13) which joins the centre line of said two horizontal lines.

[0036] Advantageously, the sending to the user of one or more corrective information relating to the position that the mobile device must assume with respect to a reference plane is sent before the photograph is taken.

[0037] Advantageously, the taking of the photograph can be inhibited if the one or more sensors detect an incorrect position of the mobile device.

[0038] Advantageously, the photo can be taken by voice command.

[0039] Advantageously, the photo, once taken and analysed through the aforementioned reference points, is processed to calculate body composition, in particular the fat and/or lean mass is calculated.

[0040] Advantageously, image processing can include sending the image acquired by the user to a cloud system.

[0041] Preferably, advantageously, the subject is a person.

[0042] The present invention also includes a computer program suitable for performing one or more of the method steps indicated above.

[0043] The invention also includes a computer-readable storage medium/data carrier comprising the above-mentioned program.

Brief description of the drawings

[0044] Further features and advantages of the present device, and related method according to the invention, will become clearer with the following description of an embodiment thereof, given by way of example and not of limitation, with reference to the attached drawings, in which:

[0045] Figure 1 shows an image framed by a mobile phone device and highlights a subject (in this example case a test doll) placed in a correct image acquisition position;

[0046] Figure 2 shows an image framed by a mobile phone device and highlights a subject (in this example case a test doll) placed in an incorrect image acquisition position; in particular, the subject is not aligned on the vertical axis and the camera is tilted downwards;

[0047] Figure 3 shows an image framed by a mobile phone device and highlights a subject (in this example case a test doll) placed in an incorrect image acquisition position; in particular, the subject is not aligned on the vertical axis and the camera is tilted downwards;

[0048] Figure 4 shows an image framed by a mobile phone device and highlights a subject (in this example case a test doll) placed in an incorrect image acquisition position; in particular, the subject is not aligned on the vertical axis and the camera is tilted upwards, with part of the subject outside the camera's field of view;

[0049] Figure 5 shows an image framed by a mobile phone device and highlights a subject (in this example case a test doll) placed in an incorrect image acquisition position; in particular, the subject appears to be moved to the left of the visual field and the camera is tilted upwards;

[0050] Figure 6 shows, once the subject image has been obtained, the predefined "landmark" points, an example case with segments relating to the arms (upper arm and forearm), torso and legs from the hip to the ankle including the feet. In particular, the figure shows an incorrect positioning of the subject's arm and wrist.

[0051] Figure 7 shows, once the subject image has been obtained, the predefined "landmark" points, an example case with segments relating to the arms (upper arm and forearm), torso and legs from the hip to the ankle including the feet. In particular, the figure shows an incorrect positioning of the subject.

Description of some preferred embodiments

[0052] Mobile telephone devices, for example of the I-Phone or Smart-Phone type or a portable device equipped with a digital camera and connectable to the smart phone via cable, Blue-Tooth, WiFi such as Tablet, portable PC, digital cameras and similar, have been known for some time - usable in the present method, exactly as already discussed in the previous publications cited in the prior art.

[0053] These devices, especially mobile phones, have become commonplace and are increasingly replacing older cell phones with more limited functionality.

[0054] That being said, figure 1 represents a shot that can be obtained with a normal mobile telephone device (for example, an I-PHONE, SMARTPHONE and similar).

[0055] Frame 100 shows a video screen of the mobile device showing a subject 200 framed by the camera with which the mobile phone device is equipped.

[0056] The image therefore portrays a subject 200, for example a person, within the image together obviously with a possible background which is not of interest.

[0057] At this point, having framed the subject in the image, it would be possible to take the shot to acquire the image and send it to be processed through the calculation algorithm that calculates a body composition, for example a fat mass as described in the prior art.

[0058] The image that is sent for processing can then be processed in the cloud and then the result returned to the user, always on his mobile device.

[0059] The body composition algorithm can reside in the phone itself or, preferably, it can be present in the cloud or on a remote server and therefore processed remotely and then return the result.

[0060] In this way the user has a service where he can download an app to acquire the image as described below, and then send the acquired image to a remote or cloud system that determines the required body composition, for example fat or lean mass, and returns the result.

[0061] In any case, as mentioned, the calculation algorithm relating, for example, to fat and/or lean mass is known and described, for example, in previous publications. For this reason, it is not further detailed here.

[0062] In accordance with the invention, however, the method described here mainly offers a service that helps and guides the user in the correct acquisition of the image in order to then obtain a much more precise subsequent processing.

[0063] The user, who can for example download an "App" specifically onto the mobile device, will have references that will help him in the correct acquisition of the image.

[0064] In particular, as shown in figure 1, there is now a reference frame (11, 12, 13), which functions as a diopter.

[0065] It is made up of two horizontal lines 11 and 12 placed respectively on the upper and lower side of the image visible on the screen of the mobile device (see figure 1 for example). These two lines 11 and 12 are therefore parallel to each other and include an upper horizontal line 12 and a lower horizontal line.

[0066] The upper one is therefore the one parallel to the upper image frame of the mobile device (for example, as mentioned, a cell phone or mobile telephone device) while the lower one is the one parallel to the lower image frame.

[0067] They are then connected by a transverse line 13.

[0068] The transverse line 13 is orthogonal to the two horizontal lines.

[0069] The presence of this reference which acts as a diopter allows, very simply, the user to have a framing reference with the subject portrayed in the image which must be contained within the two horizontal lines and preferably with the transverse line 13 which acts as the centre line of the profile of the subject itself.

[0070] The user simply has to use the frame as a firearm sight (diopter) and simply ensure that the profile is contained between the two horizontal lines (11, 12) and with the transverse axis 13 which must somehow intercept the body of the subject portrayed as if it were a centre line.

[0071] This simple device, i.e. the presence of the frame in the frame with the function of a diopter, therefore helps the user to take an initial shot that is much more precise than in the case of shots without reference.

[0072] However, the reference frame alone would not be sufficient since, as previously stated in the prior art, the user could still tilt the phone and still obtain a distorted image.

[0073] The frame introduced above therefore helps to optimise a shot by avoiding, for example, that the mobile device appears rotated too much to the right or left.

[0074] However, there remains a problem related to the forward or backward rotation which would always lead to a distorted image acquisition and in any case even the lateral rotations are not entirely resolvable with the frame alone.

[0075] In particular, the frame alone is not able to guarantee perfect alignment of the cell phone.

[0076] In this way, therefore, the frame described certainly helps to have an initial reference but then, as described immediately below, the sensors with which the mobile device is already equipped are used to verify the correctness of the framing and, if necessary, inhibit the taking of the photo.

[0077] More specifically, it is known that mobile telephone devices or mobile devices in general are equipped with sensors such as gyroscopes and accelerometers capable of detecting the position that the mobile phone assumes in space when held in the hand of a subject.

[0078] For example, programs that automatically "rotate" the image based on the position of the mobile phone device use these gyroscopes and/or accelerometers so that when the device rotates, the image displayed on the screen (for example a photo or a video) also rotates.

[0079] Given this, the system described is able to calculate in a very simple way the spatial position of the mobile device with respect to a reference (X, Y, Z) thanks to these sensors and, consequently, corrective indications can be given.

[0080] As shown in the sequence of images from figure 2 to figure 5, there are cases in which the mobile device is rotated and not in a correct alignment position.

[0081] More specifically, as stated above, the image of figure 2 or figure 3 is within the frame but, nevertheless, the sensors of the mobile device such as the gyroscope and/or accelerometers detect a position of the mobile device that is not perfectly orthogonal to the ground and therefore, based on the detected misalignment, corrective rotation operations are suggested, for example forwards or backwards.

[0082] The example in figure 4 and figure 5 clearly shows a condition in which the framed subject has the part of the feet that go over the lower reference line 12.

[0083] With the frame reference, the user simply has to make the subject's profile fit within the frame as mentioned. In the case of figure 4, the profile is outside the frame because the mobile device is rotated excessively. The gyroscope and/or accelerometer are still able to determine the rotated position of the mobile device with respect to the horizontal plane and therefore send a warning message for a corrective rotation.

[0084] Ultimately, imagining a device held in the hand by a user to take a photo framing the subject, the software guides the user to rotate the device forwards or backwards (therefore with rotations around the X and/or Y axis) or with lateral rotations, for which, with reference to figure 1, with a rotation around the Z axis in such a way that the mobile device (as per figure 1) is perfectly orthogonal to a horizontal support plane which could be, for example, the same floor on which the user is resting while holding the mobile device.

[0085] This ensures optimal image quality and the frame helps the user to frame correctly right from the start.

[0086] The software that actually detects the position of the mobile device allows the photo to be taken only when the positioning identified through the sensors it is equipped with corresponds to the pre-established one, that is, with the mobile device orthogonal to the ground.

[0087] Otherwise the shot is inhibited.

[0088] Another important aspect also concerns the positioning of the limbs, which must be placed, as shown in figure 1, in the correct way.

[0089] More specifically, once the image has been acquired, it is processed by the software (in the form of an app downloaded onto the mobile device or in the cloud) which checks the correct positioning of the limbs.

[0090] As mentioned, in fact, limbs placed in an incorrect position could affect subsequent calculations.

[0091] The system described, therefore, inhibits image acquisition if the spatial positioning of the mobile device is incorrect (detection with the mobile device's sensors).

[0092] Once this image has been acquired, before it is processed by the algorithms for determining fat or lean mass, the software checks with the landmarks whether the position is correct.

[0093] In case of correct or incorrect posture, it sends a relevant message to the user on his mobile device who can, based on what is received, decide whether to reacquire an image or authorise the processing of the acquired image (even if the software suggests a posture correction).

[0094] If, for example, an elbow appears too protruding, this could affect the outcome of the subsequent calculation and therefore the software analyses the image to suggest to the subject whether to reacquire an image or not.

[0095] The user is still free to send the acquired image for processing even if the landmarks indicate an incorrect posture.

[0096] For this purpose, as shown in figure 6, the software, having obtained the subject's profile, is able to position predefined "reference" points, as indicated in figure 6 or 7.

[0097] It is already known from previous publications described how the software can obtain the profile of the subject by distinguishing the various parts and resulting in this in a known image processing operation.

[0098] Once the "Silhouette" of the subject has been defined with the various parts according to known techniques, the software is now able to apply references at pre-established points indicated as landmarks and highlighted in figures 6 and 7.

[0099] Both figure 6 and figure 7 show a succession of "landmark" points that can be placed by the software once the subject's profile has been defined and therefore the body parts such as the head, arms, legs and torso have been determined.

[00100] Both Figure 6 and Figure 7 show landmarks from the shoulders to the feet and landmarks on the face as well.

[00101] Each landmark can be joined to the next, thus defining segments.

[00102] For example, both Figure 6 and Figure 7 show two example cases with segments relating to the arms (upper arm and forearm), torso and legs from the hip to the ankle including the feet.

[00103] The software is therefore programmed to compare the detected angles, relative to the various joint segments of the landmarks, with predefined reference values.

[00104] Reference values may, for example, be available in appropriate software libraries.

[00105] In essence, once the predefined reference points have been built and the segments have been built on the acquired image, these segments are compared with ideal reference segments to verify whether there is an angular deviation.

[00106] If there are angular deviations that are outside a certain tolerance range, the software consequently signals the need for corrective action.

[00107] In particular, the software reports that the image is not suitable for correct calculation.

[00108] If, for example, in the case of figure 6 the forearm is bent, overall the arm is not properly extended and parallel to the torso. In fact, overall the two segments of the arm and forearm will be rotated excessively with respect to a range of admissible angles with respect to the segments taken as reference. This results, as shown in figure 6, in a request for correction.

[00109] The same applies to the example in figure 7.

[00110] Therefore, according to the operating process, a first guiding phase is foreseen to acquire the image in which the sensors on board the mobile device and the frame guide the user in the correct image acquisition.

[00111] Once the image is acquired, it is sent to the algorithm which may reside, for example, in the cloud on a dedicated server.

[00112] In a first phase, this algorithm verifies, once the landmarks have been detected, whether the angular predisposition of the limbs is correct by making a comparison with reference angles.

[00113] It then sends a message to the user who can then confirm to proceed with the next body composition calculation or who can proceed with a new image acquisition if, for example, the message indicates that the image is not suitable.

[00114] In this way the user is assisted in a correct image acquisition.

[00115] In order to facilitate the acquisition of the image once the most correct position has been found, it is possible to activate the shooting command via voice command.

Claims (10)

1. A method for correctly acquiring an image portraying a subject, such that the outline of the portrayed subject is electronically processed with an algorithm for determining a body composition, for example its fat and/or lean mass, the method comprising: - The arrangement of a mobile device equipped with a screen and a camera to frame a subject, in such a way that, once the subject has been framed by the camera, the image is visible on the screen of the mobile device, and in which the vision of a frame (11, 12, 13) is activated on the screen of the mobile device, the function of which is that of a diopter which helps the user to correctly frame the subject in the image in such a way that the subject can be intercepted by the frame and/or arranged within the limits of the frame; - The method providing for the sending to the user of one or more corrective information relating to the position that the mobile device must assume with respect to a reference plane during the framing of the subject, said corrective information being obtained through an analysis of the position of the mobile device carried out with one or more sensors present in the mobile device; - And in which, once the image has been acquired, it is processed by positioning a plurality of reference points along the subject's outline and connecting the succession of reference points with segments in such a way as to define an angular positioning of one or more parts of the subject's body, this angular predisposition of one or more parts of the body being compared with reference angles; - And in which a warning message is sent indicating whether the detected angular positioning is correct or incorrect. 2. The method according to claim 1, wherein said one or more sensors comprise at least one gyroscope and/or one or more accelerometers. 3. The method according to claim 1 or 2, wherein said diopter comprises two horizontal lines (11, 12) parallel to each other joined by a transverse line (13) which joins the centre line of said two horizontal lines. 4. The method according to one or more of the preceding claims, wherein said sending to the user of one or more corrective information relating to the position that the mobile device must assume with respect to a reference plane occurs before the photograph is taken. 5. The method according to one or more of the preceding claims, wherein the taking of the photograph is inhibited if said one or more sensors detect an incorrect position of the mobile device. 6. The method according to one or more of the preceding claims, wherein the photo is taken by voice command. 7. The method according to one or more of the preceding claims, wherein the photo, once taken and analyzed through the said landmarks, is processed to calculate body composition, in particular fat and/or lean mass is calculated. 8. The method according to claim 7, wherein image processing includes sending the image acquired by the user to a cloud system. 9. The method according to one or more of the preceding claims, wherein the subject is a person. 10. A computer program comprising instructions which, when the program is executed through a computer or processor, causes the computer or said processor to perform one or more steps of the method according to one or more of the preceding claims.