CN119149031A

CN119149031A - UI control association walk-through method, device, equipment and storage medium

Info

Publication number: CN119149031A
Application number: CN202411185390.4A
Authority: CN
Inventors: 邹越; 苏沁宁
Original assignee: Ping An Bank Co Ltd
Current assignee: Ping An Bank Co Ltd
Priority date: 2024-08-27
Filing date: 2024-08-27
Publication date: 2024-12-17

Abstract

The present application discloses a method, apparatus, device and storage medium for UI control association inspection, which captures element information corresponding to all controls contained in a front-end page, classifies the controls according to preset labels, forms a control set, further deduplicates the control set based on the area of each control in the control set, and finally associates the controls in pairs according to coordinates and preset priorities, determines problematic controls in the associated control set according to preset control comparison rules, implements conditional comparison between elements of multiple controls, provides more accurate spacing inspection, reduces manual inspection costs, and improves inspection efficiency.

Description

UI control association walk-through method, device, equipment and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a UI control association walking method, device, equipment, and storage medium.

Background

In the present digital age, front-end development has become an indispensable ring in software development. Quality control for front-end development becomes particularly important as user demands for aesthetics and functionality of web pages and application program interfaces (UIs) continue to increase. Among other things, spacing walks between page elements are one of the key factors to ensure that the user interface is clean, consistent, and easy to use.

The control walk-through mode of the development page is various, the walk-through condition is visual, and only a single control is required to be compared. For spacing such walk-through conditions requiring multiple control associations, there is currently a lack of specific solutions.

Disclosure of Invention

The application provides a UI control association walking method, a device, equipment and a storage medium, which realize condition comparison among elements of multiple controls, provide more accurate interval walking, reduce manual walking cost and improve walking efficiency.

In view of this, a first aspect of the present application provides a UI control association walk method, the method including:

step S1, capturing element information corresponding to all controls of a target page;

step S2, classifying the element information according to a preset label to form a plurality of control sets corresponding to the preset label;

Step S3, performing de-duplication processing on each control set based on the area of each control in the control set;

S4, performing pairwise association on the controls contained in the control set after duplication removal according to preset priority and coordinates to form a plurality of associated control sets corresponding to the preset labels;

And S5, comparing the associated control sets according to the preset labels based on a preset control comparison rule to determine the problem control.

Optionally, the step S3 specifically includes:

step S31, calculating the areas of the controls in the control sets;

step S32, sorting the controls in the control set according to the descending order of the areas;

step S33, constructing a relation storage array of the control set, wherein the relation storage array comprises a front control and a rear control group;

step S34, traversing according to the sequence, and assigning an ith control to the front control;

Step S35, sequentially calculating the IOU intersection ratio between the area of the ith control and the areas of the rest controls behind the ith control;

step S36, when the IOU intersection ratio is larger than a preset threshold, assigning the corresponding control after the ith control to the rear control group;

And step S37, taking out the representation control after the duplication removal according to the number of the included controls in the rear control group, and obtaining that the control set after the duplication removal contains a plurality of the representation controls.

Optionally, the step S37 specifically includes:

And when the rear control group contains more than two controls, taking out the control corresponding to the second position in the rear control group as a representative control, otherwise taking out the control corresponding to the first position in the rear control group as the representative control.

Optionally, the step S4 specifically includes:

Step S41, sorting a plurality of representative controls contained in the control set in descending order according to an ordinate;

Step S42, creating an association control set;

Step S43, traversing the representative controls contained in the control set in sequence, and adding the representative control combinations which are related in pairs to the related control set according to the abscissa and the preset priority difference.

Optionally, the step S43 specifically includes:

Step S431, traversing the representative controls contained in the control set in sequence, and adding the ith representative control into the associated control set;

Step S432, traversing the rest of the representing controls after the ith representing control, and sequentially comparing the horizontal coordinate difference value of the target positions of the ith representing control and the rest of the representing controls after the ith representing control with the preset priority difference value of the front page;

Step S433, if there is a difference value between the horizontal coordinates of the other representative control and the i-th representative control and the preset priority difference value are both lower than the corresponding preset threshold, adding the corresponding other representative control as the associated control of the i-th representative control to the associated control set, otherwise, returning to execute the step S431 after i=i+1.

Optionally, the step S431 further includes:

If the ith representative component already exists in the association control set, the step S431 is re-executed after i=i+1.

Optionally, the step S5 further includes:

and S6, after packaging the information of the plurality of problem controls, sending the information to a target user for alarming.

The second aspect of the present application provides a UI control association walkthrough device, the device comprising:

the capturing unit is used for capturing element information corresponding to all the controls of the target page;

the classifying unit is used for classifying the element information according to a preset label to form a plurality of control sets corresponding to the preset label;

the duplicate removal unit is used for carrying out duplicate removal processing on each control set based on the area of each control in the control set;

The association unit is used for associating the controls contained in the control set after the duplicate removal according to preset priority and coordinates in pairs to form a plurality of associated control sets corresponding to the preset labels;

And the comparison unit is used for comparing the associated control sets according to the preset labels based on a preset control comparison rule to determine the problem control.

Optionally, the deduplication unit is specifically configured to:

calculating the areas of the controls in the control sets;

Sorting the controls in the control set according to the descending order of the areas;

constructing a relation storage array of the control set, wherein the relation storage array comprises a front control and a rear control group;

Assigning the ith control to the front control according to the sorting traversal;

sequentially calculating the IOU intersection ratio between the area of the ith control and the areas of the rest controls behind the ith control;

When the IOU intersection ratio is larger than a preset threshold, assigning the corresponding control after the ith control to the rear control group;

And according to the number of the included controls in the rear control group, taking out the representation controls after the duplication removal, and obtaining that the control set after the duplication removal contains a plurality of the representation controls.

Optionally, according to the number of the controls included in the rear control group, taking out the representation controls after the duplicate removal, where the control set after the duplicate removal includes a plurality of representation controls specifically includes:

Optionally, the association unit is specifically configured to:

sorting a plurality of representative controls contained in the control set in descending order according to an ordinate;

Creating an association control set;

And traversing the representative controls contained in the control set in sequence, and adding the representative control combinations which are related in pairs to the related control set according to the abscissa and the preset priority difference.

Optionally, the step of sequentially traversing the representative controls included in the control set sequentially, and adding the representative control combinations associated with each other to the associated control set specifically includes:

traversing the representative controls contained in the control set in sequence, and adding the ith representative control into the associated control set;

traversing the rest of the representing controls after the ith representing control, and sequentially comparing the horizontal coordinate difference value of the target position of the rest of the representing controls after the ith representing control and the rest of the representing controls after the ith representing control with a preset priority difference value of a front page;

If the difference value of the horizontal coordinate between the rest representative control and the ith representative control and the difference value of the preset priority are lower than the corresponding preset threshold, the rest corresponding representative control is used as the associated control of the ith representative control and added into the associated control set, otherwise, i=i+1 is added, and the representative control contained in the control set is traversed.

Optionally, the association unit is further configured to:

If the ith representing component exists in the associated control set, after i=i+1, traversing the representing control contained in the control set again.

Optionally, the method further comprises:

and the alarm unit is used for packaging the information of the plurality of problem controls and then sending the information to a target user for alarm.

A third aspect of the present application provides a UI control associated walkthrough device, the device comprising a processor and a memory:

The memory is used for storing program codes and transmitting the program codes to the processor;

the processor is configured to execute the steps of the method for UI control association walkthrough according to the instructions in the program code.

A fourth aspect of the application provides a computer readable storage medium for storing program code for performing the method of the first aspect described above.

From the above technical solutions, the embodiment of the present application has the following advantages:

According to the UI control association walking method, device, equipment and storage medium, element information corresponding to all the controls contained in a front-end page is captured, the controls are classified according to preset labels to form a control set, the control set is further de-duplicated based on the area of each control in the control set, finally the controls are associated in pairs according to coordinates and preset priorities, problem controls in the associated control set are determined according to preset control comparison rules, condition comparison among elements of multiple controls is achieved, more accurate interval walking is provided, manual walking cost is reduced, and walking efficiency is improved.

Drawings

FIG. 1 is a flow chart of a method for UI control association walkthrough in an embodiment of the application;

FIG. 2 is a schematic structural diagram of a UI control association walking device in an embodiment of the application;

Fig. 3 is a schematic structural diagram of a UI control related walking device in an embodiment of the application.

Detailed Description

In order to make the present application better understood by those skilled in the art, the following description will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The application designs a UI control association walking method, a device, equipment and a storage medium, which realize condition comparison among elements of multiple controls, provide more accurate interval walking, reduce manual walking cost and improve walking efficiency.

For ease of understanding, referring to fig. 1, fig. 1 is a flowchart of a method for UI control association walking method in an embodiment of the present application, as shown in fig. 1, specifically:

it should be noted that, for the target page to be searched, the code may be injected into the target page, so as to traverse the target page to capture the element information corresponding to all the controls.

It should be noted that, control classification is performed according to preset labels in the element information corresponding to the capturing controls, so as to form a plurality of control sets.

If the control label is ambiguous or the control elements are difficult to identify, a CSS selector or other query methods can be utilized to perform more accurate query according to other characteristics (such as position, style, content and the like) of the control;

the screenshot of the page can be analyzed by adopting a computer vision technology, and control elements which are difficult to identify by a traditional method are identified;

A machine learning model can be trained to identify and classify the controls, and the model can learn the characteristics of the controls from the labeling data and gradually improve the identification accuracy;

the rules engine may also be constructed or optimized to identify and classify controls according to a complex set of rules and conditions.

Further, the control which is difficult to identify is used as feedback, and the identification algorithm and rule are continuously iterated and optimized.

S3, performing duplicate removal processing on each control set based on the area of each control in the control set;

Specifically:

Step S31, calculating the area of each control in a plurality of control sets;

S33, constructing a relation storage array of a control set, wherein the relation storage array comprises a front control and a rear control group;

Step S36, when the IOU intersection ratio is larger than a preset threshold, assigning the control after the corresponding ith control to a rear control group;

Step S37, taking out the representing control after the duplication removal according to the number of the containing controls in the rear control group, and obtaining that the control set after the duplication removal contains a plurality of representing controls;

the step S37 specifically includes:

When the rear control group contains more than two controls, taking out the control corresponding to the second position in the rear control group as a representative control, otherwise taking out the control corresponding to the first position in the rear control group as the representative control.

It should be noted that, the controls in each control set are calculated and ordered according to the area. And then constructing a relation storage array for storing the association relation between the controls. And identifying the duplicate controls by calculating the IOU value, de-duplicating according to a preset threshold value, and finally determining the representative controls in each set.

IOU (Intersection overUnion) cross ratio is an index for measuring the overlapping degree of two space regions, and is widely applied to image processing, target detection and machine vision. In the UI control association walk method, the IOU is used in the duplication removal process to help identify and merge different elements representing the same control.

The calculating method of the IOU cross ratio comprises the following steps:

given two spatial regions A and B, their intersection region is A.u.B, and their union region is A.u.B. The IOU intersection ratio can be calculated by the following formula:

Where |aΣb| represents the area of the overlapping portion of the areas a and B, and |aΣb| represents the total area covered by a and B.

In computing the IOU of a UI control, the control is typically considered a rectangular region, the area of which can be calculated from the width and height of the control. If the control has an irregular shape, a more complex geometric algorithm may need to be used to determine the area of the intersection and union.

By calculating IOU values between controls, visually overlapping or very close controls may be identified, which may be duplicates caused by different HTML elements or style hierarchies.

When the IOU value exceeds some preset threshold (e.g., 0.5 or higher), the two controls may be considered similar or different manifestations belonging to the same entity, merging them during deduplication.

After deduplication, a representative control needs to be selected from the group of controls with high IOU values. Controls with larger areas or specific features are typically selected as a representation for use in subsequent association and comparison processes.

Through the weight reduction, the number of controls to be processed in the subsequent association step can be reduced, so that the efficiency and accuracy of the association process are improved.

S4, performing pairwise association on the controls contained in the duplicate-removed control set according to a preset priority and coordinates to form a plurality of associated control sets corresponding to the preset labels;

the method comprises the following steps:

Step S41, sorting a plurality of representative controls contained in the control set in descending order according to the ordinate;

Step S42, creating an association control set;

And step S43, sequentially traversing the representative controls contained in the control set, and adding the pairwise association of the representative control combinations to the association control set according to the abscissa and the preset priority difference.

Further, step S43 specifically includes:

Step S431, sequentially traversing the representative controls contained in the control set, and adding the ith representative control into the associated control set;

Step S432, traversing the rest of the representing controls after the ith representing control, and sequentially comparing the horizontal coordinate difference values of the target positions of the rest of the representing controls after the ith representing control and the rest of the representing controls after the ith representing control with preset priority difference values of the front-end page;

Step S433, if there is a difference value between the horizontal coordinates of the other representative control and the i-th representative control and the difference value between the horizontal coordinates and the preset priority are lower than the corresponding preset threshold, the corresponding other representative control is used as the associated control of the i-th representative control and added to the associated control set, otherwise, after i=i+1, the step S431 is executed again.

Further, step S431 further includes:

If the ith representative component already exists in the associated control set, step S431 is re-executed after i=i+1.

It should be noted that, the representative controls after deduplication are sorted in descending order according to the ordinate (y-coordinate). And then creating an association control set, and carrying out pairwise association according to the abscissa and the preset priority difference. In the association process, if the horizontal distance and the priority difference of the two controls are lower than the preset threshold, the two controls are considered to be associated and added to the association control set.

If the difference in horizontal coordinates and the difference in preset priorities of the two controls are both lower than the preset value, this generally means:

the two controls are close to each other in the horizontal direction. A small difference in horizontal coordinates indicates that they are adjacently or closely arranged in position on the page, possibly based on intentional arrangements in design, to maintain the interface's cleanliness and logic;

the control may be visually a continuous part or belong to the same set of functionally associated elements. The smaller difference in horizontal coordinates helps to maintain visual smoothness and consistency;

the preset priority is typically set according to the importance or functional ordering of the controls in the page. If the priority difference of the two controls is small, this indicates that they may be functionally related or belong to the same functional module.

The ordered arrangement and prioritization of the controls helps to maintain consistency of the layout. When the controls meet the condition that the horizontal coordinate difference value and the priority difference value are lower than the preset value, they may follow a certain design specification or layout pattern.

And S5, comparing the associated control sets according to preset labels based on preset control comparison rules to determine problem controls.

It should be noted that, according to a preset control comparison rule, the formed associated control set is compared. These rules may relate to spacing between controls, alignment, etc. After comparison, it is determined which controls have problems and do not meet the design specification.

Further, step S5 further includes:

and S6, after the information of the plurality of problem controls is packaged, the information is sent to a target user for alarming.

It should be noted that, the information of the problem control is packaged and sent to the target user or the development team for alarming. This is typically done by generating a report or highlighting the problem control directly on the user interface.

Referring to fig. 2, fig. 2 is a schematic structural diagram of a UI control related walking device in an embodiment of the application, as shown in fig. 2, specifically:

A capturing unit 201, configured to capture element information corresponding to all controls of the target page;

The classifying unit 202 is configured to classify the element information according to a preset label, and form a plurality of control sets corresponding to the preset label;

The deduplication unit 203 is configured to perform deduplication processing on each control set based on the area of each control in the control set;

Specifically:

calculating the area of each control in a plurality of control sets;

When the IOU intersection ratio is larger than a preset threshold value, assigning the corresponding control after the ith control to a rear control group;

according to the number of the included controls in the rear control group, the representing controls after the duplicate removal are taken out, and a control set after the duplicate removal is obtained to contain a plurality of representing controls;

The method comprises the steps of taking out the representation control after the duplication removal according to the number of the contained controls in the rear control group, and obtaining that the control set after the duplication removal contains a plurality of representation controls specifically comprises the following steps:

The calculating method of the IOU cross ratio comprises the following steps:

The association unit 204 is configured to associate the controls included in the control set after duplication removal according to a preset priority and coordinates, so as to form a plurality of associated control sets corresponding to the preset labels;

further, the association unit 204 is specifically configured to:

sorting a plurality of representative controls contained in the control set in descending order according to the ordinate;

Creating an association control set;

And traversing the representative controls contained in the control set in sequence, and adding the representative control combinations which are associated with each other to the associated control set according to the abscissa and the preset priority difference.

Further, the representative controls contained in the control set are sequentially traversed in sequence, and the representative control combination which is associated with each other is added to the associated control set according to the abscissa and the preset priority difference, specifically:

traversing the rest of the representing controls after the ith representing control, and sequentially comparing the horizontal coordinate difference values of the target positions of the ith representing control and the rest of the representing controls after the ith representing control with the preset priority difference value of the front page;

If the difference value of the horizontal coordinate of the other representative control and the i-th representative control and the difference value of the preset priority are lower than the corresponding preset threshold, the corresponding other representative control is used as the associated control of the i-th representative control and added into the associated control set, otherwise, the representative control contained in the traversing control set is returned after i=i+1.

Further, the association unit 204 is further configured to:

If the ith representative component exists in the associated control set, after i=i+1, traversing the representative control contained in the control set again.

The comparison unit 205 is configured to determine a problem control by comparing the associated control set according to a preset label based on a preset control comparison rule.

Further, the method further comprises the following steps:

and the alarm unit is used for packaging the information of the plurality of problem controls and then sending the information to the target user for alarm.

The embodiment of the application also provides another UI control related walking device, as shown in fig. 3, for convenience of explanation, only the parts related to the embodiment of the application are shown, and specific technical details are not disclosed, please refer to the method parts of the embodiment of the application. The terminal can be any terminal equipment including a mobile phone, a tablet personal computer, a personal digital assistant (English full name: personal DigitalAssistant, english abbreviation: PDA), a sales terminal (English full name: point ofSales, english abbreviation: POS), a vehicle-mounted computer and the like, taking the terminal as an example of the mobile phone:

Fig. 3 is a block diagram showing a part of a structure of a mobile phone related to a terminal provided by an embodiment of the present application. Referring to fig. 3, the mobile phone includes a Radio Frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power source 1090. Those skilled in the art will appreciate that the handset configuration shown in fig. 3 is not limiting of the handset and may include more or fewer components than shown, or may combine certain components, or may be arranged in a different arrangement of components.

The following describes the components of the mobile phone in detail with reference to fig. 3:

The RF circuit 1010 is used for receiving and transmitting signals during a message or a call, and particularly, receives downlink information from a base station, processes the received downlink information in the processor 1080, and transmits uplink data to the base station. Generally, RF circuitry 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (English holly: lowNoiseAmplifier, english abbreviation: LNA), a duplexer, and the like. In addition, the RF circuitry 1010 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to global system for mobile communications (english: global System ofMobile communication, english abbreviation: GSM), general packet Radio Service (english: GENERAL PACKET Radio Service, GPRS), code division multiple access (english: code Division Multiple Access, english abbreviation: CDMA), wideband code division multiple access (english: wideband Code DivisionMultipleAccess, english abbreviation: WCDMA), long term evolution (english: long TermEvolution, english abbreviation: LTE), email, short message Service (english: shortMessaging Service, SMS), etc.

The memory 1020 may be used to store software programs and modules that the processor 1080 performs various functional applications and data processing of the handset by executing the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a storage program area which may store an operating system, an application program required for at least one function (such as a sound playing function, an image playing function, etc.), etc., and a storage data area which may store data created according to the use of the mobile phone (such as audio data, a phonebook, etc.), etc. In addition, memory 1020 may include high-speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state memory device.

The input unit 1030 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the handset. In particular, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also referred to as a touch screen, may collect touch operations thereon or thereabout by a user (e.g., operations of the user on the touch panel 1031 or thereabout using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a predetermined program. Alternatively, the touch panel 1031 may include two parts, a touch detection device and a touch controller. The touch controller receives touch information from the touch detection device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 1080, and can receive and execute commands sent by the processor 1080. Further, the touch panel 1031 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. The input unit 1030 may include other input devices 1032 in addition to the touch panel 1031. In particular, other input devices 1032 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a track ball, a mouse, a joystick, etc.

The display unit 1040 may be used to display information input by a user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041, and optionally, the display panel 1041 may be configured in the form of a Liquid crystal display (english full name: liquid CRYSTAL DISPLAY, english abbreviation: LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, the touch panel 1031 may overlay the display panel 1041, and when the touch panel 1031 detects a touch operation thereon or thereabout, the touch panel is transferred to the processor 1080 to determine a type of touch event, and then the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in fig. 3, the touch panel 1031 and the display panel 1041 are two independent components for implementing the input and output functions of the mobile phone, in some embodiments, the touch panel 1031 and the display panel 1041 may be integrated to implement the input and output functions of the mobile phone.

The handset may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of ambient light, and the proximity sensor may turn off the display panel 1041 and/or the backlight when the mobile phone moves to the ear. The accelerometer sensor can be used for detecting the acceleration in all directions (generally three axes), detecting the gravity and the direction when the accelerometer sensor is static, and can be used for identifying the gesture of a mobile phone (such as transverse and vertical screen switching, related games, magnetometer gesture calibration), vibration identification related functions (such as pedometer and knocking), and other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors which are also configured by the mobile phone are not repeated herein.

Audio circuitry 1060, a speaker 1061, and a microphone 1062 may provide an audio interface between a user and a cell phone. The audio circuit 1060 may convert the received audio data into an electrical signal, transmit the electrical signal to the speaker 1061, and convert the electrical signal to a sound signal for output by the speaker 1061, while the microphone 1062 converts the collected sound signal into an electrical signal, receives the electrical signal from the audio circuit 1060, converts the electrical signal into audio data, outputs the audio data to the processor 1080 for processing, sends the audio data to another mobile phone, for example, via the RF circuit 1010, or outputs the audio data to the memory 1020 for further processing.

WiFi belongs to a short-distance wireless transmission technology, and a mobile phone can help a user to send and receive emails, browse webpages, access streaming media and the like through a WiFi module 1070, so that wireless broadband Internet access is provided for the user. Although fig. 3 shows a WiFi module 1070, it is understood that it does not belong to the necessary constitution of the handset, and can be omitted entirely as required within the scope of not changing the essence of the invention.

Processor 1080 is the control center of the handset, connects the various parts of the entire handset using various interfaces and lines, and performs various functions and processes of the handset by running or executing software programs and/or modules stored in memory 1020, and invoking data stored in memory 1020, thereby performing overall monitoring of the handset. Optionally, processor 1080 may include one or more processing units, and preferably, processor 1080 may integrate an application processor and a modem processor, wherein the application processor primarily processes operating systems, user interfaces, application programs, etc., and the modem processor primarily processes wireless communications. It will be appreciated that the modem processor described above may not be integrated into processor 1080.

The handset further includes a power source 1090 (e.g., a battery) for powering the various components, which may preferably be logically connected to the processor 1080 by a power management system, such as to provide for managing charging, discharging, and power consumption by the power management system.

Although not shown, the mobile phone may further include a camera, a bluetooth module, etc., which will not be described herein.

In an embodiment of the present application, the processor 1080 included in the terminal further has the following functions:

Specifically:

Step S31, calculating the area of each control in a plurality of control sets;

the step S37 specifically includes:

The calculating method of the IOU cross ratio comprises the following steps:

the method comprises the following steps:

Step S42, creating an association control set;

Further, step S43 specifically includes:

Further, step S431 further includes:

Further, step S5 further includes:

The embodiment of the application also provides a computer readable storage medium for storing program code for executing any one of the UI control association walking methods described in the foregoing embodiments.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

The terms "first," "second," "third," "fourth," and the like in the description of the application and in the above figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented, for example, in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "and/or" is used to describe an association relationship of an associated object, and indicates that three relationships may exist, for example, "a and/or B" may indicate that only a exists, only B exists, and three cases of a and B exist simultaneously, where a and B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one of a, b or c may represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. The storage medium includes various media capable of storing program codes, such as a U disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RandomAccess Memory, RAM), a magnetic disk, or an optical disk.

While the application has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art that the foregoing embodiments may be modified or equivalents may be substituted for some of the features thereof, and that the modifications or substitutions do not depart from the spirit and scope of the embodiments of the application.

Claims

1. The UI control association walk-through method is characterized by comprising the following steps of:

2. The UI control association walk-through method according to claim 1, wherein the step S3 specifically includes:

step S31, calculating the areas of the controls in the control sets;

3. The UI control association walk-through method according to claim 2, wherein the step S37 specifically is:

4. The UI control association walk-through method according to claim 2, wherein the step S4 specifically is:

Step S42, creating an association control set;

5. The UI control association walking method as claimed in claim 4, wherein the step S43 specifically includes:

6. The UI control association walkthrough method of claim 5, wherein the step S431 further includes:

7. The UI control association walk-through method according to claim 1, wherein after the step S5, further comprises:

8. A UI control association walkthrough device, comprising:

9. A UI control associated walkthrough device, the device comprising a processor and a memory:

the processor is configured to perform the UI control association walkthrough method of any one of claims 1-7 according to instructions in the program code.

10. A computer readable storage medium for storing program code for performing the UI control association walkthrough method of any one of claims 1-7.