CN114840107B - Sketch data reuse and scene sketch auxiliary construction method and system - Google Patents

Abstract

The invention discloses a method and a system for the reuse of sketch data and the auxiliary construction of a scene sketch, which belong to the field of computer vision, and extract and reuse the existing sketch materials based on sketch restoration, sketch-based image retrieval and other sketch intelligent processing technologies by utilizing the advantages of efficient information representation of sketch interaction; and assisting a user in constructing a scene sketch by using the extracted sketch materials and sketch data stored in a database, so that the method is used for subsequent applications such as sketch retrieval, video positioning and the like.

Description

Method and system for reuse of sketch data and auxiliary construction of scene sketch

technical field

The invention belongs to the field of computer vision, and in particular relates to a sketch data reuse and scene sketch auxiliary construction method and system.

Background technique

Visual media applications based on sketch interaction have always been a research hotspot in the fields of human-computer interaction, computer vision and multimedia. How to optimize the basic processing of sketch data and improve the efficiency of visual media applications based on sketches are the key issues of research. Sketch interaction is widely used in all aspects of life and work, including drawing, shorthand notes, document annotation, web user interface (UI) and concept design in the Internet industry, animation and film and television production in the film and animation industry, etc. In recent years, research and application related to sketch interaction have attracted widespread attention in both industry and academia. One of the important reasons is the vigorous development of touch-screen hardware devices (such as Microsoft Surface series touch laptops, Apple Inc. Apple pencil stylus, etc.). In the era of artificial intelligence, on the one hand, it is more convenient for users to obtain sketch data; on the other hand, the performance of sketch data algorithms based on deep learning technology continues to improve. Applications and tasks based on sketch interaction have also gained unprecedented popularity.

In terms of sketch interaction tasks and sketch data processing, great progress has been made using sketch data processing technologies based on deep learning, such as sketch recognition, sketch-based image retrieval (sketch-based image retrieval, SBIR), sketch-based image generation (sketch-based image generation, SBIG), sketch parsing (sketch parsing), sketch-based video summarization (sketch-based videosummarization), etc. Some new sketch interaction tasks are also proposed, such as sketch generation, model generation based on sketch interaction, sketch abstraction, sketch-based image editing, sketch segmentation, etc. SketchGAN (References: Liu, Fang, Xiaoming Deng, Yu-Kun Lai, Yong-Jin Liu, Cuixia Ma, and Hongan Wang."Sketchgan: Joint sketch completion and recognition with generative adversarial network."In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pp.5830-5839.2019.) proposed a sketch completion and recognition method based on a generative adversarial network, which effectively generatively completes and repairs incomplete sketch data, and utilizes the completed sketch The data is used for sketch recognition to improve the recognition accuracy of current mainstream sketch recognition algorithms for incomplete sketches.

For the research on sketch interaction, we should develop towards fine-grained methods in response to the needs of practical applications. Specifically, relative to whole-based tasks such as sketch recognition, some fine-grained sketch interaction tasks have been proposed in recent years. At present, most of the sketch-based image retrieval related technologies are based on the premise of instance-level and category-level retrieval, that is, the input sketch and the image object to be retrieved are both single objects; Consistency in category is correct retrieval. Sketch Me that Shoe (References: Yu, Qian, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M. Hospedales, and Chen-Change Loy."Sketch me that shoe."In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.799-807.2016.) and other researches on instance-level sketch-based image retrieval not only require the retrieved image objects to be consistent with the input query sketches in category, but also require their shape, action, Details like directions are similar.

The unique high abstraction, intuition, and simplicity of sketches make sketches widely used in various fields such as human-computer interaction, computer vision, multimedia, and computer graphics. From the 1960s to the present, with the continuous improvement of data processing technology, the research and application of sketches have been continuously optimized. In the field of intelligent human-computer interaction, although many scholars have applied sketches to the research of visual media (images, videos, animations, etc.), there are still many sketch data processing and in the field of visual media in the application process. The key technology for practical application has not been effectively solved. In sketch-based interactive applications, a crucial issue is the acquisition of sketch data. Although several important sketch databases have been proposed one after another, sketch data is still relatively rare and difficult to obtain in large quantities. For non-professional painters, it is not easy to draw the exact outline and appropriate details of the objects in their minds, and it often takes a lot of time, which also affects the large number of sketch interactions in real life to a certain extent Apps get in the way.

Contents of the invention

In order to overcome the existing sketch interaction-based methods and applications that have a low reuse rate of sketch data and difficulty in inputting sketches by users, the present invention proposes a sketch data reuse and scene sketch auxiliary construction method and system, which utilizes the efficient information of sketch interaction Based on the advantages of representation, based on intelligent sketch processing technologies such as sketch repair and sketch-based image retrieval, the existing sketch materials are extracted and reused; the extracted sketch materials and the sketch data stored in the database are used to assist users in the construction of scene sketches, thereby It is used for subsequent applications such as sketch retrieval and video positioning.

The technical solution adopted by the present invention to solve the problems of the technologies described above is:

A sketch data reuse and scene sketch auxiliary construction method, the steps of which include:

Extract the sketch material selected by the user in the existing scene sketch and save it in the database, and automatically repair the incomplete sketch material;

Use existing sketch materials in the database or temporarily draw sketches to construct scene sketches or scene sketch sequences;

Perform image retrieval based on the constructed scene sketch to find the image most similar to the scene sketch;

Video localization is performed based on the constructed scene sketch sequence, and the video segment most similar to the scene sketch sequence is found.

Further, the sketch material in the database is used to construct the sketch of the scene by one or several operations of dragging, zooming and rotating.

Further, the image retrieval method is based on the SceneSketcher method and removes the class-sensitive intersection-over-union ratio loss part of the method.

Furthermore, an end-to-end image search model based on the image search method is constructed, and the image search model improves image search accuracy through training.

Further, when performing image retrieval based on the constructed scene sketch, the most similar image to the scene sketch is found by adjusting one or more of the pose, size, direction, and position of the scene sketch.

Further, the step of performing video positioning includes: extracting scene sketch visual features based on convolutional neural network; extracting video action features based on optical flow method; extracting video features based on C3D network; according to the scene sketch visual features, video action features and video Features utilize regression algorithms for video localization.

Further, when performing video positioning based on the constructed scene sketch sequence, the most similar video segment to the scene sketch sequence is found by adjusting one or more of the pose, size, direction, and position of the scene sketch.

A sketch data reuse and scene sketch auxiliary construction system, including a top-level interactive interface and a bottom-level data processing algorithm module. The interactive interface includes a sketch data extraction page, a scene sketch construction and image retrieval page, and a scene sketch-based video positioning page. and a page switching button for switching pages;

The sketch data extraction page includes: scene sketch browsing panel and sketch data extraction panel;

The scene sketch construction and image retrieval page includes: scene sketch construction canvas, scene sketch editing tool panel, sketch material browsing and selection panel, and image retrieval result display panel;

The video positioning page based on the scene sketch includes: the scene sketch construction canvas, the scene sketch editing tool panel, the sketch material browsing and selection panel, the sketch sequence display panel and the video positioning result display panel;

Among them, the scene sketch browse panel is used to display the current scene sketch, frame-select sketch object elements and fit the scene sketch; the sketch data extraction panel is used to extract the frame-selected sketch objects, and automatically repair the incomplete sketch materials; the scene sketch construction canvas It is used to support dragging, zooming and rotating operations on the scene sketch; the scene sketch editing tool panel is used to provide tools for drawing and editing sketch elements; the sketch material browsing and selection panel is used to display the list of sketch materials recommended by the system; The image retrieval result display panel is used to display the image retrieval results; the sketch sequence display panel is used to display the sketch sequence; the video positioning result display panel is used to display the positioned video clips;

The data processing algorithm module is used to realize all interactions and data processing in the above-mentioned interactive interface.

Further, the scene sketch editing tool panel provides brush, eraser, undo, delete, clear and retrieve tools.

Further, the data processing algorithm module contains an end-to-end image search model based on the image search method, and the image search model improves image search accuracy through training; the image retrieval method is based on the SceneSketcher method and removes the category-sensitive cross-merge ratio in the method Lost part of the resulting method.

Compared with prior art, the beneficial effect of the present invention is:

1. The present invention first repairs the sketch object selected by the user, then assists the user to draw a scene sketch, and performs image retrieval according to the scene sketch.

2. The present invention supports personalized sketch interaction: 1) The user can obtain more accurate and ideal retrieval results by reselecting sketch materials and adjusting the posture and position of sketch objects. Moreover, the system also provides sketching tools such as Pen and Eraser to support users to manually extract objects in the scene sketch. 2) Users can use the sketch materials in personal favorites or public databases to construct scene sketches, and can also use the created scene sketches to search for images. 3) The system allows the user to reselect the sketch material, adjust each object in the scene sketch to optimize the current scene sketch to retrieve a more expected image or locate a more accurate video clip.

Description of drawings

FIG. 1 is an interactive interface diagram of a sketch data reuse and scene sketch auxiliary construction system in Embodiment 1 of the present invention.

Fig. 2 is a flow chart of sketch data reuse and scene sketch assistant construction in Embodiment 2 of the present invention.

FIG. 3 is an interactive interface diagram of video positioning based on a sketch sequence in Embodiment 3 of the present invention.

Figure 4 is a diagram of the baseline system interaction interface.

Fig. 5 is a box plot of each indicator in the questionnaire survey of the system of the present invention and the baseline system.

Fig. 6 is an example of a sketch constructed by the tested user in the experiment of the present invention.

Detailed ways

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail in conjunction with the accompanying drawings.

Example 1

This embodiment proposes a system and method for sketch data reuse and scene sketch auxiliary construction, which is aimed at the problems of sketch data reuse and scene sketch auxiliary construction. Technical solutions for positioning.

A kind of sketch data reuse and scene sketch auxiliary construction system proposed in this embodiment includes a top-level interactive interface and a bottom-level data processing module. An example in the embodiment), which includes: a draft data extraction page, a scene sketch construction and image retrieval page, and a scene sketch-based video positioning page, and the user can switch between the three pages through the page switching button. The upper picture in Figure 1 is the sketch data extraction page, which includes ②scene sketch browsing panel and ③sketch data extraction panel; the lower picture is the scene sketch construction and image retrieval page, which includes ④scene sketch construction canvas, ⑤scene sketch editing tool panel , ⑥ sketch material browsing and selection panel and ⑦ image retrieval result display panel. The underlying data processing module is used to process the related data interaction operation and data calculation of the top interface.

Among them, ①page switching button: switch directly on the page; ②scene sketch browsing panel: displays the current scene sketch, at this time, the user can draw a rectangular frame to frame the interested sketch object elements, and can use this panel The left and right arrows of the left and right arrows switch between different scene sketches (as shown in the middle picture in Figure 1); ③Sketch data extraction panel: When the objects in the scene sketch are mutually occluded or overlapped, the system supports automatic repair of the sketch objects and saves them to the user ④Scene Sketch Construction Canvas: Supports related operations for users to build scene sketches (including dragging, zooming, rotating, etc.); ⑤Scene Sketch Editing Tool Panel: Contains Pen( Brush), Eraser (eraser), Undo (undo), Delete (delete, that is, delete a single object), Clear (clear, that is, clear all objects) and Retrieval (retrieval) and other buttons, support the drawing and editing of sketch elements; Users can click the Retrieval button to retrieve images based on the currently constructed scene sketch; ⑥Sketch material browsing and selection panel: display the list of sketch material recommended by the system; ⑦Image retrieval result display panel: display image retrieval results, among which the best match is displayed on the top Larger version of the image, with other images similar to the currently built scene sketch listed below. This system is based on the website architecture and can adapt to various devices such as mobile phones, laptops, and digital whiteboards. It also supports the use of mouse and fingers or pens on touch screen devices for input.

A sketch data reuse and scene sketch auxiliary construction method proposed in this embodiment is based on the above-mentioned system, which includes the following steps:

S1: Extract objects from existing scene sketches: This system supports users to interactively extract and save interested sketch materials from scene sketches. When the user selects the object of interest from the existing scene sketch, the system can repair the incomplete material. This system optimizes the sketch interaction mode as follows: First, when the user browses the scene sketch (such as a sketch pictorial or browses a storyboard), the scene sketch browsing panel displays the current scene sketch, and the user can select the interested sketch object elements. The elements in the scene sketch generally have the phenomenon of mutual occlusion or overlapping. This system can automatically repair the sketch and store it in the user's personal favorites (Favorits); when the user is not satisfied with the effect of the automatically repaired sketch, he can use this system Provides sketching tools such as pen and eraser to manually modify objects in the scene sketch. Secondly, in the face of ordinary users who do not have professional drawing skills, instead of requiring users to draw a complete sketch in their minds by hand, the system provides some drawing tips or suggestions, or utilizes the existing sketches in the user's personal favorites and the system's public database Materials help users quickly build new sketches. Again, in addition to the sketch input tools such as paintbrush and eraser provided by the usual pen-style interactive interface, more interactive editing tools can be provided to assist users in drawing sketches. Finally, the system can support more flexible sketching of multiple objects, and at the same time facilitate the user to draw more detailed information (such as the direction of the object, the background, etc.).

S2: Scene sketch construction: This system supports users to reuse the existing sketch materials in the database, and quickly and efficiently construct scene sketches by dragging, zooming and rotating the sketch materials.

S3: Image retrieval based on the scene sketch: This system provides the function of image retrieval based on the scene sketch constructed by the user, and the image can be retrieved according to the currently constructed scene sketch by clicking the Retrieval button. The user can adjust the posture, size, direction and position of the sketch material object by dragging, zooming, rotating and other operations on the scene sketch construction canvas, and constantly adjust the currently constructed scene sketch (the scene sketches in the middle picture of Figure 1 ), and use the tools in the scene sketch editing tool panel to draw and edit the sketch elements to search for images that are more in line with expectations. The image retrieval method used in this system is based on SceneSketcher (reference: Liu, Fang, et al. "Scenesketcher: Fine-grained image retrieval with scene sketches." European Conference on Computer Vision. Springer, Cham, 2020.) Removed the original method The category-wise IoU loss (category-wise IoU loss) part in the method builds an end-to-end sketch-based image retrieval model to speed up the training speed of the model while ensuring that the accuracy of the retrieval model does not drop significantly.

S4: Video positioning based on the scene sketch sequence: the present invention provides the function of video positioning based on the scene sketch sequence constructed by the user; the user can continuously adjust the currently constructed scene sketch, specifically, on the scene sketch construction canvas by dragging, Scale, rotate and other operations, adjust the pose, size, direction and position of the sketch material object, so as to locate the video clip more in line with expectations. The video localization methods used include extracting visual features of scene sketches based on convolutional neural network, extracting action features based on optical flow method, and based on C3D network (reference: Tran, Du, et al."Learning spatial temporal features with 3d convolutional networks."Proceedings of theIEEE international conference on computer vision.2015.) Extract video features, and finally use regression algorithm for video positioning.

Example 2

This embodiment provides a specific application example of providing a sketch data reuse and scene sketch auxiliary construction system and method, as shown in Figure 2, the method includes the following steps:

S1: When the user browses the scene sketch, the user selects the interested sketch object, and the system repairs the sketch and stores it in the user's personal favorites (see (1) in Figure 2). Occasionally, when the algorithm is not effective, the system Provides a sketch drawing tool to support users to manually extract objects in the scene sketch.

S2: The user uses the sketch material in personal favorites or public databases to construct the scene sketch (see (2) in Figure 2), and the user can also use the created scene sketch to search for images (see (3) in Figure 2).

S3: The system allows the user to reselect the sketch material, adjust each object in the scene sketch to optimize the current scene sketch (see (4) in Figure 2) to retrieve an image that is more in line with expectations (see (5) in Figure 2).

Example 3

This embodiment proposes a sketch data reuse and scene sketch auxiliary construction system and method, which is based on the sketch for video positioning, and the video positioning page based on the scene sketch is used, as shown in Figure 3, the page includes: ① Scene sketch construction Canvas; ②Scene sketch editing tool panel; ③Sketch material browsing and selection panel; ④Sketch sequence display panel; ⑤Video positioning result display panel. Among them, ①②③ are the same as those in Embodiment 1, ④the sketch sequence display panel is used to display the sketch sequence, and ⑤the video positioning result display panel is used to display the positioned video clips. Video targeting consists of three steps:

S1: Scene sketch construction. Users build scene sketches based on the target segments of video positioning. This system allows users to reuse existing sketch materials to speed up the construction process through operations such as dragging, moving, zooming, and rotating.

S2: Sketch sequence generation. Synthesize a sketch sequence by repeating step S1, which represents the keyframe of the target segment to be located in the video, and the change of elements in the sketch sequence represents the motion of the corresponding object in the video.

S3: Video positioning. The system finds the most similar segments in the video based on the sketch sequence created by the user.

The following is an experimental test of a sketch data reuse and scene sketch auxiliary construction system proposed by the present invention:

A baseline system is now designed as a control system to evaluate the system of the present invention. The system uses the scene sketch construction task, questionnaire survey and interview methods for research. Compared with the system of the present invention, the baseline system does not have scene sketch material extraction and reuse functions. As mentioned above, the existing mainstream sketch interactive application systems generally require users to draw complete sketches as input; similarly, in the baseline system, users need to draw sketch objects in the scene by hand to construct a scene sketch to retrieve the target image. Figure 4 shows the baseline system interface, including: ①Scene sketch construction canvas; ②Scene sketch editing tool panel, which also includes Pen, Eraser, Undo, Delete, Clear, and Retrieval from left to right; ③Image retrieval result display panel.

(1) Participants

16 participants were recruited, 8 males and 8 females. The average age of the tested users is 26.313 years old, and the variance is 3.260 (male: average age=27.250, variance=3.845; female: average age=25.375, variance=2.446). Before the test, this experiment explained the purpose and process of the experiment to the tested users in detail, and let the participants familiarize themselves with the system first. Each experiment lasted about half an hour, and none of the participants had professional sketching skills.

(2) Experimental tasks and steps

The tested users need to use the system of the present invention and the reference system to complete the experimental tasks respectively. In each experiment, a color image was randomly shown to the test users as a target (the display time was 2 seconds), and the test users were required to remember the key features of the image during the display time, and then according to the impression in their minds, use this The invention system or the baseline system constructs a scene sketch, and the system performs image retrieval from the database according to the scene sketch constructed by the tested user. The baseline system only provides the tested users with Pen and Eraser tools to draw sketches; the system of the present invention allows the tested users to reuse the sketch objects in the sketch material library to assist the construction process.

Each test user conducted 6 experiments (4 of which were to construct scene sketches given the target image, and 2 free to draw); the experiment collected 96 scene sketches drawn by 16 test users.

(3) Questionnaire survey

When the tested users use the system of the present invention and the benchmark system to complete the test, a questionnaire survey is conducted on the tested users to quantitatively evaluate the tested user experience. The volume title of the questionnaire covers the usability of the system (usability metric for user experience, UMUX, reference: FINSTAD K.The usability metric for user experience[J].Interacting with Computers,2010,22(5):323-327 .), TAM Technology Acceptance Model (Perceived Utility (PU) and Perceived Ease of Use (PEOU), References: DAVIS F D, BAGOZZI R P, WARSHAW P R. User acceptance of computer technology: a comparison of two theoretical models[ J].Management science,1989,35(8):982-1003.), and AI interface issues.

UMUX consists of four metrics:

Effectiveness: the function (of the current system) meets the user's requirements;

Satisfaction: using (the current system) is a frustrating experience;

Overall: (the current system) is easy to use;

Efficiency: Users have to spend too much time making corrections (on the current system).

The TAM technology acceptance model mainly evaluates the user's acceptance of technology. This experiment tests the TAM metrics in the user scene sketch construction phase and the image retrieval phase respectively, which contains two aspects: Perceived Usefulness (PU) and Perceived Ease of Use (PEOU).

There are four indicators in PEOU:

Easy to learn: In the stage of (scene sketch construction/image retrieval), the operation of the system is easy to understand;

Easy to use: the system is easy to use during the (scene sketch construction/image retrieval) stage;

Comfortable: During the (scene sketch construction/image retrieval) phase, the system can be used without significant cognitive load;

Communicative: In the (scene sketch construction/image retrieval) stage, the system knows what the user wants.

There are three indicators in PU:

Useful: In the (scene sketch construction/image retrieval) stage, the system can provide assistance;

Time-saving: In the (scene sketch construction/image retrieval) stage, the system saves time;

Fulfilling: In the (scene sketch construction/image retrieval) phase, the system completes tasks very accurately.

In addition, this experiment also introduces three commonly used indicators in AI user interface (references: OH C, SONG J, CHOI J, et al. I lead, you help but only with enough details: Understanding user experience of co -creation with artificial intelligence[C]//Proceedings of the 2018CHI Conference on Human Factors in Computing Systems.2018:1-13.):

Controllability;

Creativity;

Degree of Freedom.

Finally, this experiment introduces the Net Promoter Score (NPS) indicator, which is to investigate whether users are willing to recommend the system to others.

When each test user completes all the experimental tasks, they need to fill in the Likert scale of 0-7 points for scoring; for the net promoter score, the test users score between 0-10.

(4) Interview

In order to understand the user experience in the system of the present invention in more detail, this experiment conducted semi-structured interviews with the tested users. In the interviews, the experiment asked the test users about their thoughts on the two different systems and their experiences using sketches as a means of human-computer interaction. Each interview lasts about half an hour.

(5) Experimental results

This experiment carried out quantitative and qualitative analysis on the system of the present invention and the baseline system respectively based on the data in the questionnaire survey and user interviews.

(5.1) Quantitative analysis

In this experiment, a paired t-test was used to test whether there was a significant difference between the index scores of the inventive system and the baseline system. In this experiment, the significance level is set at 0.05, that is, if p<0.05, it is considered that there is a significant difference between the two. Fig. 5 is the box diagram of each index in the questionnaire survey of the system of the present invention and the baseline system, as can be seen, except Satisfaction, Easy to Learn, Controllability and Degree of Freedom four indexes, other indexes of the system of the present invention and the baseline system all exist Significant differences.

Usability measurement of user experience: on the three indicators of effectiveness (t=3.60, p<.01), efficiency (t=5.16, p<.01), and overall rating (t=-3.06, p<.01), the present invention There are significant differences in the scores of the system and the baseline system, and the index scores of the system of the present invention are better than the baseline system. The scores of the system of the present invention and the baseline system on the satisfaction index are (M=4.25, SD=1.48) and (M=3.50, SD=1.79), respectively, and there is no significant difference in the satisfaction index (t=1.44, p>.05).

Perceived Ease of Use (PEOU): There is no significant difference between the system of the present invention and the baseline system on the easy to learn (t=-0.22, p>.05) index, that is, the tested users think that the system of the present invention (M=6.00, SD =1.10) and the baseline system (M=6.06, SD=.68) were both easy to learn. The system of the present invention found significance on the ease of use index score (t=3.88, p<.01): the tested users scored (M=6.19, SD=.98) on the ease of use index for the system of the present invention significantly Higher than the baseline system score (M=5.00, SD=1.32). Significance also exists in comfortability (t=3.67, p<.01), and the score of the system of the present invention (M=5.75, SD=.77) is better than that of the baseline system (M=4.38, SD=1.20). In addition, the score (M=5.81, SD=.98) of the inventive system on the communicative (t=4.58, p<.01) index was also significantly higher than that of the baseline system (M=4.25, SD=1.06).

Perceived usefulness (PU): For the usefulness indicator (t=5.23, p<.01), the score of the inventive system (t=5.23, p<.01) is higher than that of the baseline system (M=4.00, SD=1.21) high. Also, the inventive system (M=6.13, SD=.89) was more time-saving (t=7.67, p<.01) than the baseline system (M=3.25, SD=1.13). Moreover, there are also significant differences in the scores of the two systems on the fulfilling (t=5.57, p<.01) index, which are the system of the present invention (M=5.81, SD=.66), the baseline system (M=3.75, SD = 1.29).

AI user interface commonly used indicators: there is a significant difference between the score of the inventive system and the baseline system on the indicator of creativity (t=-4.14, p<.01), and the score of the inventive system (M=5.13, SD=.89) is higher than that of the baseline The system scored low (M=6.13, SD=.89), which also partly indicates that material reuse, while facilitating the construction of scene sketches, limits user creativity. In addition, the scores of the two systems on controllability (t=1.86, p>.05) and degree offreedom (t=0.64, p>.05) are similar, and there is no significance.

Net Promoter Score (NPS): The NPS of the system of the present invention is divided into 93.75\%, which means that 15 tested users scored above 9 points; the Net Promoter Score of the baseline system is -37.5\%, which means that only 1 One of the tested users scored above 9 points, and 8 users scored 7 or 8 points.

(5.2) Qualitative Analysis

In qualitative analysis, this experiment aims to understand users' subjective thoughts, comments and suggestions when using the inventive system and the baseline system. Figure 6 shows an example of some scene sketches constructed by the tested users in the user experiment and the corresponding image retrieval targets. It can be seen that the sketches constructed by users based on the system (SketchMaker) of the present invention are of higher quality. This experiment summarizes the more common and important views of the test users in the interviews:

The system user interface of the present invention is easy to use;

The system user interface of the present invention is more friendly and uses efficient inspiration and creativity;

In the interviews, this experiment found that users have different views on whether the system can enhance creativity, but users are more inclined to use the system of the present invention when retrieving complex images.

Although the present invention has been disclosed as above with the embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent replacements to the technical solutions of the present invention by those of ordinary skill in the art shall fall within the protection scope of the present invention. The scope of protection of the invention is defined by the claims.

Claims (9)

1. The method for reuse of the sketch data and auxiliary construction of the scene sketch is characterized by comprising the following steps:

extracting sketch materials selected by a user from the existing scene sketch, storing the sketch materials into a database, and automatically repairing incomplete sketch materials in the sketch materials;

constructing a scene sketch or a scene sketch sequence by utilizing the existing sketch materials in the database or temporarily drawing the sketch;

performing image retrieval based on the constructed scene sketch, and finding out an image most similar to the scene sketch;

video positioning is carried out based on the constructed scene sketch sequence, and video fragments most similar to the scene sketch sequence are found; the step of performing video localization includes: extracting visual features of a scene sketch based on a convolutional neural network; extracting video action features based on an optical flow method; extracting video features based on a C3D network; and carrying out video positioning by using a regression algorithm according to the visual features of the scene sketch, the video action features and the video features.

2. The method of claim 1, wherein the scene sketch is constructed by one or more of dragging, zooming, and rotating using sketch material already in the database.

3. The method of claim 1, wherein the image retrieval method is a method based on a sceneskeyer method and wherein the class-sensitive cross-ratio loss portion of the method is removed.

4. The method of claim 1, wherein an end-to-end image search model based on an image search method is constructed, the image search model improving image search accuracy and speed through training.

5. The method of claim 1, wherein the image most similar to the scene sketch is found by adjusting one or more of a pose, a size, a direction, and a position of the scene sketch when performing image retrieval based on the constructed scene sketch.

6. The method of claim 1, wherein in video localization based on the constructed sequence of scene sketches, video clips most similar to the sequence of scene sketches are found by adjusting one or more of the pose, size, direction, and position of the scene sketches.

7. A sketch data reuse and scene sketch auxiliary construction system for implementing the method according to any of claims 1-6, characterized by comprising a top layer interactive interface and a bottom layer data processing algorithm module, the interactive interface comprising a sketch data extraction page, a scene sketch construction and image retrieval page and a scene sketch based video positioning page, and a page switching button for switching pages;

the sketch data extraction page includes: a scene sketch browsing panel and a sketch data extraction panel;

the scene sketch construction and image retrieval page comprises: a scene sketch construction canvas, a scene sketch editing tool panel, a sketch material browsing and selecting panel and an image retrieval result display panel;

the video positioning page based on the scene sketch comprises: the scene sketch construction canvas, the scene sketch editing tool panel, the sketch material browsing and selecting panel, the sketch sequence display panel and the video positioning result display panel;

the scene sketch browsing panel is used for displaying a current scene sketch, selecting sketch object elements and cutting the scene sketch; the sketch data extraction panel is used for extracting sketch objects selected by a frame and automatically repairing incomplete sketch materials; the scene sketch construction canvas is used for supporting dragging, zooming and rotating operations for constructing the scene sketch; the scene sketch editing tool panel is used for providing tools for drawing and editing sketch elements; the sketch material browsing and selecting panel is used for displaying a sketch material list recommended by the system; the image retrieval result display panel is used for displaying image retrieval results; the sketch sequence display panel is used for displaying sketch sequences; the video positioning result display panel is used for displaying the positioned video clips;

the data processing algorithm module is used for realizing all interactions and data processing in the interaction interface.

8. The system of claim 7, wherein the scene sketch editing tool panel provides a brush, eraser, undo, delete, purge, and retrieve tool.

9. The system of claim 7, wherein the data processing algorithm module comprises an end-to-end image search model based on an image search method that improves image search accuracy and speed through training; the image retrieval method is a method based on a SceneSkeyer method and obtained by removing a category-sensitive cross ratio loss part in the method.