KR100711950B1 - Real-time tracking of objects of interest using hybrid optics and virtual zooming devices - Google Patents

Abstract

The video processing system tracks an object of interest using a hybrid combination of (i) pan-tilt-zoom (PTZ) camera and (ii) virtual zooming of the image generated by the camera. The object of interest 22-k is initially detected within the image 40 generated by the camera 18. Optical zooming operation 34 then adjusts the pan and tilt settings to frame the object of interest 22-k, and on the object of interest 22-k until one or more specified stop criteria are satisfied. Zoom in. Virtual zooming operation 36 processes the resulting optically-zipped image 44 to identify and extract a particular region of interest 47, and then virtually-zipped image 46. Interpolate the extracted region of interest to generate. The specified stop criterion may indicate, for example, that the optical zooming continues until the object of interest 22-k occupies a fixed or dynamic percentage of the resulting optically-zipped image. Can be.

PTZ camera, virtual zooming, stop criteria

Description

Real-time tracking of an object of interest using a hybrid optical and virtual zooming mechanism}

FIELD OF THE INVENTION The present invention generally relates to the field of video signal processing, and more particularly to techniques for tracking people or other objects of interest using a video camera such that a desired video output is achieved.

Tracking people or other objects of interest is an important aspect of video camera based systems such as video conferencing systems and video surveillance systems. For example, in a video conferencing system, it is often desirable to frame the head and shoulders of a particular conference participant in the resulting output video signal, while in a video surveillance system, it may be necessary to enter a restricted area monitored by the system. It may be desirable to frame the whole body of a person coming or going, for example.

The systems generally use one of two distinct approaches to perform tracking of an object of interest. The first approach uses a pan-tilt-zoom (PTZ) camera that allows the system to position and optically zoom the camera to perform the tracking task. The problem associated with this approach is that in some cases the tracking device is not strong enough to abruptly change the position of the object of interest. This may be due to the fact that the camera is often zoomed-in too far to respond to the sudden change. For example, in a video conferencing system, it is common for a participant to move out of their seat, for example, to lean forward or backward, or to one side or the other. If the PTZ camera is zoomed in too far on a particular participant, the relatively small movement of the participant will prevent the PTZ camera from tracking the participant and will cause the viewer to confuse the viewer of the resulting output video signal. It requires re-tracking.

) 또는 전자 줌(Aelectronic zoom

)으로 언급된다. 상기 접근법에서, 하나 이상의 카메라들로부터의 비디오 정보는 전자적으로 처리되어서 관심있는 물체는 상기 물체가 어떤 특정 카메라의 시야의 중심에 있지 않을 수 있다는 사실에도 불구하고 상기 출력 비디오 신호내의 바람직한 구성에서 보이게 되도록 전자적으로 처리된다. U.S. 특허 제 5,187,574 호는 상기 접근법의 예를 개시하는데, 여기서 도착하는 손님의 영상은 감시 시스템의 고정된 텔레비젼 카메라에 의해 픽업된다. 상기 영상은 손님의 머리 부분이 항상 상기 모니터 스크린의 중앙에서 디스플레이되도록 보장하기 위해 검출, 추출 (extraction), 및 보간(interpolation) 동작을 사용하여 처리된다. 상기 접근법은 상기 비디오 출력이 팬(pan), 틸트(tilt), 또는 줌(zoom) 동작의 필요없이 바람직한 형태를 갖도록, 예를 들어, 관심있는 물체의 중심에 있도록 보장한다. 결과로서, 상기 접근법은 고정된 카메라로 동작할 수 있으며, 이것은 상기 언급된 PTZ 카메라보다 일반적으로 상당히 가격이 싸다. 그러나, 상기 접근법은 많은 응용들에서 요구되는 출력 영상 품질을 제공하지 못한다. 예를 들어, 상기 가상 주밍과 연관된 추출 및 보간 동작들은 결과적인 출력 비디오 신호에서 감소된 해상도와 영상 품질을 발생시키므로, 비디오 회의나 다른 유사한 응용들에서 적합하지 않을 수 있다.The second approach is virtual zoom

Or Aelectronic zoom

). In this approach, video information from one or more cameras is processed electronically so that the object of interest is visible in the desired configuration in the output video signal despite the fact that the object may not be in the center of the field of view of any particular camera. It is processed electronically. US Pat. No. 5,187,574 discloses an example of this approach, wherein the image of the arriving guest is picked up by a fixed television camera of the surveillance system. The image is processed using detection, extraction, and interpolation operations to ensure that the guest's head is always displayed in the center of the monitor screen. The approach ensures that the video output has the desired shape, for example in the center of the object of interest, without the need for pan, tilt, or zoom operations. As a result, the approach can operate with a fixed camera, which is generally considerably cheaper than the PTZ cameras mentioned above. However, this approach does not provide the output image quality required for many applications. For example, extraction and interpolation operations associated with the virtual zooming may not be suitable in video conferencing or other similar applications as they result in reduced resolution and image quality in the resulting output video signal.

As is evident from the above, there is a need for an improved tracking technique that can provide the output video signal quality and resolution associated with the PTZ camera approach as well as the flexibility of the virtual zoom approach, while the approaches and general There is also a need to avoid related problems.

The present invention utilizes a hybrid combination of (i) optical zooming with a pan-tilt-zoom (PTZ) camera and (ii) virtual zooming of images generated by the camera, thereby real-time the object of interest in the video processing system. It is to provide a method and apparatus for tracking. In an exemplary embodiment of the invention, the object of interest is initially detected in the image generated by the camera. The optical zooming operation then adjusts the pan and tilt settings to frame the object of interest and zooms in the object of interest until one or more specified stopping criteria are met. The virtual zooming operation processes the resulting optically zoomed image to identify and extract a particular region of interest, and then interpolates the extracted region of interest to generate a virtually zoomed image.

According to one aspect of the invention, the specified stop criteria is such that the optical zooming continues until the object of interest occupies a fixed or dynamic percentage of the resulting optically zoomed image, for example. Can be displayed. In the case of a dynamic percentage, the percentage can vary as a function of the detected quality associated with the object of interest. Embodiments of the detected qualities include a certain level of movement, the use of certain audibly-detectable keywords or other cues, and changes in intensity, pitch or other voice quality.

According to another aspect of the present invention, if the level of movement of the object of interest exceeds a first specified threshold, the resultant optically zoomed using the same pan, tilt, and zoom settings installed in the optical zooming operation. Can be repeated in the image. If the level of movement of the object of interest exceeds a second specified threshold above the first threshold, the optical zooming operation itself may be repeated to set new pan, tilt, and zoom settings for the camera. .

The hybrid optical and virtual zoom device of the present invention provides a number of significant advantages over conventional approaches. For example, the hybrid device accommodates any amount of movement of the object of interest without having to determine new optical pan, tilt, and zoom settings, and also provides the desired output image quality level. By preventing the PTZ camera from zooming in too far, the present invention ensures that the PTZ camera settings are adjusted less frequently, whereby the computational load on the system processor is required by conventional optical zoom approaches. Reduced compared to the In addition, the hybrid device of the present invention provides an improved compression rate for image transmission. These and other features and advantages of the present invention will become more apparent from the accompanying drawings and the following detailed description.

1 is a block diagram of a video processing system according to an exemplary embodiment of the present invention.

2 is a functional block diagram illustrating hybrid real time tracking video processing implemented in the system of FIG.

1 illustrates a video processing system 10 according to an exemplary embodiment of the present invention. The system 10 includes a processor 12, a memory 14, an input / output (I / O) device 15, and a controller 16, all connected to communicate via a system bus 17. have. The system 10 further includes a pan-tilt-zoom (PTZ) camera 18 coupled to the controller 16 as shown. In the exemplary embodiment, the PTZ camera 18 is a video conference in which the table 20 accommodates a number of conference participants 22-1, ..., 22-k, ..., 22-N. Used in applications. In operation, the PTZ camera 18 is of interest corresponding to a particular participant 22-k in the embodiment application, as managed by the controller 16 in accordance with instructions received from the processor 12. Track an object. The PTZ is connected to FIG. 2 to perform the real time tracking function using a hybrid optical and virtual zooming device described in more detail below.

)는 하이브리드 광학 및 가상 주밍 장치와 결합하여 사용될 수 잇는 어떤 형식의 영상 포착 장치를 포함하도록 의도된다. While the present invention will be illustrated in the context of an application of video conferencing, it should be understood that the video processing system 10 can be used in a variety of other applications. For example, part 24 of the system 10 may be used in video surveillance applications, and other types of video conferencing applications, such as applications including parliamentary seat arrangements, circular or rectangular table arrangements. Can be. More generally, the portion 24 of system 10 may benefit from the improved tracking functionality provided by hybrid optical and virtual zooming devices. The portion 26 of the system 10 may therefore be, for example, another video conferencing arrangement, a video surveillance arrangement, or any one or more objects of interest to be tracked using the portion 24 of the system 10. Can be replaced with another arrangement. It will also be apparent that the present invention can be used with PTZ cameras and other image capture devices. Thus, the term camera as used herein (Acamera)

) Is intended to include any form of image capture device that can be used in combination with hybrid optical and virtual zooming devices.

It should be noted that the devices or groups of devices of the system 10 may represent portions or combinations of the and other processing devices as well as corresponding devices of other conventional desktop or portable computers. Moreover, in other embodiments of the present invention, some or all of the functions of the processor 12, controller 16, or other devices of the system 10 may be combined into a single device. For example, one or more devices of system 10 may be implemented as an application specific integrated circuit (ASIC) or circuit card to be integrated into a computer, television, set-top box, or other processing device. The term processor, as used herein, is intended to include a microprocessor, central processing unit, microcontroller, or any other data processing device that may be used in a given data processing device. In addition, it should be noted that the memory 14 may represent an electronic memory, an optical or magnetic disk based memory, a tape based memory, as well as combinations or portions of the and other types of storage devices.

2 is a functional block diagram illustrating the hybrid optical and virtual zoom device 30 performed in the system 10 of FIG. 1. Again, although illustrated in the context of video conferencing applications, it will be apparent that the techniques described above are readily applicable to any other tracking application. As shown in FIG. 2, the hybrid optical and virtual zoom device 30 includes a detection and tracking operation 32, an optical zooming operation 34, and a virtual zooming operation 36. The operations will be described with reference to images 40, 42, 44, and 46 that correspond to images generated for an example video conferencing application in portion 26 of system 10. The operations (32, 34, and 36) may be performed in system 10 by processor 12 and controller 16, stored in the memory 14 or from the local or remote storage device. Use one or more software programs accessible via the O device 15.

In operation, the PTZ camera 18 is an image 40 comprising an object of interest, ie, a video conference participant 22-k, and an additional object, ie, another participant 22-k + 1 adjacent to the object of interest. ). The image 40 is fed to the detection and tracking operation 32 as a video input and detects and tracks the object of interest 22-k using well-known conventional detection and tracking techniques.

For example, in the video conferencing application, the object of interest 22-k may correspond to a current speaker. In this case, the detection and tracking operation 32 uses techniques such as audio location to determine which meeting participant is the current speaker, which meeting participant speaks, gestures, shakes hands, and in a particular manner. Techniques such as motion detection can be used to detect and track the object of interest 22-k to determine whether to move, speak in a particular manner, and the like.

In video surveillance applications, the object of interest may be a person doing a particular action, for example, a person entering, leaving or suspecting a restricted area, a child moving in a home room, a vehicle entering or leaving a parking lot, and the like. . The output of the detection and tracking operation 32 includes information that identifies the particular object 22-k of interest, shown as hatched within the image 42.

IEEE Trans.PAMI,19(7):780-785,July 1997;H.Rowley,S.Bluja,T.Kanade,ARotation Invariant Neural Network-Based Face Detection,

Proc.IEEE Conf.on Computer Vision,pp.38-44, June 1998; and A.Lipton, H.Fujiyoshi, R.Patil,AMoving Target Classification and Tracking from Real-Time Video,

Proc.IEEE Workshop on Application of Computer Vision,pp.8-14,Oct 1998.에 기술된 것을 포함한다.The particular type of detection and tracking device used in act 32 will generally vary depending on the application. Conventional detection and tracking techniques that can be used in operation 32 include, for example, C.Wren, A. Azarbayejani, T. Darrell, A. Pentland, APfinder: Real-time Tracking of the Human Body,

IEEE Trans. PAMI, 19 (7): 780-785, July 1997; H. Rowley, S. Bluja, T. Kanade, AROTATION Invariant Neural Network-Based Face Detection,

Proc. IEEE Conf. On Computer Vision, pp. 38-44, June 1998; and A. Lipton, H. Fujiyoshi, R. Patil, AMoving Target Classification and Tracking from Real-Time Video,

Proc. IEEE Workshop on Application of Computer Vision, pp. 8-14, Oct 1998.

The optical zooming operation 34 of FIG. 2 provides a sufficient amount of zooming to ensure that the desired output image quality is achieved, while also allowing some amount of movement of the object of interest. The optical zooming operation 34 includes a framing portion having pan and tilt operations for framing the object of interest 22-k, followed by a zooming portion having a zooming operation that continues until a specified stop criterion is satisfied. Follow.

Assuming that the radial distortion of the camera lens is negligible, the following approach can be used to determine the required amount of pan and tilt in the framing portion of operation 34. Assume that object 22-k of interest is detected at operation 32 as located at pixel coordinate position (x, y) in image 42. The framing portion of operation 34 adjusts the pan and tilt of camera 18 such that the object of interest appears in the center of the image (c _x , c _y ). As previously determined when the camera zoom factor ZF = 1, ZF is the current zoom factor, α _P ^C is the current camera pan angle, α _T ^C is the current camera tilt angle, and D is the pixel per pixel. It is called the number of degrees. The new pan angle α _P ^N and the new tilt angle α _T ^N are then given as:

α _P ^N = α _P ^C + D * ((xc _x ) / ZF),

α _T ^N = α _T ^C + D * ((yc _y ) / ZF).

Other techniques may also be used to determine appropriate pan and tilt adjustments for the framing portions of act 34. For example, techniques for determining pan and tilt in the presence of radial distortion of the camera lens will be apparent to those skilled in the art.

After completion of the framing portion of operation 34, the zooming portion of operation 34 begins. As already mentioned, the portion of operation 34 includes optical zooming that continues until one or more specified stop criteria are met. There are a number of different types of stop criteria that can be used. In a fixed stationary reference approach, the optical zooming continues until the object of interest occupies a fixed percentage of the image. For example, in a video conferencing system, the optical zooming may continue until the head of the current speaker occupies between 25% and 35% of the vertical size of the image. Of course, the specific percentage used will vary depending on the fracting application. Such specific percentages suitable for a particular application can be determined by a person skilled in the art in a correct manner.

In the dynamic stop reference approach, the optical zooming continues until the object of interest reaches a specified percentage of the image, but the percentage in this approach is a function of other detected quality associated with the object of interest. For example, the percentage may vary as a function of qualities such as the level of apparent movement, the use of certain keywords or other audio or voice cues, changes in intensity, pitch or other voice quality, and the like. Again, the particular percentage and how they vary based on the detected qualities will generally depend on the particular tracking application and can be determined in a correct manner by one skilled in the art.

The result of the optical zooming operation 34 is an optically-zipped image 44, wherein the object of interest 22-k is centered within the image and determined based on the fixed or dynamic stop criteria described above. Likewise, it occupies the desired percentage of the image. The image 44 may be stored by the system 10, for example in the memory 14.

The virtual zooming operation 36 is then performed on the optically zoomed image 44. The virtual zooming operation first extracts an area of interest from the image 44. For example, in the video conferencing application, the region of interest 47 can be identified as the head and shoulder of the object 22-k that is currently of interest. In video surveillance applications, the area of interest may be the hand, foot, head, torso, or other designated portion of the object of interest. The identification of the region of interest may be a dynamic process, for example it may be selected by the operator based on the current tracking goal. The region of interest can be identified and extracted using known techniques, for example, the techniques described in the references referenced above in connection with the detection of an object of interest. The extracted region of interest is then interpolated using well-known image interpolation techniques to generate a video output comprising a virtually-zipped image 46. The image 46 thus represents a virtual zoom of the optically-zipped image 44.

It should be noted that the virtual zooming operation 36 may be performed in a different system than the detection and tracking operation 32 and the optical zooming operation 34. For example, the image 44 may be compressed and then transmitted from the system 10 via an I / O device 15, wherein the virtual zooming operation is performed in the signal processing device of the corresponding receiver.

Advantageously, the hybrid device 30 allows some amount of movement on the portion of the object of interest, while preserving the desired level of image quality in the video output. For example, if the object of interest 22-k moves, the virtual zooming operation 36 may be repeated using the same pan, tilt, and zoom settings determined in the optical zooming operation 34. In this case, the extraction and interpolation operations of the virtual zoom may generate an output image in which the object of interest 22-k is substantially at the center of the image.

The hybrid device 30 may incorporate a number of thresholds for determining when the virtual zooming and optical zooming operations should be repeated. For example, if a given amount of movement of an object of interest exceeds a first threshold, the virtual zooming operation 36 may be repeated without changing the pan, tilt, and zoom settings of the camera. If the given amount of movement exceeds the second higher threshold, the optical zooming step 34 may be repeated to determine new pan, tilt, and zoom settings, and then the virtual zooming operation 36 ) Is repeated to obtain the desired output image 46. A feedback path 48 is included between the optical zooming operation 34 and the detection and tracking operation 32 such that the detection and tracking operation is, if necessary, for example, of the object of interest. It can be repeated if it detects no substantial movement and no longer tracks the object.

The hybrid optical and virtual zoom device of the present invention provides a number of important advantages over conventional approaches. As already described, the hybrid device accommodates several movements of the object of interest without having to determine new optical pan, tilt, and zoom settings, while also providing a desirable level of output image quality. By preventing the PTZ camera from zooming in too far, the present invention allows the PTZ camera settings to be adjusted less frequently and the computational load on the system processor is thereby reduced relative to what is required by conventional optical zoom approaches. In addition, the hybrid device of the present invention can provide an improved compression rate for image transmission. For example, as described above, the virtual zoom operation may be performed after an image is transmitted from the system 10 to the receiver via the I / O device 15. As a result, the proportion of objects in the transmitted image is lower than when using the conventional approach, thereby allowing for less compression and improved compression speed.

The above-described embodiments of the present invention are intended to be illustrative only. For example, the present invention can be used to perform real time tracking of any desired object of interest and includes video conferencing systems, video surveillance systems, and other camera based systems in a wide variety of applications. In addition, although the use of a system with a single PTZ camera has been illustrated, the invention may be applied to systems with multiple PTZ cameras and to systems with other formats and arrangements of image capture devices. Moreover, the present invention can use many different types of techniques to detect and track objects of interest, and to extract and interpolate regions of interest. The invention may be performed at least in part in the form of one or more software programs stored on an electronic, magnetic or optical medium and executed by a processing device, for example the processor 12 of the system 10. These and many other embodiments within the scope of the following claims will be apparent to those skilled in the art.

Claims (14)

A method for tracking an object of interest 22-k in a video processing system 10, Detecting the object of interest in a first image (40) generated by a camera (18); Performing an optical zooming operation (34) to set at least one zoom setting for the camera according to one or more specified stop criteria based on the object of interest; And Performing a virtual zooming operation (36) of a second image (44) generated by the camera at the set setting. A device for tracking an object of interest 22-k in a video processing system 10, Camera 18; And A processor 12 coupled to the camera and operative to detect the object of interest in a first image 40 generated by the camera, The processor, (i) performing an optical zooming operation 34 to set at least one zoom setting for the camera according to one or more specified stop criteria based on the object of interest; And (ii) directing the performance of the virtual zooming operation (36) of the second image (44) generated by the camera at the set setting. The method of claim 2, The camera is a pan-tilt-zoom (PTZ) camera with adjustable pan, tilt, and zoom settings. The method of claim 3, wherein The optical zooming operation, Framing the object of interest by adjusting the pan and tilt settings of the camera; And Performing optical zoom on the framed object of interest until the specified stop criterion is met. The method of claim 2, The specified stop criterion indicates that the optical zooming continues until the object of interest occupies a percentage of the resulting image. The method of claim 5, And said percentage is a fixed percentage. The method of claim 5, The percentage varies as a function of detected quality associated with the object of interest. The method of claim 7, wherein And the detected quality associated with the object of interest comprises at least one of an apparent level of movement, the use of certain audible-detectable cues, and a change in speech quality. The method of claim 2, The virtual zooming operation, Identifying an area of interest (47) in the second image; Extracting the region of interest; And Interpolating the extracted region of interest to generate a third image (46). The method of claim 3, wherein The processor manages the repetition of the virtual zooming operation of the second image using pan, tilt, and zoom settings set in the optical zooming operation if the level of movement of the object of interest exceeds a first threshold. A tracking device further operative to. The method of claim 10, The processor is configured to manage repetition of the optical zooming operation to set at least one new setting for the camera if the level of movement of the object of interest exceeds a second threshold that is higher than the first threshold. Tracking device. The method of claim 2, And the video processing system comprises a video conferencing system. The method of claim 2, And the video processing system comprises a video surveillance system. A computer readable recording medium having one or more programs recorded thereon, Detecting an object of interest 22-k in the first image 40 generated by the camera 18; Performing an optical zooming operation (34) to set at least one zoom setting for the camera according to one or more specified stop criteria based on the object of interest; And And the one or more programs executing the step of performing a virtual zooming operation (36) on a second image (44) generated by the camera at the set setting.

Images