DE102015015510A1 - A method of recognizing an object based on a location of the object in an image, apparatus and motor vehicle - Google Patents

Abstract

The invention relates to a method for detecting an object (8) in a surrounding area (9) of a camera (3), in which an image (16) representing the object (8) is defined in a ground reference frame at a position (10) (11) positioned camera (3) is selected, and the object (8) is selected in the image (16), wherein a direction (15) of the object (8) from the position (10) by means of a layer (18). of the object (8) in the image (16) is determined and by means of the direction (15) and the position (10) a location (17) of the object (8) in the ground reference frame (11) is determined, wherein a plurality of reference locations to which a reference object is assigned, is provided, and the location (17) is compared with the reference sites and the object (8) is identified as one of the reference sites as the reference object assigned to that reference site, if the site (17) matches.

Description

The invention relates to a method for detecting an object in a surrounding area of a camera. An image representing the objects is recorded by means of the camera positioned at a position defined in a ground-based reference frame. Furthermore, the object in the image is selected. The invention also relates to a device with a camera, which is designed to record an image with an object from a surrounding area of the camera, and to a position determination unit, which is designed to determine a position of the camera defined in an earth-fixed reference system. Furthermore, the invention relates to a motor vehicle with a corresponding device.

Methods for detecting an object in a surrounding area of a camera are known from the prior art. That's how it describes US 8,131,118 B1 a method for determining the location at which an image was taken. A device may capture an image of a location, and a central system may compare the submitted image with images in an image library to identify matches. The location of the match can then be provided to the device.

The DE 20 2013 007 354 U1 discloses a computer program product in which it is determined whether a traffic sign is located in an environment of a vehicle. This is done based on data from an optical camera. A corresponding traffic sign recognition system recognizes the traffic signs with the aid of the front camera of the vehicle and, by means of a text recognition algorithm in conjunction with stored information, identifies the point of interest mentioned on the traffic sign and displays these, for example, in an infotainment system of the vehicle.

The US 2015/0063642 A1 describes a method in which objects are detected and isolated in a uploaded image by means of machine vision algorithms. The isolated objects are compared with photographs from a database.

It is an object of the invention to provide a method, a device and a motor vehicle to recognize an object in a surrounding area of a camera easier and more reliable.

This object is achieved by a method by a device and by a motor vehicle with the features according to the respective independent claims.

In a method according to the invention, an object is detected in a surrounding area of a camera. An image representing the object is recorded by means of the camera positioned at an, in particular three-dimensional, position in a ground-based reference frame. Furthermore, the object in the image is selected. An essential idea of the invention is to be seen in that a direction of the object is determined from the position based on a position of the object in the image and, based on the position of the camera based on the direction, a location of the object in the ground reference frame, in particular with a three-dimensional coordinate, is determined. Furthermore, a plurality of, in particular in the Erdfesten reference system defined, reference locations are provided to which each a reference object is assigned. The location is compared to the reference locations, and if the location matches one of the reference locations, the object is recognized as the reference object assigned to that reference location.

The method according to the invention is based on the knowledge that, with the knowledge of the position of the camera and the direction of the object from the camera, a line or a beam can be generated unambiguously from the position of the camera in a coordinate system of the earth-fixed reference system. In the coordinate system of the earth-fixed reference system, reference objects may then be present which can be cut by the beam. If, for example, several reference objects are cut by the beam, then preferably the first cut reference object, that is to say the camera closest to the reference object, is selected. By virtue of the method, the object can be reliably detected in the surrounding area in a simple and effective manner.

In particular, it is provided that the position of the camera in the earth-fixed reference system is defined in three dimensions. The position of the camera can be determined, for example, by means of a global navigation satellite system (GNSS) of a device comprising the camera. The ground-based reference system can be determined, for example, as WGS84 (World Geodetic System 1984). The direction determination on the basis of the position of the object in the image can take place, for example, by determining an alignment of the camera at the time the image is taken. The orientation relates in particular to the reference system. For example, a horizontal angle and a vertical angle of the object are determined based on the position of the camera. By the position and the direction of the location of the object can then be determined by the resulting by the position and the direction beam with For example, a digital terrain model (DTM) of the surrounding area of the camera is cut. The digital terrain model is also present in particular in the Erdfesten reference system. The digital terrain model provides a three-dimensional image of the earth's surface. For example, in another embodiment, a digital surface model (DOM) may also be chosen to determine the location of the object in the ground reference frame. The digital surface model describes the surface of the earth, the vegetation, and the development through the three-dimensional coordinates of a representative set of ground and non-ground points.

The camera is preferably arranged on a motor vehicle. In particular, an external orientation of the camera is known. The outer orientation provides a transformation law between an image coordinate system of the image and the ground reference system. The reference sites with the assigned reference objects are preferably stored in a database, which can be arranged for example in the camera or the motor vehicle or else in a vehicle-external device, which is then accessed, for example, wirelessly.

As already stated, it is preferably provided that the direction of the object from the camera is determined by a horizontal angle and / or a vertical angle. The horizontal angle is determined for example by a compass and is also referred to as azimuthal angle. The horizontal angle is an angle measured in a horizontal plane. The vertical angle is measured in a vertical plane. For the vertical angle, for example, the zenith, the nadir and the horizon can be selected as the starting point. Thus, the vertical angle can be indicated, for example, as a zenith angle or as an elevation angle or as a depth angle or as a nadir angle. A reference surface for the vertical angle can also be determined, for example, by an electronic spirit level. In particular, it is provided that the direction is determined by both the horizontal angle and the vertical angle. However, it can also be provided that the direction is determined solely by the horizontal angle alone or the vertical angle alone. For example, there are situations in which there is only one reference object at a certain height or in a specific orientation to the camera in the surrounding area of the camera, in particular in that part of the digital terrain model that is visible from the position of the camera. Thus, the reference object can be unambiguously assigned to the object during the comparison, since only this one reference object is suitable for the present vertical angle or the present horizontal angle. In particular, this has two advantages: on the one hand, the direction of the object can be determined more simply and more quickly and, on the other hand, the comparison of the location with the reference locations can be carried out more simply and more quickly. It is advantageous in determining the direction by the horizontal angle and the vertical angle together that one of the reference locations can be assigned to the location of the object with particular reliability.

Furthermore, it is preferably provided that the image is displayed on a display unit and the object in the image is selected by a user. By displaying the image on the display unit, the user can select the object more easily and reliably. Furthermore, for example, the image of the surrounding area can be displayed live on the display unit and the user can view the image of the surrounding area live. Thus, the user can select the object in the picture currently being taken by the camera. For example, the user can simultaneously view the object real in the surrounding area and as an image in the picture.

In one embodiment, it is preferably provided that the object is selected by means of gesture control by recognition of a non-contact gesture of the user. For example, the user can make a gesture in free space, based on which then the object is selected in the image. The gesture can be performed, for example, with one hand of the user. Gesture control makes it easier to select the object.

In a further embodiment, it is preferably provided that the object is selected by the user by touching the display unit embodied as a touch-sensitive screen. The display unit can thus be designed, for example, as a touchscreen. The display unit is preferably arranged in a motor vehicle or a mobile device. The user can now, for example, tap the display unit with a finger to select the object in the image displayed on the display unit. By selecting the object by the touch, the selection can be made easier and more comfortable for the user.

Furthermore, it may be provided that the location with the reference location is assumed to match if a predetermined match criterion is met. In particular, the location with the reference location is assumed to be coincidental if a distance from the location to the reference location is less than a distance limit. Thus, by the predetermined match criterion For example, a tolerance for a deviation of the location from the reference location can be provided. For example, it may be that the coordinates of the location do not exactly match the coordinates of the reference site. This can be compensated for example by the predetermined match criterion. For example, a radius around the location or reference location within which the location and reference location are assumed to be consistent may be determined. Preferably, a probability value based on a distance from the location to the reference location can also be used for this purpose. For example, it can be assumed that the greater the distance between the location and the reference location, the less likely it is to assume that a match exists. On the other hand, the closer the location and the reference location are, the smaller the distance between the location and the reference location, the more likely to be a match. For example, the least squares method can be used to weight the distance. In this method the distance is weighted non-linearly. The conformity of the location with the reference location can thus be determined more reliably. However, the match criterion can also be adjusted, for example, depending on the situation, depending on an accuracy with which the position and / or the direction is determined. The matching criterion can also be predetermined, for example, individually for the respective reference location.

In a further embodiment, it is preferably provided that a detail object that describes the reference object is assigned to a respective reference object of the plurality of reference objects, and the detailed information of the reference object that corresponds to the object is output for describing the object. Thus, the reference objects, in particular stored in the database, are preferably assigned detailed information. The respective detailed information is then output to the user when the object matches the reference object, ie when the location of the object matches the reference location of the reference object. Based on the detailed information, the user can then, for example, receive information about a landscape, a castle, a mountain, a beach, a house or an estuary. The detailed information may include, for example, historical data or a current state of the object, such as opening times.

Furthermore, it is preferably provided that the detailed information is output by voice and / or text and / or graphics and / or video. For example, the detail information may be displayed to the user in the form of a sound output, a written text displayed on the display unit, a graphic, for example, also displayed on the display unit, and / or a video, which in turn is also displayed on the display unit can be spent. As a result, the detailed information can be provided to the user in a variety of situations. For example, providing the detail information by voice is advantageous if the user is a driver of a motor vehicle and this is focused on a traffic event. For example, the detailed information provided by the graphic or video is particularly illustrative and easy to understand.

The invention also relates to a device having a camera for taking a picture with an object from a surrounding area of the camera and a position-determining unit for determining a position of the camera defined in an earth-fixed reference frame. An essential idea of the invention is that the device has an evaluation unit which is designed to determine a direction of the object from the camera based on a position of the object in the image, and a location based on the direction and the position of the object in the ground-based frame of reference. The location is then comparable to a plurality of reference locations in the fixed ground reference system. If the location matches the reference site, the object is recognizable as a known reference object associated with the reference site.

The device may for example be comprised by a motor vehicle or a portable device, in particular a smartphone.

Furthermore, the invention relates to a motor vehicle with a device according to the invention. A camera of the device is preferably connected to the motor vehicle and has at least one motor vehicle fastening element for attachment to the motor vehicle. A position determination unit of the device and an evaluation unit of the device are preferably realized by a GNSS receiver and a computing unit of the motor vehicle. The position determination unit and / or the evaluation unit can be arranged manifold on or in the motor vehicle. The position determination unit and / or the evaluation unit can also be integrated into the camera, for example, or be designed separately from the camera.

The preferred embodiments presented with reference to the method according to the invention and their advantages apply correspondingly to the device according to the invention and to the motor vehicle according to the invention.

Further features of the invention will become apparent from the claims, the figures and the description of the figures. The features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of the figures and / or in the figures alone can be used not only in the respectively specified combination but also in other combinations or in isolation, without the scope of To leave invention. Thus, embodiments of the invention are to be regarded as encompassed and disclosed, which are not explicitly shown and explained in the figures, however, emerge and can be produced by separated combinations of features from the embodiments explained. Embodiments and combinations of features are also to be regarded as disclosed, which thus do not have all the features of an originally formulated independent claim.

The embodiments of the invention will be explained in more detail with reference to schematic drawings.

Showing:

1 a schematic plan view of an embodiment of a motor vehicle according to the invention with a device with a camera;

2 a schematic representation of an image taken with the camera from a surrounding area of the camera; and

3 a schematic representation of a digital terrain model in a terrestrial reference system.

In the figures, identical or functionally identical elements are provided with the same reference numerals.

In 1 is schematically a motor vehicle 1 with a device 2 shown. The device 2 includes a camera 3 , a position determination unit 4 , an evaluation unit 5 , a direction determination unit 6 and a display unit 7 ,

The camera 3 is on a front 8th of the motor vehicle 1 arranged, however, the camera can 3 also at any other position of the motor vehicle 1 be arranged. The camera 3 captures an object 8th in a surrounding area 9 the camera 3 , The camera 3 For example, it can be designed as a CMOS camera (complementary metal-oxide-semiconductor) or as a CCD camera (charge-coupled device) or as any type of image capture device. The camera 3 preferably provides an image sequence of images. The image sequence of the images is then, for example, in real time by the evaluation unit 5 processed.

The position determination unit 4 For example, it can be designed as a global navigation satellite system (GNSS), in particular GPS, Galileo, GLONASS or Beidou. Furthermore, the position determination unit 4 For example, be designed as a relative global navigation satellite system or as a supported Global Positioning System (A-GPS-Assisted Global Positioning System), are used in which correction data for position improvement. By the position determination unit 4 becomes a position 10 the camera 3 a ground-based frame of reference 11 certainly. The position 10 is determined in particular three-dimensional, that is, for example, with three Cartesian coordinates. The earth-proof frame of reference 11 may for example be designed as WGS84.

The direction determination unit 6 is in particular adapted to a horizontal angle 12 and a vertical angle 13 from the camera 3 to the object 8th to determine. The horizontal angle 12 extends in the horizontal plane and denotes the angle between a main reference point of the terrestrial reference system 11 and the object 8th , The main reference point 14 may be, for example, the magnetic north pole or the geographic north pole. The vertical angle 13 may be formed as zenith angle, elevation angle, depth angle or nadir angle. Accordingly, the vertical angle refers to the zenith, nadir or horizon. A reference plane for the vertical angle 13 can be provided for example by a spirit level.

The evaluation unit 5 is designed to one direction 15 of the object 8th from the position 10 from based on a location of the object 8th in one - in 2 shown - picture 16 to determine. The picture 16 is with the camera 3 added. From the camera 3 In particular, an external orientation is known which indicates a relationship between the position of the object 8th in the picture 16 and a location 17 of the object 8th in the earth-fixed frame of reference 11 allows. So can from an image coordinate system of the image 16 based on the knowledge about the external orientation, and for example an inner orientation of the camera, the location 17 of the object 8th in the earth-fixed frame of reference 11 be determined. It can then be a beam starting from the position 10 the camera 3 by the direction 15 clearly in the earth-fixed frame of reference 11 be determined. This beam then cuts, for example, one through a digital terrain model in the ground reference frame 11 given earth surface. Where the beam with the shortest distance to the position 10 cut from the Earth's surface of the digital terrain model becomes the location 17 certainly.

The site 17 is then compared to a plurality of reference sites. Each of the reference sites is assigned a reference object. If the location matches 17 with one of the reference locations becomes the object 8th recognized as the reference object assigned to the reference location.

The match between the location 17 and the reference location can be done for example via a three-dimensional coordinate comparison. In the comparison, for example, a predetermined matching criterion can also be used if the coordinates of the location 17 and the reference site do not exactly match. For example, a tolerance range around the location 17 or the reference location. For example, the tolerance range can be a radius around the location 17 or the reference location be formed. For example, a distance from the location 17 be used to the reference location. If this distance is then, for example, smaller than a distance limit value, then a correspondence between the location and the reference location can be assumed.

The position determination unit 4 , the evaluation unit 5 and the direction determination unit 6 can both in the camera 3 be integrated as well as separately, for example in the motor vehicle 1 be arranged.

2 shows the picture 16 , The picture 16 in particular on the display unit 7 output. In the picture 16 are several objects 8th shown. Each of the objects 8th is at an individual location 18 in the picture 16 shown.

A user of the device 2 can the respective object 8th now for example by contactless gesture in free space or by touching the display unit 7 at the respective location 18 of the object 8th in the picture 16 choose. The object selected by the user 8th then, for example, in the picture 16 highlighted graphically, for example, by a change in color of the object shown 8th in the picture 16 or by marking the picture 16 represented object 8th with, for example, a circle 19 which the object 8th surrounds.

Becomes the object 8th now by matching location comparison 17 detected with the reference location as a reference object, the user is preferably output a detail information about the reference object. For example, the detail information may be an explanation of properties of the object 8th be. The detailed information can then be output to the user, for example by voice and / or text and / or graphics and / or video. So the user can see the object live in the camera 3 taken picture 16 select and obtain, after comparing the location of the object with the reference locations and the location matches one of the reference locations, the detailed information about the reference object assigned to that reference location.

3 shows a digital terrain model 19 in the earth-fixed frame of reference 11 from the perspective of the camera 3 , From the camera 3 can now out due to the position 10 and the direction 15 a ray 20 from the camera 3 to be sent out. The beam 20 is merely a mathematical description in the earth-fixed frame of reference 11 to understand. Through the beam 20 becomes the digital terrain model 19 in the location 17 cut. The site 17 is in particular the cut with the digital terrain model 19 , which next to the position 10 is the shortest distance between the point of intersection and the position 10 having.

The direction 15 of the beam 20 becomes through the horizontal angle 12 and the vertical angle 13 certainly. The horizontal angle 12 and the vertical angle 13 be by means of the direction determination unit 6 certainly. The position 10 is determined by the position determination unit 4 certainly. Schematically are in 3 the position determination unit 4 as well as the direction determination unit 6 represented and their products, as the position 10 , the horizontal angle 12 and the vertical angle 13 ,

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 8131118 B1 [0002]
• DE 202013007354 U1 [0003]
• US 2015/0063642 A1 [0004]

Claims (10)

Method for recognizing an object ( 8th ) in a surrounding area ( 9 ) of a camera ( 3 ), in which an object ( 8th ) image ( 16 ) by means of at a position ( 10 ) defined in a terrestrial reference system ( 11 ) positioned camera ( 3 ) and the object ( 8th ) in the picture ( 16 ), characterized in that one direction ( 15 ) of the object ( 8th ) from the position ( 10 ) based on a location ( 18 ) of the object ( 8th ) in the picture ( 16 ) and the direction ( 15 ) and the position ( 10 ) a location ( 17 ) of the object ( 8th ) in the earth-fixed frame of reference ( 11 ) is determined, wherein a plurality of reference sites, to each of which a reference object is assigned, is provided and the location ( 17 ) is compared with the reference locations, and the object ( 8th ) if the location matches ( 17 ) is detected with one of the reference locations as the reference object assigned to this reference location. Method according to claim 1, characterized in that the direction ( 15 ) of the object ( 8th ) from the camera ( 3 ) from a horizontal angle ( 12 ) and / or a vertical angle ( 13 ) is determined. Method according to claim 1 or 2, characterized in that the image ( 16 ) on a display unit ( 7 ) and the object ( 8th ) in the picture ( 16 ) is selected by a user. Method according to claim 3, characterized in that the object ( 8th ) is selected by gesture control by recognizing a non-contact gesture of the user. Method according to claim 3 or 4, characterized in that the object ( 8th ) by touching the display unit designed as a touch-sensitive screen ( 7 ) is selected by the user. Method according to one of the preceding claims, characterized in that the location ( 17 ) is assumed to coincide with the reference site if a predetermined match criterion, in particular a distance from the site ( 17 ) to the reference location is less than a distance limit. Method according to one of the preceding claims, characterized in that a detail of the reference object is assigned to a respective reference object of the plurality of reference objects, and the detailed information of that reference object associated with the object ( 8th ), to describe the object ( 8th ) is output. A method according to claim 7, characterized in that the detailed information is output by voice and / or text and / or graphics and / or video. Contraption ( 2 ) with a camera ( 3 ) to take a picture ( 16 ) with an object ( 8th ) from a surrounding area ( 9 ) the camera ( 3 ) and a position determination unit ( 4 ) for determining in a ground reference system ( 11 ) defined position ( 10 ) the camera ( 3 ), characterized in that the device ( 2 ) an evaluation unit ( 5 ), which is adapted to a direction ( 15 ) of the object ( 8th ) from the camera ( 3 ) based on a location ( 18 ) of the object ( 8th ) in the picture ( 16 ) and the direction ( 15 ) and the position ( 10 ) a location ( 17 ) of the object ( 8th ) in the earth-fixed frame of reference ( 11 ), the location ( 17 ) with a plurality of reference locations in the ground reference frame ( 11 ), and if the location matches ( 17 ) with the reference location, the object ( 8th ) is recognizable as a reference object associated with the known reference object. Motor vehicle ( 1 ) with a device ( 2 ) according to claim 9.

Images