DE102018109680A1 - Method for distinguishing lane markings and curbs by parallel two-dimensional and three-dimensional evaluation; Control means; Driving assistance system; as well as computer program product - Google Patents

Abstract

The invention relates to a method for distinguishing lane markings (7) and curbs (8) in an environment (U) of a motor vehicle (1) with a camera system (2) of the motor vehicle (1). The following steps make this distinction possible with improved accuracy: provision of two images (4) of the environment (U) by the camera system (2) of the motor vehicle (1), wherein a respective perspective of the two images (4) differs, Determining at least one two-dimensional feature in at least one of the two images, wherein the at least one two-dimensional feature represents an object in the surroundings of the motor vehicle; Evaluating the at least one two-dimensional feature (42) by means of an artificial neural network (33), extracting depth information from the two images (4) of the environment (U), and classifying the object (6) in the environment either as a lane marking ( 7) or as a curb (8) both on the basis of a result (40) in the evaluation of the two-dimensional feature (42) and on the basis of the depth information.

Description

The invention relates to a method for distinguishing lane markings and curbs in an environment of a motor vehicle with a camera system of the motor vehicle, wherein at least two images of the environment are provided, which are recorded with the camera system, wherein a respective perspective of the at least two images differs. The invention also relates to a control device for assessing objects in an environment of a motor vehicle and a driver assistance system having such a control device and a computer program.

From the prior art it is known to detect an environment of a motor vehicle by means of one or more cameras. For example, images of the environment are captured and evaluated for nearby objects. Examples include cameras at the front and / or at the rear of the vehicle, which detect an area ahead or rearward of the vehicle. Information, in particular about objects in the vicinity of the motor vehicle, can be provided to a driver assistance system of the motor vehicle. Examples of objects that can be detected include road markings, curbs or curbs, pedestrians, other road users or any kind of obstacles in the vicinity of the motor vehicle.

A key problem in detecting the environment of the motor vehicle is the distinction of road markings and curbs. Road markings and curbs can often be difficult to distinguish from each other due to similar features.

In order to allow an improved distinction of lane markings and curbs, discloses the US 2016/0104047 A1 a method in which a curb can be detected on the basis of two images of an environment of the motor vehicle. In this case, lines are recognized in two consecutively recorded top-view images and an optical flow between the two consecutively recorded images is calculated for them. On the basis of this optical flow curbs and lane markings can be distinguished.

It is an object of the present invention to allow a distinction of lane markings and curbs in an environment of a motor vehicle with improved accuracy.

This object is achieved by the subject matters of the independent claims. Advantageous embodiments with expedient developments are the subject of the dependent claims.

• Bereitstellen zweier Bilder der Umgebung durch das Kamerasystem des Kraftfahrzeugs, wobei sich eine jeweilige Perspektive der zwei Bilder unterscheidet, Bestimmen zumindest eines zweidimensionalen Merkmals zumindest in einem der zwei Bilder, wobei das zumindest eine zweidimensionale Merkmal ein Objekt in der Umgebung des Kraftfahrzeugs repräsentiert, Auswerten des zumindest einen zweidimensionalen Merkmals mittels eines künstlichen neuronalen Netzes, Extrahieren einer Tiefeninformation aus den zwei Bildern der Umgebung und Klassifizieren des Objekts in der Umgebung entweder als Fahrbahnmarkierung oder als Bordstein sowohl anhand eines Ergebnisses beim Auswerten des zweidimensionalen Merkmals als auch anhand der Tiefeninformation.

The invention is based on a method for distinguishing lane markings and curbs or curbs in an environment of a motor vehicle with a camera system of the motor vehicle. The method is based on the following steps:

• Providing two images of the environment through the camera system of the motor vehicle, wherein a respective perspective of the two images differs, determining at least one two-dimensional feature in at least one of the two images, wherein the at least one two-dimensional feature represents an object in the vicinity of the motor vehicle, evaluating the at least one two-dimensional feature by means of an artificial neural network, extracting depth information from the two images of the environment and classifying the object in the environment either as a lane marking or as a curb on the basis both of a result in the evaluation of the two-dimensional feature and on the basis of the depth information.

In particular, the two images of the environment are a so-called front view, that is to say a representation of a region ahead of the motor vehicle. Thus, the respective perspective of the two images can be aligned according to a direction of travel of the motor vehicle. A detection area of the camera system may essentially comprise an area of the surroundings ahead of the motor vehicle in the direction of travel. In other words, an area of the environment ahead of the motor vehicle is advantageously detected on the basis of the two images of the surroundings.

The two-dimensional feature is in particular a feature in the two-dimensional image plane of the respective image. In the image plane, features can only be determined and evaluated two-dimensionally. Two-dimensional means in particular that an expansion or an evaluation of the feature is present only with respect to two independent (ie in particular right-angled) spatial direction. These two spatial directions (defined for example by an x-axis and a y-axis) can span the image plane of the respective image. In other words, these two spatial directions (in particular x-axis and y-axis) lie in the image plane of the respective image. A third spatial direction (defined, for example, by a z-axis) which is independent of the aforementioned axes (x-axis and y-axis) and in particular runs at right angles thereto can be seen in a single one of the two pictures Environment are not represented because they are two-dimensional and therefore individually have no information regarding the third spatial direction.

This information concerning the third spatial direction (z-axis) is referred to herein as the depth information. The depth information may include distances and aspect ratios relative to the third spatial direction (z-axis). The third spatial direction (z-axis) runs in particular parallel to a detection direction of the respective camera through which the respective image is detected or recorded. In other words, the depth information describes the distance of image contents or features from the camera in an abstract or concrete way. The depth information can be extracted from the two images of the environment since they each have different perspectives on the environment. Thus, each of the two images is two-dimensional in itself and without the depth information. However, when the two images are evaluated together, the depth information can be extracted therefrom.

As objects within the meaning of the present invention, in particular a roadway marking and / or a curb apply. In other words, two-dimensional features are determined in the at least one of the two images representing a pavement marker and / or a curb. These two-dimensional features are evaluated by an artificial neural network as to whether these two-dimensional features represent a lane marker or a curb. The artificial neural network may have previously been trained or learned by means of training data. The training data may include a plurality of images of lane markers and / or curbs that are modeled on the nature and perspective of the two images of the environment provided by the camera system. In the images of the training data are lane markings and curbs, for example by a user or manufacturer of a control device or a driver assistance system or a motor vehicle with such a driver assistance system already marked. Thus, in the process of learning, the artificial neural network can learn to recognize lane markings and / or curbs based on the two-dimensional features, and in particular to distinguish them from one another. To improve accuracy in teaching, mismatched two-dimensional (or mis-assigned three-dimensional) features that are misrecognized may be used for further training. This procedure is also referred to as Hard Negative Mining.

The determination of the at least one two-dimensional feature in the at least one of the two images can be performed by a recognition unit or an input part of the artificial neural network. Such an input part of the artificial neural network is also referred to by the technical term CNN encoder. This recognition unit or this input part can be designed to determine the at least one two-dimensional feature in the at least one of the two images. In addition, the input part or the recognition unit can process the at least one two-dimensional feature for evaluation by the artificial neural network. The at least one two-dimensional feature can be determined by pixel-level semantic segmentation. In particular, the recognition unit or the input part semantically segments one of the images, that is, by examining a meaning content of the respective pixels of the at least one of the images.

It is sufficient to evaluate only one of the two images of the environment in two dimensions. However, it can be provided, for example for evaluating redundant information, that both of the two images of the environment are evaluated in two dimensions. In this case, in particular, both images of the environment are independently evaluated by the artificial neural network based on respective two-dimensional features.

Subsequently, the object is classified or classified in the environment either as a lane marking or as a curb. The classification or classification of the object takes place on the basis of the result in the evaluation of the two-dimensional feature and on the basis of the depth information. In other words, the object is classified or classified depending on the result in the evaluation of the two-dimensional feature by the artificial neural network and depending on the depth information. Thus, the classification of the object takes place, on the one hand, on the basis of two-dimensional information (result of the evaluation of the at least one two-dimensional feature) and on the basis of three-dimensional information (depth information). In this way, lane markings and curbs can be distinguished particularly well, since lane markings are flat and curbs on the other hand have an increase compared to a lane level of a lane on which the motor vehicle is located.

According to a further development, it is provided that at least one three-dimensional feature is determined as the depth information, wherein the at least one three-dimensional feature is the Object in the environment of the motor vehicle represents. In addition, it is provided that the at least one three-dimensional feature is likewise evaluated by means of the artificial neural network. The three-dimensional feature is to be understood analogously to the two-dimensional feature, wherein the three-dimensional feature additionally has an extension according to the third spatial direction (z-axis). The three-dimensional feature can be jointly extracted from the two images of the environment. In this case, in an intermediate step, a three-dimensional map or a distance map of the surroundings can be generated from the two images. The three-dimensional feature can be determined for example on the basis of gradients and / or edges in the three-dimensional map of the environment or in the distance map. For determining the at least one three-dimensional feature, a further recognition unit or a further input part of the artificial neural network, also called another CNN encoder, may be provided. By means of this further detection unit or the further input part, the at least one three-dimensional feature can be determined. In addition, the at least one three-dimensional feature can be prepared for evaluation by the artificial neural network.

In this case, the artificial neural network or the artificial neural network is previously trained on the basis of further training data, which pairings of two images of the environment, which are modeled on the two images of the environment contain. In these pairings, the lane markings may already be marked on the curbs, for example by the above-mentioned user or manufacturer. In this way, the artificial neural network can be trained or become aware of it on the basis of the at least one three-dimensional feature road markings and / or curbs and to distinguish them in particular. Characterized in that the at least one two-dimensional feature and the at least one three-dimensional feature are evaluated jointly by means of the artificial neural network, road markings and curbs can be distinguished particularly well. In addition, both the evaluation of the at least one two-dimensional feature and the evaluation of the at least one three-dimensional feature benefit from the high adaptability and learning capability of the artificial neural network.

According to a development it is provided that the at least one two-dimensional feature and the at least one three-dimensional feature are evaluated jointly by means of the artificial neural network. In other words, the at least one two-dimensional feature and the at least one three-dimensional feature are jointly evaluated by the artificial neural network. In this way, the accuracy in distinguishing lane markings and curbs can be further improved, since a particularly early fusion of the different image information, so the two-dimensional features and three-dimensional features is guaranteed. In other words, the at least one two-dimensional feature and the at least one three-dimensional feature are combined in a particularly early processing step, namely here before the evaluation by means of the artificial neural network.

According to a development, it is provided that the distance map is determined as the depth information. The distance map can specify a distance of regions of the environment, and thus of the object, from a camera plane of the camera system. In this case, the distance map can be formed from the two images of the environment. The distance map may be aligned parallel to the at least two images of the environment with regard to two spatial directions (for example x-axis and y-axis). In contrast to the two images of the environment, however, the distance map also has an extension in the third spatial direction (z-axis). The depth information is thus provided by the extension of the distance map in the third spatial direction (z-axis). The expansion of the distance map into the z-axis can be provided, for example, by associating with each pixel of the distance map a respective distance value indicating a distance of the part of the environment represented by the respective pixel from a reference point of the camera system. This reference point can be defined for example by the position of a camera of the camera system. The pixels of the distance card can be arranged according to a grid which is defined or defined by the first two spatial directions (x-axis and y-axis). In other words, the grid of the image plane may correspond to one of the first two images. By determining the distance map, a three-dimensional map of at least part of the environment is provided in a particularly effective and simple manner.

According to a further development, it is provided that a representation of the object in the distance map is sought, in particular determined, by means of machine vision, in particular using mathematical algorithms. In other words, in the distance map by machine vision, also referred to by the English term computer vision, a representation of the object is sought. As a representation of the object in particular a pattern in the distance map denotes, which represents the object in the environment. In other words, a pattern is searched in the distance map, which is characteristic of the object. In particular, for this purpose, it is possible to search for gradients, lines and geometric shapes in the distance map which respectively correspond to a predetermined pattern. Machine vision, also referred to as image or computer vision, is the computer-aided solution to problems that are based on the capabilities of the human visual system. In this case, it is known to the person skilled in the art that the image evaluation by machine vision fundamentally differs from the image evaluation by means of artificial neural networks. While the latter method is trained on the basis of large data sets (training data), optical information from the underlying image / image is preferably acquired within the scope of machine vision, utilizing mathematical algorithms and pattern recognition. Examples of such mathematical algorithms as machine vision tools are adhesive transformation, contrast analysis, optical flow, Sobel operator, wavelets, Gauss-Laplace pyramid, and similar methods for contrast analysis, motion extraction, and edge detection, for example. Due to the pattern recognition or predetermined patterns that are searched, this requires a great deal of prior knowledge about the objects to be recognized. The combination of machine vision and artificial neural network makes the differentiation of lane markings particularly robust and yet particularly easy to adapt to different application scenarios.

According to a development, it is provided that the classification of the object additionally takes place on the basis of the search of the representation. In other words, the classification of the object takes place, on the one hand, on the basis of the result in the evaluation of the two-dimensional feature by the artificial neural network and, on the other hand, on the basis of the search of the representation. For example, when searching for the representation of the object, at least one predetermined pattern for a curb and at least one predetermined pattern for a lane mark are searched in the distance map. The at least one predetermined pattern for the curb may be predetermined according to a typical appearance of a curb in the distance map. The at least one predetermined pattern for a lane marking can be predefined on the basis of a typical appearance of a lane marking in the distance map. Depending on whether the predetermined pattern of a lane marking or a curb is found when searching for the representation of the object, the object can be classified either as a lane marking or as a curb.

According to a further development, it is provided that the classification of the object by means of the artificial neural network and the machine vision initially takes place independently of each other and respective results of the classification are subsequently fused, in particular based on probability. In other words, the evaluation of the two images is based on two-dimensional information, namely the at least one two-dimensional feature, and three-dimensional information, namely the depth information, largely separated from each other. The respective results (ie the result of evaluating the at least one two-dimensional feature and the result when evaluating the depth information, in particular the result when searching for the representation of the object) are then merged, in particular based on probability. The probability-based merger can be based on a confidence level of the respective results. In other words, both when evaluating the two-dimensional feature by the artificial neural network and when searching for the representation of the object or when searching for one of the predetermined patterns, a respective confidence level for the respective result is calculated. The respective confidence level can indicate with which probability the respective result is correct. In other words, the confidence level can indicate how reliably the respective result could be determined. This confidence level can be used to merge the different results of the probability-based merger. For example, in the context of the probability-based merger, one of the results is weighted higher than the other of the results if its confidence level is higher, ie the result could be determined more reliably. In this way, the distinction between curb and lane marking can be ensured even more reliable and robust. In addition, the different evaluation branches (artificial neural network and machine vision) can be easily further developed independently.

According to a development, it is provided that the classification of the object on the basis of the artificial neural network takes place exclusively on a two-dimensional plane and the classification of the object on the basis of machine vision exclusively on a three-dimensional plane. In other words, only two-dimensional features are evaluated by the artificial neural network. In addition, the classification of the object based on the machine vision takes place exclusively taking into account the depth information. In some embodiments, classifying the object is based on machine vision using only the distance map. In this case, it is taken into account that the artificial neural network is particularly well suited for evaluating the two-dimensional features and, in contrast, machine vision is particularly suitable for evaluating the depth information.

According to a further development, the two images of the environment are detected offset in time by means of the same camera of the motor vehicle. In other words, the two images are captured by the same camera of the camera system, wherein due to the drive of the motor vehicle, this camera is in a respectively different position when capturing the two images. This is a method which is also referred to as motion stereo, to German motion stereo. The two images of the environment in this example can be related in particular by movement data of the motor vehicle with respect to their respective perspective. In this case, the movement data of the motor vehicle can be connected by odometry with the respective perspective of the two images. In other embodiments of the invention, the camera system of the motor vehicle has two cameras, wherein a first of the two images are detected by a first of the two cameras and a second of the two images by a second of the two cameras. In this case, the two images can be detected in particular at the same time. The two cameras can both have a detection range, which lies ahead of the motor vehicle essentially in the direction of travel. The different perspective of the two images is ensured in this case by the different position of the cameras on the motor vehicle.

A second aspect of the invention relates to a control device for distinguishing lane markings and curbs in an environment of a motor vehicle based on images of a camera system of the motor vehicle with a receiving unit for providing two images of the environment from the camera system of the motor vehicle, wherein a respective perspective of the two images differs and an identification unit for determining at least one two-dimensional feature in at least one of the two images, wherein the at least one two-dimensional feature represents an object in the surroundings of the motor vehicle, and an artificial neural network for evaluating the at least one two-dimensional feature, and an extraction unit for Extracting depth information from the two images of the environment, and a classification unit for classifying the object in the environment either as a lane marking or as a curb s owohl based on a result in the evaluation of the two-dimensional feature as well as the depth information. The recognition unit can be an input part of the artificial neural network, which is also called the English-language technical term CNN encoder. The extraction unit for extracting the depth information may be another input part of the artificial neural network. This further input part of the artificial neural network can be designed to determine at least one three-dimensional feature in the two images, wherein the at least one three-dimensional feature represents the object in the environment. In this case, the artificial neural network can additionally evaluate the at least one be formed three-dimensional feature. Alternatively, the extraction unit for extracting the depth information is a machine vision evaluation unit. Both embodiments are already described in the context of the previously explained method, which is why a renewed description is omitted here. All advantageous developments and features of the method according to the invention apply analogously to the control device according to the invention. Advantageously, the control device is adapted to perform a method according to one or more of the embodiments described above.

A further aspect of the invention relates to a driver assistance system having a camera system which has at least one camera for capturing two images of the surroundings as well as the abovementioned control device. The driver assistance system can be arranged on a motor vehicle. Thus, a motor vehicle with such a driver assistance system is part of the present invention. The features of the method according to the invention and the control device according to the invention thus also apply analogously to the driver assistance system and the motor vehicle.

Another aspect of the invention relates to a computer program product having program code means stored in a computer readable medium for performing the method of distinguishing lane markers and curbs according to one or more embodiments of the above-described method when the computer program product is executed on a processor of an electronic controller , The invention thus also includes a computer-readable medium, particularly in the form of a computer-readable floppy disc, CD, DVD, memory card, USB memory device, or the like, stored in the program code means for discriminating lane markers and curbs according to one or more of the above mentioned embodiments perform when the program code means are loaded into a memory of an electronic control unit and processed on a processor of the electronic control unit.

Further features of the invention will become apparent from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and feature combinations mentioned below in the description of the figures and / or shown alone in the figures can be used not only in the respectively specified combination but also in other combinations, without departing from the scope of the invention , Thus, embodiments of the invention are to be regarded as encompassed and disclosed, which are not explicitly shown and explained in the figures, however, emerge and can be produced by separated combinations of features from the embodiments explained. Embodiments and combinations of features are also to be regarded as disclosed, which thus do not have all the features of an originally formulated independent claim. Moreover, embodiments and combinations of features, in particular by the embodiments set out above, are to be regarded as disclosed, which go beyond or deviate from the combinations of features set out in the back references of the claims.

• 1 in einer schematischen Draufsicht ein Kraftfahrzeug in dessen Umgebung sich eine Fahrbahnmarkierung und ein Bordstein befinden, welche durch ein Kamerasystem des Kraftfahrzeugs erfasst werden;
• 2 in einem Blockdiagramm eine erste Ausführungsform einer Rechenstruktur für das vorliegende Verfahren; und
• 3 in einem Blockdiagramm eine zweite Ausführungsform einer Rechenstruktur für das vorliegende Verfahren.

Showing:

• 1 in a schematic plan view of a motor vehicle in the vicinity of which are a lane marking and a curb, which are detected by a camera system of the motor vehicle;
• 2 a block diagram of a first embodiment of an arithmetic structure for the present method; and
• 3 in a block diagram, a second embodiment of an arithmetic structure for the present method.

1 shows a motor vehicle 1 , which according to a direction of travel 11 a roadway 12 is traveling. A camera system 2 of the motor vehicle 1 has a camera in the present case 20 for capturing images 4 the environment U on. A detection direction 22 the camera 20 is essentially parallel to the direction of travel 11 of the motor vehicle 1 aligned. This captures the camera 20 a the motor vehicle 1 in the direction of travel 11 lying area of an environment U of the motor vehicle 1 , The camera 20 can therefore also be referred to as a front camera. Such front cameras are known from the prior art. The camera system 2 or the camera 20 is part of a driver assistance system 10 of the motor vehicle 1 , The driver assistance system 10 further comprises a control device 3 , The control device 3 is divided into several functional units. Concretely, the control device comprises 3 in the present case, a receiving unit 31 for receiving and / or providing images 4 the environment U from the camera system 2 or the camera 20 , a recognition unit 32 , an artificial neural network 33 , an extraction unit 34 and a classification unit 35 , The function of the individual functional units will be explained in more detail below.

In the environment U there are objects 6 , in this case a road marking 7 as well as curb 8th , The driver assistance system 10 or the control device 3 is designed to curbs 8th and road markings 7 to distinguish. This is especially important because both lane markings 7 as well as curbs 8th each elongated objects 6 , which can easily be confused in the context of an image analysis. The driver assistance system 10 recognizes objects 6 , in this case road markings 7 and curbs 8th , in the environment U. Also, the control device 3 or the driver assistance system 10 adapted to road markings 7 and curbs 8th to distinguish. Data obtained in this way can be used by other driver assistance systems of the motor vehicle 1 to be provided. Based on this data about objects 6 For example, driver assistance functions can be provided. Examples of driver assistance functions are lane departure warning, autopilot or autonomous control of the motor vehicle 1 and collision avoidance wizards.

The 2 and 3 each show, by way of example, an embodiment of an arithmetic structure for carrying out a method for distinguishing lane markings 7 and curbs 8th , The in the 2 and 3 Example computational structures shown in the control device 3 of the motor vehicle 1 be realized or programmed. Both embodiments own two pictures 4 the environment U through the receiving unit 31 the control device 3 out of the camera 20 or from the camera system 2 be received. In the two pictures 4 is the environment U captured in a different perspective. In other words, the two pictures show 4 a respective representation of the environment U from a different perspective. The different perspective of the two pictures 4 in the present case is due to a movement of the motor vehicle 1 in the direction of travel 11 conditions. By the movement of the motor vehicle 1 changes the camera 20 their position regarding the environment U , whereby the respective perspective of the delayed taken pictures 4 different. This Method is also referred to as motion stereo or motion stereo.

In the two embodiments according to 2 and 3 is provided in each case that the two pictures 4 be evaluated together on a three-dimensional level and a picture 5 which is preferably one of the two images 4 is evaluated on a two-dimensional plane. In other words, one of the two pictures 4 as the picture 5 evaluated in two dimensions. The evaluation on two-dimensional level of the image 5 takes place in both embodiments according to 2 and 3 by means of the artificial neural network 33 , In some embodiments of the invention, both images may also be used 4 independent or redundant as the respective image 5 be evaluated on a two-dimensional level. However, this is not necessarily necessary. In the present case, only one of the two images is used to save computing capacity 4 as the picture 5 evaluated in two dimensions.

First, the embodiment according to 2 to be discribed. The approach underlying this embodiment may be referred to as late fusion, since the evaluation on two-dimensional level and three-dimensional level is done separately and only in a final step, the respective results 40 . 47 be merged. Through the detection unit 32 is in the picture 5 at least a two-dimensional feature 42 certainly. The recognition unit 32 is especially as an input part of the artificial neural network 33 executed. In particular, it is the recognition unit 32 a so-called CNN (Convolutional Neural Network) encoder, so an encoder for an artificial neural network. Through the detection unit 32 and the artificial neural network 33 becomes the picture 5 semantically segmented. Thereby two-dimensional features become 42 which are in the detection unit 32 be determined, grouped into segments. The semantic segmentation is preferably done at the pixel level. In other words, the picture becomes 5 segmented semantically at the pixel level. In doing so, semantically related areas of the image become 5 based on the two-dimensional image information of the image 5 grouped in segments. In the context of this semantic segmentation may be lane markings 7 and curbs 8th be recognized. Because the lane markings 7 and the curbs 8th however, often lack distinctive features, it is often difficult to distinguish them. The artificial neural network gives as a result 40 the evaluation of the two-dimensional features 42 a semantic card. The classification unit 35 classifies or classifies the objects 6 based on the result 40 , in particular the semantic map, either as a lane marking 7 or as a curb 8th ,

For an improved distinction between lane markings 7 and curb 8th In addition, in the present case, depth information from the two pictures is allowed 4 provided. The depth information is in the pictures 4 because of their different perspectives. In the embodiment according to 2 becomes a distance map as the depth information 41 certainly. The distance map 41 can each pixel of the distance card 41 a respective distance from the camera plane 21 assign. The distance map 41 thus includes respective distances of regions of the environment U, which by the distance map 41 (and thus also through the pictures 4 ). In other words, the distance map arranges 41 the pixel-wise by the represented regions of the environment U to a respective distance value. This distance value is in particular parallel to the detection direction 22 the camera 20 to understand. In other words, the respective distance values give the distance map 41 the respective distance of the respective region of the surroundings U from the camera plane 21 parallel to the detection direction 22 at.

By means of machine vision, a representation of the object can be evaluated by evaluating mathematical methods 6 in the distance map 41 be searched. For example, this can be a segmentation based on the distance values of the distance map 41 be provided. In other words, contiguous regions of pixels with the same or similar distance values are combined in one segment. For example, distance values may be considered similar if and only if they deviate from each other by at most one predetermined limit value or from a common average value. Based on this segmentation method, the distance map 41 be further processed to a segmented distance map. For example, in the distance map 41 or in the segmented distance map after a representation of the object 6 searched. In particular, then a representation in the distance map 41 or segmented distance map detected when a predetermined pattern in the distance map 41 or the segmented distance card is detected. For example, at least one predetermined pattern for a lane marking 7 and / or at least one predetermined pattern for a curb 8th specified. If one of these predetermined patterns is detected, a representation of the object applies 6 as recognized. For example, lane markings 7 to be considered flat. In this case, there is no predetermined pattern for a lane marking 7 intended. road markings 7 can in this example only by the distance map 41 or the segmented distance map are not recognized. In this case, the distance map 41 or the segmented distance card only one Representation of a curb 8th as an object 6 be recognized. Regardless of the exact design, lane markings 7 and curbs 8th by the distance map 41 or the segmented distance map are distinguished particularly well.

In the classification unit 35 will be the result 40 when evaluating through the artificial neural network 33 as well as the result 47 when evaluating the distance map 41 for classifying or classifying the object 6 either as a lane marking 7 or curb 8th used. It can by the classifying unit 35 a probability-based fusion approach should be provided. In other words, the result 40 as well as the result 47 when evaluating the distance map 41 for classifying the object 6 probability-based merged. For example, when evaluating through the artificial neural network 33 as well as when evaluating the distance map 41 determines a respective confidence level. The confidence level can indicate a quality of the evaluation. When merging or classifying, that of the results 40 . 47 weighted higher whose confidence level is higher. A higher confidence level gives a higher quality in the respective evaluation or a higher probability for the correctness of the respective result 40 . 47 at.

In the embodiment according to 3 is the extraction unit 34 as another input part for the artificial neural network 33 educated. In other words, the extraction unit 34 as another CNN encoder. Through the extraction unit 34 or the other CNN encoder is at least a three-dimensional feature 43 from the pictures 4 the environment U determined. For determining the at least one three-dimensional feature 43 can the pictures 4 initially as part of a three-dimensional evaluation 37 be evaluated in perspective. Only through this three-dimensional evaluation 37 can the depth information, in this case the at least one three-dimensional feature 43 , from the two pictures 4 be extracted or determined. As part of the three-dimensional evaluation 37 can also be generated a distance map, which is analogous to the distance map 41 to understand. In this case, this can be at least a three-dimensional feature 43 from within the framework of the three-dimensional evaluation 37 created distance map determined or extracted.

Both two-dimensional features 42 as well as three-dimensional features 43 can be determined within semantic segmentation. In particular, according to the embodiment 3 semantic segmentation on two-dimensional level and three-dimensional level continuously by means of a learning system, namely by the artificial neural network 33 , intended. In this example, the training of the artificial neural network takes place 33 not just on the basis of the two-dimensional features 42 but also on the basis of depth information, namely on the basis of the at least one three-dimensional feature 43 , In this way, a learning algorithm of the artificial neural network 33 learn, two-dimensional features 42 as well as three-dimensional features 43 to connect with each other.

As part of a link 45 the two-dimensional features 42 and three-dimensional features 43 the respective output values of the two CNN encoders (in this case the recognition unit 32 and the extraction unit 34 ) are normalized. This normalization is present as part of a learning process of the artificial neural network 33 carried out. Within this normalization become the two-dimensional features 42 and the three-dimensional features 43 weighted relative to each other. This weighting can be adjusted in real time during application of the method. For example, the weighting is done on the basis of respective confidence levels, which for the two-dimensional features 42 and the three-dimensional features 43 be determined. Subsequently, the two-dimensional features 42 and the three-dimensional features 43 as summarized features 38 through the artificial neural network 33 be evaluated.

The entire learnable pipeline, which is controlled by the two CNN encoders (detection unit 32 and extraction unit 34 ) to the output of the artificial neural network 33 is given, can be taught throughout (ie end-to-end). In this case, respective training data, which pairings of images, which are analogous to the two images 4 are constructed, include, used for learning. In the training data, respective road markings and curbs can be marked. In addition, areas of the images of the training data may be marked, which include neither a lane marking nor a curb (so-called negatives). In a further embodiment, training data can be evaluated by the adaptive pipeline and incorrectly classified objects can be correctly marked for further training. Incorrectly classified objects are in particular wrongly positively recognized road markings or curbs. This procedure is also referred to as Hard Negative Mining, for example using incorrect assignments.

Overall, the two embodiments show how linking the evaluation of images to differentiating road markings 7 and curbs 8th on a two-dimensional level and on a three-dimensional level is possible. In particular, this shows how an accuracy in distinguishing lane markings and curbs can be improved. Position and type of object 6 (Road marking 7 or curb 8th ), as a result 36 , in particular for a driver assistance function of the motor vehicle 1 , provided.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 2016/0104047 A1 [0004]

Claims (12)

Method for distinguishing lane markings (7) and curbs (8) in an environment (U) of a motor vehicle (1) with a camera system (2) of the motor vehicle (1), comprising the steps: Providing two images (4) of the surroundings (U) by the camera system (2) of the motor vehicle (1), wherein a respective perspective of the two images (4) differs, Determining at least one two-dimensional feature in at least one of the two images, wherein the at least one two-dimensional feature represents an object in the surroundings of the motor vehicle; Evaluating the at least one two-dimensional feature (42) by means of an artificial neural network (33), Extracting depth information (41, 43) from the two images (4) of the environment (U), and Classifying the object (6) in the environment either as a lane marking (7) or as a curb (8) both on the basis of a result (40) when evaluating the two-dimensional feature (42) and on the basis of the depth information (41, 43). Method according to Claim 1 , characterized in that - as the depth information at least one three-dimensional feature (43) is determined, wherein the at least one three-dimensional feature (43) represents the object (6) in the environment (U) of the motor vehicle (1), and - that at least a three-dimensional feature (43) is also evaluated by means of the artificial neural network (33). Method according to Claim 2 , characterized in that the at least one two-dimensional feature (42) and the at least one three-dimensional feature (43) are evaluated jointly (38) by means of the artificial neural network (33). Method according to one of the preceding claims, characterized in that a distance card (41) is determined as the depth information. Method according to Claim 4 , characterized in that by means of machine vision, in particular using mathematical algorithms, a representation of the object (6) in the distance map (41) is sought, in particular determined, becomes. Method according to Claim 5 characterized in that classifying the object (6) is additionally based on searching the representation. Method according to Claim 5 or 6 , characterized in that the classification of the object (6) by means of the artificial neural network (33) and the machine vision initially takes place independently of one another and respective results (40, 47) of the classification are subsequently fused, in particular based on probability. Method according to Claim 7 , characterized in that the classification of the object (6) on the basis of the artificial neural network (33) takes place exclusively on a two-dimensional plane and the classification of the object (6) on the basis of machine vision exclusively on a three-dimensional plane. Method according to one of the preceding claims, characterized in that the two images (4) of the environment (U) by means of the same camera (20) of the motor vehicle (1) are detected offset in time. Control device (3) for distinguishing lane markings (7) and curbs (8) in an environment (U) of a motor vehicle (1) with images (4) of a camera system (2) of the motor vehicle (1) a receiving unit (31) for providing two images of the surroundings (U) from the camera system (2) of the motor vehicle (1), wherein a respective perspective of the two images (4) differs, a recognition unit for determining at least one two-dimensional feature in at least one of the two images, wherein the at least one two-dimensional feature represents an object in the surroundings of the motor vehicle, an artificial neural network (33) for evaluating the at least one two-dimensional feature (42), an extraction unit (34) for extracting depth information (41, 43) from the two images (4) of the environment, and - A classification unit (35) for classifying the object in the environment either as a lane marking or curb both on the basis of a result (40) in the evaluation of the two-dimensional feature as well as the depth information (41, 43). Driver assistance system (10) with - a camera system (2) having at least one camera (20) for detecting two images (4) of the environment (U), and - a control device (3) according to Claim 10 , Computer program product comprising program code means stored in a computer readable medium for the method of distinguishing lane markings (7) and Curbs (8) according to one of the preceding Claims 1 to 10 perform when the computer program product is executed on a processor of an electronic control device (3).

Images