WO1995023364A2 - Ccd sensor system for tracking manipulators - Google Patents

Abstract

The invention concerns a rapid sensor system for the determination of the paths of robots. The system consists of an imaging sensor (4, 5) and a projector (7 to 11) for illuminating the surface features of the object. This system is capable of measuring five degrees of freedom in real time.

Description

CCD sensor system for path tracking of handling systems

Today's robot systems are faced with ever increasing demands regarding their machining flexibility. Tasks for which the robot has to master all six spatial degrees of freedom occur more and more frequently. A path control that calculates the contour to be traversed solely from the data of a CAD system or from subsequently learned base values can no longer meet the increased demands on flexibility. The problem often arises that the parts to be machined are positioned differently due to the process, so that the path contour is subject to scattering, or the contour to be traversed is not known at all beforehand. Then, at the latest, a sensor-assisted path correction or a sensor-guided path tracking is required. For this purpose, e.g. B. laser distance sensors or CCD line scan cameras are used. These sensors have the disadvantages that they can either measure only a few degrees of freedom at the same time or achieve a measurement rate that is too low for real-time requirements. In order to compensate for these deficits, systems with several identical or different sensors are sometimes built. Such "multi-sensor systems" quickly reach a level of complexity that can hardly be mastered.

It is therefore u. a. an object of the invention to improve the speeds (or: sampling rates) of known sensors without increasing the measurement or evaluation effort and getting to certain image areas as often as possible; consequently, only the accelerated acquisition of the required image content (lines or partial images).

To this end, the invention proposes a multitude of possibilities (claim 1, claim 10, claim 13, claim 15, claim 20 and claim 21), which are further specified with the respective dependent claims. All of these solutions have in common that it is avoided to first read out those image contents from the CCD charge image area (image zone) in order to subsequently convert a large part of the time-consuming image information into measured values. Rather, only the required partial area, which can also be individual lines or parts of lines, is read out, while the rest of the image is still deleted in the image zone.

The anti-blooming electrode can be used for deletion, contrary to its actual purpose.

In order to accelerate the reading out, the memory zone of the CCD image sensor can be omitted and the readout register can be arranged directly on the image zone (claim 13).

If the image section essential for the measurement changes (claim 22), it can be in its size and position can be tracked. The "dynamically changeable image windows" formed in this way are each read out in such a way that time-consuming reading and moving from the image zone into the storage zone is kept as short as possible. The change in the readout direction of the readout register can also be used for acceleration.

The invention proposes the fast measuring system for the autonomous detection of trajectories by robots; the measuring system consists of an image sensor and a projector for structured illumination of the object surface. With this measuring system, five degrees of freedom in space can be measured in real time.

With the image sensor proposed here, any two image lines can be read out in a time pattern of 1 ms. The intended goal of building a fast measuring system for (autonomous) contour tracking with an industrial robot is thus achieved according to the invention. In addition, the camera (the "image sensor") is equipped with maximum flexibility, so that the built-up sensor can be used universally. Pattern recognition, quality control and other typical tasks for CCD image sensors are areas of application of the invention. The high level of flexibility also opens up new areas of application for the use of CCD image sensors, partly in areas that were previously only reserved for CCD line sensors. The large gap between image and line sensors can thus be almost completely eliminated.

Using a transputer as a microcontroller offers a simple and fast interface for coupling the sensor to a transputer network or to any other device that has a link interface (e.g. a PC with a link adapter) via a link connection.

A camera can be used as the sensor, which is modified and adapted in accordance with the proposals of the invention, and a bare lens with a CCD chip arranged behind it can also be provided.

With few changes, the control electronics can also be used for other CCD sensors, which could be shown in practical tests with the TH 7864A (from Thomson). A "universal sequencer" can be implemented that flexibly supports most common CCD sensors. Hardware changes would then only result in the implementation of different sensor interface components.

At the core of a camera is a CCD image sensor with, for example, 768 * 576 pixels, control electronics specially developed for the CCD chip, and coordinated electronics digital evaluation electronics. These components enable targeted and time-optimal reading of the required image sections from the image matrix. So z. B. the first image line can be read up to 2500 times per second. All other image lines are already deleted in the sensor in order to save the otherwise required readout time (claim 2). Two freely selectable image lines can be read out and processed every millisecond.

The projector projects e.g. six (thin) measuring lines arranged substantially perpendicular to the image lines on the surface of the object to be measured. A line detection algorithm determines the high-resolution positions of the

Measuring line centers. The coordinates of points on the object surface can be calculated from the intersections between the measurement lines displayed on the sensor and the evaluated image lines using a simple triangulation method - similar to a laser distance sensor. The measuring system thus simulates a matrix of n * m laser distance sensors, where n is the number of projected measuring lines and m is the number of evaluated image lines.

In addition, the position of a significantly stronger line on the object (path marking or object contour) can be recognized. If the coordinates of several object points in the vicinity of this line have been measured, the coordinates of a line point on the object can be determined with high accuracy by interpolation between the corresponding image points and the intersection of the line with an image line on the sensor.

The measuring method is used for the autonomous tracking of web markings on spatial surfaces with an industrial robot. Five spatial degrees of freedom can be measured and made available for control in the Cartesian control loop or for path planning.

The measurable degrees of freedom are:

* the lateral deviation of the robot hand from the path marking (y component)

* the distance from the robot hand to the object surface (z component)

* the complete orientation of the robot hand with respect to a path line on the object (angle, ß and γ corresponding to the rotations around the coordinate axes x _h , y _h and z _{h of} the hand-specific coordinate system).

The remaining sixth degree of freedom - the feed direction v along the path marking B - is not regulated. A feed rate v is explicitly specified by the path control. The orientation of the robot hand 1 can e.g. B. be regulated so that a Tool is always aligned perpendicular to the object surface and the x-axis of the hand-specific coordinate system points exactly in the direction of movement along path B.

To measure the five degrees of freedom, the evaluation of only two image lines is sufficient. The actual position of the robot hand 1 relative to the path marking B is detected with the first image line. The position of a point of path B lying further ahead can be determined from the second, leading image line (the point is formed by the intersection of the scanning line and the path curve). This data can be used for path planning and pilot control concepts. The quality of the feedforward control can be increased by evaluating a third, trailing line.

The measuring frequency of 1 kHz achieved with the measuring system corresponds to the interpolation cycle (lms) of an experimental robot controller based on transputer.

The following examples are intended to deepen the understanding of the invention:

Figure 1.1 shows schematically the viewing angles of the projector for the measuring lines and the camera with the CCD sensor. The viewing angles of the camera and projector are at a fixed angle and look at the object along the path tracking line, which is to be recognized and tracked, with an essentially identical section. The dark area in Figure 1.1 projects measurement lines or other symmetrical structures onto the surface of the measurement object. The camera detects these measuring lines together with the path curve running between two of the measuring lines (cf. also FIG. 2.2). The scanning direction a of the line-oriented CCD chip runs perpendicular to the measuring line direction and approximately perpendicular to the path line direction B. The viewing angle of the optics of the camera is shown in light gray in Figure 1.1.

Figure 1.2 illustrates the movement of the object with its surface and the resulting overlap of the viewing angles of the projector and camera.

FIG. 2.1 is an enlargement in a schematic representation of the front end of a robot hand 1. The camera 4 outlined in the previously mentioned figures with a CCD image sensor 5 and lens 6 is arranged on it, while at the angle to the central axis of the camera 4 the measuring line Projector housing 8 is arranged with its condenser lens 9, the measuring slide 10 defining the line structure and the imaging lens 11. Behind the condenser lens 8 there is a field with a large number of luminous LED diodes 7 which stroboscopically measure the measuring lines of the slide 10 onto the object surface project. The reading of the image zone of the CCD chip 5 is synchronized with the stroboscopic illumination of the surface, so that the influence of extraneous light can be suppressed. If the robot hand 1 moves, the measuring device 4 and 8 moves together with it, since the projector and camera are arranged in a common housing 3 or on a common mounting plate of this housing 3. The mounting plate 3 is immovably fixed to the robot hand via a flange 2.

The view of the measuring surface is explained in FIG. 2.2, the measuring lines l _j to lg can be dark lines, while the remaining field is brightly illuminated. The contour of the trajectory B runs within two measuring lines, the course between the measuring lines I3 and I4 is shown. The length of the measuring lines in the direction of movement v of the robot hand 1 can be made dependent on how strongly curved the trajectory B is.

When using a CCD line sensor for autonomous line tracking, the question arises of how to attach the sensor - with or without a lead, optimal lead distance. The type of attachment influences the properties of the system, whereby it should be optimized from the point of view of achievable accuracy, maximum travel speed and possibilities for path planning and pilot control. A dynamic variation of the line feed is desirable, but technically difficult to achieve.

By evaluating two lines of the CCD image sensor, all necessary information can be obtained in order to completely determine the position and orientation of the robot hand 1 with respect to the path B. The disadvantages of the higher readout time of a CCD image sensor and its predetermined rigid (usual) readout procedure to comply with the CCIR TV standard of the output signal curve would reduce the maximum achievable image sequence to 25 frames or 50 fields per second if not the speed with the invention would be increased considerably. The invention also "handles" the large amount of data that occurs in image sensors, which has to be processed or at least recorded, without problems.

In the following, the accelerating suggestions are presented by way of example in order to enable time-optimal reading out of only a few required image lines from a CCD image sensor. The procedures are e.g. particularly efficient with the THX 7868A or TH 7864A from Thomson.

Image sensors are generally adapted to the duration of the display of a television picture. The image is read out in two successive fields (interline method), the readout time being 20 ms for each field. 40ms are required for a complete picture. This factor alone limits the measuring rate to a maximum of 25 measurements per second - theoretically. In addition, there is the processing time for the data (in) quantities. The processing of the data usually takes a multiple of the readout time. So far, these measuring rates have not been suitable for a high-quality robot control with an interpolation cycle of a few milliseconds. Measurement rates that ideally correspond to the interpolation cycle or are at least less than an order of magnitude lower would be required.

By flexible control of a suitable CCD image sensor, i. H. by abandoning the rigid control according to the CCIR-TV standard, such a high measuring rate can be achieved with reasonable effort. Thereby all are not needed

Image information already deleted in the sensor in order to save the readout and processing time. This means that access (only) to the required data is time-optimized.

The proposals of the inventions were tested with a six-axis articulated arm robot. For the acquisition of five spatial degrees of freedom, the image information from two to three (preferably arbitrarily selectable) image lines are required. At feed speeds of up to 1m / sec (36km / h), the path contour B or measuring line is resolved with sufficient accuracy (1 measuring point / mm). This results in a measuring rate of 1000 measurements per second. This measuring rate corresponds to the interpolation cycle of the aforementioned control on a transputer basis.

Background of the CCD image sensors, which are divided into three groups:

* Interline transfer sensors

* Frame transfer sensors * Frame interline transfer sensors

A typical frame transfer image sensor is shown in simplified form in FIG. 3.1. It consists of the following functional units:

an image zone which is formed from a two-dimensional matrix of individual image elements (pixels);

* an equally large storage zone;

* the readout register directly adjacent to the memory zone, the length of which corresponds to the number of pixels of a pixel line of the image and memory zone;

* the output stage connected to the first pixel element of the readout register.

The image and storage zones are largely identical. However, the storage zone is covered by an opaque layer, so that incident light has no effect there. Furthermore, all devices for exposure control are missing in the storage zone since they are not required there. The charges generated in the image zone by the action of light (charge image) can be shifted vertically within the two zones using clock signals (usually four phase-shifted by 90 °) according to the bucket chain principle. Separate clock signals (P clocks and M clocks) are available for the image and memory zone, so that the charges in the memory zone can also be moved separately without influencing the charges in the image zone.

The charges from the line that directly adjoins the readout register can be pushed into the readout register in a single step. In the readout register, they can then be transported horizontally to an output stage by means of further clock signals (L clocks). The output stage converts the charge packets into a voltage proportional to the amount of charge, which can be tapped at the video output of the sensor.

Figure 3.2 shows the chronological sequence of an image recording and the subsequent reading process.

Starting from the initial situation (a), in which both the image and the storage zone are free of charge carriers, the sensor is exposed for the first time. In the image zone, the charge carriers generated by exposure to light are so-called.

Potential pots integrated (b to d). The integration time is almost 20ms according to the CCIR TV standard. In many modern sensors, however, it can be reduced as desired using a suitable clock control (electronic shutter).

After the exposure time has elapsed, the charges are transferred vertically from the image zone to the storage zone (e and f). The transfer typically takes about 350μs. During the vertical transfer, charges are still generated by exposure to light, which are also integrated and smear the image slightly (smearing effect). The pollution is of course not noticeable in the first line and is strongest in the last line. It is usually. low because the transfer time / integration time ratio is very small (350/20000 = 0.0175). However, the influence of the smear effect becomes more important the further the exposure time is reduced with the help of the electronic shutter. Extremely bright pixels can also cause a clearly visible tail (drag effect).

After the vertical transfer has been completed, the image is completely in the storage zone, the first image line being located directly on the readout register. This is followed by line-by-line reading of the image from the memory zone via the read-out register to the output stage. In the time required for this - in parallel to this - the next picture in the Image zone integrated (g to i). After the reading process has ended, the state shown in (d) is reached again and the cyclical process begins again.

Once charges have been generated in the image zone of the sensor by the incidence of light, they can only be eliminated by time-consuming, sequential pushing out of all image lines via the readout register into the output stage.

Figure 3.3 illustrates the results when the entire middle third of the image is pushed together in the readout register in order to access the image content in the last third of the image more quickly. The area shown in black in the picture represents the collapsed area. Although the picture as a whole is quite dark, a large part of the last section of the picture is "flooded" with charges.

In order to get the information from just any two "useful lines" - the image section - as quickly as possible without the content; of the useful lines to destroy or impair are carried out:

1. Vertical transfer of the image from the image zone to the storage zone.

2. Push the lines that are not required down to about eight lines before the first useful line. In the meantime a fast horizontal transfer takes place. 3. Reading out seven lines with fast horizontal transfer.

4. Reading out the last line before the actual useful line with a normal, quality-preserving shift frequency.

5. Reading out the first useful line with normal shift frequency.

6. Repeat steps 2 to 5 for the second useful line. 7. Delete any possibly following lines in any way.

The last step can also be omitted.

In the image and in the storage zone, horizontally adjacent pixels are separated by vertically extending, fixed potential barriers which are generated by means of a gate. The height of the potential wall can be set via the potential applied to the gate. Even if the height of the potential wall is set high, it can happen that overexposure of the sensor overflows the potential wells and charges flow into neighboring, less strongly exposed picture elements. The associated disturbing impairments of the image are called "blooming". An anti-blooming unit to prevent this effect is therefore usually integrated in modern CCD sensors. For this purpose, the gate between two adjacent pixel columns is divided into two individual electrodes, a so-called anti-blooming drain being inserted in the intermediate space. The anti-blooming drain picks up the excess electrons from the overfilled potential wells would otherwise flow into the neighboring potential wells.

An electronic shutter for setting the exposure time can be implemented by means of the anti-blooming unit controlling the drain. By raising the potential well bottoms to the level of the potential wall - the potential wells then disappear completely - all the charges generated by exposure to light flow immediately into the anti-blooming drain. The potential well bottoms are raised by placing all four P cycles at the same, appropriately selected potential. The integration of the optically generated charges only begins when the potential well bottoms are lowered again. With the electronic shutter, extremely short exposure times of a few microseconds can be achieved.

Activating the electronic shutter function immediately before the vertical transfer of the image from the image to the storage zone provides an absolutely black image. This could be demonstrated by tests on the sensors TH 7864A and THX 7868A. This opens up the possibility of activating the shutter function during the vertical transfer, immediately after the transfer of a specific, freely selectable line from the image zone to the memory zone, which results in the deletion of the residual image that is still in the image zone. The part of the image already in the storage zone is in no way affected. The shutter function can thus be used to set the number of m rows to be read (0 <m <292 for TH 786A).

Since the charge of the (292-m) remaining image lines can be completely eliminated, only the charges of the first m lines have to be pushed out of the memory zone via the readout register. The time required to process the last (292-m) non-relevant lines is therefore completely eliminated. If in particular only the first image line is required, i.e. The image sensor is used as a line sensor, so image sequences of up to 2500 images per second can be achieved. The "misuse" of the anti-blooming electrode to delete the image zone is not intended by the manufacturer.

Figure 3.4 illustrates the reading cycle described, in which, for example, only the first half of the image is read out. Starting from a completely exposed image in the image zone (a), a vertical transfer of the first 146 lines into the storage zone is first carried out (b). As soon as the last line to be read is in the memory zone, the P clocks are stopped and the remaining field in the image zone is deleted with the anti-Bloomin unit (c). During the period of only a few microseconds

Deletion process, the drawing file to be read is shifted further in the direction of the reading register (d). Deleting effectively does not contribute to the total readout time. After the deletion process, the integration of the next image begins in the image zone. This process in turn runs simultaneously with the reading out of the first half of the image from the memory zone (e, f).

With the measures discussed so far, reading out any two image lines would also be conceivable with the TH 7864A in only about three to six milliseconds.

A special feature of the THX 7868A CCD image sensor is a so-called windowing device (windowing device), with which any image lines can be completely deleted individually and in just 2μs. In contrast, reading out a line takes approx. 64μs.

The windowing unit is a quenching device arranged parallel to the readout register. It essentially consists of the windowing drain, which is separated from the readout register by the windowing gate. During normal operation, a potential is applied to the windowing gate, which creates an insurmountable potential barrier for the charges contained in the readout register. When the potential barrier is lowered by a suitable voltage control of the windowing gate, the charges flow out of the readout register into the windowing drain, which is equivalent to deleting the line information. A targeted control of the windowing gate consequently allows the quick deletion of individual image lines and areas, which saves the high expenditure of time for the serial reading out of the image lines in question.

Experiments have shown that the windowing unit can be used to clean parts of a line properly. This results in the possibility of only reading out the first n pixels (0 <n <845) of a line which contain relevant information. The part of the line that is not required (the last 845-n pixels) can be deleted in order to save even more time.

As a further special feature, the THX 7868 A has a second output stage, which is constructed analogously to the first stage, but with the difference that it is connected to the last instead of the first picture element of the readout register. The horizontal transfer direction can be influenced by suitable control of the shift clocks on the readout register. By reversing the normal shift direction, the line information is passed to the second output stage in reverse order. An image read out via this output appears horizontally mirrored. If the desired image contents are more in the left half of the image, the lines are read out normally. However, if the information is predominantly on the right in the picture, the lines are read out mirrored.

In combination with the option to delete parts of a line, the Mirroring function can be used to get the desired information of an image even faster.

Since only a few, but widely spaced, lines have to be read out in a very short time for rapid contour tracking, the windowing function of the sensor is advantageous for the intended application.

The so-called tracking function may be mentioned here as an example. With the tracking function a z. B. particularly bright point can be tracked due to the fast achievable image sequence with a high temporal and spatial resolution. The image section can be tracked in real time - following the position of the point - the required window size and position can be selected depending on the current speed and direction of movement (Figure 3.6). Interesting areas of application are conceivable for this, for example motography.

Since in the planned application of the camera, due to the selective reading out of partial images, the entire normally available exposure time for the next image is no longer available, stroboscopic illumination with intense flashes of light can be provided in order to increase the sensitivity.

If only stroboscopic lighting is used, the storage zone can be omitted.

An overview of the circuit components of all camera electronics combining all features is given in the block diagram in FIG. 4.1. Regarding the technical implementation of the camera sequencer, it should be noted that the sequence control units were designed as simple Mealy machines, which are housed in a total of five AMD type MACH210-15 PLD modules.

The camera electronics contain their own microcomputer based on transputer. The IMS T222 is above the system logic in the hierarchical structure of a parent

Transputer node (up system) embedded as a subordinate node (down system). The transputer receives control commands from a controlling transputer via link 0. This is generally the transputer of the frame grabber assembly. The camera can also be controlled by any other transputer module or by a PC with a link adapter. The only requirement for this is that the link connection must be made differentially, since the link drivers are specified in accordance with the RS422 standard. Differential transmission was chosen because it is particularly robust against electromagnetic interference. It ensures trouble-free operation in the severely disturbed environment. The camera sequencer can be programmed and controlled by the transputer via the 2048- 16 bit (4 kByte) large two-port memory and the four 16 bit registers. For practical use of the camera, this means that the reading of each image from the CCD sensor can be programmed individually. The mentioned tracking function is thus directly supported by the sequencer.

The digital clock signals generated by the sequencer for the CCD sensor are transmitted to the camera via coaxial cables. There they are converted by the sensor interface to the voltage level as specified for the sensor control in the data sheet. The sensor interface also ensures that the required steepness is observed. To generate all the clock sequences, a camera sequencer was built, which can perform several simple basic functions in three time levels, the sequence of which can be modified by programming. The three time levels are justified as follows:

* Cycles for processing individual pixels (approx. 67ns)

* Cycles for processing individual image lines (up to approx. 61μs)

* Cycles for processing fields (up to approx. 20ms)

Figure 4.4 gives an overview of the function blocks of the camera sequencer. The

Sequencer is made up of two control units, which are arranged hierarchically, but otherwise work largely independently of one another. It generates 16 different digital control clocks for time-optimized reading of the relevant image sections and for deleting all unnecessary image areas. Different sub-functions of the control units are generated depending on programmable input parameters and input flip-flops.

The shutter register

The two shutter registers are used for electronic exposure time control. At the beginning of an image cycle, the exposure counter is loaded with the content of a shutter register. The exposure counter is then counted down to zero with a clock cycle of approximately 533 ns. As long as the counter content is not zero, the formation of potential pots is prevented by suitable control of the P clocks, so that all generated charges flow into the anti-blooming drain and the integration of an image is prevented (electronic shutter). As soon as the value zero is reached, the integration of the image is released. The exposure release is also used as an external signal to control a stroboscopic light source 7, so that the flash light can be fired at exactly the right time - at the beginning of the exposure. The two shutter registers can be programmed independently of one another for delay times of different lengths. For example, it is conceivable to record a scene in two successive fields under different lighting conditions. Is z. For example, if a shutter register is initialized with the value zero and the other with a value that leads to an exposure time of only 100 μs, the image scene is recorded in one image in daylight or ambient light, while the scene in the following image with - possibly structured - flash light is recorded. The influence of daylight and ambient light is practically completely suppressed when taking flash pictures due to the short exposure time.

The image parameter register

Apart from the exposure delay, all image parameters can be programmed via the two 16-bit wide image parameter registers. Figure 4.5 shows the assignment of the individual bits of an image parameter register.

begin

The initialization of the sequencer by setting this bit after switching on the camera or after a reset ensures a defined start of the control units and is used for synchronization with the transputer.

Reset

Reset the sequencer. The sequence control unit, the clock control unit and all counters are reset to the basic state. All clocks and other output signals are set to a defined, non-critical resting potential. After the reset, the start bit must be set for a defined start of the sequencer.

Figure 4.5 shows the assignment of the bits of a single line parameter register.

Number of pixels from the line (10 bits) The least significant 10 bits are used to load the pixel counter that decides the number of pixel output cycles for the current line. After the programmed number of pixels has been output, the windowing unit is activated and the rest of the line is deleted.

Left shift Each line can be output to either the left or right sensor output. The bit informs the control unit of the desired shift direction. If the bit is set, the line is read out in the normal direction via VOS1. If the bit is deleted, the line is read out in the opposite direction via VOS2. The control unit automatically takes into account the different number of irrelevant pixels at the beginning and end of the line. A picture line sets namely consists of 845 pixels, with only 768 pixels containing image information. The remaining pixels are used as so-called pre-scan, post-scan and dark reference pixels. In

They appear in a different order depending on the direction of the slide

Sensor output:

Output VOS 1 output VOS2

* 14 post-scan pixels * 13 pre-scan pixels

* 50 dark reference pixels * 768 image pixels

* 768 image pixels * 50 dark reference pixels

* 13 pre-scan pixels * 14 post-scan pixels

The sequencer only signals the output of the relevant image pixels via the interface to the "frame grabber" (partial image capturer). The frame grabber does not process the irrelevant pixels at all.

Last line

The bit must be set in the last line to be processed because it instructs the sequencer to end the frame cycle and start the next frame cycle.

Read out The bit signals the sequencer whether the line should be read out or deleted. If the bit is set, the image line is read out up to the programmed number of pixels. Otherwise the line is deleted immediately instead of the horizontal transfer by activating the windowing unit. The next line is then processed.

The control unit reads the image from the CCD sensor in a time-optimized manner as follows:

* Eject lines 1 to 50 from the image zone into the memory zone.

* Delete the remaining lines in the image zone. From now on, the exposure of the next image can be carried out in parallel with all subsequent readout operations. * Move the 50 lines in the memory zone to the output register.

* Read out line 1.

* Delete lines 2 to 49 using the windowing unit.

* Read line 50.

After the last step, the storage zone is completely cleared. The next image has already been exposed in the image zone and can be read out.

After programming the sequencer, the readout process described runs periodically. The external synchronization mechanism for triggering the readout process (signal start image cycle in Figure 4.1) and the signal for synchronous activation of a Stroboscopic lighting can be used to implement an operation that is synchronized with the scanning and regulating cycle of the robot controller.

The functions of the flexible camera described are particularly effective with a suitably tailored "universal frame grabber" for digitizing the image data. Such a frame grabber must be able, in deviation from the "grabbing" of a full image, also to correctly process the image segments supplied by the camera and in some cases not connected. In order to achieve this goal, a suitable interface between camera and grabber is being worked out. The interface between the camera and the grabber consists of seven signal lines (Figure 5.2). In addition to the actual video signal, five other control signals are transmitted from the camera to the grabber so that the sequence control unit of the grabber can understand the image output from the camera in all possible operating modes. The five control lines have the following functions:

(a) New image (image synchronizing signal)

An image synchronizing signal is transmitted at the beginning of each image cycle (for one field). The signal causes the controller to begin "grabbing" a new image. The line counter of the control unit is reset.

(b) Field 1

The signal is required to be able to distinguish between the two successive fields (with even and odd picture lines in CCIR TV mode). If field 1 is active, the digitized image data are written in odd line positions in the image memory, otherwise the even line positions are written.

(c) New line (line synchronizing signal)

A line synchronizing signal is transmitted at the beginning of each line cycle. The signal causes the grabber to increment the line counter and reset the pixel counter. The subsequent digitized pixels are thus entered in the next "line" in the grabber's image memory.

(d) PHI (pixel shift clock)

In order to obtain a defined sampling time with the lowest possible sampling jitter, the pixel shift clock PHI was derived from the signal F2L, which is generated by the camera control electronics. This synchronization signal is applied directly to the clock input of the A / D converter of the grabber. The signal can be varied in phase to compensate for runtime effects in the transmission line from the camera to the grabber. This ensures an exact pixel-synchronous sampling of the video signal. (e) Pixel valid

The signal is required to distinguish between valid pixels and pre, post or dark reference pixels. The incoming pixels are only relevant pixels if this signal is active. The signal is deactivated during the output of the pre-, post- and dark reference pixels of a line, so that these pixels are not digitized by the grabber or transferred to the image memory.

Another control signal is transmitted from the grabber to the camera: camera start

If the signal is active, the camera sequencer can read out the CCD image sensor cyclically. If the signal is deactivated, the camera sequencer stops after processing the current image cycle. With a short pulse on this signal line, a single image cycle of the camera can be triggered from the grabber. This means that the frame rate of the camera can be set as long as the period is not chosen to be shorter than the time required by the camera sequencer for the output of all desired parts of the image.

The image sequence can be z. B. synchronize with the sampling and control cycle of the robot controller, or an extremely short sequence of images can be realized, for. B. for time-lapse recordings.

The frame grabber contains a programmable, autonomously operating sequence control unit for the cyclical acquisition of the image information supplied by the sensor. The video signal is digitized in 256 shades of gray. The frame grabber maps the physical pixel matrix of the CCD sensor as a virtual pixel matrix in a 1 Mbyte two-port memory. One byte in the frame grabber's memory thus corresponds to each pixel of the pixel matrix in the CCD sensor. The grabber is also equipped with a transputer (IMS T805) and 4 MB of RAM, so that it can program and monitor the camera functions via a link. The transputer has direct access to the digitized image data via the two-port memory. He can transfer them over several link connections with a considerable transmission bandwidth into another transputer network for real-time data processing (transputer 1, 2, n).

The complete reading of an image is announced to the transputer by means of an image interrupt generated by the sequence control unit in order to achieve short run times and thus short response times of the control.

Reading the image data into the memory of the frame grabber can be compared to writing to a paper page with a printer (eg a typewriter printer): * After receiving the image synchronization signal New Image, the grabber positions the write pointer at the first pixel position in the first line of the virtual pixel matrix - depending on the signal field on the first odd or on the first even line. This step is comparable to feeding paper into the printer and aligning the printhead to the first writing position.

* Digitized pixels are written sequentially in successive byte positions until a line synchronization signal new line (corresponding to a line return at the printer) is received. Under no circumstances does a complete image line always have to be transmitted. Rather, the number of pixels read in each image line may vary as desired.

* If several line synchronization signals arrive immediately one after the other, the corresponding image lines in the grabber's two-port memory are skipped. Only the line counter is incremented (corresponding to the insertion of blank lines during a printing process).

In contrast to conventional frame grabbers, the "universal frame grabber" presented here is able to correctly read any image format from an 1 * 1 image matrix to a 1024 * 1024 image matrix without reprogramming the control unit. It thus directly supports different CCD sensors, especially the THX 7868A with 768 * 576 pixels. Successive lines may vary in length as desired.

The control unit automatically writes the image information precisely to the positions in the two-port memory which correspond to the physical pixel positions in the CCD image sensor.

In conjunction with the interface, the grabber and camera form a particularly powerful unit for advanced real-time image data acquisition and image processing.

The test pattern in FIG. 6.1 shows that each line can be programmed individually for non-mirrored reading via VOS1 or mirrored reading via VOS2. In the top section of the picture all lines were read out normally, in the second fifth of the picture each straight line was read out mirrored. In the center of the picture, all lines, in the following part of the picture only the odd lines were read in mirror image. The bottom section contains only non-mirrored picture lines. The mirroring function is e.g. of interest for the optimal implementation of the tracking function (see Figure 3.6).

The function of the windowing unit is illustrated in FIG. 6.2. The enlargement of an image shows the transition area between the completely deleted half of a camera image and the subsequent second half of the image, that was read out. The first line read is not lighter than the following lines, which suggests that all charges from the deleted lines have actually been completely deleted (see Figure 3.3; problems with the TH 7864 without a windowing unit). However, the first line read is also not darker than the following lines, which proves that the windowing function switches between deleting and reading out in good time, so that the information in the first useful line is not or is not affected by the deletion of the previous line partially deleted with.

Figure 6.4 illustrates that the number of pixels to be read out can be set individually for each line. The black areas indicate parts of the image that have not been read from the CCD sensor but have already been deleted in the sensor. The readout times saved for each incompletely read line add up, so that the readout time is shorter, the larger the black areas. The areas of interest in an image can thus always be read out in a time-optimized manner. In particular, the effective pixel matrix of the sensor can be programmed between 1 * 1 and 768 * 576 pixels depending on the task and real-time requirement.

In extreme cases, the number of rows read out can be zero, which of course makes no sense. "Scanning" with any line - or even with only part of a line - is a very interesting application. Figure 6.5 was created by scanning with (only) line 147. The image sequence was set to 1000 image cycles per second by synchronization with a robot controller - this corresponds to the interpolation cycle of the robot controller. In front of the camera, the playing card was swiftly swiped from top to bottom. When the edge of the card reached the "sensitive" line 147, the brightness reflex was registered by the grabber and the evaluation of 1024 successive image lines was triggered every millisecond. The frame grabber digitized the incoming image lines and copied them from the image memory into successive line positions in the main memory. The sequence of 1024 image lines resulted in the displayed scan image with 1024 * 768 pixels.

The fact that it is actually a scan image can be recognized from the fact that, due to the inconsistent speed at which the card was moved in front of the sensitive image line during scanning, the upper half of the image is compressed and the lower half appears stretched. When scanning with the first image line, a scan frequency of 2500 Hz can be achieved, which is in the order of magnitude of common CCD line sensors.

Figure 6.6 shows the time required to read 1 to 3 image lines, depending on the number of the last image line read out. FIG. 6.7 shows the time required for reading out any n as well as the n first lines in a time-optimized manner. In the example, the measuring system based on the camera uses only three freely selectable image lines of the line-oriented image sensor (CCD sensor) for the measurement of five spatial degrees of freedom with a measuring rate of 1000 measurements per second (sampling frequency of 1 kHz).

(A) Data for efficient path planning and Cartesian pilot control will be obtained from a leading image line.

(B) The middle line records the actual position of the tool reference point of the robot, the position of which is regulated relative to the path line.

(C) A trailing line finally supports the measurement of some parameters, whereby the measurement accuracy can be increased.

The choice of the leading and trailing line can be dynamically adapted to the course of the web and the current web speed.

The principle of how a line detection algorithm with a high sampling frequency can determine the position of the trajectory B between two measuring lines I3 and I4 has been explained with reference to FIG. 2.2, although only a single transverse scan line is read from the CCD and processed.

This performance result is potentiated by an interpolation. It is used to measure and process the distance between the point Z of the trajectory B lying in the scanning line a between various of the other measuring lines l _j ~> 1: In the example in Figure 2.2 there are 18 possibilities to calculate the position of Z (from l _j to I4, from I2 to I4, from I3 to I4; from l _j to I5, from I2 to I5, from I3 to I5; etc.). If an average is formed from this multitude of measured values - one interpolates - the position of Z can even be given on the basis of the center point of the width of the trajectory B. Even wide, even irregular train lines are compensated. One (of many) algorithms for interpolation is given in Figure 1.1.

Due to its enormous flexibility, the frame grabber can also be used for a wide variety of other tasks, especially for tasks that require the rapid provision of data from selected image areas or from a few freely selectable lines. Especially for the construction of fast and highly precise optical measuring systems, the camera has decisive advantages over conventional CCD full-frame and line cameras due to its high flexibility.

Claims

Expectations: 1. Autonomous detection of path markings or object contours of spatial surfaces for controlling robots in Cartesian coordinates, in which (a) measurement lines (l are projected onto a surface of the spatial object essentially perpendicular to the scanning of an image sensor, in particular a CCD image sensor; (b ) two image lines of the image sensor are read out and processed; 2. The method according to claim 1, in which unnecessary image lines of the image sensor are (already) deleted in the sensor itself. 3. The method according to claim 1 or 2, in which a third image line is read out and - to increase the measuring accuracy - evaluated. 4. The method according to any one of the preceding claims, wherein the first image line is a lead line and the third image line is a trailing line, while the second image line is expanded to calculate the actual position of the tool reference point of the robot to control its position relative to the path line or is to be regulated. 5. Method according to one of the claims mentioned, in which the image sensor is controlled or read out, in deviation from the CCIR television standard, approximately in the time grid of the interpolation or sampling clock of the robot controller and / or robot controller. 6. The method according to claim 5, wherein the image sensor is deleted in the time interval of the interpolation or the scanning of the control. 7. The method according to any one of the claims mentioned; in which the distance between different measuring lines (lj) and the trajectory (B) is determined in order to determine the center (Z) of the trajectory (B) by means of an interpolation. 8. The method according to any one of the claims mentioned, in which the measuring lines (l _j ) are rectilinear, parallel and in particular equidistant. Method according to one of the mentioned claims, in which an equal number of measuring lines (l _j ) are projected on both sides of the path curve (B). 10. Use the anti-blooming electrode of a CCD sensor to delete the CCD content (the charge on it) before the CCD content has been read out (completely). 11. Use according to claim 10, wherein the charge image is deleted with the anti-blooming electrode after reading out a few lines. 12. Use according to claim 10 or 11, wherein the charge image of the sensor is deleted after (only) one line has been read out. 13. CCD image sensor without memory zone and with a serial readout register arranged directly on the (charge) image zone. 14. CCD image sensor according to claim 13, wherein a window masking unit is provided. 15. Measuring device for handling systems in the (a) a line-oriented image sensor, in particular a CCD sensor, carried along with the robot hand is provided; (b) a read-out device is provided for individual lines of the line-oriented image sensor; (c) a charge erasing device erases the entire content of the sensor after one, two or three lines have been read out and before the entire image content of the sensor (5) has been read line by line. 16. Measuring device according to claim 15, in which a projection device projects thin, essentially straight measuring lines onto the spatial object to be processed or gripped by the handling system. 17. Measuring device according to claim 15 or 16, in which the measuring lines (l are projected in pulses onto the object. 18. Measuring device according to one of claims 15 to 17, in which the projection of the measuring lines and the "exposure" of the CCD chip (release and push out) are synchronized by a synchronous unit. 19. Measuring device according to claim 18, wherein the synchronous unit has two shift registers for controlling the delay and the time of pushing out the charge image. 20. Method for time-optimal reading of arbitrarily selectable image sections on a line-oriented chip without reading out the overall image from the image zone of the chip, in which (a) one or more lines are read out in whole or in part; (b) the read analog values of the lines are digitized and the digitized values are stored in a digital image memory in the address area which corresponds to the charge image image section which was read out; (c) the residual or total charge image is deleted immediately on the chip. 21. The method in particular according to claim 20, in which the reading and the integration are interleaved in a CCD chip, wherein (a) the sub-image shifted into the memory zone of the CCD chip is (further) shifted to the read-out register; (b) the charge image (still) in the image zone of the CCD chip is deleted via the "anti-blooming drain" of the CCD chip; (c) a new one (already) while moving or reading out according to (a) Charge image is built up in the image zone of the CCD chip. 22. The method according to claim 20 or 21, in which the arbitrarily selectable image section is changed in size and / or position on the image zone of the CCD chip. 23. The method according to any one of claims 20 to 22, wherein the change is dependent on the movement of a striking - in particular bright - point in the image section. 24. The method according to claim 20 to 23, in which the readout register is read out from the readout register to one side or the other depending on the position of the partial image in the readout direction (Figure 3.6).