CN100559850C - Method used for dominant color extraction - Google Patents

Abstract

Extracting and processing video content encoded in a rendered color space for emission simulation by an ambient light source includes extracting dominant color information from the video signal and converting the color information through an uncolored color space using a tristimulus dominant matrix to form a second rendered color space for ambient distribution. The steps include quantizing the rendered color space to form a specified color distribution, such as by reducing possible color states, or storing to form a super pixel; dominant colors are selected using a pattern, average, median, or weighted average of pixel chromaticities. The colors of interest may be further analyzed to generate true dominant colors, and previous video frames may be used to guide the selection of dominant colors in later frames.

Description

Method used for dominant color extraction

technical field

The present invention relates to the use of multiple light sources to generate and set ambient light effects, typically based on, or in conjunction with, eg video content from a video display. More particularly, it relates to a method of sampling or subsampling video content in real-time to extract dominant color information, and performing a color mapping conversion from the color space of the video content to a color space that preferably allows driving multiple ambient light sources.

Background technique

Engineers have been exploring for a long time to expand the sensory experience by sampling video content, such as expanding the viewing screen and projection area, modulating the sound into a true 3D effect, and improving the video image, including a wide video color gamut, resolution High-definition (HD) digital television and video systems, for example. Moreover, film, television and video players also attempt to use visual and auditory means to influence the viewer's perception, such as through the clever use of color, scene cuts, perspective, surrounding scenes and computer-aided graphic representation. This will also include theater stage lighting. Lighting effects, such as light effects, are usually choreographed in sync with video or theatrical scenes, reproduced with the aid of machines or computers programmed with appropriate scene scripts encoded in the intended scheme.

In existing technologies in the digital domain, including unplanned or unscripted scenes, the adaptation of lighting in response to rapidly changing scenes is not easy to coordinate in large scenes because the additional high-bandwidth bitstream needs to utilize current system.

Philips (Netherlands) and others have disclosed methods for improving video content by changing the ambient or ambient lighting using a separate light source remote from the video display, which is applied in a typical home or business, and for many applications, the desired light effect Perform prior arrangement or encoding. It has been shown that ambient lighting applied to a video display or television can reduce visual fatigue and increase realism and the perception of depth.

Sensory perception is a natural function of the human visual aspect that utilizes a large and complex sensory tube and nervous system to produce the perception of color and light effects. Humans can distinguish about 10 million different colors. In the human eye, for color reception or photopic vision, there are three groups of approximately 2 million sensory bodies called cones, which have light wavelength peak distributions at 445nm, 535nm, and 565nm absorption distributions with a large amount of overlap. These three types of cones form what is known as the trichromatic system, also known for historical reasons as B (blue), G (green), R (red); the peaks do not have to correspond to any of the dominant colors that are used in displays, For example, commonly used RGB phosphors. There are also interactions for dark adaptation, or so-called night vision bodies called rods. The human eye typically has 120 million retinal rods, which affect video perception, especially in low light conditions, such as in a home theater.

Color video is based on the laws of human vision. It is well known that the three-color and opposite-channel theory of human vision has been combined to understand how to influence the eye to see the desired color. The original signal or the expected image have high fidelity color and effect. . In most color models and spaces, three dimensions or coordinates are used to describe the human visual perception.

Color video relies entirely on metamerism, which allows color perception to be produced with a small amount of reference chromaticity, rather than actual light of desired colors and characteristics. In this way, the full gamut of colors is reproduced in the human mind using a limited number of reference chromaticities, such as the well known RGB (Red, Green, Blue) tristimulus system utilized worldwide in video reproduction. It is known, for example, that almost all video displays display a yellow scene by producing approximately equal amounts of red and green light in each pixel or picture element. The pixels are small compared to the solid angle they subtend, and the eye is mistaken to perceive yellow; it cannot perceive the green or red light that is actually emitted.

There are many color models and ways of specifying colors, including the well-known CIE (Commission Interationale de I'eclairage) color coordinate system, which is used to describe and specify colors for video reproduction. Instant Creation can utilize any number of color models, including the use of uncolored oppositional color spaces such as the CIE L*U*V* (CIELUV) or CIE L*a*b* (CIELAB) systems. The CIE, established in 1931, is the basis for the management and reproduction of all colors, resulting in a chromaticity diagram utilizing three coordinates x, y, and z. This three-dimensional system is commonly used to describe color in terms of x and y in the region of maximum luminance, this region, known as the 1931x,y chromaticity diagram, which is believed to describe all human-perceivable colors. This is in contrast to color reproduction, where metamerism tricks the eye and brain. Today, many color models or spaces in use reproduce colors by utilizing three primary colors or phosphors, among them Adobe RGB, NTSC RGB, etc.

In particular, it should be noted that the range of all possible colors that video systems can exhibit by utilizing these tristimulus systems is limited. The NTSC (National Television Standards Committee) RGB system has a relatively wide range of usable colors, but this system can only reproduce half of all the colors that humans can perceive. The range of blues and violets, cyan and orange/red cannot be adequately reproduced with the available range of conventional video systems.

Furthermore, the human visual system is endowed with compensatory and recognition properties, an understanding of which is essential for designing any video system. Human colors can appear in several display modes, among which are target mode and glow mode.

In target mode, the light stimulus is perceived as light reflected from an object illuminated by the light source. In luminescent mode, the optical stimulus is considered a light source. Luminescence patterns include stimuli in complex fields that are brighter than other stimuli. It does not include stimuli known as light sources, such as video displays, whose brightness or luminance is equal to or less than the full brightness of the scene or viewing site, so that such stimuli are displayed in the target mode.

It's worth noting that there are a number of colors that only appear in target mode, among them, brown, olive, chestnut, gray, and beige flesh. There are no lamps, such as brown traffic lights, for example, as sources of brown light emission.

For this reason, the ambient light supplemented to the video system to increase the color of the object cannot use bright light as a direct light source in this way. A combination of bright red and green lights at close range can't reproduce browns or maroons, so the options are rather limited. Only the spectral colors of the rainbow, with varying intensities and saturations, can be reproduced by direct observation of light from bright sources. This emphasizes the need for fine control over ambient lighting systems, such as providing low-level light output from light sources with attention to tone management. With current data structures, this fine-grained control cannot yet be addressed in fast-changing and fine-grained ambient lighting modes.

Video reproduction can take many forms. Spectral color reproduction allows accurate reproduction of the spectral energy distribution of the original excitation, but this is not achievable in any video reproduction utilizing the three primary colors. Accurate color reproduction replicates the tristimulus values of human vision, yielding homogeneity to the original match, but overall viewing conditions must be similar for the image and original scene to obtain a similar display. The overall viewing conditions of the image and the original scene include the corners of the image, surrounding brightness and chromaticity, and glare. One reason accurate color reproduction cannot often be obtained is because of limitations on the maximum brightness that can be produced on a color monitor.

Chroma color reproduction provides a useful alternative when tristimulus values are proportional to the chroma of the original scene. The chromaticity coordinates are reproduced accurately, but with proportionally reduced brightness. Chroma color reproduction is a good reference standard for a video system if the original and reproduced reference whites have the same chroma, the viewing conditions are the same, and the system has an overall uniform gamma. Due to the limited luma produced in a video display, an equivalent color reproduction that matches the chroma and luminance of the original scene cannot be obtained.

Most video reproductions in practice attempt to achieve a corresponding color reproduction, where the reproduced colors will have the same color as the original scene if the original scene were lit to produce the same average luminance level and the same base white chromaticity as in the reproduction. color performance. However, the ultimate goal of much debate is to show the system's preferred color reproduction in practice, where observer preference affects color fidelity. For example, a tanned skin color is preferably the average color of real skin, and the sky is preferably a bluer color and leaves are greener than they actually are. Even if the corresponding color reproduction is accepted as design criteria, some colors are more important than others, such as flesh tones, which are the subject of special treatment in many reproduction systems such as the NTSC video standard.

Chromatic adaptation for white balance is important in reproducing scene lighting. With a properly adjusted camera and monitor, whites and neutral grays are typically reproduced at the chromaticity of the CIE standard daylight illuminant D65. By always reproducing white surfaces at the same chromaticity, the system mimics the human visual system, which inherently adapts perception so that white surfaces always present the same display, regardless of the chromaticity of the light source, so that a sheet of white paper , appearing white whether on a sunny beach or under incandescent lighting in an indoor scene. In color reproduction, white balance adjustments are usually achieved with gain controls in the R, G and B channels.

The light output of a typical color receiver is typically not linear, but follows a power law relationship to the applied video voltage. The light output is proportional to the video drive voltage raised to a power gamma, where gamma is typically 2.5 for a color CRT (cathode ray tube) and 1.8 for other types of light sources. This factor is compensated by three main gamma correctors in the camera video processing amplifier so that the main video signal encoded, transmitted and decoded is not actually R, G and B, but R ^1/( , G ^{1 /(} and B ^1/( . Chroma color reproduction requires uniform gamma across the video reproduction - including cameras, monitors and any gamma adjustment electronics - but the brightness of the environment takes precedence when attempting to reproduce the color accordingly For example, a dark environment requires a gamma of approximately 1.2, and a dark environment requires a gamma of approximately 1.5 for optimal color reproduction. For the RGB color space, gamma is an important implementation issue.

Most color reproduction codes utilize standard RGB color spaces such as sRGB, ROMM RGB, Adobe RGB98, Apple RGB and video RGB spaces such as those utilized in the NTSC standard. Typically, images are captured to a sensor or source device space, which is specific to the device and image. It can be converted to an uncolored image space, which is a standard color space representing raw chromaticity (see Definitions section).

However, video images are often converted directly from the source device space to a rendered image space (see Definitions section), which represents some real or virtual output device, such as the color space of a video display. Most of the existing standard RGB color spaces are rendered image spaces. For example, the source and output spaces produced by cameras and scanners are not CIE-based color spaces, but spectral spaces defined by spectral sensitivity and other characteristics of the camera or scanner.

A shaded image space is a device-specific color space based on the chromaticity of real or virtual device characteristics. Images can be converted from shaded or unshaded image space to shaded space. These transformations can vary in complexity and can include complex image-dependent algorithms. This conversion is irreversible and discards or compresses some information encoded in the original scene to accommodate the dynamic range and color gamut of a particular device.

Currently there is only one uncolored RGB color space, which is in the process of becoming a standard. The ISO RGB defined in ISO17321 is mostly used for the color characteristics of digital cameras. In most applications today, images including video signals are converted to a rendered color space for archiving and data conversion. Converting from one rendered image or color space to another can cause severe image artifacts. The more gamut and white point mismatches there are between the two devices, the greater the negative impact.

A disadvantage of prior art ambient light display systems is that the extraction of representative colors for ambient emissions from video content is problematic. For example, color averaging of pixel chromaticity often results in gray, brown or other color casts that are not a perceived representation of a video scene or image. Colors derived from simple chromaticity averaging often look blurry and misselected, especially when contrasted with image features such as bright fish or a dominant background such as a blue sky.

Another problem with prior art ambient light display systems is that no special method has been provided for real-time synchronized operation to convert the rendered tristimulus values from video to ambient light for proper chromaticity and presentation. For example, light emitted from LED ambient light sources is often harsh and has a limited or skewed color gamut—often, hue and chromaticity are difficult to evaluate and reproduce. For example, U.S. Patent 6,611,297 to Akashi et al. deals with the realism of ambient light, but provides no special method to ensure correct and satisfactory chromaticity, and Akashi's 297 patent does not allow real-time analysis of video, but requires script or its equivalent.

Additionally, settings for ambient light sources that utilize the gamma of the corrected video content color space often result in dazzling and bright colors. Another serious problem with the prior art is the need to convert the large amount of information used to drive the ambient light source as a function of real-time video content to the expected rapidly changing ambient light environment where good color matching is desired.

Therefore, it would be beneficial to expand the possible gamut of colors produced by ambient lighting in conjunction with typical three-color video display systems, when exploiting the characteristics of the human eye, such as the change in relative visual luminosity of different colors as a function of brightness level , by adjusting or changing the color and light characteristics conveyed to video users utilizing ambient lighting systems, using it to enhance beneficial compensation effects, sensitivity and other characteristics of human vision.

It is also beneficial to produce a quality ambience without the effects of gamma-induced distortion. It is a further need to be able to provide a method for providing better ambient lighting by extracting dominant colors from selected video regions with the savings of data streams encoding average or qualitative color values. It is further desired to reduce the size of the required data stream.

2003。Information on video and television engineering, compression techniques, data transmission and coding, human vision, color scenes and perception, color spaces, chromaticity, coloring of images, including video reproduction, is cited in the following references, which are The combination of these information as a whole: refer to [1] Color Perception, Alan R. Robertson, Physics Today, December 1992, Volume 45, No. 12, pp. 24-29; refer to [2] The physics and Chemistry of Color , 2nd, Kurt Nassau, John Wiley & Sons, Inc., New York 2001; Reference [3] Principles of Color Technology, 3ed, Roy S. Berns, John Wiley & Sons, Inc., New York, 2000; Reference [4] Standard Handbook of Video and Television Engineering, 4ed, Jerry Whitaker and K. BlairBenson, McGraw-Hill, New York

2003.

Contents of the invention

Various embodiments of the present invention present methods that use pixel-level statistics or equivalents to determine or extract one or more dominant colors in a way that requires as little computation as possible, but at the same time, provides a choice based on perceptual laws as The comfort and suitable shade of the main color.

The present invention relates to a method of extracting dominant colors from video content encoded in a rendered color space, with ambient light sources being used to generate dominant colors for simulation. The method steps include: [1] quantizing the rendered color space by quantizing at least some pixel chromaticities of the video content in the rendered color space to form a distribution of specified colors; [2] performing dominant color extraction from the distribution of specified colors to obtain Either of the following produces the dominant color: [a] the mode of the specified color; [b] the median value of the specified color; [c] the weighted average of the chromaticities of the specified color; [d] the weighted average using a weighting function; and [3] Converts the primary color from the shading color space to a secondary shading color space that allows driving ambient light sources.

Quantization of pixel chromaticities (or shading color spaces) can be achieved in many ways (see Definitions section), the goal of which is to reduce the computational burden by seeking simplifications among the possible color states, e.g. from a large number of chromaticities allocated (e.g. degree) to a smaller number of specified shades or colors; or to reduce the number of pixels by a selection process that singles out selected pixels; or binning to produce representative pixels or superpixels.

If the quantization of the shaded color space is performed in part by binning pixel chromaticities to at least one superpixel, the superpixels thus produced may have a size, orientation, shape or position consistent with image features. The color specified by the quantization process may be selected as an area color vector, which does not have to be in the rendered color space, for example may be in the second rendered color space.

Once the dominant color has been selected from the specified color distribution, it is then possible to return to the actual pixel chromaticity to refine the dominant color. For example, at least one interesting color can be determined in the specified color distribution, and the pixel chromaticity specified here can be extracted to obtain the finally specified real main color as the main color. This way, when the specified color can be a rough approximation of the video content, the true dominant color provides the correct chromaticity for the ambient send, but still saves computation that would otherwise be required. Primary colors can also include primary color palettes.

Other possible embodiments for conversion step [3] includes [3a] converting the dominant color from the shaded color space to the unshaded color space; then [3b] converting the dominant color from the space of the unshaded color to the second shaded color space. This additionally includes matrix transformations, using the first and second tricolor primary matrices, converting the primaries of the shaded color space and the second shaded color space to the unshaded color space; Matrix multiplication is performed on the inverse matrices of the two and three color matrices to obtain the conversion of the color information to the second coloring color space.

The pixel chromaticity of step [1] may be obtained from the extraction region, and the additional step [4] may include emitting dominant-color ambient light from an ambient light source adjacent to the extraction region.

A reasonable weighting function of step [2] allows multiple pixel chromaticities to be extracted from image features, and previous video frames can be used to guide the selection of dominant colors in subsequent video frames. Any extracted region can be selected as an image feature extracted from a video frame.

These steps can be combined in many ways, the uncolored color space can be CIE XYZ; ISO RGB as defined in ISO standard 17321; photographic YCC; CIE LAB; or any one of the other uncolored spaces. And steps [1], [2] and [3] are substantially synchronous with the video signal, using the color information in the second rendering color space to emit ambient light from or around the video display.

Description of drawings

Figure 1 represents a front surface view of a simple video display of the present invention, showing color information extraction regions and associated ambient light emission from six ambient light sources;

Figure 2 shows a top view, part schematic and part cross-sectional view, of a room in which ambient light emitted from multiple ambient light sources is generated using the present invention.

Figure 3 shows a system according to the present invention for extracting color information and effecting color space conversion to allow driving ambient light sources;

Figure 4 represents an equation for calculating the mean value of color information from a video extraction region;

Fig. 5 represents the matrix equation of prior art conversion coloring main RGB to uncolored color space XYZ;

Figures 6 and 7 represent matrix equations for mapping video and ambient light shaded areas to unshaded areas, respectively;

Fig. 8 represents the way to obtain the ambient light tristimulus value R'G'B' from the uncolored color space XYZ by known matrix reverse conversion;

Figures 9-11 show that the white point method is used to derive the three-color main matrix M in the prior art;

Figure 12 shows a system similar to that shown in Figure 3, additionally including a gamma correction step for ambient emissions;

Fig. 13 represents the schematic diagram of the overall conversion process used by the present invention;

Fig. 14 shows the processing steps of obtaining ambient light source conversion matrix coefficients utilized in the present invention;

Figure 15 shows the processing steps for estimated video extraction and ambient light rendering used by the present invention;

Fig. 16 represents the schematic diagram that extracts according to the video frame of the present invention;

Figure 17 shows simplified colorimetric evaluation processing steps according to the present invention;

Figure 18 shows the extraction steps as shown in Figures 3 and 12, to drive the ambient light source, utilize the frame decoder, set the frame extraction rate and perform output calculations;

Fig. 19 and Fig. 20 represent the processing steps of color information extraction and processing in the present invention;

Figure 21 shows a schematic diagram of the general process according to the invention, including dominant color extraction and conversion to ambient light color space;

Figure 22 schematically represents a possible method of quantizing pixel chromaticity from video content by specifying pixel chromaticity to specified color;

Figure 23 schematically represents an example of a possible method of quantization by storing pixel chromaticity to superpixels;

Figure 24 represents a storage process similar to that of Figure 23, but here the size, orientation, shape or position of the superpixels can be formed consistently with the image features;

Figure 25 shows area color vectors and their color or chromaticity coordinates on a standard Cartesian CIE color map, where a color vector lies outside the color gamut obtained by the PAL/SECAM, NTSC and Adobe RGB color generation standards ;

Figure 26 shows a close-up of a portion of the CIE diagram shown in Figure 25, additionally showing pixel chromaticity and its distribution on area color vectors;

Figure 27 shows a histogram showing one possible way of specifying patterns of color distribution according to the present invention;

Figure 28 shows one possible method of specifying the median value of a color distribution according to the present invention;

Fig. 29 shows one possible method according to the present invention by specifying the mathematical sum of the weighted average of the chromaticity of the color;

Figure 30 shows one possible method according to the present invention of using a pixel weighting function to pass the mathematical sum of the weighted average of the chromaticity of a given color;

Fig. 31 represents the schematic diagram of determining the color of interest in the specified color distribution, and then extracting the specified pixel chromaticity there, to obtain a real main color and designate the main color;

Fig. 32 schematically shows that dominant color extraction according to the present invention can be performed multiple times or respectively simultaneously to provide a series of dominant colors;

Figure 33 shows a simple front surface view of the video display shown in Figure 1, showing an example of applying different weights to preferred spatial regions in accordance with the method demonstrated in Figures 29 and 30;

Fig. 34 provides a simple front surface diagram shown in a video as shown in Fig. 33, and the graphical representation extracts an image feature according to the purpose of main color extraction of the present invention;

Figure 35 presents an illustration of another embodiment of the present invention, video content is decoded into a set of frames, thereby allowing the dominant color of the resulting frame to depend at least in part on the dominant color of the previous frame;

Figure 36 shows the process steps of the omission process for selecting a dominant color in accordance with the present invention.

Detailed ways

definition

The following definitions apply throughout:

- Ambient light sources - will be included in the following claims to include any light generating circuitry or drivers needed to affect light generation.

- Ambient Space - shall mean any and all volumes of material or air or space external to the Video Display Unit.

- Specify Color Distribution - will represent a set of colors chosen to represent (eg for computational purposes) the full range of pixel chromaticities found in a video image or video content.

- Chromaticity - in the context of driving ambient light sources, shall denote a mechanical, numerical or physical way of specifying the color characteristics of light production, and shall imply no particular methodology, eg for NTSC or PAL television broadcasting.

- color information - will include all or one of chromaticity and lightness, or functionally equivalent quantities;

-Computers- shall include not only all processors, such as CPUs (Central Processing Units) utilizing known structures, but also any intelligent device that allows encoding, decoding, reading, processing, executing set codes or changing codes, such as can A digital optical device or an analog circuit that performs the same function.

- Predominant Color - shall mean any shade chosen to represent video content for ambient emission purposes, including any color chosen to illustrate the methods disclosed herein;

-Extract Regions - will include any entire video image or subset of frames.

- Frames - shall include image information that occurs in chronological order within the video content, consistent with the term "frame" used in the industry, but shall also include any partial (e.g. interleaved) or full Or deliver video content at any interval.

- Angularity - will refer to the property of a given different color or shade as a function of viewing angle or viewing angle, eg produced by a rainbow.

- Goniophotometry - will refer to the properties of a given different luminosity, transmission and/or color as a function of viewing angle or viewing angle, such as found in the phenomena of radiance, sparkling or retroreflection.

- Interpolation - will include linear or mathematical interpolation between two sets of values, as well as functional descriptions of set values between two sets of known values;

- light characteristics - in a broad sense means, for example, descriptions of properties of any light produced by ambient light sources, including all descriptions other than luminance and chromaticity, such as the degree of light transmission or reflection; or descriptions of angular-metric properties, including The degree of color, flare, or other known phenomenon produced when viewing an ambient light source as a function of viewing angle; the direction of light output, including directionality imparted by a Poynting or other propagation vector; or a description of the distribution of light angles , such as the solid angle or the solid angle distribution function. It can also include a coordinate specifying its position in an ambient light source, such as the position of a unit pixel or a light.

- Brightness - shall denote any parameter or measured brightness, intensity or equivalent measure, and shall not impose a particular method or measurement or psycho-biological interpretation of light production.

- Pixels - will refer to real or virtual video pixels, allowing equivalent information for pixel information deviations.

- quantized color space - within the scope of the specification and claims, shall refer to a reduction of possible color states, e.g. resulting from a specified number of chromaticities (e.g. pixel chromaticities) to a small number of specified chromaticities or colors; or selected by picking Pixel selection process, the number of pixels is reduced; or binned to produce representative pixels or superpixels.

- Rendered color space - shall represent an image or color space captured from a sensor or a specific light source or display device as a device and a specific image. Most RGB color spaces are rendered image spaces, including the video display space used to drive the video display D. In the appended claims, the specific color space of the video display and ambient light source 88 is a rendered color space.

- Transformation of color information to an uncolored color space - in the appended claims will include either direct conversion to an uncolored color space, or utilize or benefit from an inverse transformation by utilizing a three-color primary matrix by transforming to an uncolored color space It is obtained by coloring a color space ((M ₂ ) ^-1 as shown in FIG. 8 ).

- Uncolored color space - will denote a standard or device-neutral color space, such as those that utilize standard CIE XYZ colorimetry to describe raw images; such as ISO RGB as defined in the ISO17321 standard; photographic YCC; and CIE LAB color spaces .

- Video - shall designate any visual or light-generating device, whether active device requiring energy to generate light, or any transmission medium for transmitting image information, such as windows in an office building, or optical waveguides for image information obtained from a distance.

- Video signal - shall indicate the signal or information transmitted to control the video display unit, thus including any audio part. It is contemplated that video content analysis includes possible audio content analysis for the audio portion. In general, video signals may comprise any type of signal, such as radio frequency signals utilizing any number of known modulation techniques; electrical signals, including analog and quantized analog waveforms; digital (electrical) signals, such as those utilizing pulse width modulation, Pulse Number Modulation, Pulse Position Modulation, PCM (Pulse Code Modulation) and Pulse Amplification Modulation; or other signals such as auditory signals, sound signals and optical signals, all of which can utilize digital techniques. Data in which only sequential or other information, such as in computer-based applications, may also be utilized.

-Weighting-shall refer to any equivalent method given here to give priority status or higher mathematical weighting for particular chromaticities or spatial locations.

specific description

Ambient lighting derived from video content in accordance with the present invention is formed, if desired, to allow a high degree of fidelity to the original video scene lighting, yet maintaining a high degree of ambient light freedom characteristics requires only a low computational burden. This allows ambient light sources with small color gamuts and luma reduction headroom to simulate video scene lighting from more advanced light sources with relatively large color gamuts and luma response curves. Possible light sources for ambient lighting can include any number of known light emitting devices, including LEDs (Light Emitting Diodes) and related semiconductor radiation sources; electroluminescent devices including non-semiconductor types; Advanced chemical modification types; ionizing discharge lamps, including fluorescent and neon lamps; lasers; remodulated light sources, such as by using an LCD (liquid crystal display) or other light modulator; photoluminescent emitters, or any number of known Controllable light sources, including arrays that function like displays.

The description given here will, in part, first relate to the extraction of color information from video content, and subsequently to extraction methods to obtain dominant or true colors that can represent the ambient emission of a video image or scene.

Reference is made to Fig. 1 which, for exemplary purposes only, illustrates a simple front view of a video display D according to the invention. Display D may include any number of known devices that decode video content from a rendered color space, such as NTSC, PAL, or SECAM broadcast standards, or a rendered RGB space, such as Adobe RGB. Display D may include selectable color information extraction regions R1, R2, R3, R4, R5 and R6, whose boundaries may be separated from those graphical regions. The color information extraction area can be arbitrarily predetermined and has the characteristics of generating unique ambient light A8, for example by installing a controllable ambient lighting unit (not shown) behind, which generates and emits ambient light L1, L2, L3, L4, L5 and L6, for example by leaking part of the light onto a wall (not shown) where display D is mounted. Optionally, the display frame Df as shown in the figure also itself includes an ambient lighting unit for displaying light in a simple manner, including outwardly to a viewer (not shown). If desired, each color information extraction region R1-R6 can individually affect the ambient light proximate it. For example, as shown, the color information extraction region R4 may affect the ambient light L4.

Referring to Figure 2, a top view - part schematic and part cross-sectional view - shows a venue or ambient space A0 in which ambient light from multiple ambient light sources is generated using the present invention. Seats and tables 7 as shown are provided in ambient space A0, configured to allow video display D to be viewed. A large number of ambient lighting units are also configured in the ambient space A0, which can be controlled at will by using the instant invention, including the shown light speakers 1-4, the shown floor lamp SL under the sofa or seat, and a group of configurations in the A special simulated ambient lighting unit around D is shown, namely a central light producing ambient light Lx as shown in Fig. 1. Each of these ambient lighting units can emit ambient light A8, as indicated by the shaded portion in the figure.

In conjunction with this immediate invention, ambient light can be arbitrarily generated from these ambient lighting units, with color or chromaticity emitted from these ambient lighting units but not actually displayed by the video. This allows exploiting the characteristics and visual system of the human eye. It is worth noting that the luminance function of the human visual system, which has a detection sensitivity for different visible wavelengths, varies as a function of light level.

For example, scotopic or night vision relies on retinal rods that tend to be more sensitive to blues and greens. Photopic vision using cone cells is better at detecting longer wavelengths of light, such as red and yellow. In the black space of a theater environment, by modulating or varying the colors delivered to a video viewer in the ambient space, these changes in the relative luminosity of the different colors can be somewhat offset as a function of light level. This can be done by subtracting from the ambient lighting unit, e.g. the optical speakers 1-4 using a light modulator (not shown) or by using an added element on the optical speaker, i.e. a photoluminescent emitter that changes further before ambient release. Light. Photoluminescent emitters perform color conversion by absorbing or undergoing excitation from incident light from a light source, and then re-emit light in a higher desired wavelength. This excitation and re-emission of photoluminescent emitters, such as fluorescent dyes, may allow new color rendering not present in the original video image or light source, and perhaps not within the inherent operating color or color gamut of the display D . This is helpful when the desired brightness of the ambient light Lx is low, eg in very dark scenes, and when the desired perceived level is higher than would normally be obtained without light modulation.

The creation of new colors can provide new and interesting visual effects. An illustrative example could be the generation of orange light, for example the so-called seeking orange, for which useful fluorescent dyes are well known (cf. reference [2]). Examples given include fluorescent colors as opposed to general fluorescent and related phenomena. The use of fluorescent orange or other fluorescent dye species is especially useful for low light conditions, where red and orange boosts can counteract the sensitivity of scotopic vision to long wavelengths.

Utilization of fluorescent dyes in ambient lighting units may include dyes known in the class of dyes such as perylenes, naphthalimides, coumarins, thioxanthenes, anthraquinones, thioindigo, as well as specialized classes of dyes such as those produced by Manufactured by Daylight Fluorescent Dye Colors, Cleveland, Ohio, USA. Available colors include Apache Yellow, Tigris Yellow, Prairie Yellow, Pocono Yellow, Mohawk Yellow, Potomac Yellow, Marigold Orange, Ottawa Red, Volga Red, Damascus Pink, and Columbia Blue. These dye classes can be incorporated into synthetic resins such as PS, PET and ABS using known processes.

Fluorescent dyes and materials have improved visual performance because they can be engineered to be significantly brighter than non-fluorescent materials of the same shade. The so-called durability problem of traditional organic pigments used to produce fluorescent colors has been largely solved in the past two decades. With the advancement of technology, it has led to the development of durable fluorescent pigments, which can be exposed to sunlight. Keep them realistically colored for 7-10 years. These pigments are therefore virtually immune to damage in home theater environments with minimal exposure to UV rays.

Alternatively, fluorescent photopigments can be utilized, which simply work by absorbing short wavelength light, and re-emitting this light as longer wavelength light, eg red or orange. Technological advances in inorganic pigments have enabled visible light excitation, such as blue and violet, such as 400-440nm light.

Angular and goniophotometric effects can be similarly developed to produce different color, brightness, and characteristics of light as a function of viewing angle. To achieve this effect, the ambient lighting units 1-4 and SL and Lx can individually or in combination utilize known goniophotometric elements (not shown), such as metallic and pearlescent transfer shading; Or rainbow materials with thin-film interference, such as using scale-like entities; thin-scale guanine; or 2-aminohypoxanthine with preservatives. Utilize fine mica or other substances as diffusers, such as radiant materials made of oxide, bornite, or malachite; flakes of metal, glass, or plastic; particulate matter; oils; frosted glass, and frosted plastic.

Referring to FIG. 3, there is shown a system for extracting color information (such as primary color or true color) and effecting color space conversion to drive ambient light sources according to the present invention. In a first step, color information is extracted from the video signal AVS using known techniques.

The video signal AVS may comprise frames or packets of digital data known for MPEG encoding, audio PCM encoding, and the like. Known encoding schemes may be used for the data packets, such as program streams with variable length data packets, or transport streams with evenly divided data packets, or other single program transport stream schemes. Alternatively, the functional steps or block diagrams disclosed herein may be emulated using computer code or other communication standards, including asynchronous protocols.

As a general example, the shown video signal AVS may be subjected to the shown video content analysis CA, possibly using known methods to record and transfer selected content to and from the shown hard disk HD, possibly using the shown library of content types or Other information stored in memory MEM. This can allow selection of individual, parallel, direct, delayed, continuous, periodic or aperiodic switching of video content. From these video contents, the feature extraction FE shown can be performed, eg extraction of color information in general (eg dominant colors), or feature extraction from an image. The color information is also encoded in the shaded color space and then transformed to an unshaded space, such as CIE XYZ using the RUR mapping transformation circuit 10 shown. Here RUR represents the desired conversion type, i.e., shaded-uncolored-colored, such that the RUR mapping conversion circuit 10 further converts the color information into a second rendering color space formed to allow driving the ambient light source 88. RUR conversion is preferred, but other mappings can be used as long as the ambient light generating circuit or its equivalent can be used to receive information in the second rendering color space.

The RUR mapping conversion circuit 10 may be functionally included in a computer system, which uses software to perform the same function, but in the case of decoding packet information transferred by a data transfer protocol, there is a memory inside the circuit 10, which contains or Updated to include information that correlates or provides shading color space coefficients, etc. The newly created second rendered color space is appropriate and intended to be used to drive the ambient light source 88 (shown in FIGS. 1 and 2 ) and is supplied with the encoding of the ambient light generating circuit 18 as shown. The ambient light generation circuit 18 obtains the second rendering color space information from the RUR mapping conversion circuit 10, and then accounts for any input from any user interface and any composite preference memory (shown with U2), which may result in ambient light as shown in reference The (secondary shading) color space look-up table LUT is used to develop real ambient light output control parameters (eg applied voltage). The ambient light output control parameters generated by the ambient light generating circuit 18 are supplied to the illustrated lamp interface driver D88 to directly control or supply the illustrated ambient light source 88, which may comprise individual ambient lighting units 1-N, such as those previously referenced Ambient light speakers 1-4 as shown in Figures 1 and 2, or an ambient light center light Lx.

To reduce any real-time computational burden, the color information removed from the video signal AVS may be omitted or limited. Referring to Fig. 4, an equation representing the calculation of average color information from a video extraction region is used for discussion. It is contemplated, but not required, that the video content of video signal AVS will comprise a series of time-sequential video frames, as described below (see FIG. 18). For each video frame or equivalent time frame, average or other color information can be extracted from each extraction region (eg R4). Each extraction region can be set to have a specific size, for example, to 100×376 pixels. Assuming, for example, that the frame rate is 25 frames per second, for each of the three main colors of video RGB, the total data of the synthesis of the region R1-R6 extracted before extracting the average value (assuming that only 1 byte needs to specify 8 bits of color) will be That's 6 x 100 x 376 x 25 or 5.64 Mbytes/sec. This kind of data flow is very large, and it is difficult to handle in the RUR mapping conversion circuit 10. Therefore, the extraction of the average color of each extraction region R1-R6 can be used in the feature extraction FE. In particular, it is shown that the RGB color channel values (such as R _ij ) can be summed for each pixel of each m×n pixel extraction region, and the mean value of each main RGB is reached by the number of m×n pixels, For example R _avg is shown in red. This repeats the summation for each RGB color channel, and the mean value of each extracted region will be a three-bit byte R _AVG = |R _avg , G _avg , B _avg |. Repeat the same process for all extracted regions R1-R6 and for each RGB color channel. The number and size of the extraction regions can be different from what is shown, or can be divided as desired.

The next step in the color mapping transformation performed by the RUR mapping transformation circuit 10 may be illustrative as shown and represented using the three-color primary matrix shown, for example as shown in FIG. The color space is converted using a three-color master matrix M with elements Xr _,max , Yr _,max , Zr _,max, where Xr _,max is the three-color value initially output by R at its maximum value.

The conversion from shaded color space to unshaded, device-only space can be image and/or device-specific - known linearisation, pixel reconstruction (if needed) and white point selection steps can be performed, followed by matrix conversion. In this case, we simply choose to use the shaded video output space as the starting point for the colorimetric conversion to the unshaded color space. Unrendered images need to undergo an additional transformation to a second rendering space in order to make them viewable or printable, and such RUR transformations include transformations to the second rendering space.

In a first possible step, Figures 6 and 7 represent the matrix equations for drawing the video shading color space, denoted by the primary R, G, B and ambient shading color spaces, respectively, denoted by the primary R', G', B', to the unshaded color space X, Y, Z shown, where tristimulus _M1 converts the video RGB to unshaded XYZ and trismatrix _M2 converts the ambient light source R'G'B' to the shown Uncolored XYZ color space. Equalizing the shading spaces RGB and R'G'B' shown in FIG. 8, using the first and second tricolor primary matrices (M ₁ , M ₂ ), allows matrix transformation of the shading (video) color space and the second shading ( Primaries RGB and R'G'B' of ambient) color space to said unrendered color space (RUR mapping conversion); by rendering the primary colors RGB of video color space, the inverse of the first three-color matrix M1 and the second three-color matrix The matrix (M ₂ ) ^-1 performs matrix multiplication to obtain the conversion of the color information into the second coloring color space (R'G'B'). However, the three-color main matrix of the known display device is easy to obtain, and those skilled in the art can use the known white point method to determine the ambient light source.

Referring to Figs. 9-11, it shows that the prior art uses the white point method to obtain the general three-color main matrix M. In Figure 9, the quantities S _r X _r represent the primary tristimulus values at maximum output for each (ambient light source), S _r represents the white point amplitude, and X _r represents the chromaticity of the primary light produced by the (ambient) light source. Using the white point method, using the known inverse of the chromaticity matrix of the light source, the matrix equation equates S _r to the white point reference value vector. Figure 11 is an algebraic process to suggest that a white point reference value, such as _Xw , is the product of the white point amplitude or luminance and the chromaticity of the light source. Throughout, tristimulus X is set equal to chroma x; tristimulus Y is set equal to chroma y; tristimulus Z is constrained to equal 1-(x+y). The primary and reference white components of the second colored ambient light source color space can be obtained using known techniques, for example by using a color spectrometer.

A similar amount can be found for the first rendering video color space. For example, contemporary studio monitors are known to have slightly different standards in North America, Europe, and Japan. But, for example, on high-definition television (HDTV), there has been international agreement on basic standards that closely represent the characteristics of studio monitors in terms of studio video, computing, and computer graphics. This standard formally represents the ITU-R recommendation BT.709, which includes the required parameters, where the three-color main matrix (M) for RGB is:

0.640 0.300 0.150 Matrix M of ITU-R BT.709

0.330 0.600 0.060

0.030 0.100 0.790

The value of the white point is also known.

Referring to FIG. 12 , similar to the system shown in FIG. 3 , after the feature extraction step FE for ambient light emission, a gamma correction step 55 is additionally included. Alternatively, the gamma correction step 55 may be performed between the steps performed by the RUR mapping conversion circuit 10 and the ambient light generation circuit 18 . The optimum gamma value for LED ambient light sources has been found to be 1.8, so negative gamma values to offset video color spaces with a typical gamma of 2.5 can be performed using known mathematically derived gamma values Correction.

Generally, the RUR mapping conversion circuit 10, which can be a functional block acting on any known suitable software platform, performs general RUR conversion as shown in FIG. The video signal AVS in the shaded color space of , and convert it to an uncolored color space such as CIE XYZ; then to the second shaded color space (ambient light source RGB). After RUR conversion, in addition to signal processing, an ambient light source 88 may be driven as shown.

Fig. 14 shows the processing steps of obtaining the conversion matrix coefficients of the ambient light source by using the present invention, as shown in the figure, wherein the steps include driving the ambient light unit; and checking the output linearity in the field shown. If the primary color of the ambient light source is stable, (as shown by the bifurcation on the left, stable primary color), the conversion matrix coefficient can be obtained by using a color spectrometer; on the other hand, if the primary color of the ambient light source is unstable, (as shown by the bifurcation on the right, unstable primary color), you can reset the previously given gamma correction (as shown in the figure, reset the gamma curve).

In general, it is desirable to extract color information from each pixel in an extraction region such as R4, but this is not required. Instead, polling of selected pixels may allow rapid evaluation of the average color, or rapid generation of extraction region color features, if desired. happened. Figure 15 shows the processing steps of video extraction evaluation and ambient light rendering using the present invention, where the steps include [1] preparing evaluation of video rendering chromaticity (from rendered color space, such as video RGB); [2] converting to uncolored color space; and [3] transformed chromaticity evaluation for ambient rendering (second rendering color space, eg LEDRGB).

In accordance with the present invention, it has been found that the data bitstream required to support the extraction and processing of video content (eg dominant colors) from video frames (see Figure 18 below) can be reduced by judicious sub-sampling of video frames. Referring to FIG. 16, a diagram illustrating video frame extraction according to the present invention is shown. A series of independent consecutive video frames F are shown, ie video frames _F1 , _F2 , _F3 and so on - eg specified video frames with independent interleaving or without interleaving as specified by NTSC, PAL or SECAM standards. By performing content analysis and/or feature extraction—such as extracting dominant color information—from selected consecutive frames, such as frames F ₁ and F _N , the data load or overhead can be reduced while maintaining acceptable responsiveness, realistic sexiness and fidelity. It has been found that N=10 gives good results, ie subsampling one frame out of 10 consecutive frames works. This provides a refresh period P between frame fetches with low processing overhead, where the interframe interpolation process can provide a good approximation of the chrominance in the display D over time. Extraction of selected frames _F1 and _FN as shown (extraction) and intermediate interpolation values of chroma parameters, shown as _G2 , _G3 and _G4 , provide the necessary color information to inform the previously cited environment The driving process of the light source 88. This avoids the need to simply freeze or keep the same color information in frame 2 to frame N-1. For example in case the overall chrominance difference between the extracted frames F ₁ and F _N extends over the interpolated frame G, the interpolation value can be determined linearly. Alternatively, a function may extend the chrominance difference between frames _F1 and _FN extracted in any other way, such as a higher order approximation suitable for temporal visualization of the extracted color information. The result of the interpolation can be used to influence interpolated frames by previsiting frame F (e.g. in a DVD player), or alternatively the interpolation can be used to influence future interpolated frames when frame F is not preselected (e.g. application in).

Figure 17 shows simplified colorimetric evaluation processing steps in accordance with the present invention. Higher order analysis of frame extraction can expand the refresh period P and expand N relative to what would otherwise be possible. During frame extraction, or in temporal polling of selected pixels in the extraction region _Rx , the simplified chrominance evaluation shown can be processed, which will result in either a delay in the extraction of the next frame, as shown on the left, or make the entire frame Extraction begins, as shown on the right. In either case, the interpolation continues (interpolation), with a delay in fetching the next frame resulting in freezing, or increasing the chrominance value utilized. This would provide even more economical operation in terms of bitstream or overhead bandwidth.

Figure 18 represents the top of Figures 3 and 12, where an optional extraction step is shown next to the frame decoder FD utilized, at step 33 shown, allowing extraction of region information from the extraction region (eg R1). Further processing or composed step 35 includes evaluating the chrominance differences and using this information to set the video frame extraction rate as indicated. Prior to data transmission to the ambient lighting and generation circuitry 18 shown above, the next processing step is performed as shown, which performs an output calculation 00, such as the averaging process of FIG. 4, or dominant color extraction as discussed below.

As _shown in Figure 19, the general processing steps representing the color information extraction and processing of the present invention include obtaining _the video signal AVS; interpolation between video frames; RUR mapping conversion; optional gamma correction; and using this information to drive ambient light sources (88). As shown in Figure 20, after the region extraction from the selected frame information: two further processing steps can be inserted: a chromaticity difference evaluation between the selected frames _F1 and _FN can be performed, and relying on predetermined criteria, one can be as Indicates to set a new frame fetch rate. Thus, if the chroma difference between consecutive frames _F1 and _FN is large, or increases rapidly (e.g. large first order derivative), or satisfies some other criterion, e.g. based on chroma difference history, then frame extraction can be increased rate, which reduces the refresh period P. However, if the chromaticity difference between consecutive frames F ₁ and F _N is small and stable or does not increase rapidly (for example, the absolute value of the first derivative is low or zero), or satisfies some other criteria, such as based on chromaticity The degree difference history can then save the required data bit stream and reduce the frame extraction rate, thus improving the refresh period P.

Referring to Figure 21, there is shown a general processing in accordance with one aspect of the present invention. As shown, as an optional step, possibly reducing the computational burden, [1] the rendering color space corresponding to the video content is quantized (QCS, Quantized Color Space), for example by using the method given below; then [2 ] select the dominant color (or palette of dominant colors) (DCE, dominant color extraction); and [3] color mapping conversion, such as performing RUR mapping conversion (10) (MT mapping conversion to R'G'B') to Improves the realism, range, and fit of the resulting ambient light.

An optional quantization of the color space can reduce the number of possible color states and/or pixels to be measured and can be performed with different methods. As an example, Figure 22 schematically shows a possible method of quantizing pixel chromaticity of video content. Here, as shown, illustrative video master values R range from values 1 to 16, with any one of these master values R arbitrarily assigned a given color AC. Thus, for example, whenever any red pixel chromaticity or value from 1 to 16 is encountered in the video content, it will therefore replace the specified color AC, resulting in a factor of 16 reduces the red primaries individually. In this example, such a reduction in possible color states would result in a reduction by a factor of 16x16x16, or 4096 - in the number of colors used for the calculation - for all three primary colors. This is extremely useful in reducing the computational burden in primary color determination in many video systems, such as those with 8-bit colors, which represent 256x256x256, or 16.78 million possible color states.

Another method of quantizing video color space is shown in Figure 23, which schematically represents another example of quantized rendering color space by binning pixel chrominance from a number of pixels Pi (eg 16 shown) to superpixels XP. Binning itself is a method by which adjacent pixels are mathematically (or computed) added together to form a superpixel, which itself is used for further computation or representation. Thus, where video formats typically have, for example, 0.75 megapixels, the number of superpixels chosen to replace video content can reduce the number of pixels used for calculations to 0.05 megapixels or any other desired small number.

The number, size, orientation, shape or position of such superpixels may vary as a function of video content. Here, for example, it is advantageous during the feature extraction FE to ensure that the superpixels XP are only extracted from image features and not from edge regions or backgrounds, the superpixels XP being formed accordingly. Figure 24 shows a binning process similar to that of Figure 23, but here the size, orientation, shape or position of the superpixels can be formed consistent with the pixel feature J8 shown. Image feature J8 is shown as jagged or irregular, with no straight horizontal or vertical edges. As shown, the selected superpixel XP mimics or mimics the image feature shape accordingly. In addition to having a custom shape, the location, size and orientation of the superpixels can also be influenced by image features J8 using known pixel-level computing techniques.

Quantization can make the pixel chromaticity consistent with an alternate specified color (such as specified color AC). Those specified colors can be specified arbitrarily, including using preferred color vectors. Thus, rather than utilizing an alternative or uniform set of specified colors, at least some video image pixel chromaticities can be set to preferred color vectors.

Figure 25 shows the area color vectors and their color or chromaticity coordinates on a standard Cartesian CIE x-y chromaticity diagram or color map. This graph represents all known or perceived colors at maximum luminance as a function of chromaticity coordinates x and y, with nanometer light wavelengths and the CIE standard luminescence white point shown as reference. The 3 area light vectors V are shown on this diagram, here, one can see that a color vector V lies outside the color gamut which is produced by the PAL/SECAM, NTSC and Adobe RGB color standards (color shown domain) obtained.

For clarity, Figure 26 shows a close-up of a portion of the CIE diagram of Figure 25, additionally showing pixel chromaticities Cp and their assigned area color vectors V. The criteria for specifying an area color vector can vary, using known calculation techniques, and can include Euclidean calculations or distances from other specific color vectors V. The marked color vector, V, lies outside the rendering color space or color gamut of the display system; this may allow the preferred chromaticity easily produced by the ambient lighting system or light source 88 to be converted into a quantized rendering (video) color space A specified color for .

Once the distribution of specified colors has been obtained using one or more of the methods given above, the next step is to perform dominant color extraction from the specified color distribution by extracting any of the following: [a] the way in which the color is specified; [b] the specified The median value of a color; [c] a weighted average of the chromaticity of a specified color; or [d] a weighted average using a weighting function.

For example, a histogram method can be used to select the specified color that occurs with the highest frequency. The histogram represented in Fig. 27 gives the assigned pixel color or the most frequently occurring (see coordinates, pixel percentage) color (assigned color), ie the manner in which assigned colors are distributed. This way or a majority of the ten utilized specified colors can be selected as the dominant color DC (shown) for utilization or simulation by the ambient lighting system.

Likewise, the median value of a given color distribution can be chosen as or to help influence the choice of dominant color DC. Fig. 28 schematically shows the median value of the assigned color distribution, where the displayed median or middle value (interpolated for an even number of assigned colors) is selected as the dominant color DC.

Alternatively, the summation of the specified colors may be performed using a weighted average in order to influence the selection of the dominant color, possibly more suited to the strength of the ambient lighting system's color gamut. Figure 29 shows the mathematical summation of weighted averages of the specified color chromaticities. For clarity, a single variable R is shown, but any number of dimensions or coordinates (eg, CIE coordinates x and y) could be utilized. The chromaticity variable R is expressed in pixel coordinates (or superpixel coordinates, if desired) i and j, which in this example take values between 1 and n and 1 and m, respectively. The chrominance variable R with indices i and j as shown is multiplied by the pixel weighting function W and summed overall; the result is divided by the number of pixels n x m to obtain a weighted average.

A similar weighted average using a pixel weighting function is shown in Figure 30, similar to Figure 29, except that W shown is also a function of pixel positions i and j shown, which allows for a spatial principal function. By also weighting the pixel positions, the center or any other part of the display D can be emphasized during selection or extraction of the dominant color DC, as will be discussed below.

The weighted summation may be performed by the given extract region information step 33 given above, W may be selected and stored in any known manner. The pixel weighting function W can be any function or operator, so that the whole can be included, and zero can be excluded for a specific pixel position. Image features can be identified using known techniques, as shown in Figure 34, and W can be changed accordingly to serve the larger purpose.

Using the above method, or any equivalent method, once the specified color has been selected as the dominant color, a better assessment of the appropriate chromaticity for representation can be performed by the ambient lighting system, especially when considering if all chromaticities and and/or all video pixels require fewer computational steps than they would otherwise require. Fig. 31 shows that the color of interest is determined in the distribution of specified colors, and then the specified pixel chromaticity is extracted there to obtain a real dominant color as the specified primary color. It can be seen that the pixel chromaticity Cp is assigned to the two assigned colors AC; the assigned color AC shown at the bottom of the figure is not selected as the dominant color, however the upper assigned color is considered the dominant color (DC) and is selected as Color of interest COI indicated. Pixels of the assigned color AC assigned to (or at least partially) considered the color COI of interest can then be further examined, and read out their chromaticity directly (e.g. using the mean value, as shown in Figure 4, or for a particular Purpose, within a small area already given, performing the dominant color extraction step), a better reproduction of the dominant color can be obtained, shown here as the true dominant color TDC. Any processing steps required for this purpose may be accomplished using the steps and/or components given above, or by using a separate true color selector, which may be a known software program or subroutine or task circuit or its equivalent.

As shown in FIG. 32 , the extraction of dominant colors according to the present invention can be performed multiple times or provide a palette of dominant colors independently and in parallel, wherein the dominant colors DC can include the dominant colors DC1+DC2+DC3.

As mentioned in Figure 30, a weighting function or equivalent may provide weighting by pixel location to allow special consideration or emphasis on certain display regions. Figure 33 shows a simple front surface view of the video display shown in Figure 1 and shows an example of providing unequal weights to pixels Pi at preferred spatial regions. For example, as shown, certain regions C of the display may be weighted with a numerically large weighting function W, while an extracted region (or any region, such as the scene background) may be weighted with a numerically small weighting function W.

Such weighting or emphasis can be applied to image features J8 as shown in FIG. 34, which gives a simple front surface view of the video display shown in FIG. Image feature J8 (a fish) is selected (see Figures 3 and 12). The image feature J8 may be only the video content utilized in the dominant color extraction DCE as shown or described above, or a portion of the video content utilized.

Referring to Figure 35, it can be seen that utilizing the method presented here allows obtaining the dominant color chosen for a video frame by relying at least in part on at least one dominant color of the previous frame. The illustrated frames F ₁ , F ₂ , F ₃ and F ₄ undergo the process of obtaining the dominant color extraction DCE as shown in the figure, the purpose of which is to extract the shown dominant colors DC1, DC2, DC3 and DC4 respectively, wherein, by Computationally, the dominant color chosen for the frame, denoted DC4, can be established as a function of the shown dominant colors DC1, DC2 and DC3 (DC4 = F(DC1, DC2, DC3)). This allows any simplification of the selection of the dominant color DC4 for frame F4, or better informs the selection of the dominant color of the previous frames _F1 , _F2 , _F3 therein, helping to influence the selection of the dominant color DC4. This simplification process is shown in Figure 36 and is used here to reduce the computational burden. Temporary dominant color extraction DC4* utilizes chrominance evaluation, which is then assisted in the next step by the dominant color extracted from the previous frame (or a single previous frame). , to help prepare the selection of DC4 (Prepare DC4 using the simplified procedure).

In general, ambient light sources 88 may incorporate various diffuser effects to produce light mixing, as well as translucency or other phenomena, such as by utilizing lamp structures with frosted or smooth surfaces; ribbed glass or plastic; or aperture structures , for example by utilizing metal structures surrounding individual light sources. To provide these interesting results, any number of known diffusing and scattering materials or phenomena can be utilized, including those obtained by utilizing scattering from small suspended particles; shaded plastics or resins, ready to utilize colloids, latex, or water beads 1- 5:m or less, such as less than 1:m, including long-term organic mixtures; gels; and sols, the production and manufacture of which are known to those skilled in the art. Scattering phenomena may include Rayleigh scattering at visible wavelengths, such as blue production to enhance the blue color of ambient light. The resulting color can be defined regionally, for example an overall bluish hue in certain regions or a regional hue, for example as an upper part of the blue light generation (ambient light L1 or L2).

Ambient lights can also be fitted with goniophotometric elements, such as cylindrical prisms or lenses, which can be formed, integrated, or inserted inside the light structure. This can allow special effects when the resulting light characteristics vary as a function of the viewer's position. Other light shapes and forms may be utilized, including rectangular, triangular, or irregularly shaped prisms or shapes, which may be placed over or make up an ambient lighting unit. Is the result an isotropic output, the effect obtained can be varied infinitely, such as bands of light of interest projected onto surrounding walls, objects and surfaces around ambient light sources, when scene elements, colors and intensities are in The video display changes on the unit, creating a light display in a dark room. Such an effect could be a theater ambient light element that perceptibly changes the light signature as a function of the viewer's position - when viewing a home theater, when the viewer stands up from a chair or moves the viewing position - such as watching a bluish Sparks, then red light. The number and type of goniophotometric elements can be utilized virtually unlimited, including sheets of plastic, glass, and optical effects resulting from scratches and moderately destructive manufacturing techniques. Unique ambient lights can be made, even interchangeable for different theatrical effects. These effects may be modulatable, for example by varying the amount of light allowed to pass through the goniophotometric element, or by varying different lit portions of the ambient lighting unit (eg, with sub-lamps or groups of LEDs).

Thus, the ambient light generated in L3 shown in FIG. 1 to simulate extracted region R3 may have a chromaticity that provides a perceptual extension of phenomena in that region, such as the moving fish shown. This can increase the visual experience and provide a suitable and not harsh or normal matching color tone.

The video signal AVS may of course be a digital data stream and contain synchronization bits and concatenation bits; parity bits; error codes; "Storm"; "Sunrise" etc.) and functional steps that would be implemented by those skilled in the art, for the sake of clarity, are given here for illustrative purposes only and do not include routine steps and data.

GUI & preference memory as shown in Figures 3 and 12 can be used to vary ambient lighting system behavior, e.g. vary the degree of color fidelity of the video content for the desired video display D; vary gorgeously, including any fluorescent color or color gamut The extent to which additional colors are emitted into the ambient space, or how quickly and how much the ambient light changes based on changes in the video content, such as by increasing the varying brightness or other properties in the light script command content. This can include advanced content analysis, which can create a flat tone for a movie or specific characteristics of content. Video content that includes many dark scenes in the content can affect the behavior of the ambient light source 88, causing a dim emission of the ambient light, while flashy or bright tones can be used for some other content, like many flesh-toned or bright scenes (sunlight a shining beach, a tiger on the savannah, etc.).

The description given here will enable any person skilled in the art to utilize the present invention. Many configurations are possible using the present teachings, these configurations and arrangements given are illustrative only. Not all of the objects sought here need to be demonstrated, for example, without departing from the present invention, special transformations of the second rendering color space can be excluded from the teachings given here, especially when the rendering color spaces RGB and R'G'B' is similar or same when. In practice, the methods taught or claimed may be displayed as part of a larger system, be it an entertainment center or a home theater center.

It is well known that the functions and calculations illustratively taught herein can be functionally reproduced and simulated using software or machine code, and those skilled in the art will be able to utilize these teachings regardless of the control of the manner of encoding and decoding taught herein. This is especially true when considering that it is not strictly necessary to frame the decoded video information in order to perform pixel-level statistics.

Based on these teachings, those skilled in the art can modify the apparatus and methods taught and claimed herein, for example, rearrange steps or data structures to suit a particular application, creating a system that may include few similar options for illustrative purposes.

The invention disclosed using the above examples can be implemented using some of the features recited above. Likewise, nothing that is not taught or required herein shall not preclude the addition of other structural or functional elements.

Obviously, modifications and variations of the present invention are possible in light of the above teaching. It is therefore to be understood that within the scope of the appended claims, the invention may be utilized rather than as specifically described or suggested herein.

Claims (19)

1, a kind ofly extract mass-tone to produce the method by the mass-tone of environment light source (88) simulation from the video content of encoding the painted color space, it comprises:

1) quantification at least some pixel colourities from described video content in the described painted color space are to form the distribution of designated color;

2) from the distribution of described designated color, carry out mass-tone and extract, so that any one produces mass-tone below extracting: a) designated color that takes place with highest frequency; B) intermediate value of described designated color; C) weighted average of the colourity of described designated color; D) utilization is to weighting function (W (i, j, R)) the weighted average of location of pixels weighting;

3) from the painted color space of the described mass-tone to the second of described painted color space transformation, this second painted color space is formed and allows to drive described environment light source.

2, the method for claim 1, wherein said quantification comprise a plurality of described pixel colourities are assigned in the described designated color one of them.

3, the method for claim 1, wherein said quantification comprise cases described pixel colourity at least one super pixel.

4, method as claimed in claim 3, the size of wherein said super pixel, direction, shape, or any one and characteristics of image form adaptably in the position.

5, method as claimed in claim 3 comprises a plurality of described pixel colourity in the described super pixel is assigned in the described designated color one of them in addition.

6, the method for claim 1, at least one is the field color vector in the wherein said designated color, it is optional in the described painted color space.

7, method as claimed in claim 6, wherein said field color vector is arranged in the described second painted color space.

8, the method for claim 1 is included in addition and determines in the distribution of described designated color that at least a interested color and extraction specify in pixel colourity there, to obtain a kind of true mass-tone that will be appointed as described mass-tone.

9, the method for claim 1, wherein said mass-tone comprises the palette of mass-tone.

10, the method for claim 1, wherein step 3) comprises

3a) from the described mass-tone of described painted color space transformation to the not painted color space;

3b) from the described mass-tone of described not painted color space transformation to the described second painted color space.

11, method as claimed in claim 10, wherein step 3a) and 3b) also comprise step 3c): utilize the one three and the 23 look principal matrix (M ₁, M ₂), the primary colors of the described painted color space of matrix conversion and the second painted color space is to the described not painted color space; With inverse matrix (M by the described primary colors of the described painted color space, described the one or three look principal matrix and described the two or three look principal matrix ₂) ^-1Carry out matrix multiple, obtain the conversion of colouring information to the described second painted color space.

12, the method for claim 1 wherein obtains the pixel colourity of step 1), and comprises in addition from extract the zone:

4) from broadcast the surround lighting of described mass-tone adjacent to the described described environment light source that extracts the zone.

13, the method for claim 1, wherein step 1) comprises in addition described video content is decoded into a framing, and wherein a plurality of pixel colourity obtains from the characteristics of image that extracts by the frame the described framing.

14, the method for claim 1, wherein step 1) comprises in addition described video content is decoded into a framing, and wherein by depending on from least one mass-tone of former frame and the small part that arrives obtains the mass-tone of frame.

15, a kind ofly extract mass-tone to produce the method by the mass-tone of environment light source (88) simulation from the video content of encoding the painted color space, it comprises:

1) in the described painted color space, described video content is decoded into a plurality of frames, and in one of them of described frame, quantizes at least some, to form the distribution of designated color from pixel colourities of extracting the zone;

2) from the distribution of described designated color, carry out mass-tone and extract, so that any one produces mass-tone below extracting: a) designated color that takes place with highest frequency; B) intermediate value of described designated color; C) weighted average of the colourity of described designated color; D) utilization is to weighting function (W (i, j, R)) the weighted average of location of pixels weighting;

3a) from the described mass-tone of described painted color space transformation to the not painted color space;

3b) from the described mass-tone of described not painted color space transformation to the described second painted color space, assist by the following step:

3c) utilize the one three and the 23 look principal matrix (M ₁, M ₂), the primary colors of the described painted color space of matrix conversion and the second painted color space is to the described not painted color space; With inverse matrix (M by the described primary colors of the described painted color space, described the one or three look principal matrix and described the two or three look principal matrix ₂) ^-1Carry out matrix multiple, obtain the conversion of colouring information to the described second painted color space;

4) from broadcast the surround lighting of described mass-tone adjacent to the described described environment light source that extracts the zone.

16, method as claimed in claim 15, wherein said extraction zone is selected to the characteristics of image that extracts from a frame.

17, method as claimed in claim 15 is included in addition and determines in the distribution of described designated color that at least a interested color and extraction specify in pixel colourity there, to obtain a kind of true mass-tone that will be appointed as described mass-tone.

18, method as claimed in claim 15, at least one in the wherein said designated color are the field color vectors, and it is optional in the described painted color space.

19, method as claimed in claim 15, wherein said frame comprises first and second frames, and by depending on from least one mass-tone of former frame and the small part that arrives obtains the mass-tone of frame.

Images