Psych221 Winter 2007

Final Project Report: Camera Phone Color Appearance Utility

Chung-Ching Chang, Shu-Lu Chen, Sufen Fong

n Introduction

Since the first functional camera phone was built in 1997, the market for camera phones has grown rapidly. The fact that 460 million camera phones were reported to have been sold in 2006 underscores the increasing manifestation of camera phones in our daily lives. The main purpose of having a camera within a phone is to enable people to capture moments in their lives on a daily basis wherever they are, without having to lug around bulky equipment.

Besides capturing visual memories for people, another potential application for camera phones would be to act as a visual aid for a blind/color blind person. Specifically, the camera phone would function as a device which identifies the color of an object of interest to the user. For example, a blind/color blind person could take a picture of a shirt which he picks up and the phone would possess software which does color recognition of the shirt. After a color match has been made, a pre-recorded voice would speak the color of the shirt. This sequence of events would enable a blind/color blind person to pick out a shirt which is of the user¡¦s desired color.

The scope of our project focuses on capturing an image and processing the image to identify the color. To correctly identify a color in an image captured by a camera phone, there are many difficulties which must be overcome before such an application can be feasibly implemented.

Difficulties

Color balancing under different lighting conditions is a challenge because camera sensors are unable to accurately account for different luminance conditions, unlike the human vision system. The neural system in humans computes information from the cones such that objects retain their color appearance whether we encounter them indoors or outdoors. The light incident at our cones depends on the properties of the objects that reflect light and the wavelength composition of the illuminant.

Different lighting conditions have very different spectral power densities. Hence, the colors of two camera phone images of the same colored card captured under two different lighting conditions may appear to have different saturations. In addition, objects with different textures have different surface reflectance. An object which does not have a uniform texture may also cause problems in color identification since it may have different saturation values at different parts of the object surface.

Isolating an object of interest may also prove hard when many objects clutter the background, or if the color of the object of interest is too similar to the colors of adjacent objects or the background.

Assumptions

In view of these difficulties, we made the following assumptions in our project:

1)The object of interest has a uniform texture.

2)The image only comprises of two colors. (One of the colors is white, which is used as our reference color to normalize the RGB values for our color segmentation methods)

3)Our image is always two-dimensional.

Experimental Set up

A piece of white paper which was folded twice was laid in the center of a colored paper.

We chose 18 colors to work with and matched these colors, by visual inspection, to web safe colors to obtain RGB values. These RGB values are the reference values which will be compared against the RGB values obtained after processing the images captured under different light conditions and different white balance settings.

Color name	R	G	B	Color Sample
Light yellow	255	255	102	Color Sample
Dark yellow	255	204	0	Color Sample
Bright orange	255	102	0	Color Sample
Orange	255	102	51	Color Sample
Light pink	255	204	204	Color Sample
Salmon pink	255	153	102	Color Sample
Pink	255	153	153	Color Sample
Bright pink	255	51	153	Color Sample
Red	204	0	0	Color Sample
Light green	204	255	153	Color Sample
Bright green	204	255	0	Color Sample
Dark green	51	153	0	Color Sample
Light blue	102	255	204	Color Sample
Blue	102	204	204	Color Sample
Dark blue	102	102	153	Color Sample
Light purple	255	153	204	Color Sample
Grey	204	204	204	Color Sample
Black	0	0	0	Color Sample

The images were captured by using a Nokia N80 camera phone. The phone had a 3.0 megapixel camera with a 20x digital zoom. There were eight shooting modes ¡V automatic, user, portrait, landscape, sports, night landscape, night portrait and candlelight. The camera also allowed users to adjust the brightness, contrast, white balance, color tone and exposure values. The default or automatic settings for all the features (except the white balance) were used at all times. The camera flash was also turned off at all times so that the only illumination would come from our desired light source.

Experimental Settings

There were four different lighting conditions under which we captured the images: sunny, cloudy, incandescent and fluorescent.

Sunny	This condition was obtained by going outdoors at around 3pm to take pictures under the sun.
Cloudy	This condition was obtained by taking pictures outdoors around 6pm but not directly under the sun (in the shade).
Incandescent	This condition was obtained by switching off the room lights at night and capturing an image of the paper placed under a table lamp.
Fluorescent	This condition was obtained by taking pictures at night in the room under the fluorescent ceiling lights.

There were five different white balance settings of the camera: automatic, sunny, cloudy, and incandescent. For each lighting condition and color, we captured 5 different images using each of the different white settings that were available in the camera. The figures below show an example of the 5 different images obtained under sunny conditions for a bright green paper. A grand total of 360 images were used as experimental data (4x5x18).

n Methods

Once the image has been captured, it is processed through four-steps: color segmentation, color statistics, color balancing, and model training. All these steps are further explained in the following subsection.

Fig. 4. Data Processing flow chart

Color Segmentation

In our experiment design, the captured image is composed of two regions: reference white and the test color. The color segmentation algorithm clusters the captured image into these two regions.

Fig. 5. Color Segmentation flow chart

To implement color segmentation, we first remove the boundary region of the reference white according to the sobel edge detection followed by morphological processing. This is necessary since those edges along with the shadows usually are not representative of the color of the region. The sobel 2D filter, as in the below, is designed to detect horizontal and vertical edges through 2D convolution.

The first filter(matrix) is used to detect a horizontal edge and the second one is used to detect a vertical edge. The idea of using weight value 2 is to achieve smoothing by giving more importance to the center point [4]. A morphological operation is next applied to the edge mask to reduce false alarm caused by shot noise (and usually composed of few pixels), and to thicken the edges to avoid using the pixels neighboring to the edge.

After removing the edge regions, we apply the Kmeans algorithm to cluster the rest of the images pixels into two clusters, reference white and test color, according to the color of each pixel. The idea of Kmeans algorithms is illustrated in the example as shown in the following figure.

Fig. 6. Kmeans algorithms illustration

The algorithm Kmeans is named after its assumption. As in the example, we first have to guess k, the number of clusters, which is 2 in the example. Initially, we randomly select k points as seeds and clustering the rest of the points according to the distance to those seeds. Next, we calculate the mean of each cluster as the seeds in the next iteration and continue clustering by distance and calculating means until it converges. A more sophisticated clustering called Expectation Maximization (EM) considers not only the mean but also the covariance of each cluster. However, it requires far more computation than Kmeans and is not suitable in mobile computation and, therefore, is not adopted here.

In our experiment, we consider the RGB value of each pixel of a captured image as a data in three dimensional coordinate and clustering each pixel to be either reference white region or test color region, which also implies k = 2.

Color Statistics

Knowing the region of the reference white and the test color, we want to calculate the color statistics for further color balancing. The mean color and the covariance between the color elements can be calculated by the following formula.

Choosing the color space is very important since some transformations between color spaces are non-linear, and thus makes it hard to transform the covariance matrix. If the transformation is linear, such as RGB to XYZ, once we know the mean and the covariance of one color space, we can calculate the other by factor analysis.

Color Balancing

In the color balancing part of the image processing, the ultimate goal is to match two samples under different lighting conditions to the same points on the gamut plot. In our experimental setup, we use different colors of paper with one same white rectangular region on each of it. Under different lighting condition, the same colored sample looks different. The following picture is taken based on different lighting conditions with the same sample:

As these pictures show, the blue paper looks very different and the reference white even look like yellow in the first case.

The previous section describes how we distinguish these two different colors, ie, reference white and blue in this sample, through the Kmeans method and machine learning.

The output from our color segmentation will give 2 sets of XYZ values, ie: reference white (x,y,z) , and blue (x,y,z) for each lighting condition. To make the comparison, here are the two gamuts without any modification and plotted directly using the xyz values from the above two pictures.

Fig. 7. Left, sample1; right, sample2

On the color balancing part of our project, we try three methods to modify the color difference which is due to the lighting condition. The first method is based entirely on the XYZ space only while the second and third method are based on two spaces: RGB and XYZ. Going from method 1 to method 3 is actually an evolution from simple ¡§number scaling (0D)¡¨ to the ¡§vector 3X1 scaling (1D)¡¨ then to the most complete ¡§matrix 3X3 scaling (2D)¡¨ model. It turns out simple number scaling gives the worst color balancing while the most robust 3X3 model performs the best balancing. The detailed experiments using these 3 models are illustrated in the next section, along with corresponding discussions and gamut plots.

Method 1: reference white luminance normalization (scaling number ¡V 1 X 1)

This method normalizes the luminance component, namely the y value in the xyz space. The whole process flow can be summarized in the following chart.

To make it easier to compare, we use the mean value. The result will be a point on the gamut plane. In the complete design, we also include the variance value for each color, which results in an ellipse on the gamut plane. The complete plot will be shown in Fig. 12. The red point represents the reference white and the green point represents the blue color. From the result, it seems that the points actually do not move. We can now explain these results from the mathematical formula. Since the chromaticity transformation can be described by the following formula: (x,y ¡V chromaticity; XYZ- xyz value)

;.

If we scale both X ,Y, and Z at the same number, it causes no effect on the gamut plot. This is explained from the upper figure in Fig. 8.

The chromaticity moves the point in the 3-D xyz space along the [1 1 1] direction. So after the projection to the (1 1 1) plane, which is what the chromaticity function does, the results are the same. It turns out that this simple method does nothing for color balancing. Hence, we need to try more completes model to correct this.