Dr.B.A.T.

University, Lonere

Image Cartoonize with Stegnography

CHAPTER 1 MULTIPASS BILATERAL FILTER CARTOONIZER

1.1Introduction
This application applies Multipass Bilateral Filter to colour Images. Bilateral filtering is an Edge-preserving smoothing filter. This technique extends the concept of Gaussian smoothing by weighting the filter coefficients with their corresponding relative pixel intensities. Pixels that are very different in intensity from the central pixel are weighted less even though they may be in close proximity to the central pixel. This is effectively a convolution with a non-linear Gaussian filter, with weights based on pixel intensities. This is applied as two Gaussian filters at a localized pixel neighbourhood , one in the spatial domain, and one in the intensity domain.

1.1.1 Changes
V6 Better Contour Algorithm; Luminance segmentation; Bilateral Filter optimization; Interface look.

1.1.2 Cartoon
To obtain cartoon-like results, these new effects have been implemented:

1.1.3 Contour
Contour, when used, is applied to the Bilateral Filter Output in the CieLAB colour space. It has 2 parameters: General Amount, and how much it is based on Luminance (L) or Hue (AB). It's created with a Sobel filter applied in separated channels: L and A & B (of the CieLAB colourspace).

Dept of Information Technology

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

1.2 Luminance Segmentation

This is a nice way to obtain a more cartoon-like output. (Like contour this works with CieLAB colour space.)

1.3 Algorithm:
It's built an histogram of the Luminance channel (L) of Bilateral filtered picture. Then this histogram is divided by N segments (in a way so that in each segment there's the same number of intensities values). To each segment it is assigned a value of Luminance given by the weighted average of the Histogram segment values. (Fig. Example Yellow lines) Then these segments are merged with original bilateral output by a value called Presence. Note that we are talking only about Luminance values. 0% Presence means that the bilateral filter output luminance has no variation. (No Segmentation applied) 25% Presence means that the Luminance of the output picture is given by 25% of the Segmented Luminance and by 75% of the bilateral filter output luminance and so on 100% Presence means that the Luminance of the output picture is fully given by Segmented Luminance. The best way to have an idea of how this work is to tweak these parameters and to watch the output results.

Dept of Information Technology

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

1.4 Flow

As we can see, before applying Bilateral Filter it's possible to apply a pre-Effect. The pre-Effext can be one of this: 1. Auto Histogram Equalize 2.Exposure Control 3.Brightness, Contrast & Saturation. After the Bilateral filter there are 4 possible paths to follow:

1.4.1 Pure Bilateral (black)

0-1-6 The Output Picture is the Output of Bilateral. To follow this path set Contour Amount = 0 and Segmentation Presence = 0

1.4.2 Add a Contour (red)

0-1-2-5-6 [1-2] A contour filter is applied to the Bilateral Output [1-5] Contour is applied to the bilateral output To follow this path set Contour Amount > 0 and Segmentation Presence = 0

1.4.3 Luminance Segmentation (blue)

0-1-3-4-6 [1-3] A Luminance Segmentation is applied to the Bilateral Output. (The Number of Segments is customizable)

Dept of Information Technology

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

[1-4] The Output is the result of the merging of the Luminance Segmented picture with the bilateral Output. This merge is customizable with Presence % slidebar. To follow this path set Contour Amount = 0 and Segmentation Presence > 0.

1.4.4 CARTOON-like: Contour and Luminance Segmentation (green)

0-1-2-3-4-5-6 [1-2] A contour filter is applied to the Bilateral Output [1-3] A Luminance Segmentation is applied to the Bilateral Output. (The Number of Segments is customizable) [1-4] We merge the Luminance Segmented picture with the bilateral Output. This merge is customizable with Presence % slidebar. [4-5] Contour is applied to the output of above merging. To follow this path set Contour Amount > 0 and Segmentation Presence greater > 0. --

1.4.5 Parameters Details

Parameters are saved in the files LastSettings.txt and files LastSettingsEX.txt for easy reading. Not to loose them, make a copy.

1.5 Pre-EFFECT
As seen above it can be: Auto Equalize Exposure BCS Brightness, Contrast, Saturation

1.6 BILATERAL FILTER

1.6.1 Colour Space This is the Colour space the bilateral filter will be applied to. -RGB Bilateral filter is computed from all channels Red,Green and Blue. -CieLAB. Here the bilateral filter is computed only from the Luminance (L) channel of the CieLab Colour Space faster. 1.6.2 Radius Radius of Intensity and Spatial Domains. R=1 means that for each pixel the computed pixel will be the result of ((R*2)+1)^2 = 9 neighborhood pixels R=2 means that for each pixel the computed pixel will be the result of ((R*2)+1)^2 = 25 neighborhood pixels
Dept of Information Technology 4

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

R=3 means that for each pixel the computed pixel will be the result of ((R*2)+1)^2 = 49 neighborhood pixels R=4 means that for each pixel the computed pixel will be the result of ((R*2)+1)^2 = 81 neighborhood pixels R=5 means that for each pixel the computed pixel will be the result of ((R*2)+1)^2 = 121 neighborhood pixels 1.6.3 Intensity Mode This is the type of decay curve for the intensity space, it can be 0 Gaussian1 1 Gaussian2 2 InvProportional 3 Linear 1.6.4 Intensity Sigma This is how fast the intensity space decrease 1.6.5 Spatial Sigma This is how fast the Spatial space decrease going far from central pixel. 1.6.6 Iterations How many times Bilateral Filter is applied.

1.7 CONTOUR
Amount Contour Darkness Lum/Hue How much the contour is based on Luminance or Hue.

1.8 LUMINANCE SEGMENTATION

Segments Number of Luminance Segments

Samples This are good settings for cartoon-like results PreEFFECT: Brightness:110
Dept of Information Technology 5

Dr.B.A.T.University, Lonere Contrast :0 Saturation:100

Image Cartoonize with Stegnography

BILATERAL FILTER: Colour Space: CieLAB Radius: 5 Intensity Mode : 1 Gaussian2 Intensity Sigma: 0,355 Spatial Sigma: 1000 Iterations: 3

CONTOUR: Amount : 100 Lum/Hue: .5

LUMINANCE SEGMENTATION: Segments: 4 Presence: 60%

Dept of Information Technology

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

CHAPTER 2 COLOURIZATION USING OPTIMIZATION

2.1 Digital Image Processing
2.1.1 Background: Digital image processing is an area characterized by the need for extensive experimental work to establish the viability of proposed solutions to a given problem. An important characteristic underlying the design of image processing systems is the significant level of testing & experimentation that normally is required before arriving at an acceptable solution. This characteristic implies that the ability to formulate approaches &quickly prototype candidate solutions generally plays a major role in reducing the cost & time required to arrive at a viable system implementation. 2.1.2 What is DIP? An image may be defined as a two-dimensional function f(x, y), where x & y are spatial coordinates, & the amplitude of f at any pair of coordinates (x, y) is called the

intensity or gray level of the image at that point. When x, y & the amplitude values of f are all finite discrete quantities, we call the image a digital image. The field of DIP refers to processing digital image by means of digital computer. Digital image is composed of a finite number of elements, each of which has a particular location & value. The elements are called pixels. Vision is the most advanced of our sensor, so it is not surprising that image play the single most important role in human perception. However, unlike humans, who are limited to the visual band of the EM spectrum imaging machines cover almost the entire EM spectrum, ranging from gamma to radio waves. They can operate also on images generated by sources that humans are not accustomed to associating with image. There is no general agreement among authors regarding where image processing stops & other related areas such as image analysis& computer vision start. Sometimes a distinction is made by defining image processing as a discipline in which both the input & output at a process are images. This is limiting & somewhat artificial boundary. The area of image analysis (image understanding) is in between image processing & computer vision.
Dept of Information Technology 7

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

There are no clear-cut boundaries in the continuum from image processing at one end to complete vision at the other. However, one useful paradigm is to consider three types of computerized processes in this continuum: low-, mid-, & high-level processes. Low-level process involves primitive operations such as image processing to reduce noise, contrast enhancement & image sharpening. A low- level process is characterized by the fact that both its inputs & outputs are images. Mid-level process on images involves tasks such as segmentation, description of that object to reduce them to a form suitable for computer processing & classification of individual objects. A mid-level process is characterized by the fact that its inputs generally are images but its outputs are attributes extracted from those images. Finally higher- level processing involves Making sense of an ensemble of

recognized objects, as in image analysis & at the far end of the continuum performing the cognitive functions normally associated with human vision. Digital image processing, as already defined is used successfully in a broad range of areas of exceptional social & economic value.

2.1 What is an image?

An image is represented as a two dimensional function f(x, y) where x and y are spatial co-ordinates and the amplitude of f at any pair of coordinates (x, y) is called the intensity of the image at that point.

1 Gray scale image: A grayscale image is a function I (xylem) of the two spatial coordinates of the image plane. I(x, y) is the intensity of the image at the point (x, y) on the image plane. I (xylem) takes non-negative values assume the image is bounded by a rectangle [0, a] [0, b]I: [0, a] [0, b] [0, info)

Dept of Information Technology

Dr.B.A.T.University, Lonere 2 Color image:

Image Cartoonize with Stegnography

It can be represented by three functions, R (xylem) for red, G (xylem) for green and B (xylem) for blue. An image may be continuous with respect to the x and y coordinates and also in amplitude. Converting such an image to digital form requires that the coordinates as well as the amplitude to be digitized. Digitizing the coordinates values is called sampling. Digitizing the amplitude values is called quantization.

Figure 2.1: Given a grayscale image marked with some colour scribbles by the user (left), our algorithm produces a colourized image (middle).

Dept of Information Technology

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

Abstract:
Colourization is a computer-assisted process of adding colour to a monochrome image or movie. The process typically involves segmenting images into regions and tracking these regions across image sequences. Neither of these tasks can be performed reliably in practice; consequently, colourization requires considerable user intervention and remains a tedious, time-consuming, and expensive task.In this paper we present a simple colourization method that requires neither precise image segmentation, nor accurate region tracking. Our method is based on a simple premise: neighboring pixels in space-time that have similar intensities should have similar colours. We formalize this premise using a quadratic cost function and obtain an optimization problem that can be solved efficiently using standard techniques. In our approach an artist only needs to annotate the image with a few colour scribbles, and the indicated colours are automatically propagated in both space and time to produce a fully colourized image or sequence. We demonstrate that high quality colourizations of stills and movie clips may be obtained from a relatively modest amount of user input. CR Categories: I.4.9 [Image Processing and Computer Vision]: Keywords: colourization, recolouring, segmentation

2.2 Introduction
Colourization is a term introduced by Wilson Markle in 1970 to describe the computer-assisted process he invented for adding colour to black and white movies or TV programs [Burns]. The term is now used generically to describe any technique for adding colour to monochrome stills and footage. Colourization of classic motion pictures has generated much controversy which partially accounts for the fact that not many of these movies have been colourized to date. However, there are still massive amounts of black and white television shows that could be colourized: the artistic controversy is often irrelevant here, while the financial incentives are substantial, as was succinctly pointed out by Earl Glick1 in 1984: .You couldn't make Wyatt Earp today for $1 million an episode. But for $50,000 a segment, you can turn it into colour and have a brand new series with no residuals to pay. Colourization of still images also appears to be a topic of considerable interest among users of image editing software, as evidenced by multiple colourization tutorials on the World Wide Web .A major difficulty with colourization, however, lies in the fact that it is an expensive and time-consuming process. For example, in order to colourize a still image an
Dept of Information Technology 10

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

artist typically begins by segmenting the image into regions, and then proceeds to assign a colour to each region. Unfortunately, automatic segmentation algorithms often fail to correctly identify fuzzy or complex region boundaries, such as the boundary between a subject's hair and her face. Thus, the artist is often left with the task of manually delineating complicated boundaries between regions. Colourization of movies requires, in addition, tracking regions across the frames of a shot. Existing tracking algorithms typically fail to robustly track non-rigid regions, again requiring massive user intervention in the process. In this paper we describe a new interactive colourization technique that requires neither precise manual segmentation, nor accurate tracking. The technique is based on a unified framework applicable to both still images and image sequences. The user indicates how each region should be coloured by scribbling the desired colour in the interior of the region, instead of tracing out its precise boundary. Using these user supplied constraints our technique automatically propagates colours to the remaining pixels in the image sequence. This colourization process is demonstrated in Figure 1. The underlying algorithm is based on the simple premise that nearby pixels in space-time that have similar gray levels should also have similar colours. This assumption leads to an optimization problem that can be solved efficiently using standard techniques. Our contribution, thus, is a new simple yet surprisingly effective interactive colourization technique that drastically reduces the amount of input required from the user. In addition to colourization of black and white images and movies, our technique is also applicable to selective recolouring, an extremely useful operation in digital photography and in special effects.

2.3 Previous work:

In Markle's original colourization process [Markle and Hunt 1987] a colour mask is manually painted for at least one reference frame in a shot. Motion detection and tracking is then applied, allowing colours to be automatically assigned to other frames in regions where no motion occurs. Colours in the vicinity of moving edges are assigned using optical flow, which often requires manual fixing by the operator. Although not much is publicly known about the techniques used in more contemporary colourization systems used in the industry, there are indications that these systems still rely on defining regions and tracking them between the frames of a shot.

Dept of Information Technology

11

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

Black Magic, a commercial software for colourizing still images, provides the user with useful brushes and colour palettes, but the segmentation task is left entirely to the user. describe a semi-automatic technique for colourizing a grayscale image by transferring colour from a reference colour image. They examine the luminance values in the neighbourhood of each pixel in the target image and transfer the colour from pixels with matching neighbourhoods in the reference image. This technique works well on images where differently coloured regions give rise to distinct luminance clusters, or possess distinct textures. In other cases, the user must direct the search for matching pixels by specifying swatches indicating corresponding regions in the two images. While this technique has produced some impressive results, note that the artistic control over the outcome is quite indirect: the artist must and reference images containing the desired colours over regions with similar textures to those that she wishes to colourize. It is also difficult to retune the outcome selectively in problematic areas. In contrast, in our technique the artist chooses the colours directly, and is able to retune the results by scribbling more colour where necessary. Does not explicitly enforce spatial continuity of the colours, and in some images it may assign vastly different colours to neighbouring pixels that have similar intensities.

2.4 Algorithm:
We work in YUV colour space, commonly used in video, where Y is the monochromatic luminance channel, which we will refer to simply as intensity, while U and V are the chrominance channels, encoding the colour [Jack 2001]. The algorithm is given as input an intensity volume Y(x; y; t) and outputs two colour volumes U(x; y; t) and V(x; y; t). To simplify notation we will use boldface letters (e.g. r; s) to denote (x; y; t) triplets. Thus, Y(r) is the intensity of a particular pixel. As mentioned in the introduction, we wish to impose the constraint that two neighbouring pixels r; s should have similar colours if their intensities are similar. Thus, we wish to minimize the difference between the colour U(r) at pixel r and the weighted average of the colours at neighbouring pixels:

where wrs is a weighting function that sums to one, large when Y(r) is similar to Y(s), and small when the two intensities are different. Similar weighting functions are used extensively in image segmentation algorithms where they are usually referred to as affinity functions. We
Dept of Information Technology 12

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

have experimented with two weighting functions. The simplest one is commonly used by image segmentation algorithms and is based on the squared difference between the two intensities:

A second weighting function is based on the normalized correlation between the two intensities:

where mr and sr are the mean and variance of the intensities in a window around r. The correlation affinity can also be derived from assuming a local linear relation between colour and intensity. Formally, it assumes that the colour at a pixel U(r) is a linear function of the intensity Y(r): U(r) = aiY(r)+bi and the linear coefficients ai;bi are the same for all pixels in a small neighbourhood around r. This assumption can be justified empirically and intuitively it means that when the intensity is constant the colour should be constant, and when the intensity is an edge the colour should also be an edge (although the values on the two sides of the edge can be any two numbers). While this model adds to the system a pair of variables per each image window, a simple elimination of the ai;bi variables yields an equation equivalent to equation 1 with a correlation based affinity function. The notation r belongs N(s) denotes the fact that r and s are neighbouring pixels. In a single frame, we define two pixels as neighbours if their image locations are nearby. Between two successive frames, we define two pixels as neighbours if their image locations, after accounting for motion, are nearby. More formally, let vx(x; y); vy(x; y) denote the optical flow calculated at time t. Then the pixel (x0; y0; t) is a neighbour of pixel (x1; y1; t +1) if:

The flow field vx(x0); vy(y0) is calculated using a standard motion estimation algorithm [Lucas and Kanade 1981]. Note that the optical flow is only used to define the neighbourhood of each pixel, not to propagate colours through time. Now given a set of locations ri where the colours are specified by the user u(ri) = ui; v(ri) = vi we minimize J(U); J(V) subject to these constraints. Since the cost functions are quadratic and the constraints are linear, this
Dept of Information Technology 13

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

optimization problem yields a large, sparse system of linear equations, which may be solved using a number of standard methods. Our algorithm is closely related to algorithms proposed for other tasks in image processing. In image segmentation algorithms based on normalized cuts [Shi and Malik 1997], one attempts to find the second smallest eigenvector of the matrix D -W where W is a n pixels matrix whose elements are the pair wise affinities between pixels (i.e., the r; s entry of the matrix is wrs) and D is a diagonal matrix whose diagonal elements are the sum of the affinities (in our case this is always 1). The second smallest eigenvector of any symmetric matrix A is a unit norm vector x that minimizes xTAx and is orthogonal to the first eigenvector. By direct inspection, the quadratic form minimized by normalized cuts is exactly our cost function J, that is xT(D-W)x = J(x). Thus, our algorithm minimizes the same cost function but under different constraints. In image denoising algorithms based on anisotropic diffusion one often minimizes a function similar to equation 1, but the function is applied to the image intensity as well.

Dept of Information Technology

14

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

2.5 RESULTS:
The results shown here were all obtained using the correlation based window (equation 3, or equivalently using the local linearity assumption). The mean and variance m;s for each pixel were calculated by giving more weight to pixels with similar intensities. Visually similar results were also obtained with the Gaussian window (equation 2). For still images we used Matlab's built in least squares solver for sparse linear systems, and for the movie sequences we used a multigrid solver. Using the multigrid solver, the run time was approximately 15 seconds per frame. The threshold T in equation 4 was set to 1 so that the window used was 3*3*3. Figure 2 shows some still gray scale images marked by the user's colour scribbles next to the corresponding colourization results. Since automating the choice of colours was not our goal in this work, we used the original colour channels of each image when picking the colours. As can be seen, very convincing results are generated by our algorithm even from a relatively small number of colour scribbles. Typically, the artist may want to start with a small number of colour scribbles, and then _ne-tune the colourization results by adding more scribbles. Figure 3 demonstrates such a progression on a still image. Figure 4 shows how our technique can be applied to recolouring. To change the colour of an orange in the top left image to green, the artist first defines a rough mask around it and then scribbles inside the orange using the desired colour. Our technique is then used to propagate the green colour until an intensity boundary is found. Specifically, we minimize the cost (equation 1) under two groups of constraints. First, for pixels covered by the user's scribbles, the final colour should be the colour of the scribble. Second, for pixels outside the mask, the colour should be the same as the original colour. All other colours are automatically determined by the optimization process. In this application the affinity between pixels is based not only on similarity of their intensities, but also on the similarity of their colours in the original image. Note that unlike global colourmap manipulations, our algorithm does not recolour the other orange in the image, since colours are not propagated across intensity boundaries. The bottom row of the figure shows another example. Figures 5 and 6 show selected frames from colourized movie clips. Even though the total number of colour scribbles is quite modest, the resulting colourization is surprisingly convincing. We have also successfully colourized several short clips from the television show .I Love Lucy. and from Chaplin's classic movie Modern Times. The original clips were obviously in black and white, so in these examples we did not have a colour reference to pick the colours from. Figures 7 and 8 compare our method to two alternative methods. In figure 7 the alternative method is one where the image is first

Dept of Information Technology

15

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

segmented automatically and then the scribbled colours are used to flood fill each segment. Figure 7a shows the result of automatic segmentation computed using a version of the normalized cuts algorithm [Shi and Malik 1997]. Segmentation is a very difficult problem and even state-of-the-art methods may fail to automatically delineate all the correct boundaries, such as the intricate boundary between the hair and the forehead, or the low contrast boundary between the lips and the face. Consequently, the colourization achieved with this alternative method (figure 7b) is noticeably worse than the one computed by our method (figure 7c). In both cases, the same colour scribbles were used. Distinctive colours were deliberately chosen so that flaws in the colourization would be more apparent. Figure 8 compares our method for colourizing image sequences to an alternative method where a single frame is colourized and then optical flow tracking is used to propagate the colours across time. Since our method uses optical flow only to define the local neighbourhood, it is much more robust to tracking failures. In both cases, either using automatic segmentation or using tracking to propagate colours across time, the results could be improved using more sophisticated algorithms. In other words, if the automatic segmentation had been perfect then flood filling segments would have produced perfect results. Likewise, if dense optical flow had been perfect then propagating colours from a single frame would have also worked perfectly. Yet despite many years of research in computer vision, state-of-the-art algorithms still do not work perfectly in an automatic fashion. An advantage of our optimization framework is that we use segmentation cues and optical low as hints. for the correct colourization but the colourization can be quite good even when these hints are wrong.

Dept of Information Technology

16

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

Figure 2: Still image colorization examples. Top row: the input black-white image with scribbled colors. Bottom row: resulting color image

(a1)

(b1)

(c1)

(a2)

(b2)

(c2)

Dept of Information Technology

17

Dr.B.A.T.University, Lonere

Image Cartoonize with Stegnography

Figure 3: Progressively improving a colorization. The artist begins with the scribbles shown in (a1), which yield the result in (a2). Note that the table cloth gets the same pink colour as the girl's dress. Also, some colour is bleeding from the cyan pacifier onto the wall behind it. By adding colour scribbles on the table cloth and on the wall (b1) these problems are eliminated (b2). Next, the artist decides to change the colour of the beads by sprinkling a few red pixels (c1), yielding the final result (c2). Note that it was not necessary to mark each and every bead

(a)

(b)

(c)

Figure 4: A comparison with automatic segmentation. For visualization purposes disctinctive colors were used. (a) Segmented image. (b) Result of colouring each segment with a constant color. Segmenting fuzzy hair boundary is a difficult task for typical segmentation methods. (c) Our result.

Dept of Information Technology

18