Image Restoration

A Seminar Report submitted

by

Vikas Meena
18MA60R32

in Partial Fulfillment for the award

of the degree of
MASTER OF TECHNOLOGY
in
COMPUTER SCIENCE AND DATA
PROCESSING
at

DEPARTMENT OF MATHEMATICS
INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR
WEST BENGAL-721302, INDIA
CONTENTS

Contents
1 Introduction 2

2 Literature Review 3

3 Convolutional Neural Network 3

3.1 Convolution Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.2 Strides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.3 Padding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.4 Non Linearity (ReLU) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.5 Pooling Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.6 Fully Connected Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4 Blurring process model 10

4.1 Filters Used for Blurring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.1.1 Gaussian Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

5 Denoising of Image using Geometric Mean Filter 12

6 Results and Analysis 13

7 Conclusion and Future works 15

8 References 16
CONTENTS

Abstract

Image restoration is the process of recovering the original image from the degraded image. Aspire of
the project is to restore the blurred/degraded images using Deconvolution algorithm. The fundamental
task of Image deblurring is to de-convolute the degraded image with the PSF that exactly describe the
distortion. Firstly, the original image is degraded using the Degradation Model. It can be done by
Gaussian filter which is a low-pass filter used to blur an image. In the edges of the blurred image, the
ringing effect can be detected using Canny Edge Detection method and then it can be removed before
restoration process. Deconvolution algorithm is applied to the blurred image. It is possible to renovate
the original image without having specific knowledge of degradation filter, additive noise and PSF.

1
1 Introduction
Blur is a common and unwanted artifact of image acquisition. There are many reasons why images
become blurred such as movement, slow shutter speed or incorrect focal distance. Because blurry
images are confusing, they are often less appealing and so it is desirable to deblur images for purely
aesthetic reasons. Outside of photography, variable blurring creates problems for feature tracking
algorithms which rely on the existence of consistent image structures. Blur removal may be used to
precondition these algorithms and greatly improve performance.

The most common approach of blur removal is to treat blur as a noisy convolution operation. This
model has both its advantages and disadvantages which we discuss in greater detail in the subsequent
sections. From the perspective of convolution, there are two basic types of blur removal: non-blind
and blind, with the main difference being that in the case of non-blind deconvolution the blur kernel
is known.

Many people think that blurring is an irreversible operation and the information in this case is lost
for good, because each pixel turns into a spot, everything mixes up, and in case of a big blur radius
we will get a flat color all over the image. But it is not quite true - all the information just becomes
redistributed in accordance with some rules and can be definitely restored with certain assumptions.
An exception is only borders of the image, the width of which is equal to the blur radius - no complete
restoration is possible here.

After blurring the value of each pixel is added to the value of the left one: x’i = xi + xi-1. Nor-
mally, it is also required to divide it by 2, but we will drop it out for simplicity. As a result we have a
  
blurred image with the following pixel values: x1 + x0  x2 + x1  x3 + x2  x4 + x3... - Blurred image

Now we will try to restore it, we will consequentially subtract values according to the following scheme
- the first pixel from the second one, the result of the second pixel from the third one, the result of
 
the third pixel from the fourth one and so on, and we will get the following: x1 + x0  x2 - x0  x3 +

x0  x4 - x0... - Restored image

As a result, instead of a blurred image, we got the source image with added to the each pixel an
unknown constant x0 with the alternate sign. This is much better - we can choose this constant
visually, we can suppose that it is approximately equal to the value x1, we can automatically choose
it with such a criteria that values of neighboring pixels were changing as little as possible, etc. But
everything changes as soon as we add noise (which is always present in real images). In case of the
described scheme on each stage there will accumulate the noise value into the total value, which fact
eventually can produce an absolutely unacceptable result, but as we already know, restoration is quite
possible even using such a primitive method.

2
2 Literature Review
Process of Image Restoration:-

• Image Blurring:-First of all the image is converted into blurred image for training the data so it
can learn the parameters. The Gaussian Kernel is taken in account for the blurring the image.

• Image Deblurring:-After blurring out the image we will use Gaussian Filter to deblur the image
using deconvolution.

3 Convolutional Neural Network

In neural networks, Convolutional neural network (ConvNets or CNNs) is one of the main categories
to do images recognition, images classifications. Objects detections, recognition faces etc., are some
of the areas where CNNs are widely used. CNN image classifications takes an input image, process it
and classify it under certain categories (Eg., Dog, Cat, Tiger, Lion). Computers sees an input image
as array of pixels and it depends on the image resolution. Based on the image resolution, it will see
h * w * d( h = Height, w = Width, d = Dimension ). Eg., An image of 6 * 6 * 3 array of matrix of
RGB (3 refers to RGB values) and an image of 4 * 4 * 1 array of matrix of grayscale image.

Technically, deep learning CNN models to train and test, each input image will pass it through a
series of convolution layers with filters (Kernals), Pooling, fully connected layers (FC) and apply Soft-
max function to classify an object with probabilistic values between 0 and 1. The below figure is a
complete flow of CNN to process an input image and classifies the objects based on values.

Array of RGB Cell

3
3.1 Convolution Layer

Neural network with many convolutional layers

3.1 Convolution Layer

Convolution is the first layer to extract features from an input image. Convolution preserves the
relationship between pixels by learning image features using small squares of input data. It is a math-
ematical operation that takes two inputs such as image matrix and a filter or kernal

Image matrix multiplies kernel or filter matrix

Consider a 5 x 5 whose image pixel values are 0, 1 and filter matrix 3 x 3 as shown in below

Image matrix multiplies kernel or filter matrix

4
3.1 Convolution Layer

Then the convolution of 5 x 5 image matrix multiplies with 3 x 3 filter matrix which is called
“Feature Map” as output shown in below

3 x 3 Output matrix

Convolution of an image with different filters can perform operations such as edge detection, blur
and sharpen by applying filters. The below example shows various convolution image after applying
different types of filters (Kernels).

Some common filters

5
3.2 Strides

3.2 Strides
Stride is the number of pixels shifts over the input matrix. When the stride is 1 then we move the
filters to 1 pixel at a time. When the stride is 2 then we move the filters to 2 pixels at a time and so
on. The below figure shows convolution would work with a stride of 2.

Stride of 2 pixels

Stride is the number of pixels a convolutional filter moves, like a sliding window, after passing on the
weighted average value of all the pixels it just covered.The old weighted average value becomes one pixel
in the featuremap in the next layer. The next weighted average comes from a new group of pixels and
it forms the next pixel in the featuremap in the next layer.

6
3.3 Padding

3.3 Padding
Sometimes filter does not fit perfectly fit the input image. We have two options:

• Pad the picture with zeros (zero-padding) so that it fits Drop the part of the image where the
filter did not fit.

• This is called valid padding which keeps only valid part of the image.

3.4 Non Linearity (ReLU)

ReLU stands for Rectified Linear Unit for a non-linear operation. The output is

f (x) = max(0, x)

Why ReLU is important : ReLU’s purpose is to introduce non-linearity in our ConvNet. Since, the
real world data would want our ConvNet to learn would be non-negative linear values.

ReLU operation

There are other non linear functions such as tanh or sigmoid can also be used instead of ReLU. Most
of the data scientists uses ReLU since performance wise ReLU is better than other two.

3.5 Pooling Layer

Pooling layers section would reduce the number of parameters when the images are too large. Spatial
pooling also called subsampling or downsampling which reduces the dimensionality of each map but
retains the important information. Spatial pooling can be of different types:

• Max Pooling

• Average Pooling

• Sum Pooling

Max pooling take the largest element from the rectified feature map. Taking the largest element could
also take the average pooling. Sum of all elements in the feature map call as sum pooling.

7
3.6 Fully Connected Layer

Max Pooling

Max Pooling also performs as a Noise Suppressant. It discards the noisy activations altogether and
also performs de-noising along with dimensionality reduction. On the other hand, Average Pooling
simply performs dimensionality reduction as a noise suppressing mechanism. Hence, we can say that
Max Pooling performs a lot better than Average Pooling.
The Convolutional Layer and the Pooling Layer, together form the i-th layer of a Convolutional Neural
Network. Depending on the complexities in the images, the number of such layers may be increased
for capturing low-levels details even further, but at the cost of more computational power.

3.6 Fully Connected Layer

The layer we call as FC layer, we flattened our matrix into vector and feed it into a fully connected
layer like neural network.

After pooling layer, flattened as FC layer

In the above diagram, feature map matrix will be converted as vector (x1, x2, x3, . . . ). With the
fully connected layers, we combined these features together to create a model. Finally, we have an
activation function such as softmax or sigmoid to classify the outputs as cat, dog, car, truck etc.,

8
3.6 Fully Connected Layer

Complete CNN architecture

9
4 Blurring process model
Now let’s pass on to more formal and scientific description of these blurring and restoration pro-
cesses. We will consider only grayscale images, supposing that for processing of a full-color image it
is enough to repeat all required steps for each of the RGB color channels. Let’s introduce the following
definitions:
f(x, y) - source image (non-blurred)
h(x, y) - blurring function
n(x, y) - additive noise
g(x, y) - blurring result image

We will form the blurring process model in the following way:

g(x, y) = h(x, y) * f(x, y) + n(x, y)

The task of restoration of a blurred image consists in finding the best approximation f ’(x, y) to the
source image. Let’s consider each component in a more detailed way. As for functions f(x, y) and g(x,
y), everything is quite clear with them. In the process of blurring the each pixel of a source image turns
into a spot in case of defocusing and into a line segment (or some path) in case of a usual blurring
due to movement. Or we can say otherwise, that each pixel of a blurred image is ”assembled” from
pixels of some nearby area of a source image.

All those overlap each other, which fact results in a blurred image. The principle, according to which
one pixel becomes spread, is called the blurring function. Other synonyms - PSF (Point spread func-
tion), kernel and other. The size of this function is lower than the size of the image itself.

10
4.1 Filters Used for Blurring

4.1 Filters Used for Blurring

4.1.1 Gaussian Filter

A Gaussian filter is a filter whose impulse response is a Gaussian function (or an approximation to it,
since a true Gaussian response is physically unrealizable). Gaussian filters have the properties of having
no overshoot to a step function input while minimizing the rise and fall time. This behavior is closely
connected to the fact that the Gaussian filter has the minimum possible group delay. Mathematically,
a Gaussian filter modifies the input signal by convolution with a Gaussian function.

Shape of the impulse response of a typical Gaussian filter

A Gaussian filter is a linear filter. It’s usually used to blur the image or to reduce noise. If you use
two of them and subtract, you can use them for ”unsharp masking” (edge detection). The Gaussian
filter alone will blur edges and reduce contrast.
The Median filter is a non-linear filter that is most commonly used as a simple way to reduce noise
in an image. It’s claim to fame (over Gaussian for noise reduction) is that it removes noise while
keeping edges relatively sharp.
One advantage of Gaussian filter has over a median filter is that it’s faster because multiplying and
adding is probably faster than sorting.

11
5 Denoising of Image using Geometric Mean Filter

Filteration of three band separately

While facing Gaussian noise, each pixel of the image will be affected. Noise is an unavoidable side
effect. Noise separates the red, green and blue channels. It is followed by introducing a gain to compen-
sate the attenuation resulting from the filter. Each filtered channel is then combined to form resulting
colored image.
Geometric mean filter replaces the colour value of each pixel with the geometric mean of colour pixel
values from a larger region surrounding it, based on filter size (3x3 or 5x5) and yields a stronger filter
effect. The geometric mean filter performed better than conventional methods such as arithmetic filters
to remove Gaussian type noise and preserve edge features.
Geometric filter is a simplest form of mean filter. Let’s Sxy represents the set of coordinates in a
sub-image window (neighborhood) of size m x n where m and n are equal, centered at point (x, y). The
local image function f(x, y) is filtered image and g(s,t) is input image. In Geometric mean filter each
restored pixel is given

mathtools f(x, y)=[ (s, t)Sxy g(s, t)][ 1/mn].

Q

12
6 Results and Analysis
The Data set taken are from public record.I has used Python to implement our code. We have used
TensorFlow, numpy, Keras and any other libraries which were required during the implementation
time of the code. OpenCV is a popular framework for computer vision and image processing. I has
used OpenCV to process the Blurry images.
Keras is a deep learning framework written in Python. It makes it easy to define, train and use deep
neural networks.
TensorFlow is Googles library for machine learning. I has been code in Keras, but Keras does not
actually implement the neural network logic itself. Instead, it uses Googles TensorFlow library behind
the scenes to do the heavy lifting.
Exploration of different neural networks and observation: We have used Convolutional Neural Network
due to their ability to work on the raw pixels of the image.
CNN Architecture used:-

• Image=64*64

• Stride=1

• Padding=’SAME’

• Gaussian mean=0.1

• Gaussian variance=0.01

• Filter size of 3*3

• flattern=4096

• ReLu=[32]

• alpha=beta=0.1

• Accuracy=67%

13
The following is output of a blurry image

Blurred image

Restored image

14
7 Conclusion and Future works
This paper deals with Image restoration caused by different kinds of blurration of image like motion
blur,environment blur and so on.I have applied the Deconvolution Technique to restore the image using
various filters and among those filters the gaussian filter works very well.The whole algorithm give a
accuracy of 67% which is less comapred to other image restoration techinques.
In the next half I will try to use advanced technique called Deep Convolutional Generative Adversarial
Network(DCGAN) which uses high-level architecture of the deep learning model is to use a generative
adversarial network which can be used for image denoising/restoration,image super-resolution and
many more things.

15
8 References

References
[1] https://www.cs.auckland.ac.nz/courses/compsci373s1c/PatricesLectures/Gaussian

[2] https://towardsdatascience.com/can-you-enhance-that-image-restoration-with-1-training-image-
b54989a57b4d

[3] https://en.wikipedia.org/wiki/Gaussianfilter

16