1|Page

AUTHENTICATION

This is to certify that the project entitled “Lane Detection Self Driving Car
System” has been carried out by Ayan Kumar Jana , Indrani Bardhan ,
Sanchari Roy in a practical laboratory of department of Information
Technology in Narula Institute Of Technology under guidance of faculty
members.

Submitted By
Name : Ayan Kumar
Jana Roll :
430416010011
Name : Indrani
Bardhan Roll :
430416020018
Name : Sanchari
Roy Roll :
430416020036
2|Page
3|Page

ACKNOWLEDGEMENT

I take this opportunity to express my profound gratitude and deep

regards to my faculty for his exemplary guidance, monitoring and
constant encouragement throughout the course of this project. The
blessing, help and guidance given by him time to time shall carry me a
long way in the journey of life on which I am about to embark.

I am obliged to my project team members for the valuable information

provided by them in their respective fields. I am grateful for their
cooperation during the period of my assignment.

Ayan Kumar Jana

Indrani Bardhan
Sanchari Roy

1. INTRODUCTION
4|Page

Index
1.1 History Of Self Driving Car

1.2 INTRODUCTION TO MACHINE LEARNING

1.3 NEED OF MACHINE LEARNING

1.3.1 KINDS OF MACHINE LEARNING

1.4 INTRODUCTION OF COMPUTER VISION

1.5 INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORK

2. SURVEY
3. PROJECT OBJECTIVES

4. SOFTWARE REQUIREMENTS SPECIFICATION

4.1 IMAGE PROCESSING MODEL
4.1.1 OPEN CV LIBRARY
4.2 MACHINE LEARNING MODEL
4.2.1 MACHINE LEARNING MODEL TENSORFLOW LIBRARY
4.2.2 KERAS IN PYTHON

5. Data Set
5.1 STREET SIGN DATASET

6. DATA PREPROCESSING
6.1 IMAGE COLOUR CONVERSION

6.1.1 RGB To Grey Scale

6.1.2 RGB To YUV Scale
5|Page

6.2 Edge Detection

6.3 Lane Detection
6.4 Line Detection
6.5 Display Lines Above The Main Image

7. System Block Diagram

8. Conclusion

9. References
6|Page

1. INTRODUCTION
1.1 History Of Self Driving Car

One of the fundamental challenges that computers face that humans have never
faced is object comprehension. When we look at the world, we are able to identify
objects such as books and phones and other things that we recognize. Computers, on
the other hand, have a hard time doing the same thing. This is because, fundamentally,
computers only recognize images and videos as a collection of pixels rather than
objects as a whole.

This problem is crucial as it is one of the first obstacles we have to overcome if we are to
make self driving cars a reality. The first thing that it has to do is identify lane lines on
the road so that it knows where to drive.

Automobile manufacturers such as Ford, General Motors, Tesla, and other companies
such as NVIDIA are investing billions of dollars in self-driving car research. Governments
of countries in Europe and of the USA are creating regulations for self-driving cars, as a
result of the latest advances of the industry.

The benefits of self-driving vehicles can go way beyond removing the need of a human
driver. Transportation services such as Uber will start using self-driving cars instead of
human drivers, and might become a cheaper and better alternative for end
consumers than owning a car. This will represent a shift on the way cities are
planned, as fewer parking places will be needed, and most importantly, in a smart
city with most of its vehicles being connected and autonomous, traffic optimization
will be able to be heavily applied by coordinating movement. This will result in a
major decrease on travel time, and it will also save lives as emergency services will
be able to reach their destinations faster.

In contrast, there are many problems that need to be addressed regarding self-
driving vehicles. Tesla’s car autopilot first fatal crash in 2016 [9], for example, brought up
discussions about reliability, safety and legal liability regarding autonomous vehicles
among researchers. Other important topics, such as security and transparency will need
to be heavily discussed as these vehicles become mainstream.

Another problem of this relatively new field is the lack of data sharing between
vehicles. If the data is shared between the vehicles it could be used for coordinating
movement, which could increase safety on cities and highways.
7|Page

1.2 INTRODUCTION TO MACHINE LEARNING

Machine Learning is an idea to learn from examples and experience, without be-
ing explicitly programmed. Instead of writing code, you feed data to the generic
algorithm, and it builds logic based on the data given. For example, one kind of
algorithm is a classification algorithm. It can put data into different groups. The
classification algorithm used to detect handwritten alphabets could also be used to
classify emails into spam and not-spam. As stated by Tom M. Mitchell A computer
program is said to learn from experience E with some class of tasks T and perfor-
mance measure P if its performance at tasks in T, as measured by P, improves
with experience E.. Consider playing checkers.

E=the experience of playing many games of

checkers T=the task of playing checkers
P=the probability that the program will win the next game

There are many examples of machine learning. Here are a few examples of
classification problems where the goal is to categorize objects into a fixed set of
categories---
Face detection: Identify faces in images (or indicate if a face is present).

Email filtering: Classify emails into spam and not-spam.

Medical diagnosis: Diagnose a patient as a sufferer or non-sufferer of some

disease.

Weather prediction: Predict, for instance, whether or not it will rain tomorrow.
8|Page

1.3 NEED OF MACHINE LEARNING

Machine Learning is a field which is raised out of Artificial Intelligence(AI). A p-plying
AI, we wanted to build better and intelligent machines. But except for few mere
tasks such as finding the shortest path between point A and B, we were unable to
program more complex and constantly evolving challenges.There was a realisation that
the only way to be able to achieve this task was to let machine learn from itself. This
sounds similar to a child learning from its self. So machine learning was developed
as a new capability for computers. And now machine learning is present in so
many segments of technology, that we do not even realise it while using it.
Finding patterns in data on planet earth is possibly only for human brains.The data
being very massive,the time taken to complete is increased and this is where machine
learning comes into action to help people with large data in minimum time.
If big data and cloud computing are gaining importance for their
contributions, machine learning as technology helps analyse those big
chunks of data, easing the task of data scientists in an automated process and
gaining equal importance and recognition.

The techniques we use for data mining have been around for many years, but
they were not effective as they did not have the competitive power to run
the algorithms.If you run deep learning with access to better data, the
output we get will lead to dramatic breakthroughs which is machine
learning.

1.3.1KINDS OF MACHINE LEARNING

There are three kinds of Machine Learning Algorithms.

a. Supervised Learning
b. Unsupervised Learning
c. Reinforcement Learning
9|Page

1.4 INTRODUCTION OF COMPUTER VISION

Humans use their eyes and their brains to see and visually sense the world
around them. Computer vision is the science that aims to give a similar, if
not better, capability to a machine or computer.

Computer vision is concerned with the automatic extraction, analysis and

un- derstanding of useful information from a single image or a sequence of
images. It involves the development of a theoretical and algorithmic basis
to achieve automatic visual understanding.

Computer vision is an interdisciplinary field that deals with how computers can
be made to gain high-level understanding from digital images or videos.
From the perspective of engineering, it seeks to automate tasks that the
human visual system can do. Computer vision tasks include methods for
acquiring, processing, an-alyzing and understanding digital images, and
extraction of high-dimensional data from the real world in order to produce
numerical or symbolic information, e.g., in the forms of decisions.
Understanding in this context means the transformation of visual images
(the input of the retina) into descriptions of the world that can interface with
other thought processes and elicit appropriate action. This image
understand-ing can be seen as the disentangling of symbolic information from
image data using models constructed with the aid of geometry, physics,
statistics, and learning theory.

As a scientific discipline, computer vision is concerned with the theory

behind artificial systems that extract information from images. The image data
can take many forms, such as video sequences, views from multiple
cameras, or multidimensional data from a medical scanner. As a
technological discipline,
10 | P a g e

computer vision seeks to apply its theories and models for the construction
of computer vision systems.

In the late 1960s, computer vision began at universities that were pioneering artificial
intelligence. It was meant to mimic the human visual system, as a stepping stone to
endowing robots with intelligent behaviour. In 1966, it was believed that this could be
achieved through a summer project, by attaching a camera to a computer and having
it describe what it saw.

What distinguished computer vision from the prevalent field of digital image
processing at that time was a desire to extract three-dimensional structure from
images with the goal of achieving full scene understanding. Studies in the 1970s
formed the early foundations for many of the computer vision algorithms that exist
today, including extraction of edges from images, labelling of lines, non-polyhedral and
polyhedral modelling, representation of objects as interconnections of smaller
structures, optical flow, and motion estimation.

The next decade saw studies based on more rigorous mathematical analysis and
quantitative aspects of computer vision. These include the concept of scale-space, the
inference of shape from various cues such as shading, texture and focus, and contour
models known as snakes. Researchers also realized that many of these mathematical
concepts could be treated within the same optimization framework as regularization and
Markov random fields. By the 1990s, some of the previous research topics became
more active than the others. Research in projective 3-D re-constructions led to better
understanding of camera calibration. With the advent of optimization methods for
camera calibration, it was realized that a lot of the ideas were already explored in
bundle adjustment theory from the field of photogrammetry. This led to methods for
sparse 3-D reconstructions of scenes from multiple images. Progress was made on the
dense stereo correspondence problem and further multi-view stereo techniques. At
the same time, variations of graph cut were used to solve image segmentation. This
decade also marked the first time statistical learning techniques were used in practice to
recognize faces in images. Toward the end of the 1990s, a significant change came
about with the increased interaction between the fields of computer graphics and
computer vision. This included image-based rendering, image morphing, view
interpolation, panoramic image stitching and early light-field rendering.
11 | P a g e
12 | P a g e

1.5 INTRODUCTION TO NEURAL NETWORK

Neural networks are a set of algorithms, modeled loosely after the human brain,
that are designed to recognize patterns. They interpret sensory data through a
kind of machine perception, labeling or clustering raw input. The patterns they
recognize are numerical, contained in vectors, into which all real-world data, be it
images, sound, text or time series, must be translated.
Neural networks help us cluster and classify. You can think of them as a clustering
and classification layer on top of the data you store and manage. They help to
group unlabeled data according to similarities among the example inputs, and they
classify data when they have a labeled dataset to train on. (Neural networks can
also extract features that are fed to other algorithms for clustering and
classification; so you can think of deep neural networks as components of larger
machine-learning applications involving algorithms for reinforcement learning,
classification and regression.
What kind of problems does deep learning solve, and more importantly, can it
solve yours? To know the answer, you need to ask questions: What outcomes do I
care about? Those outcomes are labels that could be applied to data: for example,
spam or not spam in an email filter, good guy or bad guy in fraud detection, angry
customer or happy customer in customer relationship management
13 | P a g e

2. SURVEY

According to a recent technical report by the National Highway Traffic Safety

Administration (NHTSA), 94% of road accidents are caused by human errors. Automated
driving systems (ADSs) are being developed with the promise of preventing accidents,
reducing emissions, transporting the mobility-impaired and reducing driving related
stress. Annual social benefits of ADSs are projected to reach nearly $800 billion by
2050 through congestion mitigation, road casualty reduction, decreased energy
consumption and increased productivity caused by the reallocation of driving time.
Eureka Project PROMETHEUS was carried out in Europe between 1987-1995,and it
was one of the earliest major automated driving studies. The project led to the
development of VITA II by Daimler-Benz, which succeeded in automatically driving on
highways. DARPA Grand Challenge, organized by the US Department of Defense in
2004, was the first major automated driving competition where all of the attendees
failed to finish the 150-mile off- road parkour. The difficulty of the challenge was due
to the rule that no human intervention at any level was allowed during the finals.
Another similar DARPA Grand Challenge was held in 2005. This time five teams
managed to complete the off-road track without any human interference.
Common practices in system architecture have been established over the years. Most of
the ADSs divide the massive task of automated driving into subcategories and employ an
array of sensors and algorithms on various modules. More recently, end-to-end
driving systems started to emerge as an alternative to modular approaches. Deep
learning models have become dominant in many of these tasks.
The accumulated knowledge in vehicle dynamics, breakthroughs in computer vision
caused by the advent of machine learning and availability of new sensor modalities,
such as lidar, catalysed ADS research and industrial implementation. Furthermore, an
increase in public interest and market potential precipitated the emergence of ADSs
with varying degrees of automation. However, robust automated driving in urban
environments has not been achieved yet. Accidents caused by immature systems
undermine trust, and furthermore, they cost lives. As such, a thorough investigation of
unsolved challenges and the state-of-the-art is deemed necessary here.
14 | P a g e

3. PROJECT OBJECTIVES

In this project we are trying to creat a protocol that specifies how self-driving cars
can share their sensor data and use this information to collaboratively optimize the line
on the road and increase safety, without needing a central server. The idea is to
propose a standard set of messages for cars and rules for their behaviour. These
rules will take into account, for each vehicle, its received messages, its intent and its
sensing information.
In addition to developing the protocol specification, the simulation environment for
the protocol will need to be created. This will involve modifying a self-driving car simulator
to use the protocol.The path planner that controls the self-driving car was written using
the Python programming language with the help of machine learning and AI.
In short, the main goals of the project are:
• develop the specific protocol to do the project
• develop a environment(i.e we are creating a video clip to run the program) where
cars can use the protocol to communicate
• assess if by using the protocol the mean travel time of the vehicles is reduced
15 | P a g e

4. SOFTWARE REQUIREMENTS SPECIFICATION

4.1. IMAGE PROCESSING MODEL

4.1.1. OPEN CV LIBRARY

OpenCV (Open Source Computer Vision Library) is released under a BSD license and hence
its free for both academic and commercial use. It has C++, Python and Java
interfaces and supports Windows, Linux, Mac OS, iOS and Android. OpenCV was
designed for computational efficiency and with a strong focus on real-time applications.
Written in optimized C/C++, the library can take advantage of multi-core processing.
Enabled with OpenCL, it can take advantage of the hardware acceleration of the
underlying heterogeneous compute platform. Adopted all around the world, OpenCV has
more than 47 thousand people of user community and estimated number of downloads
exceeding 14 million. Usage ranges from interactive art, to mines inspection, stitching
maps on the web or through advanced robotics.

4.2. MACHINE LEARNING MODEL

4.2.1MACHINE LEARNING MODEL TENSORFLOW LIBRARY

TensorFlow is an open source software library for high performance numerical

computation. Its flexible architecture allows easy deployment of computation across a
variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to
mobile and edge devices. Originally developed by researchers and engineers from the
Google Brain team within
16 | P a g e

Googles AI organization, it comes with strong support for machine learning and deep
learning and the flexible numerical computation core is used across many other scientific
domains.

4.2.2KERAS IN PYTHON

Keras is a high-level neural networks API, written in Python and capable of running on
top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast
experimentation. Being able to go from idea to result with the least possible delay is
key to doing good research. Use Keras if you need a deep learning library that: Allows
for easy and fast prototyping (through user friendliness, modularity, and extensibility).
Supports both convolutional networks and recurrent networks, as well as
combinations of the two. Runs seamlessly on CPU and GPU. Read the documentation
at Keras.io. Keras is compatible with: Python 2.7- 3.7
17 | P a g e

5. Data Set

5.1 STREET SIGN DATASET

The above figure shows a portion of the data set we used for street sign
detection, It is the German traffic sign data-set open sourced on GITHUB. It
contains a total of around 56000 different images with labels and has 48
different classes for classi-fication. It has One directory per class, Each
directory contains one CSV file with annotations (”GT-¡ClassID¿.csv”) and the
training images, The training images are grouped by tracks, Each track contains
30 images of one single physical traffic sign
18 | P a g e

6. DATA PREPROCESSING
The image data we collected from the simulator is raw and we cannot use this
data directly to create a model, We need to preprocess this data in order to
extract important features from the data there are many steps in
preprocessing the image data in our project.

6.1 IMAGE COLOUR CONVERSION

6.1.1RGB To Grey Scale

The images from our dataset are coloured but we convert the image to gray
image first, we get more accurate features of the image because gray
scale image use one channel conversation , so we use the python commands
to convert the image to grayscale images. The below figues shows the
difference between the original and converted images.
19 | P a g e

Code for RGB to GRAY convertion

import cv2

import numpy as np

#from matplotlib import pyplot as pp

def gray(image):

gray_image = cv2.cvtColor(convert_lane_image,cv2.COLOR_RGB2GRAY)

return gray_image

image = cv2.imread(‘test.png’)

convert_lane_image = np.copy(image)

cv2.imshow(‘result_window’,gray(convert_lane_image))

cv2.waitKey(0)
20 | P a g e

6.1.2RGB To YUV Scale

The images from our dataset are coloured but we convert the
image to YUV image so we use the python commands to convert the
image to YUV images. The below figues shows the difference between
the original and converted images

import cv2
import numpy as np

def yuv(image):
yuv_image = cv2.cvtColor(convert_lane_image,cv2.COLOR_RGB2YUV)
return yuv_image

image = cv2.imread(‘test.png’)
convert_lane_image = np.copy(image)
cv2.imshow(‘result_window’,yuv(convert_lane_image))
cv2.waitKey(0)
21 | P a g e

6.2 Edge Detection

Code for Canny Edge Detection

import cv2
import numpy as np
from matplotlib import pyplot as pp

def canny(image):
gray_image = cv2.cvtColor(convert_lane_image,cv2.COLOR_RGB2GRAY)
blur_image = cv2.GaussianBlur(gray_image,(5,5),0)
canny_edge = cv2.Canny(blur_image,40,110)
return canny_edge

image = cv2.imread('test.png')
convert_lane_image = np.copy(image)
canny_image = canny(convert_lane_image)

pp.imshow(canny(canny_image))
pp.show()
22 | P a g e

6.3 LANE DETECTION

The above figure show how the lane markings are seen by a computer
vision the openCV tool is used along with python language to read the frames
of a live video feed and then perform some operations like gaussian blur,
edge detection and hough transform to get the lane markings accurately.
Code Of Main Region
import cv2
import numpy as np
from matplotlib import pyplot as pp
def canny(image):
gray_image = cv2.cvtColor(convert_lane_image,cv2.COLOR_RGB2GRAY)
blur_image = cv2.GaussianBlur(gray_image,(5,5),0)
canny_edge = cv2.Canny(blur_image,40,110)
return canny_edge
def main_region(image):
height = image.shape[0]
region = np.array([
[(134,height),(982,height),(579,312)]])
mask = np.zeros_like(image)
cv2.fillPoly(mask,region,255)
masked_image = cv2.bitwise_and(image,mask)
return masked_image
image = cv2.imread('test.png')
convert_lane_image = np.copy(image)
canny_image = canny(convert_lane_image)
area_image = main_region(canny_image)
cv2.imshow('result_window',area_image)
23 | P a g e

6.4 Line Detection

Code Of Line Detection

import cv2
import numpy as np
from matplotlib import pyplot as pp

def canny(image):
gray_image = cv2.cvtColor(convert_lane_image,cv2.COLOR_RGB2GRAY)
blur_image = cv2.GaussianBlur(gray_image,(5,5),0)
canny_edge = cv2.Canny(blur_image,40,110)
return canny_edge

def main_region(image):
height = image.shape[0]
region = np.array([
[(134,height),(982,height),(579,312)]
])
mask = np.zeros_like(image)
cv2.fillPoly(mask,region,255)
masked_image = cv2.bitwise_and(image,mask)
return masked_image

def line_display(image,lines):
line_image = np.zeros_like(image)
if lines is not None:
for line in lines:
x1,y1,x2,y2 = line.reshape(4)
cv2.line(line_image,(x1,y1),(x2,y2),(255,0,0), 8)
return line_image
24 | P a g e

image = cv2.imread('test.png')
convert_lane_image = np.copy(image)
canny_image = canny(convert_lane_image)
area_image = main_region(canny_image)
lines = cv2.HoughLinesP(area_image,2,np.pi/180,100,np.array([]), minLineLength = 30 , maxLineGap = 20
cv2.imshow('result_window',line_image)
cv2.waitKey(0)

6.5 Display Lines Above The Main Image

25 | P a g e

Code For Displaying lines over The Image

import cv2
import numpy as np
from matplotlib import pyplot as pp

def canny(image):
gray_image = cv2.cvtColor(convert_lane_image,cv2.COLOR_RGB2GRAY)
blur_image = cv2.GaussianBlur(gray_image,(5,5),0)
canny_edge = cv2.Canny(blur_image,40,110)
return canny_edge

def main_region(image):
height = image.shape[0]
region = np.array([
[(134,height),(982,height),(579,312)]
])
mask = np.zeros_like(image)
cv2.fillPoly(mask,region,255)
masked_image = cv2.bitwise_and(image,mask)
return masked_image

def line_display(image,lines):
line_image = np.zeros_like(image)
if lines is not None:
for line in lines:
x1,y1,x2,y2 = line.reshape(4)
cv2.line(line_image,(x1,y1),(x2,y2),(255,0,0), 8)
return line_image

image = cv2.imread('test.png')
convert_lane_image = np.copy(image)
canny_image = canny(convert_lane_image)
area_image = main_region(canny_image)
lines = cv2.HoughLinesP(area_image,2,np.pi/180,100,np.array([]), minLineLength = 30 , maxLineGap = 20)
line_image = line_display(convert_lane_image , lines)

final_line_image = cv2.addWeighted(convert_lane_image,0.8, line_image,1,1)

cv2.imshow('result_window',final_line_image)
cv2.waitKey(0)
26 | P a g e

7. SYSTEM BLOCK DIAGRAM

Open Cv Lane detection

Back Propagation
Manual training by Convolutional
driving car Neutral Network

Machine Learning
Algorithm Tensorflow,Keras

Adjustment in the car

steering

Autopilot Car

The Above block diagram shows the basic block diagram of our system where in we take
the input from the camera, The image from the camera is the live from the camera mounted
on the bonnet of the car so the image will be of the road lane in front of the car in which the
is being driven. The image is processed using Open cv lane detection algorithm and the
output of the detected lane is fed as input to the Machine learning algorithm which will
receive another input from the convolutional neural network. The convolutional neural
network is designed to keep the car inside the desired lane. The convolutional neural
network is trained using back propagation technique in which we drive the car multiple times
through different tracks and keep recording the video and updating the data set. The
Machine learning algorithm after receiving all the required inputs will the accurately
calculates the position of the lanes and the degree to which we need to steer the car the
output is corrected after each training cycle and we get the autopilot control.
27 | P a g e

Conclusion
After implementation of this project we conclude that the ’Autopilot Vehicles’
are the future Which can be a useful resource if used with care and
precautions are taken while making the algorithm and adequate amount
of training on different environments are done with care. We are done till
the part of lane detection and street sign detection and are planning to do
the total self driven car.

References
https://towardsdatascience.com/

https://opencv.org/

https://opencv-python-
tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_canny/py_canny.html

https://keras.io/
https://youtube.com