Institut fr Informatik

Classification using
Logistic Regression
Ingmar Schuster
Patrick Jhnichen
using slides by Andrew Ng
Logistic regression 2
This lecture covers

Logistic regression
hypothesis

Decision Boundary

Cost function
(why we need a new one

!implified Cost function "

#radient Descent

$dvanced %ptimi&ation
$lgorithms

'ulticlass classification
Logistic regression 3
Logistic regression
Hypothesis Representation
Logistic regression 4
Classification (ro)lems

Classification

malignant or benign cancer

Spam or Ham

Human ace or no human ace

Positi!e Sentiment"

Binary Decision Task

(in most simple case

*ant

Data point )elongs to class

if close to +

Doesn,t )elong to class

if close to -
Logistic regression 5
Logistic .unction (!igmoid .unction

maps into interval #$%&'

$ asymptote for

& asymptote for

!igmoid .unction (!/shape
Logistic .unction
Logistic regression 6

0ypothesis

Interpretation

Because pro)a)ilites should sum to &1 define

If interpret as 2-3 chance data point )elongs to class

If classify as positive sentiment, malignant tumor, ...

Logistic regression 7
Logistic regression
(ecision boundary
Logistic regression 8

If
or e4uivalently
predict y ) &

If
or e4uivalently
predict y ) $
Logistic regression 9
56ample

If
and

(rediction y 7 + whenever
Logistic regression 10
56ample

If
and

(rediction y 7 + whenever
Logistic regression 11
Logistic regression
*ost +unction
Logistic regression 12
Training and cost function

Training data wih m datapoints1 n features

where

$verage cost
13
Reusing Linear Regression cost

Cost from linear regression

with logistic regression
hypothesis

leads to non/conve6 average
cost

Conve6 J easier to optimi&e

(no local optima
All unction !alues below
intersection with any line
All unction !alues below
intersection with any line
Logistic regression 14
Logistic Regression Cost function

If y ) & and h,-. ) &1 *ost ) $

But for

Corresponds to intuition8
if prediction is h,-. ) $ )ut
actual value was y ) &1
learning algorithm will )e
penali&ed )y large cost
Logistic regression 15
Logistic Regression Cost function

If y ) $ and h,-. ) $1 *ost ) $

But for
Logistic regression 16
Logistic regression
Simpliied *ost +unction /
0radient (escent
Logistic regression 17
!implified Cost .unction (+

%riginal cost of single training e6ample

Because we always have y = 0 or y = 1 we can simplify

the cost function definition to

To convince yourself1 use the simplified cost function to

calculate
Logistic regression 18
!implified Cost .unction (9

Cost function for training set

.ind parameter argument that minimi&es J8

To make predictions given new x output

Logistic regression 19
#radient Descent for logistic regression

#radient Descent to minimi&e logistic regression cost function

with identical algorithm as for linear regression
Logistic regression 20
1eyond 0radient (escent
2 Ad!anced 3ptimi4ation
Logistic regression 21
$dvanced %ptimi&ation $lgorithms

#iven functions to compute

an optimi&ation algorithm will compute

$dvantages

%ften faster convergence

:o learning rate to choose

Disadvantages

Comple6
%ptimi&ation $lgorithms

(Gradient Descent)

Con;ugate #radient

B.#! " L/B.#!

Logistic regression 22
(reimplemented $lorithms

$dvanced optimi&ation algorithms e6ist already in 'achine

Learning packages for important languages

%ctave<'atla)

=ava

Rapidminer > under the hood

Logistic regression 23
5ulticlass *lassiication
,by cheap trickery.
Logistic regression 24
'ulticlass classification pro)lems

Classes of 5mails8 *ork1 .riends1 Invoices1 =o) %ffers

'edical diagnosis8 :ot ill1 $sthma1 Lung Cancer

*eather8 !unny1 Cloudy1 Rain1 !now

:um)er classes as +1 91 ?1 @@@

Logistic regression 25
Binary vs@ 'ulticlass Classification
Logistic regression 26
%ne versus all
Logistic regression 27

Train logistic regression classifier

for each class i to predict pro)a)ility of y = i

%n new x predict class i which satisfies

Logistic regression 28
This lecture covered

Logistic regression
hypothesis

Decision Boundary

Cost function(why we need a

new one

!implified Cost function "

#radient Descent

$dvanced %ptimi&ation
$lgorithms

'ulticlass classification
Machine Learning Introduction 29
(ictures

Tumor picture by lickr2user bc

the path6 License CC SA NC

Lightbulb picture rom

openclipart7org6 public domain