Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Lecturers:
Goal (Lectures): To present basic theoretical concepts and key algorithms that
form the core of machine learning
Material:
More Info:
https://www.ibug.doc.ic.ac.uk/courses
Maja Pantic
Maja Pantic
Maja Pantic
Maja Pantic
Maja Pantic
Maja Pantic
Maja Pantic
Maja Pantic
Machine Learning
Learning Intelligence
(Def: Intelligence is the ability to learn and use concepts to solve problems.)
Machine Learning Artificial Intelligence
Def: AI is the science of making machines do things that require
intelligence if done by men (Minsky 1986)
Def: Machine Learning is an area of AI concerned with development of
techniques which allow machines to learn
Why Machine Learning? Why Artificial Intelligence?
Maja Pantic
Machine Learning
Maja Pantic
Machine Learning
Learning Intelligence
(Def: Intelligence is the ability to learn and use concepts to solve problems.)
Machine Learning Artificial Intelligence
Def: AI is the science of making machines do things that require
intelligence if done by men (Minsky 1986)
Def: Machine Learning is an area of AI concerned with development of
techniques which allow machines to learn
Why Machine Learning? Why Artificial Intelligence?
To build machines exhibiting intelligent behaviour (i.e., able to reason,
predict, and adapt) while helping humans work, study, and entertain
themselves
Maja Pantic
Machine Learning
Machine Learning Artificial Intelligence
Machine Learning Biology (e.g., Neural Networks, Genetic Algorithms)
Machine Learning Cognitive Sciences (e.g., Case-based Reasoning)
Machine Learning Statistics (e.g., Support Vector Machines)
Machine Learning Probability Theory (e.g., Bayesian Networks)
Machine Learning Logic (e.g., Inductive Logic Programming)
Machine Learning Information Theory (e.g., used by Decision Trees)
Maja Pantic
Machine Learning
Maja Pantic
Well-posed
Problem?
Determine type of
training examples
Determine
Target Function
Choose Target F-on
Representation
Choose Learning
Algorithm
Maja Pantic
Well-posed
Problem?
Determine type of
training examples
Determine
Target Function
Choose Target F-on
Representation
Choose Learning
Algorithm
Maja Pantic
Concept Learning
Concept learning
supervised, eager learning
target problem: whether something belongs to the target concept or not
target function: V: D {true, false}
Underlying idea: Humans acquire general concepts from specific examples
(e.g., concepts: beauty, good friend, well-fitting-shoes)
(note: each concept can be thought of as Boolean-valued function)
Concept learning is inferring a Boolean-valued function from training data
concept learning is the prototype binary classification
Maja Pantic
Aim: Find a hypothesis h H such that (d D) h(d) c(d) < 0, where H is the
set of all possible hypotheses h a1, a2, a3, a4, a5, a6, where each ak, k = [1..6], may
be ? ( any value is acceptable), 0 ( no value is acceptable), or a specific value.
Maja Pantic
instances
error rate
target f-on representation: h c: a1, a2, a3, a4, a5, a6 {0, 1}
training examples D: positive and negative examples of target function c
Aim: Find a hypothesis h H such that (d D) h(d) c(d) < 0, where H is the
set of all possible hypotheses h a1, a2, a3, a4, a5, a6, where each ak, k = [1..6], may
be ? ( any value is acceptable), 0 ( no value is acceptable), or a specific value.
h ?, ?, ?, ?, ?, ?
h 0, 0, 0, 0, 0, 0
Maja Pantic
h ?, ?, yes, ?, ?, ?
Machine Learning (course 395)
instances
Aim: Find a hypothesis h H such that (d D) h(d) c(d) < 0, where H is the
set of all possible hypotheses h a1, a2, a3, a4, a5, a6, where each ak, k = [1..6], may
be ? ( any value is acceptable), 0 ( no value is acceptable), or a specific value.
concept learning searching through H
Maja Pantic
General-to-Specific Ordering
General-to-Specific Ordering:
h1 precedes (is more general than) h2 (d D) (h1(d) = 1) (h2(d) = 1)
(e.g., h1 ?, ?, yes,?, ?, ? and h2 ?, ?, yes,?, ?, yes h1 >g h2 )
h1 and h2 are of equal generality (d D) { [(h1(d) = 1) (h2(d) = 1)]
[(h2(d) = 1) (h1(d) = 1)] h1 and h2 have equal number of ? }
(e.g., h1 ?, ?, yes,?, ?, ? and h2 ?, ?, ?, ?, ?, yes h1 =g h2 )
h2 succeeds (is more specific than) h1 (d D) (h1(d) = 1) (h2(d) = 1)
(e.g., h1 ?, ?, yes,?, ?, ? and h2 ?, ?, yes,?, ?, yes h2 g h1 )
Maja Pantic
Find-S Algorithm
1. Initialise h H to the most specific hypothesis: h a1,,an, (i) ai = 0.
2. FOR each positive training instance d D, do:
FOR each attribute ai, i = [1..n], in h, do:
IF ai is satisfied by d
THEN do nothing
ELSE replace ai in h so that the resulting h >g h, h h.
3. Output hypothesis h.
Maja Pantic
hair
body
likesSimon
pose
smile
smart
blond
thin
yes
arrogant
toothy
no
brown
thin
no
natural
pleasant
yes
blond
plump
yes
goofy
pleasant
no
black
thin
no
arrogant
none
no
blond
plump
no
natural
toothy
yes
h 0,0,0,0,0,0
h d1
h blond, ?, yes, ?, ?, no
Maja Pantic
Find-S Algorithm
Find-S is guaranteed to output the most specific hypothesis h that best fits positive
training examples.
The hypothesis h returned by Find-S will also fit negative examples as long as
training examples are correct.
However,
Find-S is sensitive to noise that is (almost always) present in training examples.
there is no guarantee that h returned by Find-S is the only h that fits the data.
several maximally specific hypotheses may exist that fits the data but, Find-S
will output only one.
Why we should prefer most specific hypotheses over, e.g., most general
hypotheses?
Maja Pantic
hair
body
likesSimon
pose
smile
smart
blond
thin
yes
arrogant
toothy
no
brown
thin
no
natural
pleasant
yes
blond
plump
yes
goofy
pleasant
no
black
thin
no
arrogant
none
no
blond
plump
no
natural
toothy
yes
Find-S Algorithm
Find-S is guaranteed to output the most specific hypothesis h that best fits positive
training examples.
The hypothesis h returned by Find-S will also fit negative examples as long as
training examples are correct.
However,
Find-S is sensitive to noise that is (almost always) present in training examples.
there is no guarantee that h returned by Find-S is the only h that fits the data.
several maximally specific hypotheses may exist that fits the data but, Find-S
will output only one.
Why we should prefer most specific hypotheses over, e.g., most general
hypotheses?
Maja Pantic
hair
body
likesSimon
pose
smile
smart
blond
thin
yes
arrogant
toothy
no
brown
thin
no
natural
pleasant
yes
blond
plump
yes
goofy
pleasant
no
black
thin
no
arrogant
none
no
blond
plump
no
natural
toothy
yes
Find-S Algorithm
Find-S is guaranteed to output the most specific hypothesis h that best fits positive
training examples.
The hypothesis h returned by Find-S will also fit negative examples as long as
training examples are correct.
However,
1. Find-S is sensitive to noise that is (almost always) present in training examples.
2. there is no guarantee that h returned by Find-S is the only h that fits the data.
3. several maximally specific hypotheses may exist that fits the data but, Find-S
will output only one.
4. Why we should prefer most specific hypotheses over, e.g., most general
hypotheses?
Maja Pantic
Candidate-Elimination Algorithm
Find-S is guaranteed to output the most specific hypothesis h that best fits positive
training examples.
The hypothesis h returned by Find-S will also fit negative examples as long as
training examples are correct.
However,
1. Find-S is sensitive to noise that is (almost always) present in training examples.
2. there is no guarantee that h returned by Find-S is the only h that fits the data.
3. several maximally specific hypotheses may exist that fits the data but, Find-S
will output only one.
4. Why we should prefer most specific hypotheses over, e.g., most general
hypotheses?
To address the last three drawbacks of Find-S, Candidate-Elimination was proposed
Maja Pantic
Main idea: Output a set of hypothesis VS H that fit (are consistent) with data D
Maja Pantic
hair
body
likesSimon
pose
smile
smart
blond
thin
yes
arrogant
toothy
no
brown
thin
no
natural
pleasant
yes
blond
plump
yes
goofy
pleasant
no
black
thin
no
arrogant
none
no
blond
plump
no
natural
toothy
yes
G0 {?, ?, ?, ?, ?, ?} , S0 {0, 0, 0, 0, 0, 0}
Maja Pantic
hair
body
likesSimon
pose
smile
smart
blond
thin
yes
arrogant
toothy
no
brown
thin
no
natural
pleasant
yes
blond
plump
yes
goofy
pleasant
no
black
thin
no
arrogant
none
no
blond
plump
no
natural
toothy
yes
d1 is positive refine S
no g G0 is inconsistent with d1
G1 G0 {?, ?, ?, ?, ?, ?}
Maja Pantic
hair
body
likesSimon
pose
smile
smart
blond
thin
yes
arrogant
toothy
no
brown
thin
no
natural
pleasant
yes
blond
plump
yes
goofy
pleasant
no
black
thin
no
arrogant
none
no
blond
plump
no
natural
toothy
yes
d2 is negative refine G
no s S1 is inconsistent with d2
Maja Pantic
hair
body
likesSimon
pose
smile
smart
blond
thin
yes
arrogant
toothy
no
brown
thin
no
natural
pleasant
yes
blond
plump
yes
goofy
pleasant
no
black
thin
no
arrogant
none
no
blond
plump
no
natural
toothy
yes
d3 is positive refine S
two g G2 are inconsistent with d3, i.e., ?, ?, ?, arrogant, ?, ? and ?, ?, ?, ?, toothy, ?
G3 {blond, ?, ?, ?, ?, ? , ?, ?, yes, ?, ?, ? , ?, ?, ?, ?, ?, no }
add to S all minimal generalizations of s S2 such that s S3 is consistent with d3
S2 {blond, thin, yes, arrogant, toothy, no}
S3 {blond, ?, yes, ?, ?, no}
Maja Pantic
hair
body
likesSimon
pose
smile
smart
blond
thin
yes
arrogant
toothy
no
brown
thin
no
natural
pleasant
yes
blond
plump
yes
goofy
pleasant
no
black
thin
no
arrogant
none
no
blond
plump
no
natural
toothy
yes
d4 is negative refine G
no s S3 is inconsistent with d4
Maja Pantic
hair
body
likesSimon
pose
smile
smart
blond
thin
yes
arrogant
toothy
no
brown
thin
no
natural
pleasant
yes
blond
plump
yes
goofy
pleasant
no
black
thin
no
arrogant
none
no
blond
plump
no
natural
toothy
yes
d5 is negative refine G
no s S4 is inconsistent with d4
"
Maja Pantic
hair
body
likesSimon
pose
smile
smart
blond
thin
yes
arrogant
toothy
no
brown
thin
no
natural
pleasant
yes
blond
plump
yes
goofy
pleasant
no
black
thin
no
arrogant
none
no
blond
plump
no
natural
toothy
yes
Output of C-E:
version space of hypotheses VS H bound with
specific boundary S {blond, ?, yes, ?, ?, no} and
general boundary G {?, ?, yes, ?, ?, ? }
Output of Find-S:
most specific hypothesis h blond, ?, yes, ?, ?, no
Maja Pantic
hair
body
likesSimon
pose
smile
smart
blond
thin
yes
arrogant
toothy
no
brown
thin
no
natural
pleasant
yes
blond
plump
yes
goofy
pleasant
no
black
thin
no
arrogant
none
no
blond
plump
no
natural
toothy
yes
Output of C-E:
version space of hypotheses VS H bound with
specific boundary S {blond, ?, yes, ?, ?, no} and
general boundary G {?, ?, yes, ?, ?, ? }
VS {?, ?, yes, ?, ?, ? , blond, ?, yes, ?, ?, ? ,
?, ?, yes, ?, ?, no , blond, ?, yes, ?, ?, no}
Maja Pantic
Maja Pantic
Maja Pantic
Maja Pantic