Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit
Shared-Memory Computability?
Shared Memory
10011
Mathematical model of concurrent computation What is (and is not) concurrently computable Efficiency (mostly) irrelevant
Art of Multiprocessor 2 Programming
Wait-Free Implementation
Every method call completes in finite number of steps Implies no mutual exclusion
Single reader
0 1
Safe Boolean register
Art of Multiprocessor Programming 4
M-valued Boolean
Asynchronous Interrupts
??? ???
10
Heterogeneous Processors
??? ???
yawn
supercomputer
supercomputer
Art of Multiprocessor Programming
toaster
11
Fault-tolerance
??? ???
12
13
Basic Questions
Wait-Free synchronization might be a good idea in principle
14
Basic Questions
Wait-Free synchronization might be a good idea in principle But how do you do it
15
Basic Questions
Wait-Free synchronization might be a good idea in principle But how do you do it
Systematically?
16
Basic Questions
Wait-Free synchronization might be a good idea in principle But how do you do it
Systematically? Correctly?
17
Basic Questions
Wait-Free synchronization might be a good idea in principle But how do you do it
Systematically? Correctly? Efficiently?
18
q.enq( )
19
q.deq()/
20
21
22
Grand Challenge
23
Grand Challenge
24
Grand Challenge
25
Grand Challenge
Grand Challenge
Grand Challenge
28
Puzzle
While you are ruminating on the grand challenge We will give you another puzzle Consensus!
29
32
19
21
30
They Communicate
31
19
19
19
32
Formally: Consensus
Consistent:
all threads decide the same value
33
Formally: Consensus
Consistent:
all threads decide the same value
Valid:
the common decision value is some thread's input
34
35
Formally
Theorem
There is no wait-free implementation of n-thread consensus from read-write registers
36
Formally
Theorem
There is no wait-free implementation of n-thread consensus from read-write registers
Implication
Asynchronous computability different from Turing computability
37
Proof Strategy
Assume otherwise Reason about the properties of any such protocol Derive a contradiction Quod Erat Demonstrandum Enough to consider binary consensus and n=2
Art of Multiprocessor Programming 38
init state
time time
Art of Multiprocessor Programming 39
Wait-Free Computation
A moves B moves
Initial state
41
Decision Values
1
42
1
43
1
44
1
45
Summary
Wait-free computation is a tree
46
Summary
Wait-free computation is a tree Bivalent system states
Outcome not fixed
47
Summary
Wait-free computation is a tree Bivalent system states
Outcome not fixed
Univalent states
Outcome is fixed May not be known yet
48
Summary
Wait-free computation is a tree Bivalent system states
Outcome not fixed
Univalent states
Outcome is fixed May not be known yet
Claim
Some initial state is bivalent
50
Claim
Some initial state is bivalent Outcome depends on
Chance Whim of the scheduler
51
Claim
Some initial state is bivalent Outcome depends on
Chance Whim of the scheduler
52
Claim
Some initial state is bivalent Outcome depends on
Chance Whim of the scheduler
54
Must Decide 0
Must Decide 1
Critical States
critical
0-valent
Art of Multiprocessor Programming
1-valent
58
1-valent
If B goes first, protocol decides 1
59
Critical States
Claim: starting from a bivalent initial state the protocol will reach a critical state
Lets prove it
60
CB
CA
univalent univalent
CB CA
univalent
univalent 0-valent
c
1-valent
61
Critical States
Claim: Starting from a bivalent initial state the protocol will reach a critical state
Proof:
Otherwise we could stay bivalent forever And the protocol would not be wait-free
62
Model Dependency
So far, memory-independent! True for
Registers Message-passing Carrier pigeons Any kind of asynchronous computation
63
Leading to a 0 or 1 decision
Possible Interactions
x.read() x.read() y.read() x.write() y.write()
y.read()
?
? ?
?
? ?
Art of Multiprocessor Programming
?
? ?
?
? ?
68
x.write()
y.write()
Possible Interactions
A reads x
x.read() x.read() y.read() x.write() y.write()
y.read()
?
? ?
?
? ?
Art of Multiprocessor Programming
?
? ?
?
? ?
69
x.write()
y.write()
Possible Interactions
A reads x A reads y
x.read() x.read() y.read() x.write() y.write()
y.read()
?
? ?
?
? ?
Art of Multiprocessor Programming
?
? ?
?
? ?
70
x.write()
y.write()
71
72
B reads x
73
B reads x
0
1
A runs solo, eventually decides 1
74
B reads x
0
1
States look the same to A
Art of Multiprocessor Programming 75
B reads x
0
1
States look the same to A
Art of Multiprocessor Programming 76
Possible Interactions
x.read() x.read() y.read() x.write() y.write()
no
no
no
no
y.read()
no
no no
no
no no
Art of Multiprocessor Programming
no
? ?
no
? ?
77
x.write()
y.write()
78
79
B writes x
80
B writes x
81
B writes x
B writes x
82
B writes x
B writes x
A writes y
83
B writes x
B writes x
A writes y
84
B writes x
B writes x
A writes y
Same story
85
B writes x
B writes x
A writes y
Same story
86
Possible Interactions
x.read() x.read() y.read() x.write() y.write()
no
no
no
no
y.read()
no
no no
no
no no
Art of Multiprocessor Programming
no
? no
no
no ?
87
x.write()
y.write()
A writes x
0
States look the same to A
1
Art of Multiprocessor Programming
no
no
no
no
y.read()
no
no no
no
no no
Art of Multiprocessor Programming
no
no no
no
no no
89
x.write()
y.write()
90
91
Consensus Object
public interface Consensus<T> { T decide(T value); }
92
93
94
95
96
97
98
99
FIFO Consensus
101
0
8
102
103
104
Initialize Queue
public class QueueConsensus extends ConsensusProtocol { private Queue queue; public QueueConsensus() { this.queue = new Queue(); this.queue.enq(Ball.RED); this.queue.enq(Ball.BLACK); } }
8
105
Who Won?
public class QueueConsensus<T> extends ConsensusProtocol<T> { private Queue queue; public T decide(T value) { propose(value); Ball ball = queue.deq(); if (ball == Ball.RED) return proposed[i]; else return proposed[1-i]; } }
Art of Multiprocessor Programming 106
Who Won?
public class QueueConsensus<T> extends ConsensusProtocol<T> { private Queue queue; public T decide(T value) { propose(value); Ball ball = queue.deq(); if (ball == Ball.RED) return proposed[i]; else return proposed[1-ij]; } Race to dequeue } first queue item
Art of Multiprocessor Programming 107
Who Won?
public class QueueConsensus<T> extends ConsensusProtocol<T> { private Queue queue; public T decide(T value) { propose(value); Ball ball = this.queue.deq(); if (ball == Ball.RED) return proposed[i]; else return proposed[1-i]; I win if I was first } }
Art of Multiprocessor Programming 108
Who Won?
public class QueueConsensus<T> extends ConsensusProtocol<T> { private Queue queue; Other thread wins if I public T decide(T value) was { second propose(value); Ball ball = this.queue.deq(); if (ball == Ball.RED) return proposed[i]; else return proposed[1-i]; } }
Art of Multiprocessor Programming 109
110
Theorem
We can solve 2-thread consensus using only
A two-dequeuer queue, and Some atomic registers
111
Implications
Given
A consensus protocol from queue and registers
Substitution yields:
A wait-free consensus protocol from atomic registers
112
Corollary
It is impossible to implement
a two-dequeuer wait-free FIFO queue from read/write memory.
113
Consensus Numbers
An object X has consensus number n
If it can be used to solve n-thread consensus
Take any number of instances of X together with atomic read/write registers and implement n-thread consensus
114
Consensus Numbers
Theorem
Atomic read/write registers have consensus number 1
Theorem
Multi-dequeuer FIFO queues have consensus number at least 2
115
116
What about
Write multiple array elements atomically Scan any array elements
119
Proof Strategy
If we can write to 2/3 array elements
We can solve 2-consensus Impossible with atomic registers
Therefore
Cannot implement multiple assignment with atomic registers
120 (1)
Proof Strategy
Take a 3-element array
A writes atomically to slots 0 and 1 B writes atomically to slots 1 and 2 Any thread can scan any set of locations
121
Initially
A
Writes to 0 and 1
Writes to 1 and 2
Art of Multiprocessor Programming 125
Thread A wins if
A
B
Art of Multiprocessor Programming
Thread A wins if
A
B
Art of Multiprocessor Programming
Thread A loses if
A
B
Art of Multiprocessor Programming
Multi-Consensus Code
class MultiConsensus extends { Assign2 a = new Assign2(3, EMPTY); public T decide(T value) { a.assign(i, i, i+1, i); int other = a.read((i+2) % 3); if (other==EMPTY||other==a.read(1)) return proposed[i]; else return proposed[j]; }}
129
Multi-Consensus Code
class MultiConsensus extends { Assign2 a = new Assign2(3, EMPTY); public T decide(T value) { a.assign(i, i, i+1, i); int other = a.read((i+2) % 3); if (other==EMPTY||other==a.read(1)) return proposed[i]; else return proposed[j]; }}
Not seen: Extends ConsensusProtocol Decide first sets j=i-1 and proposes value
Art of Multiprocessor Programming 130
Multi-Consensus Code
class MultiConsensus extends { Assign2 a = new Assign2(3, EMPTY); public T decide(T value) { a.assign(i, i, i+1, i); int other = a.read((i+2) % 3); if (other==EMPTY||other==a.read(1)) return proposed[i]; else Three slots return proposed[j]; initialized to }} EMPTY
131
Multi-Consensus Code
class MultiConsensus extends { Assign2 a = new Assign2(3, EMPTY); public T decide(T value) { a.assign(i, i, i+1, i); int other = a.read((i+2) % 3); if (other==EMPTY||other==a.read(1)) return proposed[i]; else Assign ID 0 to entries 0,1 return proposed[j]; (or ID 1 to entries 1,2) }}
132
Multi-Consensus Code
class MultiConsensus extends { Assign2 a = new Assign2(3, EMPTY); public T decide(T value) { a.assign(i, i, i+1, i); int other = a.read((i+2) % 3); if (other==EMPTY||other==a.read(1)) return proposed[i]; else return proposed[j]; Read the register my }} thread didnt assign
133
Multi-Consensus Code
class MultiConsensus extends { Assign2 a = new Assign2(3, EMPTY); public T decide(T value) { a.assign(i, i, i+1, i); int other = a.read((i+2) % 3); if (other==EMPTY||other==a.read(1)) return proposed[i]; else Other thread didnt return proposed[j]; move, so I win }}
134
Multi-Consensus Code
class MultiConsensus extends { Assign2 a = new Assign2(3, EMPTY); public T decide(T value) { a.assign(i, i, i+1, i); int other = a.read((i+2) % 3); if (other==EMPTY||other==a.read(1)) return proposed[i]; else Other thread moved return proposed[j]; later so I win }}
135
Multi-Consensus Code
class MultiConsensus extends { Assign2 a = new Assign2(3, EMPTY); public T decide(T value) { a.assign(i, i, i+1, i); int other = a.read((i+2) % 3); if (other==EMPTY||other==a.read(1)) return proposed[i]; else return proposed[j]; }} OK, I win.
136
Multi-Consensus Code
class MultiConsensus extends { Assign2 a = new Assign2(3, EMPTY); public T decide(T value) { a.assign(i, i, i+1, i); int other = a.read((i+2) % 3); if (other==EMPTY||other==a.read(1)) return proposed[i];(1) else return proposed[j]; Other thread moved }} first, so I lose
137
Summary
If a thread can assign atomically to 2 out of 3 array locations Then we can solve 2-consensus Therefore
No wait-free multi-assignment From read/write registers
138
Read-Modify-Write Objects
Method call
Returns objects prior value x Replaces x with mumble(x)
139
Read-Modify-Write
public abstract class RMWRegister { private int value; public int synchronized getAndMumble() { int prior = value; value = mumble(value); return prior; } }
140
Read-Modify-Write
public abstract class RMWRegister { private int value; public int synchronized getAndMumble() { int prior = value; value = mumble(value); return prior; } }
141
Read-Modify-Write
public abstract class RMWRegister { private int value; public int synchronized getAndMumble() { int prior = value; value = mumble(value); return prior; } }
142
RMW Everywhere!
Most synchronization instructions
are RMW methods
The rest
Can be trivially transformed into RMW methods
143
Example: Read
public abstract class RMWRegister { private int value; public int synchronized read() { int prior = value; value = value; return prior; } }
144
Example: Read
public abstract class RMW { private int value; public int synchronized read() { int prior = this.value; value = value; return prior; } }
145
Example: getAndSet
public abstract class RMWRegister { private int value; public int synchronized getAndSet(int v) { int prior = value; value = v; return prior; } }
Art of Multiprocessor Programming 146
f(x)=v is constant
Art of Multiprocessor Programming 147
getAndIncrement
public abstract class RMWRegister { private int value; public int synchronized getAndIncrement() { int prior = value; value = value + 1; return prior; } }
Art of Multiprocessor Programming 148
getAndIncrement
public abstract class RMWRegister { private int value; public int synchronized getAndIncrement() { int prior = value; value = value + 1; return prior; } }
f(x) = x+1
149
getAndAdd
public abstract class RMWRegister { private int value; public int synchronized getAndAdd(int a) { int prior = value; value = value + a; return prior; } }
Art of Multiprocessor Programming 150
Example: getAndAdd
public abstract class RMWRegister { private int value; public int synchronized getAndIncrement(int a) { int prior = value; value = value + a; return prior; } f(x) = x+a
Art of Multiprocessor Programming 151
compareAndSet
public abstract class CASObject { private int value; public boolean synchronized compareAndSet(int expected, int update) { if (value==expected) { value = update; return true; } return false; } }
152
compareAndSet
public abstract class CASObject { private int value; public boolean synchronized compareAndSet(int expected, int update) { if (value==expected) { value = update; return true; } return false; } }
If value is as expected,
Art of Multiprocessor Programming 153
compareAndSet
public abstract class CASOBJECT{ private int value; public boolean synchronized compareAndSet(int expected, int update) { if (value==expected) { value = update; return true; } return false; } }
replace it
154
compareAndSet
public abstract class RMWRegister { private int value; public boolean synchronized compareAndSet(int expected, int update) { if (value==expected) { value = update; return true; } return false; } }
Report success
Art of Multiprocessor Programming 155
compareAndSet
public abstract class RMWRegister { private int value; public boolean synchronized compareAndSet(int expected, int update) { if (value==expected) { value = update; return true; } return false; Otherwise report failure } }
156
Read-Modify-Write
public abstract class RMWRegister { private int value; public void synchronized getAndMumble() { int prior = value; value = mumble(value); return prior; } }
Definition
A RMW method
With function mumble(x) is non-trivial if there exists a value v Such that v mumble(v)
158
Par Example
Identity(x) = x
is trivial
getAndIncrement(x) = x+1
is non-trivial
159
Theorem
Any non-trivial RMW object has consensus number at least 2 No wait-free implementation of RMW registers from atomic registers Hardware RMW instructions not just a convenience
160
Reminder
Subclasses of consensus have
propose(x) method
which just stores x into proposed[i] built-in method
161
Proof
public class RMWConsensus extends ConsensusProtocol { private RMWRegister r = v; public T decide(T value) { propose(value); if (r.getAndMumble() == v) return proposed[i]; else return proposed[j]; }}
162
Proof
public class RMWConsensus extends ConsensusProtocol { private RMWRegister r = v; public T decide(T value) { propose(value); if (r.getAndMumble() == v) return proposed[i]; else Initialized to v return proposed[j]; }}
163
Proof
public class RMWConsensus extends Consensus { Am I first? private RMWRegister r = v; public T decide(T value) { propose(value); if (r.getAndMumble() == v) return proposed[i]; else return proposed[j]; }}
164
Proof
public class RMWConsensus extends ConsensusProtocol { private RMWRegister r = v; public T decide(T value) { Yes, return propose(value); if (r.getAndMumble() == v) my input return proposed[i]; else return proposed[j]; }}
165
Proof
public class RMWConsensus extends ConsensusProtocol { private RMWRegister r = v; public T decide(T value) { propose(value); if (r.getAndMumble() == v) No, return return proposed[i]; else others input return proposed[j]; }}
166
Proof
We have displayed
A two-thread consensus protocol Using any non-trivial RMW object
167
Interfering RMW
Let F be a set of functions such that for all fi and fj, either
Commute: fi(fj(v))=fj(fi(v)) Overwrite: fi(fj(v))=fi(v)
Claim: Any set of RMW objects that commutes or overwrites has consensus number exactly 2
Art of Multiprocessor Programming 168
Examples
test-and-set getAndSet(1) f(v)=1
Overwrite fi(fj(v))=fi(v)
B about to apply fB
0-valent
1-valent
170
B applies fB
B applies fB
C runs solo
0-valent
0
Art of Multiprocessor Programming
1
1-valent
171
B applies fB
C runs solo
0-valent
0
Art of Multiprocessor Programming
1
1-valent
172
B applies fB
C runs solo
A applies fA
0
C runs solo
0-valent
Art of Multiprocessor Programming
1
1-valent
173
B applies fB
C runs solo
A applies fA
0
C runs solo
0-valent
Art of Multiprocessor Programming
1
1-valent
174
Impact
Many early machines provided only these weak RMW instructions
Test-and-set (IBM 360) Fetch-and-add (NYU Ultracomputer) Swap (Original SPARCs)
compareAndSet
public abstract class RMWRegister { private int value; public boolean synchronized compareAndSet(int expected, int update) { int prior = value; if (value==expected) { value = update; return true; } return false; } }
Art of Multiprocessor Programming 176
compareAndSet
public abstract class RMWRegister { private int value; public boolean synchronized compareAndSet(int expected, int update) { int prior = this.value; if (value==expected) { this.value = update; return true; } return false; replace value if its what we } }
expected,
177
178
179
180
181
compareAndSet,
Art of Multiprocessor Programming 182
Multiple Assignment
Atomic k-assignment Solves consensus for 2k-2 threads Every even consensus number has an object (can be extended to odd numbers)
183
Lock-Freedom
Lock-free:
in an infinite execution infinitely often some method call finishes
184
185
Lock-Freedom
Any wait-free implementation is lock-free. Lock-free is the same as waitfree if the execution is finite.
186
Lock-Free Implementations
Lock-free consensus is as impossible as wait-free consensus All these results hold for lock-free algorithms also.
187
For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to
http://creativecommons.org/licenses/by-sa/3.0/.
Any of the above conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author's moral rights.
189