Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
• Matrix-Vector Multiplication
• Matrix-Matrix Multiplication
W = n2 . (1)
Matrix-Vector Multiplication: Rowwise 1-D Partitioning
P0 0 P0 0
P1 1 P1 1
. n/p . n
. .
Pp-1 p-1 Pp-1 p-1
(a) Initial partitioning of the matrix (b) Distribution of the full vector among all
and the starting vector x the processes by all-to-all broadcast
Matrix A Vector y
P0 0 1 p-1 P0 0
P1 0 1 p-1 P1 1
. 0 1 p-1 .
. 0 1 p-1 .
Pp-1 0 1 p-1 Pp-1 p-1
(c) Entire vector distributed to each (d) Final distribution of the matrix
process after the broadcast and the result vector y
• Each process initially stores n/p complete rows of the matrix and
a portion of the vector of size n/p.
n2
TP = + ts log p + tw n. (2)
p
This is cost-optimal.
Matrix-Vector Multiplication: Rowwise 1-D Partitioning
Scalability Analysis:
P0 P1 . . P√p-1
P√ p √n
p
P2 √ p . n
.
Pp-1
(a) Initial data distribution and communication (b) One-to-all broadcast of portions of
steps to align the vector along the diagonal the vector along process columns
Matrix A Vector y
P√ p
P2 √ p .
.
Pp-1
(c) All-to-one reduction of partial results (d) Final distribution of the result vector
• The first communication step for the 2-D partitioning aligns the
vector x along the principal diagonal of the matrix.
√
• The first alignment step takes time ts + tw n/ p.
√ √
• The broadcast and reductions take time (ts + tw n/ p) log( p).
• Total time is
n2 n
TP ≈ + ts log p + tw √ log p (4)
p p
Matrix-Vector Multiplication: 2-D Partitioning
Scalability Analysis:
√
• To = pTp − W = tsp log p + tw n p log p.
• Process Pi,j initially stores Ai,j and Bi,j and computes block Ci,j
of the result matrix.
√ √
• The two broadcasts take time 2(ts log( p) + tw (n2/p)( p − 1)).
√ √ √
• The computation requires p multiplications of (n/ p) × (n/ p)
sized submatrices.
n3 n2
TP = + ts log p + 2tw √ . (5)
p p
√
• In this algorithm, we schedule the computations of the p
processes of the ith row such that, at any given time, each
process is using a different block Ai,k .
(c) A and B after initial alignment (d) Submatrix locations after first shift
(e) Submatrix locations after second shift (f) Submatrix locations after third shift
n3 √ n2
TP = + 2 pts + 2tw √ . (6)
p p
k k=1
0,1 1,1 2,1 3,1
i
(a) Initial distribution of A and B (b) After moving A[i,j] from Pi,j,0 to Pi,j,j
• The two one-to-all broadcasts take 2(ts log q+tw (n/q)2 log q) time
for each matrix.
n3 n2
TP = + ts log p + tw 2/3 log p. (7)
p p
Column k
Column j
Inactive part
Active part
• The first step of the algorithm normalizes the row. This is a serial
operation and takes time (n − k) in the kth iteration.
3 1
TP = n(n − 1) + tsn log n + tw n(n − 1) log n. (8)
2 2
(1,0) (1,1) (1,2) (1,3) (1,4) (1,0) (1,1) (1,2) (1,3) (1,4) (1,0) (1,1) (1,2) (1,3) (1,4) (1,0) (1,1) (1,2) (1,3) (1,4)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,0) (2,1) (2,2) (2,3) (2,4) (2,0) (2,1) (2,2) (2,3) (2,4) (2,0) (2,1) (2,2) (2,3) (2,4)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,0) (3,1) (3,2) (3,3) (3,4) (3,0) (3,1) (3,2) (3,3) (3,4) (3,0) (3,1) (3,2) (3,3) (3,4)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,0) (4,1) (4,2) (4,3) (4,4) (4,0) (4,1) (4,2) (4,3) (4,4) (4,0) (4,1) (4,2) (4,3) (4,4)
1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4)
0 (1,1) (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4) 0 (1,1) (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4)
(2,0) (2,1) (2,2) (2,3) (2,4) 0 (2,1) (2,2) (2,3) (2,4) 0 (2,1) (2,2) (2,3) (2,4) 0 (2,1) (2,2) (2,3) (2,4)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,0) (3,1) (3,2) (3,3) (3,4) 0 (3,1) (3,2) (3,3) (3,4) 0 (3,1) (3,2) (3,3) (3,4)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,0) (4,1) (4,2) (4,3) (4,4) (4,0) (4,1) (4,2) (4,3) (4,4) 0 (4,1) (4,2) (4,3) (4,4)
1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4)
0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4)
0 (3,1) (3,2) (3,3) (3,4) 0 0 (3,2) (3,3) (3,4) 0 0 (3,2) (3,3) (3,4) 0 0 (3,2) (3,3) (3,4)
0 (4,1) (4,2) (4,3) (4,4) 0 (4,1) (4,2) (4,3) (4,4) 0 0 (4,2) (4,3) (4,4) 0 0 (4,2) (4,3) (4,4)
1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4)
0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4)
0 0 1 (2,3) (2,4) (2,5) (2,6) (2,7) 0 1 (1,2) (1,3) (1,4) (1,5) (1,6) (1,7)
P1 P1
0 0 0 (3,3) (3,4) (3,5) (3,6) (3,7) 0 0 0 (5,3) (5,4) (5,5) (5,6) (5,7)
0 0 0 (4,3) (4,4) (4,5) (4,6) (4,7) 0 0 1 (2,3) (2,4) (2,5) (2,6) (2,7)
P2 P2
0 0 0 (5,3) (5,4) (5,5) (5,6) (5,7) 0 0 0 (6,3) (6,4) (6,5) (6,6) (6,7)
0 0 0 (6,3) (6,4) (6,5) (6,6) (6,7) 0 0 0 (3,3) (3,4) (3,5) (3,6) (3,7)
P3 P3
0 0 0 (7,3) (7,4) (7,5) (7,6) (7,7) 0 0 0 (7,3) (7,4) (7,5) (7,6) (7,7)
0 1 (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) 0 1 (1,2) (1,3) (1,4) (1,5) (1,6) (1,7)
0 0 1 (2,3) (2,4) (2,5) (2,6) (2,7) 0 0 1 (2,3) (2,4) (2,5) (2,6) (2,7)
0 0 0 (3,3) (3,4) (3,5) (3,6) (3,7) 0 0 0 (3,3) (3,4) (3,5) (3,6) (3,7)
0 0 0 (4,3) (4,4) (4,5) (4,6) (4,7) 0 0 0 (4,3) (4,4) (4,5) (4,6) (4,7)
0 0 0 (5,3) (5,4) (5,5) (5,6) (5,7) 0 0 0 (5,3) (5,4) (5,5) (5,6) (5,7)
0 0 0 (6,3) (6,4) (6,5) (6,6) (6,7) 0 0 0 (6,3) (6,4) (6,5) (6,6) (6,7)
0 0 0 (7,3) (7,4) (7,5) (7,6) (7,7) 0 0 0 (7,3) (7,4) (7,5) (7,6) (7,7)
1 (0,1) (0,2) (0,3) (0,4) (0,5) (0,6) (0,7) 1 (0,1) (0,2) (0,3) (0,4) (0,5) (0,6) (0,7)
0 1 (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) 0 1 (1,2) (1,3) (1,4) (1,5) (1,6) (1,7)
0 0 1 (2,3) (2,4) (2,5) (2,6) (2,7) 0 0 1 (2,3) (2,4) (2,5) (2,6) (2,7)
0 0 0 (4,3) (4,4) (4,5) (4,6) (4,7) 0 0 0 (4,3) (4,4) (4,5) (4,6) (4,7)
0 0 0 (5,3) (5,4) (5,5) (5,6) (5,7) 0 0 0 (5,3) (5,4) (5,5) (5,6) (5,7)
0 0 0 (6,3) (6,4) (6,5) (6,6) (6,7) 0 0 0 (6,3) (6,4) (6,5) (6,6) (6,7)
0 0 0 (7,3) (7,4) (7,5) (7,6) (7,7) 0 0 0 (7,3) (7,4) (7,5) (7,6) (7,7)
(1,0) (1,1) (1,2) (1,3) (1,4) (1,0) (1,1) (1,2) (1,3) (1,4) (1,0) (1,1) (1,2) (1,3) (1,4) 0 (1,1) (1,2) (1,3) (1,4)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,0) (2,1) (2,2) (2,3) (2,4) (2,0) (2,1) (2,2) (2,3) (2,4) (2,0) (2,1) (2,2) (2,3) (2,4)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,0) (3,1) (3,2) (3,3) (3,4) (3,0) (3,1) (3,2) (3,3) (3,4) (3,0) (3,1) (3,2) (3,3) (3,4)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,0) (4,1) (4,2) (4,3) (4,4) (4,0) (4,1) (4,2) (4,3) (4,4) (4,0) (4,1) (4,2) (4,3) (4,4)
1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4)
0 (1,1) (1,2) (1,3) (1,4) 0 (1,1) (1,2) (1,3) (1,4) 0 (1,1) (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4)
(2,0) (2,1) (2,2) (2,3) (2,4) 0 (2,1) (2,2) (2,3) (2,4) 0 (2,1) (2,2) (2,3) (2,4) 0 (2,1) (2,2) (2,3) (2,4)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,0) (3,1) (3,2) (3,3) (3,4) (3,0) (3,1) (3,2) (3,3) (3,4) 0 (3,1) (3,2) (3,3) (3,4)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,0) (4,1) (4,2) (4,3) (4,4) (4,0) (4,1) (4,2) (4,3) (4,4) (4,0) (4,1) (4,2) (4,3) (4,4)
1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4)
0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4)
0 (2,1) (2,2) (2,3) (2,4) 0 0 (2,2) (2,3) (2,4) 0 0 (2,2) (2,3) (2,4) 0 0 (2,2) (2,3) (2,4)
0 (3,1) (3,2) (3,3) (3,4) 0 (3,1) (3,2) (3,3) (3,4) 0 (3,1) (3,2) (3,3) (3,4) 0 0 (3,2) (3,3) (3,4)
(4,0) (4,1) (4,2) (4,3) (4,4) 0 (4,1) (4,2) (4,3) (4,4) 0 (4,1) (4,2) (4,3) (4,4) 0 (4,1) (4,2) (4,3) (4,4)
1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4) 1 (0,1) (0,2) (0,3) (0,4)
0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4) 0 1 (1,2) (1,3) (1,4)
0 0 (3,2) (3,3) (3,4) 0 0 (3,2) (3,3) (3,4) 0 0 (3,2) (3,3) (3,4) 0 0 0 (3,3) (3,4)
0 (4,1) (4,2) (4,3) (4,4) 0 0 (4,2) (4,3) (4,4) 0 0 (4,2) (4,3) (4,4) 0 0 (4,2) (4,3) (4,4)
0 1 (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) 0 1 (1,2) (1,3) (1,4) (1,5) (1,6) (1,7)
0 0 1 (2,3) (2,4) (2,5) (2,6) (2,7) 0 0 1 (2,3) (2,4) (2,5) (2,6) (2,7)
√n
0 0
PSfrag replacementsn
0 (4,3) (4,4) (4,5) (4,6) (4,7) 0 0 0 (4,3) (4,4) (4,5) (4,6) (4,7)
p 0 0 0 (5,3) (5,4) (5,5) (5,6) (5,7) 0 0 0 (5,3) (5,4) (5,5) (5,6) (5,7)
0 0 0 (6,3) (6,4) (6,5) (6,6) (6,7) 0 0 0 (6,3) (6,4) (6,5) (6,6) (6,7)
0 0 0 (7,3) (7,4) (7,5) (7,6) (7,7) 0 0 0 (7,3) (7,4) (7,5) (7,6) (7,7)
0 1 (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) 0 (4,4) 0 (4,5) 0 (4,6) (4,3) (4,7)
0 0 1 (2,3) (2,4) (2,5) (2,6) (2,7) 0 (1,4) 1 (1,5) (1,2) (1,6) (1,3) (1,7)
0 0 0 (3,3) (3,4) (3,5) (3,6) (3,7) 0 (5,4) 0 (5,5) 0 (5,6) (5,3) (5,7)
0 0 0 (4,3) (4,4) (4,5) (4,6) (4,7) 0 (2,4) 0 (2,5) 1 (2,6) (2,3) (2,7)
0 0 0 (5,3) (5,4) (5,5) (5,6) (5,7) 0 (6,4) 0 (6,5) 0 (6,6) (6,3) (6,7)
0 0 0 (6,3) (6,4) (6,5) (6,6) (6,7) 0 (3,4) 0 (3,5) 0 (3,6) (3,3) (3,7)
0 0 0 (7,3) (7,4) (7,5) (7,6) (7,7) 0 (7,4) 0 (7,5) 0 (7,6) (7,3) (7,7)
√
• If the matrix is partitioned by using 2-D partitioning on a p ×
√
p logical mesh of processes, and the elements of the vector
are distributed along one of the columns of the process mesh,
√
then only the p processes containing the vector perform any
computation.
• While this is not cost optimal, since this does not dominante the
overall computation, the cost optimality is determined by the
factorization.