Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

Probabilistic methods
for approximating large dimension sparse
matrix multiplication

Mihnea Simian
333CA
simian_mihnea@fotolitera.com
http://twitter.com/mihneasim
http://facebook.com/mihneasim

02.11.21 Protocoale de comunicaţie – Prezentări Laborator 1

 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

Matrix.. All over the place!

• In your math
• In your computer graphics source code
• In all other physical representations
• In digital media – sounds, images, movies, any dimensional
material gone virtual
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

Real life. Damn infinite matrix data

• Computational choice: discrete representation

• The bigger, the better
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

Multiplication. What for?

• Lots. It’s like brushing teeth for matrices
• Easier to do
– Solving equations and equation systems, approximations,
interpolations
– Geometrical transformations
– All sorts of other transformations for large data matrix (like image
filters or physical processes)
– Encryption, decryption, hashing, data mining
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

A * B = O(n^3)
complexity

can make your ambulance

miss your call
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

What do we seek for?

• An approximation P ≈ A * B with provable

bounds for the norm of the error matrix (P –
A*B)

• O(n^2) complexity (or O(mn+mp+np) for

rectangular matrices)
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

How?

• We randomly pick a constant s number of columns out of A

(m x n)
• and a constant s number of corresponding rows out of B (n x p)
• We scale the columns and rows of the resulting S (m x s) and
R (s*p) matrices in order to compensate the lost ones
• Et voila

P=S*R (m x p)
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

Heuristically compute probabilities

p1 p2 p3 p4 p5 p6

A B
Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

Throw the dice. S times

 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

And the winners are..

p1 p2 p3 p4 p5 p6

A B
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

And the winners are..

p2 p4 p5

s  p2
X
s  p4

s  p5
R

s  p2 s  p4 s  p5
S ≈AxB
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

The benefits
• We only iterate matrices twice

– To compute p probabilities
– To copy selected columns/rows

• We only keep S and R in program memory (RAM)

• Time: O ( s*m*p)

• Space: O (s*(q+r)) where q,r – max no. of not null values in any
A column or B row (presuming we use sparse representation)

• Remember! s is constant ( O(1) ). The larger, the more

accurate
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

Error?
• We’ve just thrown away O(n) A-columns and B-rows

2
1 (k )  1 2n
E ( S  R  A  B F )    A( k ) B   A F B
2 2
F
s  k 1  s
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

Let’s take another look.. And another one

2
1 n
 1
  A( k ) B ( k )  
2 2 2
E( S  R  A  B F
) A F
B F
s  k 1  s

• E(X) – estimation (average) of random variable X

• A(k ) norm of column k

• B (k ) norm of row k

A F  i , j Aij2
2
• Frobenius norm of a matrix

• A 2  max Ax , x   n s.t. x  1, 2-norm (Euclid norm) of a matrix

 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

A better approach
p1 p2 p3 p4 p5 p6

A B
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

Sounds smart, but..

• What about p? How do we choose the probabilities?
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

Biography
• http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.57.7716 - Fast
Monte-Carlo Algorithms for Approximate Matrix Multiplication
• Images
– http://xkcd.net
– http://icanhascheezburger.com
– http://dreamstime.com/free-photos

Further reading

• matrix multiplication using singular values and singular vectors

T
• A A applications
• matrix multiplication using a sub-dimension projection
• fast matrix multiplication algorithms for multiplying dense matrices
 Universitatea Politehnica Bucureşti - Facultatea de Automatica si Calculatoare

Awesome audience, Thank You!