Chapter 4.2 - Matrix Multiplication
Matrix multiplication is essentially just doing many Dot Products all at once.
Let's visualize how a 2 × 2 2 \times 2 2 × 2 2 × 2 2 \times 2 2 × 2
To find the top-left number of the answer, we take the Top Row of the first matrix and calculate the dot product with the Left Column of the second matrix.
[ A B C D ] × [ E F G H ] = [ ( A × E ) + ( B × G ) ( A × F ) + ( B × H ) ( C × E ) + ( D × G ) ( C × F ) + ( D × H ) ] \begin{bmatrix}
\color{red}{A} & \color{red}{B} \\
\color{skyblue}{C} & \color{skyblue}{D}
\end{bmatrix}
\times
\begin{bmatrix}
\color{green}{E} & \color{orange}{F} \\
\color{green}{G} & \color{orange}{H}
\end{bmatrix}
=
\begin{bmatrix}
(\color{red}{A} \times \color{green}{E}) + (\color{red}{B} \times \color{green}{G}) & (\color{red}{A} \times \color{orange}{F}) + (\color{red}{B} \times \color{orange}{H}) \\
(\color{skyblue}{C} \times \color{green}{E}) + (\color{skyblue}{D} \times \color{green}{G}) & (\color{skyblue}{C} \times \color{orange}{F}) + (\color{skyblue}{D} \times \color{orange}{H})
\end{bmatrix} [ A C B D ] × [ E G F H ] = [ ( A × E ) + ( B × G ) ( C × E ) + ( D × G ) ( A × F ) + ( B × H ) ( C × F ) + ( D × H ) ] Let's use some real numbers to see this in action. Suppose we have two 2 × 2 2 \times 2 2 × 2 X X X Y Y Y
X = [ 1 2 3 4 ] Y = [ 5 6 7 8 ] X =
\begin{bmatrix}
\color{red}{1} & \color{red}{2} \\
\color{skyblue}{3} & \color{skyblue}{4}
\end{bmatrix}
\quad \quad
Y =
\begin{bmatrix}
\color{green}{5} & \color{orange}{6} \\
\color{green}{7} & \color{orange}{8}
\end{bmatrix} X = [ 1 3 2 4 ] Y = [ 5 7 6 8 ] When we multiply X × Y X \times Y X × Y
X × Y = [ ( 1 × 5 ) + ( 2 × 7 ) ( 1 × 6 ) + ( 2 × 8 ) ( 3 × 5 ) + ( 4 × 7 ) ( 3 × 6 ) + ( 4 × 8 ) ] X \times Y =
\begin{bmatrix}
(\color{red}{1} \times \color{green}{5}) + (\color{red}{2} \times \color{green}{7}) & (\color{red}{1} \times \color{orange}{6}) + (\color{red}{2} \times \color{orange}{8}) \\
(\color{skyblue}{3} \times \color{green}{5}) + (\color{skyblue}{4} \times \color{green}{7}) & (\color{skyblue}{3} \times \color{orange}{6}) + (\color{skyblue}{4} \times \color{orange}{8})
\end{bmatrix} X × Y = [ ( 1 × 5 ) + ( 2 × 7 ) ( 3 × 5 ) + ( 4 × 7 ) ( 1 × 6 ) + ( 2 × 8 ) ( 3 × 6 ) + ( 4 × 8 ) ] X × Y = [ 5 + 14 6 + 16 15 + 28 18 + 32 ] = [ 19 22 43 50 ] X \times Y =
\begin{bmatrix}
\color{red}{5} + \color{red}{14} & \color{red}{6} + \color{red}{16} \\
\color{skyblue}{15} + \color{skyblue}{28} & \color{skyblue}{18} + \color{skyblue}{32}
\end{bmatrix}
=
\begin{bmatrix}
19 & 22 \\
43 & 50
\end{bmatrix} X × Y = [ 5 + 14 15 + 28 6 + 16 18 + 32 ] = [ 19 43 22 50 ] In Python using PyTorch, we can perform this exact same mathematical operation instantly using the @ symbol:
import torch X = torch.tensor([ [ 1 , 2 ], [ 3 , 4 ] ]) Y = torch.tensor([ [ 5 , 6 ], [ 7 , 8 ] ]) # Perform matrix multiplication using the @ operator result = X @ Y print (result) Output:
tensor([[19, 22], [43, 50]]) The Ultimate Cheat Code
Matrix multiplication (@) is the engine of all Neural Networks. Whenever you see @ in PyTorch, just remember it means: "Take each row of the matrix on the left, and compute its dot product with each column of the matrix on the right."
(AB)11 = (2 · 5 ) + (1 · 1 ) + (3 · 3 ) = 20.000