Matrix multiplication with shapes: map a batch of vectors

Matrix multiplication composes weighted sums. If X has shape (n, d) and W has shape (d, k), X @ W has shape (n, k). The shared dimension d is summed over. The remaining axes identify observations and output coordinates.

We will map two three-feature records into two output coordinates. The numbers are invented so you can check every term by hand. This is the same shape pattern used by a dense model layer, without a bias or nonlinear activation.

Read the rows and columns before calculating

Take the first record [1, 2, 3]. The first column of W is [1, 0, 2], so the first output is 1×1 + 2×0 + 3×2 = 7. The second column is [0, 1, -1], giving 1×0 + 2×1 + 3×(-1) = -1. Thus the first output row is [7, -1].

Geometrically, each output coordinate is a dot product with one column of W. Another equivalent interpretation combines the rows of W, using the input features as weights. These are two views of the same linear map, not different algorithms.

Verify the calculation and a failing orientation

import numpy as np
X = np.array([[1, 2, 3], [4, 5, 6]])
W = np.array([[1, 0], [0, 1], [2, -1]])
Y = X @ W
print('shapes:', X.shape, W.shape, '->', Y.shape)
print('mapped rows:', Y.tolist())
manual = np.array([[sum(X[i, j] * W[j, k] for j in range(3))
                    for k in range(2)] for i in range(2)])
assert np.array_equal(Y, manual)
try:
    X @ W.T
except ValueError:
    print('X @ W.T: incompatible inner dimensions')
print('one vector:', (X[0] @ W).shape)
print('one-row batch:', (X[0:1] @ W).shape)

Checked output:

shapes: (2, 3) (3, 2) -> (2, 2)
mapped rows: [[7, -1], [16, -1]]
X @ W.T: incompatible inner dimensions
one vector: (2,)
one-row batch: (1, 2)

Distinguish multiplication from reshaping

X * W requests elementwise multiplication, subject to broadcasting rules. It does not compute the sums above. In this particular case its shapes are incompatible; with other shapes it might succeed and still answer the wrong question. A successful operation is not proof that the axes have the intended meaning.

Transposing swaps axes. Reshaping changes how values are arranged into dimensions and is not a substitute for a transpose. Fixing a shape error by blindly applying .reshape can create an executable calculation with scrambled feature relationships.

NumPy also treats one-dimensional vectors specially in matrix products. X[0] has shape (3,) and produces (2,). The slice X[0:1] retains a batch axis, giving (1, 2). Keep that distinction explicit when later code expects a batch, even if the numerical values match.

Understand what the map can preserve

The three input coordinates become two output coordinates. This map cannot retain every distinction between arbitrary three-dimensional inputs: there is at least one nonzero direction that maps to zero. Compression is a property of the linear transformation, not a rounding error.

Matrix multiplication is generally not commutative. Swapping X and W may be invalid, and even when both products exist they need not represent the same map or have the same shape. In a neural network, writing features as rows rather than columns determines where the learned matrix goes.

Extend the worked example

Add a bias vector with shape (2,) and explain why it broadcasts across the two rows. Then pass an identity matrix through the map to inspect what happens to each basis vector. Finally, find an input direction whose mapped result is zero and verify the result numerically.

The Maths course connects these operations to linear algebra. Read NumPy broadcasting for elementwise shape rules, and a neural network forward and backward pass to see the same matrix pattern inside a model.

References

NumPy matmul — Matrix product signatures, one-dimensional operands and batched multiplication.