Autoencoders and reconstruction error: compare a linear bottleneck with PCA

An autoencoder learns to reconstruct its input through a constrained representation. Reconstruction error measures what that representation fails to preserve. It can be useful for inspection, but a large or small error is not automatically a correct anomaly judgment.

We will train a linear two-input, one-latent, two-output autoencoder on invented points near a line. A PCA reconstruction supplies a simpler baseline with the same one-dimensional bottleneck. Finally, we compare an unusual point far along the line with a point off the line.

Define the training and evaluation task

The data-generating relation is approximately y = 2x with small independent noise. We center both training and held-out inputs using the training mean. The network has no biases, so it learns a linear reconstruction around that training center.

The objective is mean squared error across observations and coordinates. We use a fixed seed, full batches, one CPU thread and 1,000 Adam steps. The PCA baseline fits only the centered training rows, using the first right singular vector as its retained direction.

For a linear autoencoder with squared error, an appropriate global optimum can recover the principal subspace. The learned encoder and decoder weights need not be identical to a particular PCA basis. Our finite optimizer run is checked against the baseline rather than assumed to reach the exact optimum.

Train and compare the reconstructions

import numpy as np
import torch
torch.set_num_threads(1)
torch.manual_seed(40)
rng = np.random.default_rng(40)
x = rng.uniform(-1, 1, 400)
raw = np.column_stack([x, 2*x + rng.normal(0, .03, len(x))])
mean = raw[:300].mean(axis=0)
train = torch.tensor(raw[:300]-mean, dtype=torch.float64)
test = torch.tensor(raw[300:]-mean, dtype=torch.float64)
model = torch.nn.Sequential(torch.nn.Linear(2, 1, bias=False),
                            torch.nn.Linear(1, 2, bias=False)).double()
optimizer = torch.optim.Adam(model.parameters(), lr=.03)
for _ in range(1000):
    optimizer.zero_grad(set_to_none=True)
    loss = ((model(train)-train)**2).mean()
    loss.backward()
    optimizer.step()
_, _, vh = np.linalg.svd(train.numpy(), full_matrices=False)
direction = vh[0]
with torch.no_grad():
    auto_mse = ((model(test)-test)**2).mean().item()
pca = test.numpy() @ direction[:, None] @ direction[None, :]
print(f'held-out MSE autoencoder={auto_mse:.6f} PCA={np.mean((pca-test.numpy())**2):.6f}')
probes = torch.tensor(np.array([[4., 8.], [0., 2.]])-mean, dtype=torch.float64)
with torch.no_grad():
    errors = ((model(probes)-probes)**2).mean(dim=1).numpy()
print('probe errors [far along line, off line]:', np.round(errors, 6).tolist())
assert auto_mse < .002
assert errors[0] < errors[1]

Checked output:

held-out MSE autoencoder=0.000103 PCA=0.000096
probe errors [far along line, off line]: [1e-06, 0.426298]

Explain the missed unusual point

The point (4, 8) is well beyond the training x range of -1 to 1, but it lies on the same line. A learned linear projection can reconstruct it well. The point (0, 2) is closer to the training region in some coordinates but departs from the learned relation, producing a larger reconstruction error.

The model therefore measures distance from its reconstruction subspace, not rarity along that subspace. If unusual magnitude matters, use additional evidence about the latent distribution or the original features. A reconstruction threshold alone cannot express every anomaly definition.

A more expressive nonlinear autoencoder can learn a different representation, and an overly flexible model may reconstruct unwanted cases well too. Increasing model capacity is not a general repair for anomaly detection.

Calibrate against the actual review task

Choose a threshold on appropriate development data and inspect false positives and false negatives against an externally defined anomaly label or review process. Training reconstruction error will usually be optimistic for this purpose.

Scale and units affect squared error. A coordinate with large numerical units can dominate the loss, and changing preprocessing changes the learned geometry. Keep the training-only preprocessing with the model and report whether errors are averaged or summed over coordinates.

Extend the experiment

Add a curved relationship and compare linear PCA with a small nonlinear autoencoder. Keep latent size, split and preprocessing explicit. Then introduce unusual points near the learned curve and check whether the same failure persists.

The Deep Learning syllabus covers representation learning; lessons are being published. Review outlier detection for the broader evaluation task and scaling versus normalization before interpreting reconstruction distances.

References

PCA API — Principal components, centering and reconstruction context.

PyTorch linear layer — Linear transformations and parameter shapes.