LoRA vs full fine-tuning: measure a low-rank update on a declared task

Full fine-tuning updates the selected base weights directly. LoRA freezes a base matrix and learns a low-rank update, usually represented as two smaller matrices. It changes the number and structure of trainable parameters; whether it matches a full update depends on the task, target modules, rank and optimization.

We will compare both on a four-input, four-output linear regression task with an explicitly full-rank target update. This is a controlled layer experiment on CPU, not an LLM quality, latency or GPU-memory benchmark.

Declare the update capacity

The frozen base is a 4×4 identity matrix. The target adds diagonal values [1, 0.8, 0.6, 0.4], so the required update has rank four. Full adaptation trains 16 matrix entries. Rank-one adaptation trains A with shape (1,4) and B with shape (4,1), eight entries in total, and uses W = W0 + B @ A.

The scaling factor is one for this example. Practical LoRA configurations often specify an alpha-to-rank scaling, dropout and selected target modules; record them when comparing real models. Adapter parameter savings do not equal the percentage reduction in total process memory.

B starts at zero, making the initial adapted model equal to the base model, while A starts with small random values. If both factors were zero, their product would have zero gradients with respect to both factors at initialization.

Give both methods the same declared update budget

Use Python 3.12.14, PyTorch 2.14.1+cpu, float64 and one CPU thread. Both methods receive 800 full-batch Adam steps at learning rate 0.05 on the same training rows. The held-out set is not used to choose rank or settings. Equal settings isolate the example; they do not establish the best tuning configuration for either method.

import torch
torch.set_num_threads(1)
torch.manual_seed(45)
dtype = torch.float64
X = torch.randn(400, 4, dtype=dtype)
base = torch.eye(4, dtype=dtype)
teacher = base + torch.diag(torch.tensor([1., .8, .6, .4], dtype=dtype))
y = X @ teacher.T
train, test = X[:300], X[300:]
y_train, y_test = y[:300], y[300:]
full = torch.nn.Parameter(base.clone())
A = torch.nn.Parameter(torch.randn(1, 4, dtype=dtype)*.1)
B = torch.nn.Parameter(torch.zeros(4, 1, dtype=dtype))
for name, parameters in [('full', [full]), ('rank_one', [A, B])]:
    optimizer = torch.optim.Adam(parameters, lr=.05)
    for _ in range(800):
        optimizer.zero_grad(set_to_none=True)
        W = full if name == 'full' else base + B @ A
        loss = ((train @ W.T-y_train)**2).mean()
        loss.backward()
        optimizer.step()
    with torch.no_grad():
        W = full if name == 'full' else base + B @ A
        mse = ((test @ W.T-y_test)**2).mean().item()
        rank = torch.linalg.matrix_rank(W-base, atol=1e-6).item()
    print(name, 'trainable=' + str(sum(p.numel() for p in parameters)),
          f'test_MSE={mse:.6f}', 'update_rank=' + str(rank))
assert torch.equal(base, torch.eye(4, dtype=dtype))
assert torch.linalg.matrix_rank(B @ A, atol=1e-6).item() == 1

Checked output:

full trainable=16 test_MSE=0.000000 update_rank=4
rank_one trainable=8 test_MSE=0.287489 update_rank=1

Explain the capacity limit

A rank-one update cannot represent the full-rank target update exactly. The remaining error is therefore partly a representational constraint, not evidence that low-rank adaptation is always inferior. If the task's needed change is approximately low rank, a small adapter can be a good fit.

Our full model can represent the declared target and fits it closely. This result follows from the synthetic construction and cannot be generalized to an LLM trained on language data. Real adaptation also depends on optimization, regularization, training examples and which weights are allowed to change.

Increasing rank increases adapter parameters and capacity. At sufficiently large ranks, a factorized update may no longer save parameters for a small matrix. For an out × in matrix, the basic adapter count is rank × (in + out), before any biases or additional trainable modules.

Measure the resource claim you want to make

A practical LLM comparison should record the base revision, tokenizer, dataset split, sequence lengths, target modules, rank, scaling, precision, batch accumulation, optimizer states and hardware. Measure peak memory and elapsed time directly if those are part of the claim.

A frozen base still occupies memory and participates in the forward pass. Quantization, activation storage and optimizer choices affect the final footprint. Our eight-versus-sixteen trainable count does not imply half the runtime or half the total memory.

Extend the task

Replace the teacher update with an outer product of two vectors and rerun the comparison. Rank one can then represent the target change. Next compare ranks 1 and 2 on the original full-rank target using the same declared split.

The Generative AI syllabus covers adaptation; lessons are being published. Review matrix multiplication for the factor shapes and Adam versus momentum SGD for the limits of optimizer comparisons.

References

LoRA paper — Hu and colleagues describe frozen base weights and trainable low-rank updates.

PyTorch Adam — Optimizer settings used for this controlled layer experiment.