ULMFiT · the recipe that made fine-tuning work

Three stages, one network Recomputed

Click a stage. The same stack of layers gets reused all the way through; only what feeds it and which layers are allowed to change differ. Hover any layer for what it does right now.

trainingfrozennew, from scratch

Gradual unfreezing, epoch by epoch

The three dials that make it work Recomputed

Each one is small. Together they stop catastrophic forgetting on tiny datasets and still converge fast on big ones. The ablations further down show what happens when you drop any of them.

1 · Discriminative fine-tuning

Lower layers hold general knowledge (syntax, common words); upper layers hold task-specific stuff. So don't move them equally. Pick a learning rate for the top layer, then divide by 2.6 for each layer below.

The 2.6 was found empirically by tuning only the last layer first, then setting η^l-1 = η^l/2.6. Drag to feel how quickly the embeddings become nearly frozen.

top-layer LR 0.010 ratio 2.6

Set ratio to 1.0 and this becomes ordinary fine-tuning: every layer moves at the same speed.

2 · Slanted triangular learning rate

A short, steep climb to the peak (10% of training), then a long, slow slide down. The fast start gets the model into the right region; the long decay refines without wrecking anything.

Hover the curve to read the rate at any step. Toggle cosine annealing to see the competitor that won on large data but lost on small.

cut_frac 0.10 ratio 32 cosine

3 · Gradual unfreezing

Unfreeze the top layer, train one epoch. Unfreeze the next one down, train again. Keep going until everything is thawed, then train to convergence. The layers with the most general knowledge are exposed last and least.

This is what the epoch buttons in stage c above are doing. Unlike chain-thaw, thawed layers stay thawed.

Press play, or hover any column.

One small trick for long documents Sketched

The signal in a review is usually a few words, anywhere in it. Reading only the final hidden state throws that away. The classifier instead sees three views of the whole document, concatenated. Hover each.

Drop a dial and see what breaks Reported

Validation error, lower is better. The story changes with dataset size: pick one. Hover a row for what that method means.

What forgetting looks like Sketched

Fine-tune everything at once and validation error bottoms out early, then climbs as pretrained knowledge is overwritten. ULMFiT starts slower (only the head is training) but keeps improving into late epochs.

Curve shapes follow Figure 4 of the paper; the numbers here are illustrative, not measured.

Did it beat the field? Reported

Test error against the prior state of the art on six datasets, using one plain LSTM and identical hyperparameters everywhere. Hover a bar for the model.

Error reduction vs. best prior model

With only 100 labeled examples

If you want to actually do this Reported

The paper is unusual in giving every number. Here they are, plus what survived into the transformer era.

The exact defaults

Backbone: AWD-LSTM: 400-d embeddings, 3 layers × 1150 units, BPTT 70
Pretraining corpus: WikiText-103, 28,595 articles, 103M words (done once)
Dropout: 0.4 layers, 0.3 RNN, 0.4 input embedding, 0.05 embedding, 0.5 weight-drop on hidden-to-hidden
Optimizer: Adam, β₁ = 0.7 (not 0.9), β₂ = 0.99, batch 64
Base LR: 0.004 for LM fine-tuning, 0.01 for the classifier
STLR: cut_frac 0.1, ratio 32, one cycle only
Discriminative: ÷2.6 per layer going down
Epochs: ~15 for LM fine-tuning on small data; 50 for the classifier
Classifier head: 2 linear blocks (BN + dropout + ReLU), hidden size 50, softmax out; the only weights trained from scratch
Extras: Forward + backward LMs ensembled (about 0.5 to 0.7 error points); special tokens for ALL-CAPS, elongation, repetition

When it helps most

Small or medium labeled sets, which is most commercial data. Pretraining cut TREC-6 error from 10.7% to 5.7%; on 120k-example AG it still helped, just less.
You have lots of unlabeled text in your domain. Stage b uses it for free and doubled the effective data efficiency on IMDb.
Non-English languages and new tasks with no bespoke architecture: language modeling needs no labels and no task-specific design.

Traps the paper warns about

Only training the top layer (the classic CV move) severely underfits text models: 16.1% error on TREC-6 versus 5.7%.
Fine-tuning everything at one flat rate looks great after epoch 1 and then quietly degrades. If you early-stop, you may never notice you left accuracy on the table.
Skipping LM fine-tuning on a small dataset with plain schedules can make things worse; the discriminative rates and STLR are what made stage b pay off on TREC-6.
No dropout in the LM is fine on large data and overfits on small.

What carried over to transformers

Warm-up then decay schedules are the norm for fine-tuning; STLR is the LSTM-era ancestor of that shape.
Layer-wise learning-rate decay (a smaller LR for lower layers) is a standard knob when fine-tuning encoder models. Same idea, different name.
Continued pretraining on in-domain text before task fine-tuning (stage b) is still one of the cheapest wins on specialised domains.
Gradual unfreezing shows up less often today, but the underlying caution it encoded, don't let a new randomly-initialised head send large gradients through a good backbone, survives in tricks like freezing the backbone for the first steps.