Source reading · arXiv:1802.05365 · Peters et al. 2018 · one paper read closely, with every number traced to the table it came from
This is a source reading, not a survey of contextual embeddings. One paper is the anchor, and the page's job is to separate what it measured from what people remember it measuring. The claim under test is small and specific: a word's representation should be a function of the whole sentence, produced by a deep bidirectional language model trained on raw text, and every layer of that model should be exposed to the task rather than just the top one.
Six benchmarks, one recipe: take a strong published baseline, concatenate ELMo vectors to its input, change nothing else, retrain. All six improved. What follows is those results, the probes behind the layer story, and the parts of the popular retelling the paper does not support.
Peters M, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L. 2018. Deep contextualized word representations. NAACL-HLT 2018, 2227 to 2237. arXiv:1802.05365 · doi:10.18653/v1/N18-1202
forward LM reads left to right · backward LM reads right to left · the two are trained separately and joined only at the end
Reported a figure from the paper's result tables, on a held-out test set.
Analysis from the paper's ablations and probes: development sets, single tasks, narrower evidence.
Added here an illustration or a present-day extrapolation written for this page. Not in the paper.
Before this paper the standard move was word2vec or GloVe: every word type gets a single frozen vector, wherever it appears. The paper's Table 4 makes the failure concrete by taking the word play, listing GloVe's nearest neighbours, and then listing the sentences whose biLM representation of play sits closest to each source sentence. The neighbours move when the sentence moves.
Sports-flavoured neighbours regardless of context. Theatre, the verb, and every idiom are crushed into one point, and the list is dominated by inflections of the same word rather than by senses.
"Chico Ruiz made a spectacular play on Alusik's grounder"
Nearest source sentence in the biLM's space: Kieffer, the only junior in the group, was commended for his ability to hit in the clutch, as well as his all-round excellent play.
The representation lands squarely in the sports sense. Reported
"Olivia De Havilland signed to do a Broadway play for Garson"
Nearest source sentence: they were actors who had been handed fat roles in a successful play, and had talent enough to fill the roles competently.
Same word, completely different neighbourhood. Reported
"The kids went outside to play after lunch."
A verb reading, written for this page to round out the set. The paper's table covers the two noun senses above and does not include this sentence, so treat it as an illustration of the mechanism rather than a result. Added here
Disambiguation used to be something you engineered for. Here it falls out of unsupervised pretraining: using the biLM's top layer with a simple nearest-neighbour rule scores 69.0 F1 on word sense disambiguation, against 65.9 for the WordNet first-sense baseline and 69.9 and 70.1 for two supervised WSD systems, without the language model ever seeing a sense label. Analysis
The general lesson, and the reason this paper is remembered: raw text at scale teaches things you would otherwise pay annotators for.
The nearest-neighbour table is a qualitative demonstration on one word, chosen to be readable. It shows that the representation moves with context; it does not measure how often it moves correctly. The WSD number does that, on one benchmark, and it lands just below the supervised systems rather than above them. Two senses of one word is an illustration, not an evaluation, and the paper presents it as such.
A forward LM predicts each token from everything to its left, a backward LM from everything to its right. Two layers of each, plus a context-free token layer underneath, means every token comes out with three vectors of 1024 dimensions. Select a block for what it does and its specifications.
The only trained transfer machinery in the whole method: a softmax over three scalar weights, one per layer, plus a global scale. The biLM itself stays frozen. The task model learns which layers it wants, which is four parameters standing between a frozen language model and a state of the art result.
The top recurrent layer. In the paper's probes this layer is the better one for meaning: word sense disambiguation works from here at 69.0 F1 against 67.4 from layer 1. Higher layers see further and abstract more.
The first recurrent layer, and the syntax specialist in the same set of probes: its features alone tag parts of speech at 97.3 percent accuracy against 96.8 from layer 2, close to the dedicated tagger the paper compares against at 97.78. Different depths carry different linguistics, a finding that outlived ELMo itself.
Tokens are built from characters: 2048 character n-gram convolutional filters, two highway layers, a linear projection to 512. No fixed vocabulary, so misspellings, rare words and morphology are handled without an unknown-word token. This is layer 0 of the mix, and it is the only layer that does not depend on the sentence.
Jointly maximize the forward and backward log likelihood on roughly one billion words of raw news text. After ten epochs the average of the forward and backward perplexities is 39.7. Train once, reuse everywhere, and if the target domain differs, fine-tune the language model on unlabelled in-domain text first: the paper reports that this lowers perplexity and lifts downstream scores.
Earlier transfer work, TagLM and CoVe, exposed only the top layer of a pretrained network. ELMo's argument is that the internals are the asset. On SQuAD development data, using the top layer alone scores 84.7 F1 and learning a mix over all three layers scores 85.2, so the whole all-layers idea is worth about half a point over a strong top-layer transfer, on top of the 80.8 baseline. SNLI moves 89.1 to 89.5 and SRL 84.1 to 84.8 in the same comparison. Analysis
Small in isolation, and still the first ablation worth running whenever you ship pretrained features, because it costs four parameters.
ELMo is not deeply bidirectional. The forward and backward language models are separate stacks with no connection between them at any layer; each direction is a standard unidirectional LM, and the two are concatenated only after the fact. Nothing in the model conditions on both sides at once. That is exactly the limitation BERT named when it introduced masked language modelling eight months later, and the paper itself claims no more than the shallow joining it performs.
Everything a downstream task learns about the language model is contained in four numbers: three logits that become layer weights, and one global scale. The sliders below move those numbers directly.
bar width tracks gamma · segments track s0, s1, s2 · the result is concatenated onto the task model's own embeddings
A near-uniform mix, which is where heavy regularization puts you. Safe, and it leaves task-specific signal on the table.
With JavaScript off the sliders do not move, but the equation above is the whole method: three softmaxed weights and a scale, learned by the task, applied to a frozen language model. The presets described in the buttons opposite carry their evidence in their own text.
One. Learn the mix rather than hardcoding the top layer: 84.7 to 85.2 F1 on SQuAD dev. Analysis
Two. The scale is not cosmetic. The biLM's activation statistics differ from the task model's, and the paper calls the scale of practical importance for optimization. The modern equivalent is layer-normalizing each layer before mixing. Analysis
Three. The L2 strength interpolates between a plain average and a fully task-tuned mix. The paper's own comparison of lambda = 1 against lambda = 0.001 is 85.0 against 85.2 on SQuAD dev, 89.3 against 89.5 on SNLI, 84.6 against 84.8 on SRL: real, consistent, and small. Analysis
The weights above are the interface, not the representation. Moving them changes how three fixed vectors are averaged; it does not tell you what is in those vectors, and the verdict text is a reading of the paper's probes rather than a computation. Nothing here is running a language model. The layer characterisations come from two probing tasks, POS tagging and word sense disambiguation, on one biLM. They are evidence that depth specializes, not a general law about layer 1 and layer 2.
The evaluation design is the persuasive part. For each task the authors take a strong published baseline, concatenate ELMo to its input, and retrain with no architecture search. Every task improved. Each row below is zoomed to its own scale, so bar lengths are comparable within a row and meaningless across rows. All figures are test-set results from Table 1. Reported
Axis zoomed to 79 to 87 F1. Baseline is the BiDAF-based system the paper starts from, improved with self-attention. Same architecture, same training data, same hyperparameters except those retuned for the new input. The only change is the input representation.
Axis zoomed to 88 to 94 F1. Both numbers are means over five runs, and the paper reports a standard deviation, which is why this row carries two decimal places where others carry one. The baseline is a character CNN plus biLSTM plus CRF tagger.
Axis zoomed to 79 to 86 F1. This is also the task carrying the sample-efficiency result below, and the one task in the paper's placement ablation that preferred ELMo at the input only.
Axis zoomed to 65 to 72 average F1, the mean of three standard coreference metrics. The relative error reduction is the smallest of the six despite a 3.2 point absolute gain, because the baseline sits lower.
Axis zoomed to 49 to 56 percent accuracy. The baseline here is the biattentive classification network from the CoVe paper, so this row is also the closest thing to a head-to-head against the previous generation of contextual vectors.
Axis zoomed to 86 to 90 percent accuracy, the tightest window on the page for the smallest absolute gain, 0.7 points. Means over five runs. A saturated benchmark leaves little room, which is the honest reading of a 5.8 percent error reduction.
The headline was SQuAD, plus 4.7 F1. The sleeper result is sample efficiency: on semantic role labeling, a model with ELMo trained on 1% of the labelled data reaches roughly the F1 of the baseline trained on 10%, and it passes the baseline's best development F1 at epoch 10 where the baseline needs 486. Analysis
Pretraining converts compute on raw text into labelled data you no longer have to buy. That trade is what the field has been running on ever since.
Each gain is measured against a specific 2017-era baseline on a specific benchmark, and several of those baselines were themselves near the ceiling of what the benchmark could show. The comparison is to the state of the art of the time, not to anything current. The paper reports means over five runs for NER and SNLI only; the other four rows are single numbers, so small differences in them should not be read as reliable.
The architecture is obsolete. Several of the habits are not. Each item below is a specific result in the paper followed by what it looks like now, and the second half of each is extrapolation rather than something the paper claims. Added here
Pull hidden states from every layer of your pretrained model and learn a scalar mix, or at minimum probe which layer suits the task. ELMo's version of this ablation was worth 0.5 F1 on SQuAD dev over a strong top-layer transfer, for four parameters.
Syntax and surface features sit low in the stack, meaning sits high: 97.3 percent POS accuracy from layer 1, 69.0 F1 on word sense from layer 2. Later work found the same ordering in transformer and speech encoders, and wav2vec 2.0 reuses this exact weighted-layer-sum for downstream heads.
The gamma scale exists because pretrained activations have a different scale from the task model's other inputs and optimization suffers without a correction. Today: layer-normalize each layer before the mix, or at minimum standardize frozen features before they reach a downstream head.
The paper tested ELMo at the input of the task encoder against input plus output. On dev sets, SQuAD went 85.1 to 85.6 and SNLI 88.9 to 89.5 by adding the output position, while SRL fell 84.7 to 84.3. Where you fuse pretrained features is a choice with a measurable answer, not a convention.
Fine-tuning the biLM on in-domain raw text, with no labels, lowered perplexity and improved downstream scores. The modern form is continued pretraining on domain or task text before any supervised fine-tuning, usually the cheapest gain available for a niche domain.
With a small labelled set, a better representation beats model tweaks: ELMo at 1 percent of the SRL training data roughly matches the baseline at 10 percent. Annotation budget often does less than adaptation compute, and this result is the earliest clean measurement of that trade.
A language model trained only to predict the next token learns syntax, semantics and word sense as internal structure, and exposing that structure layer by layer transfers to essentially any language task. The paper measured this on six benchmarks with a frozen two-layer LSTM. Everything since has been the same sentence at a different scale.
Scope: this is a close reading of one paper. Figures marked Reported are test-set results from its Table 1. Figures marked Analysis come from its ablations and probing experiments, which use development sets, single tasks and in most cases a single run, and are weaker evidence than the headline table. Anything marked Added here was written for this page: the verb example in tab 1, three of the four slider presets in tab 3, and the present-day half of every item in tab 5. The mixer computes a softmax over three numbers you choose and nothing else; no language model runs in this page, and no figure on it is a measurement made here. Benchmark numbers are comparisons against 2017-era baselines and say nothing about current systems.