From a web crawl to a word on your screen

Sources

• Vaswani et al. 2017. Attention Is All You Need. arXiv:1706.03762 - the decoder block the whole stack repeats.
• Shazeer et al. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538 - the MoE routing idea.
• Shoeybi et al. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053 - tensor parallelism.
• Rajbhandari et al. 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv:1910.02054 - sharded optimizer state, later FSDP.
• Lee et al. 2021. Deduplicating Training Data Makes Language Models Better. arXiv:2107.06499 - the dedup stage.
• Ouyang et al. 2022. Training language models to follow instructions with human feedback. arXiv:2203.02155 - the post-training pipeline.
• Dao et al. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135
• Leviathan et al. 2022. Fast Inference from Transformers via Speculative Decoding. arXiv:2211.17192
• Yu et al. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. USENIX OSDI '22 - continuous batching.
• Ainslie et al. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245
• Penedo et al. 2023. The RefinedWeb Dataset for Falcon LLM. arXiv:2306.01116 - a published version of the filtering pipeline.
• Kwon et al. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180 - vLLM, the KV block pool.
• Liu et al. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context. arXiv:2310.01889 - context parallelism.
• Zhong et al. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. arXiv:2401.09670
• Shao et al. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 - GRPO.
• Agrawal et al. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. arXiv:2403.02310 - chunked prefill.
• vLLM documentation, docs.vllm.ai - paged KV cache, continuous batching, prefix caching as actually implemented.
• SGLang documentation, docs.sglang.ai - RadixAttention and prefix reuse across requests.
• Common Crawl, commoncrawl.org - the raw web snapshots the corpus stage starts from.
• NVIDIA H100 Tensor Core GPU datasheet, nvidia.com/data-center/h100
• NVIDIA H200 Tensor Core GPU datasheet, nvidia.com/data-center/h200
• NVIDIA B200 and the Blackwell architecture brief, nvidia.com/data-center/dgx-b200 and the Blackwell architecture resources

Scope: this page describes the shape of a modern training and serving stack and the arithmetic that governs it. Hard numbers are datasheet figures or published results and are attributable to the sources above. Everything else - throughput, prices, latencies, model names - is a representative illustration chosen to make the mechanism concrete, not a measurement of any deployed system.