Synthesis · Training · Serving · Cost · assembled from many sources, not a reading of one paper. Model names and version numbers are hypothetical stand-ins for a 2026-class system, not any vendor's product.

From a web crawl to a word on your screen

A frontier model is two machines wearing one name. The first spends months and hundreds of millions of dollars turning a corpus into a file of numbers. The second spends 400 milliseconds turning your sentence into the next one - and has to do that a hundred thousand times a second, profitably. This walks both, in order.

The interactive version of this page lets you vary request shape, batch size, precision, and hardware, and watch the cost and latency move. Without JavaScript you get this framing and the source list below; the numbers underneath are representative of 2026-class deployments rather than any single vendor's disclosed figures.

I · Build corpus to weights

Corpus construction, tokenization, the repeated decoder block, the parallelism mesh that splits it across tens of thousands of accelerators, what breaks during the run, and the post-training that turns a text predictor into an assistant.

II · Serve request to token

One request end to end: the edge, authentication, safeguards, routing, the scheduler, prefill and decode inside the engine, and the eight tiers of infrastructure between a user and a tensor core.

III · Physics why it costs what it costs

Why decode is memory-bound and no purchase fixes it, what the KV cache costs, how continuous batching and speculative decoding buy throughput back, and where prompt caching decides the bill in an agent loop.

Sources
  • Vaswani et al. 2017. Attention Is All You Need. arXiv:1706.03762 - the decoder block the whole stack repeats.
  • Shazeer et al. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538 - the MoE routing idea.
  • Shoeybi et al. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053 - tensor parallelism.
  • Rajbhandari et al. 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv:1910.02054 - sharded optimizer state, later FSDP.
  • Lee et al. 2021. Deduplicating Training Data Makes Language Models Better. arXiv:2107.06499 - the dedup stage.
  • Ouyang et al. 2022. Training language models to follow instructions with human feedback. arXiv:2203.02155 - the post-training pipeline.
  • Dao et al. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135
  • Leviathan et al. 2022. Fast Inference from Transformers via Speculative Decoding. arXiv:2211.17192
  • Yu et al. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. USENIX OSDI '22 - continuous batching.
  • Ainslie et al. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245
  • Penedo et al. 2023. The RefinedWeb Dataset for Falcon LLM. arXiv:2306.01116 - a published version of the filtering pipeline.
  • Kwon et al. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180 - vLLM, the KV block pool.
  • Liu et al. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context. arXiv:2310.01889 - context parallelism.
  • Zhong et al. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. arXiv:2401.09670
  • Shao et al. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 - GRPO.
  • Agrawal et al. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. arXiv:2403.02310 - chunked prefill.
  • vLLM documentation, docs.vllm.ai - paged KV cache, continuous batching, prefix caching as actually implemented.
  • SGLang documentation, docs.sglang.ai - RadixAttention and prefix reuse across requests.
  • Common Crawl, commoncrawl.org - the raw web snapshots the corpus stage starts from.
  • NVIDIA H100 Tensor Core GPU datasheet, nvidia.com/data-center/h100
  • NVIDIA H200 Tensor Core GPU datasheet, nvidia.com/data-center/h200
  • NVIDIA B200 and the Blackwell architecture brief, nvidia.com/data-center/dgx-b200 and the Blackwell architecture resources

Scope: this page describes the shape of a modern training and serving stack and the arithmetic that governs it. Hard numbers are datasheet figures or published results and are attributable to the sources above. Everything else - throughput, prices, latencies, model names - is a representative illustration chosen to make the mechanism concrete, not a measurement of any deployed system.