COLING 2025 Tutorial:
Speculative Decoding for Efficient LLM Inference

Schedule

Our tutorial will be held on January 19 (all the times are based on GST = Abu Dhabi local time).

Reading List

Bold papers are discussed in detail during our tutorial.

For further information, we recommend referring to our Survey and Reading List on Speculative Decoding.

Part II: History

• Blockwise Parallel Decoding for Deep Autoregressive Models (Stern et al., 2018)
• Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (Xia et al., 2022)
• Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2022)
• Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al., 2023)
• Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (Sun et al., 2021)
• Inference with Reference: Lossless Acceleration of Large Language Models (Yang et al., 2023)
• Assisted Generation: a new direction toward low-latency text generation (Huggingface, 2023)
• Speculative Decoding with Big Little Decoder (Kim et al., 2023)

Part II: A Taxonomy of Methods

• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (Xia et al., 2024)
• Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (Zhang et al., 2023)
• REST: Retrieval-Based Speculative Decoding (He et al., 2023)
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (Cai et al., 2023)
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (Li et al., 2023)
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (Fu et al., 2024)
• Accelerating Transformer Inference for Translation via Parallel Decoding (Santilli et al., 2023)
• Layer Skip: Enabling Early Exit Inference and Self-Speculative Decoding (Elhoushi et al., 2024)
• DistillSpec: Improving Speculative Decoding via Knowledge Distillation (Zhou et al., 2023)

Part III: Cutting-edge Algorithms

• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (Li et al., 2024)
• GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding (Du et al., 2023)
• Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 Submission)
• SAM Decoding: Speculative Decoding via Suffix Automaton (Hu et al., 2024)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (Xia et al., 2024)
• OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure (Wang et al., 2024)
• Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding (Zhao et al., 2024)

Part IV: Downstream Adaptations

• TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding (Sun et al., 2024)
• Accelerating Retrieval-Augmented Language Model Serving with Speculation (Zhang et al., 2023)
• Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters (Yi et al., 2023)
• LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding (Jang et al., 2024)
• Efficient Inference for Large Language Model-based Generative Recommendation (Lin et al., 2024)
• Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (Wang et al., 2024)