A new study from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) has developed a method to significantly reduce the computational cost of running large language models (LLMs). The technique, called ‘Speculative Decoding,’ uses a smaller, faster ‘draft’ model to predict potential outputs, which are then verified in a single step by the larger, more accurate target model. This approach allows the larger model to process multiple tokens simultaneously, dramatically increasing inference speed without altering the model’s final output. The researchers demonstrated that their method could double or triple the decoding speed of models with 7-13 billion parameters, making advanced AI more accessible and efficient for real-time applications. For the full details, read the complete article at https://technologyreview.com/2024/05/15/speculative-decoding-llm-speed.