Mixture of Experts – How Modern LLMs Scale Without Proportional Cost

A useful mental model for thinking about large language models is that they are very large lookup tables. During training, they compress an enormous amount of information about…