Mixture of Experts Explained
There is a second iteration (Feb 2026) of the blog post where we cover how the
transformerslibrary has built around MoEs to make them “first class citizens” of the library and the Hub. Here is the link to the post: Mixture of Experts (MoEs) in Transformers
With the release of Mixtral 8x7B (announcement, model card), a class of transformer has become the hottest topic in the open AI community: Mixture of Experts, or MoEs for short. In this blog post, we take a look at the building blocks of MoEs, how