8 Facts About Zyphra's ZAYA1-8B: The Tiny MoE That Outperforms Giants

In a world where bigger often seems better, Zyphra AI has turned the tables with a surprisingly small language model that punches far above its weight. The ZAYA1-8B is a Mixture of Experts (MoE) model boasting only 760 million active parameters—yet it challenges and even beats behemoths like DeepSeek-R1, Gemini 2.5 Pro, and Claude 4.5 Sonnet on complex math and coding tasks. Trained from scratch on AMD hardware and released under the permissive Apache 2.0 license, this model is a game-changer for on-device AI, cost-effective inference, and open-weight research. Here are eight compelling things you need to know about ZAYA1-8B.

1. The Model That Defies Its Size

ZAYA1-8B is a compact language model with a total of 8.4 billion parameters, but only 760 million are activated per forward pass. This makes it incredibly efficient for inference, requiring far less compute and memory than dense models of similar capability. Despite its small active footprint, it achieves scores competitive with first-generation frontier reasoning models on demanding mathematical benchmarks. It’s available on Hugging Face under an Apache 2.0 license and as a serverless endpoint on Zyphra Cloud.

2. MoE: The Secret Sauce of Efficiency

Understanding the distinction between active and total parameters is key. In a standard dense model, every parameter is used for every token. In a Mixture of Experts model, only a subset—the experts—activate at inference time. ZAYA1-8B uses this design to keep its active parameter count low while retaining the representational power of a much larger network. This slashes memory bandwidth and latency, enabling deployment on devices like smartphones or edge hardware without sacrificing performance.

3. Crushing Benchmarks: Math and Coding

On the HMMT'25 mathematics benchmark, ZAYA1-8B scored 89.6, surpassing Claude 4.5 Sonnet (88.3) and GPT-5-High. It also closes in on DeepSeek-V3.2 on general math benchmarks. These results are remarkable for a model with under 1B active parameters, showing that size isn't everything. The model's test-time compute methodology, called Markovian RSA, allows it to dynamically allocate more compute to hard problems, boosting accuracy on challenging reasoning tasks.

4. Built on AMD Hardware: A First for MoE

ZAYA1-8B was trained end-to-end on AMD hardware, marking one of the first major MoE models to be fully trained on AMD GPUs. This is a significant milestone for hardware diversity in AI, proving that AMD's infrastructure can support frontier-level training. Zyphra’s choice also hints at potential cost savings and supply chain flexibility for organizations exploring alternatives to NVIDIA.

5. Markovian RSA: Smarter Test-Time Compute

Zyphra introduces a novel test-time compute method called Markovian RSA (Repeat, Select, Aggregate). This technique enables the model to iteratively refine its reasoning on difficult tokens, effectively spending more compute where it matters most. The result is a significant leap in mathematical reasoning without requiring a larger model. It's akin to giving the model a scratchpad that it can use only when needed.

6. MoE++ Architecture: Three Key Innovations

ZAYA1-8B is built on Zyphra's MoE++ architecture, which goes beyond traditional MoE designs with three specific changes. First, it uses compressed convolutional attention (CCA) for memory-efficient sequence mixing. Second, an MLP-based router with PID-controller bias balancing prevents load imbalance across experts—a common failure mode in MoE training. Third, learned residual scaling controls residual-norm growth for more stable training. Together, these innovations maximize intelligence per parameter and per FLOP.

7. Compressed Convolutional Attention (CCA)

CCA is a sequence mixing mechanism that operates in a compressed latent space, achieving 8× KV-cache compression compared to standard attention. The KV-cache stores intermediate states during inference; an 8× reduction directly lowers memory requirements and enables longer effective contexts on the same hardware. This is crucial for deploying ZAYA1-8B on devices with limited memory, making advanced reasoning accessible offline.

8. Deployment and Availability

ZAYA1-8B is available now on Hugging Face under Apache 2.0, meaning you can download, fine-tune, and integrate it into your own projects royalty-free. Zyphra also offers a serverless endpoint on Zyphra Cloud for those who prefer API access. The model's small active parameter count means it can run on consumer-grade GPUs, mobile devices, or even CPUs with optimization. Whether you need low-latency local inference or cost-effective cloud serving, ZAYA1-8B provides a versatile solution.

Zyphra's ZAYA1-8B challenges the notion that bigger models are always better. By harnessing MoE efficiency, novel attention mechanisms, and smart test-time compute, it delivers frontier-level reasoning in a fraction of the size. Whether you're a researcher, developer, or hobbyist, this model is worth exploring. Download it today and see what small-scale intelligence can do.