Hybrid model
A model that mixes attention layers with state-space or recurrent layers to get the strengths of both.
Also: hybrid architecture
Hybrid designs keep a few attention layers for precise recall and use cheaper state-space or recurrent layers for the rest. AI21 Labs' Jamba, released in 2024, interleaved transformer and Mamba layers with mixture-of-experts blocks and offered a 256,000-token context window. Other labs have built similar hybrids, including Nvidia, IBM with Granite 4.0 and several Chinese groups. The aim is to cut the memory and time cost of long contexts without losing the quality of a pure transformer.