"Google's new Titan architecture breaks Transformer memory bottleneck with 'memory on test'

The Titan architecture innovatively introduces test time computation in the field of memory, which can be scaled up to 2 million context windows and outperforms GPT4 and Llama3; designs neural long term memory module, which realizes high efficiency of memory through the gradient, momentum and forgetting mechanism of the degree of "surprise", avoiding the memorization of useless training data; proposes MAC, MAG and MAL three memory fusion variants, which may surpass Transformer and Mamba in tasks such as language modeling and general knowledge reasoning. MAC, MAG, and MAL memory fusion variants, which may surpass architectures such as Transformer and Mamba in tasks such as language modeling and common sense reasoning.