The paper presents two main contributions from Google Research: Titans, a new AI architecture for long-term memory, and MIRAS, a unified theoretical framework.
🧠 Overview
The paper introduces two key contributions from Google Research:
- Titans — a new AI architecture that enables models to maintain long-term memory while processing large amounts of information.
- MIRAS — a unified theoretical framework that explains and generalizes how different sequence models (like Transformers and RNNs) can efficiently store and update memory.
These two innovations address one of the biggest challenges in current artificial intelligence: allowing models to "remember" and use knowledge across extremely long contexts without performance loss or prohibitive computational costs.
📌 Why is long-term memory a problem?
Classical models like Transformers work well in short contexts thanks to the attention mechanism, which compares each new token with all previous ones. However, this process grows quadratically with text size, making it unfeasible to handle very extensive texts (thousands to millions of tokens).
Alternatives like RNNs or State Space Models (SSMs) scale better but compress everything into a fixed-size vector — losing much of the important information along the way.
🔍 What is Titans
Titans is an AI architecture that combines the best of both worlds:
Titans (MAC) Architecture Overview
🧠 Neural Long-Term Memory
Instead of reducing history to a fixed vector, Titans uses a deep memory module (MLP) that functions as persistent memory — not just recording what happened, but understanding and synthesizing important information as it arrives.
🔄 Runtime Learning
The big innovation is that this module is not static. It updates continuously during inference — meaning as the model receives new inputs, it decides based on relevance what should enter permanent memory, without pausing processing or requiring offline retraining.
🎯 The "Surprise" Metric
To decide what to remember, Titans uses a "surprise metric":
- If the new information is predictable → low surprise → not stored.
- If it's unexpected → high surprise → the model adjusts its memory to store it.
This idea is inspired by how humans tend to remember important or unexpected events.
⚙️ Additional Mechanisms
To keep memory useful and stable over time, Titans incorporates:
- Momentum, which helps capture relevant context linked to the hallmark event;
- Adaptive Forgetting, which allows discarding what is no longer relevant.
📘 What is MIRAS
While Titans is the "tool", MIRAS is the theoretical model behind how memory should be understood for sequence models:
MIRAS Framework Structure
📐 A Unified Vision
MIRAS proposes that virtually all sequence models can be viewed as variants of the same optimized associative memory process. It organizes ideas into four elements:
- Memory Architecture (how information is stored)
- Attentional Bias (what is prioritized)
- Retention Gate (what is forgotten or kept)
- Update Algorithm (how memory is adjusted)
🧠 Expanding the Design Space
This paves the way for new architectures that aren't just Transformers or RNNs — for example, more robust, stable, or balanced variants — all following the same underlying structure of learning and memory retention.
🚀 Importance and Results
According to the paper:
- Titans can handle extremely long contexts (the paper mentions over 2 million tokens) with much greater efficiency than classic Transformers.
- It combines speed (like RNNs) with expressiveness and precision (like Transformers).
- MIRAS provides a theoretical basis to guide future memory designs in AI, possibly changing the design of models dealing with text, code, time series, or biological data.
🧠 Impact on the Future of AI
These ideas suggest a path for models that:
- Remember important facts for long periods, without fixed context limits.
- Learn and adapt in real-time during use, without the need for lengthy retraining.
- Can work with complex tasks like extensive documents, code analysis, genomes, or continuous user dialogues.