Continual Learning Models: An Introduction to Weight Management
Abstract
Large language and multimodal models remain static after training, they retrieve and recombine patterns learned during pretraining, but they do not gain experience from deployment the way a human does on the job. This paper introduces continual learning models, a class of architectures that update their own weights at inference time. We describe the architectural shift this requires, introduce weight management as the discipline this shift creates, and walk through the internal mechanics of how such a model learns and unlearns. As proof of concept, we present results from Vision CLv1, a 550-billion-parameter continual learning model, evaluated on ARC-AGI-3. We close by situating this work within the broader space of approaches to continual learning, and test model is available to try.
1. Introduction
Current AI models are trained once and then frozen. This is a fundamental mismatch with how intelligence actually develops: a person doesn't play a new video game well on day one, but by the tenth day of practice they've figured out how to win. The improvement comes from accumulated experience, not from a single upfront training pass.
Every instance of a deployed model remains static, it doesn't personalize to the environment it lives in, and it doesn't learn over time. In practice, this shows up as a hard ceiling: performance is bound by the context window, and once an interaction runs longer than that, the model's performance starts to saturate. There's no persistent memory underneath it, and none of the cognitive continuity that lets a human accumulate experience over time. The common workaround today is to bolt orchestration onto a static model, RAG pipelines, retrieval layers, agent scaffolding, patches that manage context better without changing what the model itself is capable of. We took a different approach: work on the model architecture itself, so that the model updates its own weights at inference time, foundationally, rather than through a layer built on top of it.
We identified this gap early in our work and set out to close it. Beginning in 2024, we researched an approach founded on architectural modifications to the Transformer, pretraining a model from scratch on an architecture that supports inference-time weight updates, with compute resources from a grant. The goal was narrow and specific: a model that could update its own weights while running, without catastrophic forgetting, and without requiring a retraining cycle.
2. What a Continual Learning Model Looks Like
A continual learning model takes two kinds of input rather than one. The first is content, text, audio, video, whatever the task calls for. The second is weights, a compact representation of everything the model has learned from prior interaction. The model produces two kinds of output to match: a response, and an updated set of weights.
This is the same relationship LLMs have with context, generalized one level up. Where an LLM manages a context window, a continual learning model manages a persistent weights file. We call this discipline weight management: anyone deploying a continual learning model now has to think about how weights are stored, updated, and carried forward, the same way developers today think about context windows and token budgets. It's a new kind of state to reason about, and it's the concept this paper is centered on.
3. How the Model Works Internally
Internally, the model is built on top of the Transformer architecture, modified to accept a weights input alongside its content input. Three parts make up how a continual learning model works:
The Continual Learning Core. This is the model's ability to learn and unlearn a capability current models are missing entirely. If the model comes across a piece of information it's never seen, or recognizes that something is worth remembering, it updates its weights by adding that input. If there's nothing new, nothing changes. The reverse also holds: if you explicitly ask the model to forget something, it forgets it. In practice, this makes interacting with the model feel closer to talking to a person than querying a static system, you can ask it to learn something, and later ask it to unlearn it, and it does both.
The Environment. This is the space the model instance operates in. Each interaction is a turn: the instance takes an input, uses its weights to personalize the output, and writes back an updated weights file for the next turn based on what it learned from that exchange. A new concept follows directly from this: the exploration phase. If you want a continual learning model to get good at a particular problem or environment, you can expose it to that environment for a stretch of time before relying on it, let it try things, make mistakes, and work out an approach on its own, the same way a person would if dropped into an unfamiliar job. That exploration period is where the model's weights actually accumulate the experience it will later draw on.
The Weights File. This is where the accumulated experience lives, everything the exploration phase produced, encoded as weights. It's used in combination with the base model: the weights file is matrix-multiplied against the base 550-billion-parameter Vision CLV1 model weights, and the output is produced from that combined set. This is what makes the output hyper-personalized: it isn't the base model answering generically, it's the base model modified by everything it has learned in that specific environment.
A brief note on format: weights are, natively, floating-point numbers. Our model APIs return them as text instead, so they show up as string and can be stored as weights.txt file rather than a raw numerical array. The underlying values don't change, the floating-point weights are converted into a character-string representation via hashing. This makes the weights file something that's easy to store, move, and version like any other text file, rather than a binary blob.
This is the mechanism that lets the model behave less like a fixed lookup and more like something that develops intuition through repetition. Consider a salesperson: their improvement on the job doesn't come from re-reading a playbook before every call, it comes from accumulated instinct, knowing how to read a room, when to push and when to hold back, built up call after call. A sales agent running on a continual learning model develops the same way: with each interaction it accumulates the judgment that makes it better at closing, not because it looked anything up, but because it's carrying forward what it learned from every prior deal.
4. Proof of Concept: ARC-AGI-3
To test whether this architecture holds up outside description, we evaluated Vision CLv1 on ARC-AGI-3, a benchmark designed specifically to measure skill-acquisition efficiency over time, it requires an agent to build a world model on the fly and transfer reasoning across games and levels rather than solving each one from a blank slate. That property makes it a natural fit for testing continual learning claims directly, rather than taking them on faith.
We ran the evaluation in two phases. In Phase 1, the model explored the full public game set offline, starting from empty weights. Each call returned an updated weights blob, which was fed into the next call, so the weights accumulated latent context across the full run, carrying over not just within a game, but across game boundaries. This phase ran until per-game performance hit target, the weight size saturated, or the model's continual learning window filled. In practice, this exploration phase ran for roughly 19 hours before the model had built up enough experience to move on.
Observing Continual Learning in Vision CLv1
The exploration-phase statistics make the learning curve visible. In the first hour, the model was updating weights aggressively, every unfamiliar level triggered a fresh mistake, and every mistake triggered a fresh update. By the nineteenth hour, that rate of change had slowed to a plateau: weight updates were becoming smaller and less frequent, not because the model had stopped trying, but because there was less and less left to learn. Having worked through the full set of games and levels, Vision CLv1 had, in effect, already learned the art of solving these visual puzzle games, and its own weight file reflected that shift, from steep and constant change to something closer to steady state.
In Phase 2, we took the weights produced by the exploration phase and let the model play the games in the real apparatus, one official scorecard, only level resets permitted, no further training. Based on the experience it had accumulated in figuring out which approaches worked and which didn't, the model figured out how to play and win.
The full evaluation pipeline is open source, so you can be independently verify thhe results and model functioning: github.com/vansh-one/arc-agi-3_Vision-CLv1.
The resulting scorecard came in at 63.15%. For context, most frontier models evaluated on ARC-AGI-3 score well under 10%, and even the current best-performing frontier model on public leaderboards sits at roughly 30%, despite running on architectures with parameter counts in the trillions. That gap isn't a harness problem or a scale problem. Trillion-parameter models still fail to close it, which is the point: the difference isn't more parameters, it's the architectural shift that lets a model learn in real time. That's what separates Vision CLv1's result from the rest of the field.
The exploration phase is the part worth dwelling on here: the score isn't the product of a bigger model seeing more data upfront, it's the product of the same fixed model getting measurably better over the course of its own runtime, which is the specific claim this Vision CLV1 is built to support by inference time weight updation.
5. Where This Sits in the Broader Landscape
There are several well-known approaches people use to approximate continual learning today: RAG based systems as a way of maintaining memory, and post-training methods like reinforcement learning and fine-tuning to adapt a model after the fact. These are, effectively, orchestration layers built on top of a static LLM.
We took a different approach and worked on the model's architecture itself, building the capacity for inference-time weight updates directly into the pretraining process, rather than adding it afterward. We trained three continual learning models from scratch on this architecture.
This is what the future looks like: every deployed model gains experience wherever it's deployed, rather than shipping frozen. That removes a real barrier, LLMs today are just tools, useful but fundamentally static, and that's part of why they haven't matched human judgment in open-ended work. Continual learning models close that gap by developing intuition and taste for a problem over time, the way a person does.
Frontier research has, so far, been almost entirely a human activity, a position no AI model could take, because taking it requires exactly what static models lack: the ability to accumulate experience over time. That's starting to change. We see two major use cases for continual learning models.
The first is personalization: a personal assistant that accumulates experience with a specific person over time, becoming genuinely theirs rather than a generic interface everyone shares.
The second, and the one we want to focus on, is frontier research itself, where a model's ability to gain experience and build intuition across a range of problems is exactly what's needed to let it produce frontier breakthroughs on its own.
6. Models and Access
We've trained three continual learning models from scratch, each targeting a different deployment profile:
- Vision CLv1 (550B) : full-scale frontier reasoning
- Vision CLv1 Medium (175B) : high velocity experiential inference
- Vision CLv1 Small (45B) : for edge deployment
All three are live and available to try at Vispark lab, our model playground. Vision CLv1 sits alongside 15+ other frontier models we've trained across audio, image, video, and text, Vispark Lab is the place to try any of them.
AIVOCO | Accelerating innovation with adaptive intelligence