Why MiniMax Says Agents Need a Million-Token Window

Who is on stage, and how Thomas frames MiniMax

Olive Song’s reason for a million-token context window is the live session. She argues that the window has to hold the user, the environment, tool responses, and the rounds that follow. Taking in a book, she says, was a job earlier MiniMax models could already do. M3 is the version that puts a long window back into a model meant to code, call tools, and understand images and video.

An unnamed host brings on Thomas Wolf as co-founder and chief science officer of Hugging Face. Thomas greets Olive, and she thanks him for the invitation. The video title gives her surname as Song. He calls her Olive. The biography is his introduction. He says she came to the United States and Pennsylvania, was studying and doing a PhD at NYU in the lab the recording names as Jan LeCun, and worked on a project the recording calls J Pa. She joined MiniMax instead of the New York Hugging Face office. They agree to leave that project for another day.

Thomas then frames the slot with a leaderboard. He says the audience has just seen a GLM model that he ranks number two on an artificial-intelligence leaderboard, after he sets aside a model he calls Fable because nobody can use it, and he places MiniMax at number four so the session covers top open models in a row. He does not name the board. He calls MiniMax one of the Chinese “AI dragons,” with DeepSeek, Moonshot’s Kimi, and GLM, all competing for first place. The recording bundles Moonshot, Kimi, and GLM in one clause, so this account keeps them as separate names.

On the release, the two speakers use different clocks. Thomas says the latest MiniMax model was M3, earlier in June, and the top open-source model at that time. Olive says M3 shipped earlier in the month of this conversation. YouTube’s upload date is 4 September 2026, catalog information for the video. Nobody names a year, so neither phrasing can be pinned to a calendar from the interview alone.

• Stage interview with Thomas Wolf and Olive Song

The bet: coding, a million tokens, and vision in one model

Olive’s product bet is that three capabilities will matter together for later applications, so M3 combines them: coding and agentic ability, a longer context, and multimodal understanding. Agentic, as she uses the word, means the model works across turns with tools and an environment. A token is a small piece of text, often part of a word. A million-token context is the context window, the span of text and other inputs she wants the model to hold at once. For the agent she is describing, that span has to include the user, the environment, tool responses, and later rounds.

Her opening description is a size, a skill list, and an architecture name. She calls M3 a smaller model of around 400 billion total parameters, with 20 billion activated, capable at coding and also able to understand video and images. Later she says 428 billion parameters and 23 billion active. She supplies those two pairs and leaves “activated” and “active” undefined. The context length she names is 1 million tokens, from an architecture she calls MiniMax Sparse Attention, abbreviated MSA. She says open models usually handle coding without that combination of coding, vision, and the long window.

Thomas answers with a comparison, and he marks it as his view. He says M3 is still the only top-five open-source model that is actually multimodal, and that MiniMax was first to make a real 1 million-token window functional. He says the team published and shared the attention method extensively. The interview states the comparison. It does not show the set of models he is ranking, and it does not read a paper aloud.

Why she says agents, not book dumps, force the long window

The window argument is her reasoning. She cites no agent benchmark.

She traces long context to MiniMax M1 and MiniMax 01. Those models, she says, could perform tasks at 10 million tokens. Her example is a passive job: dump in a book, and the model returns a review. She says those models were not agentic.

The turn she makes for M3 is the live session. An agent interacts with users and with a whole environment, receives tool responses, and stacks many rounds. She says a shorter context cannot carry a complex task under that load, so for this version the lab decided the long context had to come back. The length she attaches to M3 is the 1 million tokens above. The 10 million tokens stay with M1 and 01.

Thomas supplies a history of the field for anyone who has watched attention research swing back and forth. Attention is the mechanism that lets the model weigh earlier pieces of the context while it generates. He recalls a cost that grows with the square of the sequence length, then a wave of linear-attention work. He says that line faded after FlashAttention, because what the field mainly needed was more efficient kernels, the low-level routines that run the math. He sets a 1 million-token window against GPT-2’s 1,024-token window, which he recalls people once treating as already more than enough.

He says Jeff Dean pitched him trillion-token attention shortly before the talk. Olive treats ultra-long context as research worth doing, and she says it will take architecture design together with hardware research. The trillion in that exchange is context length.

How she describes MiniMax Sparse Attention, and who she says designed it

Olive describes MiniMax Sparse Attention as a simple, scalable design with two branches. An index branch selects, at a higher level, what in the context matters more. A sparse-attention branch then computes only on the selected blocks and carries out the task. She says that split is what lets the lab scale context length and, later, model size. That is the whole of the design she gives.

Thomas calls M3 particularly cheap because of the sparse attention and because the model is small, and he still calls it very efficient. The interview gives no price and no savings figure for M3. He reaches for OpenAI as a sign that inference cost still has room to fall. He says OpenAI has been cutting its inference bill by about half, probably through more efficient attention processing, and that outsiders do not know the method. He names no OpenAI technique. Both the half-bill anecdote and the cheap-M3 remark are his.

Asked whether efficiency can go further, Olive says a lot of work remains on architecture and inference optimization so models can be more efficient. She singles out tasks she calls “task sensitive” that still require strong capability, and she says she wants the model to be efficient on that kind of task. She leaves “task sensitive” undefined and names no latency target.

Thomas asks whether an agent invented the architecture or a person did. She credits an intern on the team and leaves the intern unnamed. She says this is unusual at many labs, where interns lack access to the data and the work, and she says MiniMax is open to anyone who wants to contribute to the models.

How Olive says MiniMax turns a weakness into a training run

Thomas opens the question by saying anyone at the lab can propose a project. Olive describes a research process she says differs from school and from earlier tech companies. The priority she names is foundation and infrastructure, so people can try the model and look for an improvement.

After a release, she says, people try the model, write their own evaluations, find weaknesses, and propose a change. Others who are interested join the project. The work can run for a couple of weeks or a couple of months. When a result works, the lab uses it in the final training run and ships it.

Architecture is her example of a long project. It can take more time because it needs research, experiments, and even a redo of the evaluations used for pre-training. Thomas says he knows she cares about evaluation, agrees they could talk about it, and then turns the conversation toward multimodality.

Why she says vision has to start on step one

She calls the training approach native multimodality. Thomas says the Hugging Face model card describes M3 as multimodal from the first training step, with vision included from the start. He is paraphrasing the card. The interview includes his paraphrase and does not read the card aloud.

The pattern she calls typical is to finish text pre-training, then add vision understanding with adapters, extra modules trained on afterward. She says MiniMax found that this harms text performance, and that vision understanding fails to converge well because the model has already converged toward text. She also calls it a weak way to scale, because the lab wants to scale the data.

Some labs, she says, start the vision capability halfway through pre-training, as continued pre-training. She calls that path recipe-sensitive. The working recipe changes with the architecture, the data mixture, and the learning rate, so conclusions from small experiments did not carry over to a larger model.

Training text and vision together from the first step is what she calls the natural choice. She says labs that try it often see the model collapse after a few steps of training, on both text and vision. She says MiniMax got past that collapse with work on the vision transformer, with interleaved natural image and video data kept in the training stream instead of masked out, with cleaning and masking on the data, and with reward modeling. On her account, that run scales up and holds together. She gives no data volumes and no learning-rate schedule.

Two size pairs, then a plan to pass a trillion parameters

The interview contains two size statements from Olive, minutes apart, and it never chooses between them. Early on she says around 400 billion total parameters with 20 billion activated. When Thomas asks whether a much larger model is coming, and notes that the current one is still fairly small, she says 428 billion parameters and 23 billion active. She leaves “activated” and “active” undefined.

He asks whether they will pass a trillion parameters. She says definitely. Her reason is that many tasks will stay out of reach at smaller parameter counts, and that MiniMax is more ambitious than M3. She names no ship date and no feature list. This trillion is model size. The trillion-token attention Thomas says Jeff Dean pitched is context length.

She says the model came first and the apps followed

Thomas remembers MiniMax beginning to open-source on Hugging Face in January of the previous year, about 18 months before the conversation, with heavy usage already on some apps. The January recollection and the 18-month figure are his memory.

He guesses the company ran from apps and data toward a research team. Olive says the order was the reverse, and that the company was model-first from day one. She says the CEO planned a model that could understand vision and output every modality, as an AGI dream, before the company started and before ChatGPT. She is giving her account of the founding plan.

Apps, she says, came along so people could experience capabilities the lab already had. Not many people will use an API, she says, so the lab needed interfaces, apps, and scenarios. She then gives reach figures: the apps have covered more than 300 million people in about 200 countries, and she thinks more than a million companies as well.

Open-source intention, and what she wants users to send back

Thomas raises the open-source business question directly. A lab with that much app usage still needs revenue, he says, he guesses M3 was released for free, and he asks whether MiniMax will keep open-sourcing models or hold some back for the apps. The free characterization is his guess.

Olive answers for herself and for the model research team. She says they always hope to open-source the models, and she calls that their plan. She credits the community with making later versions better through performance feedback and pull requests, the proposed changes people send against what the lab has released. She states an intention and a feedback loop. She names no license and makes no promise that every future model will stay open.

When Thomas asks what users should send back, she asks for the issues people hit, especially multimodal ones. She says this is the first time the lab has combined these pieces, the current model may have flaws, and they intend to fix them in later versions. She also wants feature requests. Thinking effort is her example: some people ask for it, and she says the team will try to ship it in future models. She presents it as a request for later models.

The agent jobs they sketch, still described as early

Thomas says multimodal use inside coding agents still looks underexplored. Olive agrees that it is early, and says it can unlock agent applications.

The jobs they sketch are intended uses. The talk includes no demo. She wants a model that can read a PowerPoint or a report that is not very structured, understand a long video such as a long-playing clip dumped into the context, and then act with tools. Thomas asks whether an agent could finally watch a YouTube tutorial and learn from it, including how someone describes their coding tools. She agrees that she thinks so.

On internal use, she gives a few clear examples. MiniMax builds its own research harnesses, the scaffolding that runs model workflows, and she says a lot of internal workflows are already automated. Frontier models, including M3, are used for kernel optimization, for having a model build data, and for longer-horizon cases. She says the lab harnesses that capability so daily iteration goes faster.

Asked whether M3 is already building M4, she says the current effort is building M3.1. She names no ship date and no feature list for it.

What she is watching next, and what a coder can carry away

Asked what she most recently finds exciting, Olive points to multi-agent systems and model routing. A multi-agent setup, as she is using the idea, splits harder work across more than one agent. Routing sends a piece of that work to the model suited to it. She says those setups take on more complex tasks and show what current models can do and where they fall short. She is describing an interest. The interview includes no MiniMax multi-agent product. Thomas thanks her, she thanks the audience, and the session ends in applause.

A working coder can use her split. On her account, a passive long document and a multi-round agent are different context problems. She says M3 had to restore the long window, pair it with coding and agent ability, and train vision from the first step, using MiniMax Sparse Attention plus the vision-transformer, data, and reward work she listed.

The figures that sound like a spec stay with the person who said them. Olive states two parameter pairs and does not choose, and she leaves “activated” and “active” undefined. Thomas adds a June release, leaderboard ranks, a uniqueness claim, a remark that M3 is cheap, a guess that the release was free, and a paraphrase of a model card. Olive’s reach figures include a hedge on the company count. The interview states no price and no license. Thinking effort, and an agent that learns from a YouTube tutorial, are possibilities she entertains. M3.1 is the version she says is being built.