DeepSeek-R1

DeepSeek-R1 is a reasoning model released by DeepSeek in January 2025, built on DeepSeek-V3-Base: 671 billion total parameters, 37 billion active per token. Its weights are published under the MIT licence.

It earns a page of its own for one result. Its predecessor DeepSeek-R1-Zero was trained to reason by reinforcement learning alone — starting from a base model that had never been shown a single worked example of step-by-step reasoning, with no supervised fine-tuning stage at all. Long chains of thought, self-correction and the habit of checking an answer before committing to it were not demonstrated to the model. They appeared during training, because they earned reward.

In September 2025 the work was published in Nature, making it the first major open-weight large language model to go through independent peer review.

What was actually trained#

Two models, and conflating them loses the result.

DeepSeek-R1-Zero is the pure experiment: reinforcement learning applied directly to DeepSeek-V3-Base “without any SFT data” — no supervised fine-tuning (SFT), the stage where a model is shown worked examples to imitate. It is the model that demonstrates the claim.

DeepSeek-R1 is the shipped product. R1-Zero worked but was unpleasant to read — the paper says plainly that it “struggles with challenges like poor readability, and language mixing,” switching between languages mid-thought. R1 adds a four-stage pipeline around the same idea: a small cold start fine-tune on a few thousand curated examples, reasoning-oriented reinforcement learning, a round of rejection sampling and supervised fine-tuning, then a final reinforcement-learning stage covering all uses.

So the honest summary is that pure reinforcement learning produced the capability, and a conventional pipeline was still needed to produce a usable assistant.

The three choices that make it work#

The reward is a rule, not a model#

Ordinary reinforcement learning from human feedback trains a second neural network to score answers, then optimises against that scorer. R1-Zero does not. Correctness is checked mechanically: maths answers must be written inside a box so they can be string-matched against the known answer, and code is compiled and run against test cases.

The paper gives the reason, and it is a practical one rather than an aesthetic one:

We do not apply the outcome or process neural reward model in developing DeepSeek-R1-Zero, because we find that the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process, and retraining the reward model needs additional training resources and it complicates the whole training pipeline.

A learned scorer is itself a model with flaws, and a policy optimised hard enough against it will find them — producing answers that score well and are wrong. A compiler has no such flaws to find. The cost of this choice is that it only works where correctness is mechanically checkable, which is why the result is demonstrated on mathematics and code rather than on essays.

The prompt template constrains form, not content#

The training template requires the model to put its working between <think> tags and its answer after them. It says nothing whatever about how to think. The paper is explicit that this was deliberate:

We intentionally limit our constraints to this structural format, avoiding any content-specific biases—such as mandating reflective reasoning or promoting particular problem-solving strategies—to ensure that we can accurately observe the model’s natural progression during the RL process.

(RL there is reinforcement learning.)

This is what makes the result a finding rather than an artefact. Had the template instructed the model to reflect and check its work, the emergence of reflection and checking would prove nothing.

The algorithm drops the critic#

Proximal policy optimization, the standard algorithm here, needs a value network — a second model, usually about the size of the one being trained, that estimates how good a partial answer is, to serve as a baseline for whether a given result was better or worse than expected.

Group relative policy optimization (GRPO) removes it. For each question it samples a group of answers from the current model, scores them all, and uses the group’s own mean as the baseline:

1

A_i = (r_i - mean(r_1 … r_G)) / std(r_1 … r_G)

An answer is reinforced to the extent it beat the other answers to the same question. The saving is an entire model’s worth of memory and compute at training time, which is what makes reinforcement learning at this scale affordable. GRPO comes from DeepSeek’s earlier DeepSeekMath paper, not from the R1 paper.

What emerged#

The behaviour the paper reports is that response length grows on its own. Nothing in the reward mentions length:

the thinking time of DeepSeek-R1-Zero shows consistent improvement throughout the training process. This improvement is not the result of external adjustments but rather an intrinsic development within the model… This computation ranges from generating hundreds to thousands of reasoning tokens.

And the specific behaviours:

Behaviors such as reflection—where the model revisits and reevaluates its previous steps—and the exploration of alternative approaches to problem-solving arise spontaneously. These behaviors are not explicitly programmed but instead emerge as a result of the model’s interaction with the reinforcement learning environment.

On AIME 2024, a competition mathematics benchmark, R1-Zero’s score on a single attempt rose from 15.6% to 71.0% over training. R1 itself reaches 79.8%, and 97.3% on MATH-500.

A caveat on the “aha moment”#

The paper includes a widely quoted passage describing an “aha moment” in an intermediate checkpoint, illustrated with a trace containing the line “Wait, wait. Wait. That’s an aha moment I can flag here.”

This is worth reading carefully. It is a single selected trace from one intermediate version, not a measurement, and the paper’s prose around it is unusually anthropomorphic — it describes “the power and beauty of reinforcement learning” and calls the observation “an aha moment for us” as well. The underlying claim about emergent reflection is supported by the length curves and the benchmark movement. The anecdote is an illustration of it, and reproducing the anecdote as though it were the evidence overstates what was shown.

The distilled models#

DeepSeek also trained six smaller dense models on R1’s own outputs, and published them. The result is that a 32-billion-parameter dense model scores 72.6% on AIME 2024 — above OpenAI’s o1-mini at 63.6% — without any reinforcement learning of its own. Reasoning behaviour, once it exists in a large model, transfers into a small one by ordinary fine-tuning on its outputs.

The bases are not DeepSeek models. Four are built on Qwen and two on Llama:

This has a licensing consequence that catches people out. R1 is MIT-licensed; the distills are not. Each inherits its base model’s terms, so R1-Distill-Llama-70B carries Llama’s community licence and its restrictions. “R1 is MIT” is true of R1 and false of half the models named after it.

Three senses of “distillation”#

The word does three different jobs in the story around this model, and conflating them is the commonest error in coverage of it:

1. What DeepSeek published — training smaller models on R1’s outputs, described above, entirely conventional and documented in the paper.
2. What OpenAI alleged — that DeepSeek trained on outputs harvested from OpenAI’s models. This is disputed and unconfirmed.
3. Self-distillation — a model trained on its own filtered outputs, which appears in R1’s own pipeline at the rejection-sampling stage.

Only the second is contested. A sentence about “distillation” that does not say which one it means is not saying anything.

Status#

Checked 11 September 2026. R1 is superseded: R1-0528, from May 2025, was the last model released under the R name, and DeepSeek has since merged reasoning back into its general line as a per-request effort setting rather than a separate model. There is no R2.

The weights remain available and the paper remains the clearest published account of training reasoning by reinforcement learning, which is why the page is here.

Sources#

• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv:2501.12948 (2025)
• Guo et al., Nature 645(8081), 633–638 (18 September 2025) — the peer-reviewed version
• Shao et al., DeepSeekMath, arXiv:2402.03300 (2024) — introduces GRPO
• DeepSeek-R1 repository — weights, licence terms and the distilled-model base list