A visual guide to inference LLM: Exploring inference-time computing techniques with DeepSeek-R1

Original address：A Visual Guide to Reasoning LLMs

📅 Author: Maarten Grootendorst

📆 Date: February 3, 2025

📌 Introduction

How DeepSeek-R1, OpenAI o3-mini and Google Gemini 2.0 Flash Thinking use the "inference" framework to LLM (Large Language Models) A prime example of scaling to new heights.

They mark the Extended train-time compute Arrive Extended inference time compute (test-time compute) paradigm shift.

In this article we provide Over 40 custom visualizations, take you to explore in depth:

• Reasoning LLMs field
• Test-Time Compute mechanism
• DeepSeek-R1 core idea

We'll introduce concepts step-by-step to help you build an intuitive understanding of this new paradigm.

📖 What is inference LLM?

with ordinary LLM (Large Language Models, large language models) Compared toReasoning LLM Before answering a question, the question is often Break it down into smaller steps(often called Reasoning Steps or Thought Process）。

🧠 What are the “reasoning steps” or “thinking processes”?

While we can get philosophical about whether LLMs can actually think like humans, these reasoning steps actually break down the reasoning process into smaller, more structured inferences.Inference LLM uses a structured inference method, that is:

• Ordinary LLM: Directly output the answer
• Reasoning LLM: Generate answers through systematic reasoning

In other words, inference LLM is notLearn “what to answer”, butLearn "How to Answer"！

To understand how inference LLM is built, we first need to explore Train-Time Compute and Test-Time Compute the difference between.

🔍 What is Training-time Compute?

Until mid-2024, in order to Pretraining To improve the performance of LLMs, researchers typically scale up:

• Number of model parameters (# of Parameters)
• Dataset size (# of Tokens)
• Calculation amount (# of FLOPs, Floating Point Operations)

These are collectively called Train-time Compute, that is “The Fossil Fuel of AI”, refers to:

The larger the pre-training budget, the better the final model will be.

Training-Time Compute includestrainingthe calculations required, andfine-tuningrequired calculations. Improving LLM performance has long been a major focus.

🔢Scaling Laws

in LLM (Large Scale Language Model) research areas,Model scale (Scale) with Model performance (Performance) The relationship between is called Scaling Laws. These laws are often used to describe Computing resources, data size, and model parameters How it affects the overall performance of the model.

These relationships are usually Log-log (log-log) way, and usually appears as a line on the chart Approximate straight line, to highlight the huge increase in computational effort.

This picture showsThe impact of different coordinate scales (linear vs. logarithmic) on the relationship between computing resources (Compute) and model performance (Performance), emphasizing the power law relationship (Power Law) of large model growth.

• Left picture (normal linear scale - Normal Scale)

  • Horizontal axis (X-axis): Computing resources (Compute),linear scale。
  • Vertical axis (Y-axis): Performance,linear scale。
  • Curve displayDiminishing Returns, that is:As computing resources increase, performance growth slows down, but still rising.

• Right graph (log-log scale - Log-log Scale)

  • Horizontal axis (X-axis): Computing resources (Compute),Logarithmic scale。
  • Vertical axis (Y-axis): Performance,Logarithmic scale。
  • On a log-log scale, the originally curved curve becomesa straight line, indicating the relationship between computing resources and performancePower Law Relationship。

These laws usually follow Power Laws, that is:

An increase in one variable (such as the amount of computation) will cause another variable (such as performance) to change in a certain proportion.

most famous law of scale Includes:

• Kaplan's Law of Scale(Kaplan Scaling Law): When computing resources are certain,Increasing the parameter size of the model is more effective than increasing the data size. It shows that there is a power law relationship between model performance and the amount of parameters, calculation amount and training data (Tokens), that is, more parameters and more computing resources can improve performance (proposed by GPT-3 paper).
• Chinchilla's Law of Scale(Chinchilla Scaling Law): The size of the model is equally important as the size of the data. Both require Synchronous expansion To achieve the best performance (proposed by DeepMind).

This picture showsScaling Laws in large-scale AI training, indicatingComputing resources (Compute), data set size (Dataset Size) and parameter amount (Parameters)impact on model performance. The key information is as follows:

1. Vertical axis (Y-axis): Test Loss

• The goal is to reduce test loss (Test Loss), that is, improving the generalization performance of the model.
• The smaller the loss (L), the better the model performance。

2. Horizontal axis (X-axis): three key variables

• Left picture (Compute, computing resources)：

  • The X-axis is computing resources (PF-days, non-embedding).
  • With more computing resources, test loss decreases (performance increases).
  • Formula:
    $$
    L = \left( \frac{C_{\text{min}}}{2.3 \times 10^8} \right)^{-0.050}
    $$
  • Reflect the power law relationship of computing resources: Computational resources increase, losses decrease, but diminishing returns (index -0.050).

• Middle picture (Dataset Size, data set size)：

  • The X-axis is the number of Tokens in the training data.
  • The larger the data size, the lower the test loss (performance improvement).
  • Formula:
    $$
    L = \left( \frac{D}{5.4 \times 10^{13}} \right)^{-0.095}
    $$
  • Data size has a greater impact on losses(Index -0.095).

• Right picture (Parameters, parameter amount)：

  • The X-axis is the model parameter quantity (non-embedding).
  • The larger the number of parameters, the lower the test loss (the performance increases).
  • Formula:
    $$
    L = \left( \frac{N}{8.8 \times 10^{13}} \right)^{-0.076}
    $$
  • The impact of parameters on loss lies between computing resources and data size(Index -0.076).

These studies show thatModel size, data size, and computing resources must scale together to maximize model performance。

• Increased computing resources → train more powerful models
• More Tokens → Better generalization ability
• Parameters increase → but need to match the data, otherwise they will be overfitted

Kaplan's law of scale states that in fixed computing resources In the case ofPrioritize adding model parameters Often more efficient than increasing data size. And Chinchilla's Law of Scale states,Both model parameters and data size should grow simultaneously, to obtain better model performance.

However, in 2024, the researchers found that despite computational resources, data size, and model parameters continued growth, but the performance is improved Marginal Return But there gradually decrease。

This raises an important question:

❓ “Have we reached a bottleneck in LLM development?”

🚀 What is Test-time Compute?

due to The cost of computation during training is extremely expensive, researchers began to pay attention to Test-time Compute, that is:

Let LLM "think longer" when reasoning, rather than relying solely on larger models and data sets.

fornon-inferential model, they usually Directly output the answer：

1
2

Q: 8 + 5 = ?
A: 13

Andinference modelwill Use more tokens for inference, forming a systematic "thinking" process:

1
2

Q: 8 + 5 = ?
A: 8 + 5 可拆解为 8 + 2 + 3 = 10 + 3 = 13

LLM requires consuming computing resources (such as video memory computing) to generate answers. However, it would be inefficient if all computing resources were used to generate answers directly!

Instead, by generating more tokens in advance that contain additional information, relationships, and new ideas, the model can allocate more computing resources during inference to generate the final answer.

This picture shows Large Language Model (LLM) How to distribute during calculation token(mark) to optimize reasoning ability and final answer quality. The core idea is:If computing resources (such as GPU/VRAM calculations) are all used to directly generate answers and not used to think, then efficiency will be affected. Instead, increase thought process(i.e. generating more intermediate tokens), which can improve the performance of the model reasoning ability, thereby improving final answer quality。

1. Token usage and calculation amount

• LLM generates answers that are output gradually by token., each token will occupy computing resources.
• Allocate more tokens for thinking, meaning that the model can have more reasoning steps before arriving at the final answer, thereby improving the accuracy.

2. Three different calculation methods

• Scenario 1 (1 token: minimum calculation)

  • direct output “5” as an answer.
  • Minimal computational effort, the fastest.
  • If the problem is more complex, something may go wrong, because the model doesn't have enough computational time to think.

• Scenario 2 (6 tokens: medium calculation)

  • The model generates a brief inference process:

    1	Adding 3 and 2 gives 5

  • It requires more calculations than the first method, but still simpler.
  • This approach is suitable forSimple mathematical operations or logical reasoning, but errors may still occur in more complex cases.

• Scenario 3 (15 tokens: complete reasoning)

  • The model first performs detailed step-by-step reasoning:

    1	3 + 1 = 4 , 4 + 1 = 5

    The model then summarizes explicitly:

    1	the total is 5

  • The reasoning process is more detailed and takes up the largest amount of calculations.。
  • Suitable for tasks that require multi-step reasoning, such as math questions, logical reasoning questions, etc.。

🔢Scaling Laws

Compared to training-time calculations, the scaling laws for inference-time calculations are still relatively new. It is worth noting that two studies revealed the relationship between the computational scale during inference and the computational scale during training.

First, an article published by OpenAI suggests that inference-time computation may follow the same scaling trend as training-time computation.

Annotated figure from the article "LLM for learning how to reason": The red dashed line shows that the new paradigm proposed by OpenAI may be inference-time computing.
This picture shows Training time calculation (train-time compute) and test time calculation (test-time compute) pair model pass@1 accuracy The impact of Test-time calculations may be more beneficial for scaling model performance than training-time calculations。

1. Left: Training time calculation vs. accuracy

   • X-axis (horizontal axis): training time calculation (log scale, exponential scale)。
   • Y axis (vertical axis): pass@1 accuracy rate(i.e. the probability that the model gets the correct answer on one try).
   • black dot It represents the model performance under different calculation amounts, and the pink dotted line shows the general trend.
   • It can be seen that with As the amount of training calculations increases, the accuracy gradually improves., but the growth trend is relatively stable.

2. Right: Test time calculation vs. accuracy

   • X-axis (horizontal axis): test time calculation (log scale)。
   • Y axis (vertical axis): pass@1 accuracy rate。
   • Similarly, the black dots represent the model performance under different calculation amounts, and the pink dotted line shows the general trend.
   • Here you can see that with As the amount of calculation increases during testing, the accuracy of the model increases more significantly, even exceeding the effect of the amount of training calculation.。
     Therefore, they argue that expansion of computation during inference may represent a new research paradigm.

Secondly, a paper titled "Scaling Scaling Laws with Board Games" studied AlphaZero's performance in playing Hex games under different computational loads.

Annotated diagram from the article "Scaling Scaling Laws with Board Games": This diagram shows how they build training-time computation and inference-time computation at different scales. - AlphaZero Yes DeepMind developed one Reinforcement Learning (RL) Trained AI.

• This algorithm passes self-play Training, no need for human rule input, you can master itGo, chess, shogi and other games。
• it combines Neural Network Prediction and Monte Carlo Tree Search (MCTS, Monte Carlo Tree Search) to make decisions.

This picture shows AlphaZero algorithm inTraining phase (train-time compute) and test phase (test-time compute)Different applications of computing resources. Mainly emphasized:

• during training: depends onMore parameters and longer training timeto optimize the model.
• during testing: rely on Deeper tree search to improve decision-making capabilities.

Annotated diagram from the article "Scaling Scaling Laws with Board Games": This diagram shows the relationship between training-time calculations and inference-time calculations.
Research results show that training-time calculations and inference-time calculations are closely related. Each dashed line represents the minimum amount of computation required to achieve a specific ELO score.
1. Meaning of coordinate axes

• X-axis (horizontal axis): amount of calculation during training (Train-time Compute, FLOP-seconds)
• Y axis (vertical axis): amount of calculation during inference (Test-time Compute, FLOP-seconds)
• Log scale: The amount of computation increases exponentially rather than linearly.

2. Key data trends

• Curves of different colors representDifferent ELO score levels（-1500、-1250、-1000、-750、-500、-250）。
• dashed and solid lines：
  • dashed line Represents the optimal calculation boundary under a certain ELO score.
  • solid line Represents actual data trends.

1. Training computation and inference computation can be substituted for each other

   • If the amount of inference calculation increases (upper left area), then the amount of training computation required is reduced.
   • If the amount of training calculation increases (lower right area), then the amount of inference calculations required is reduced.
   • There is a negative correlation between the two。

2. Low training compute vs. high inference compute

   • in Less computation for training (such as the red circle on the left), the model can still achieve the same ELO level, but it requires Increase the amount of computation during inference(such as deeper search trees, longer thinking paths).

3. High training compute vs. low inference compute

   • in Sufficient training calculations In the case of (such as the red circle on the right), the model canReduce inference computing requirements, that is Even with less search depth, still get high performance。

4. Formula explanation

   • Formula:
     $$
     \log_{10}(\text{test compute}) = -1.2 \cdot \log_{10}(\text{train compute}) + 0.004 \cdot \text{elo} + 29
     $$
   • This means:
     • When the training compute (train compute) increases, the inference compute (test compute) decreases (coefficient -1.2)。
     • Higher ELO (stronger AI) requires additional calculations (factor 0.004)。

With inference-time computation scaling similar to train-time computation, the research paradigm is moving in the direction of "inference" models leveraging more inference-time computation. With this paradigm shift, these "inference" models no longer focus purely on training-time computation (pre-training and fine-tuning), but balance training and inference.

Inference-time calculations can even scale with length:

This is something we will also explore in the DeepSeek-R1 study!

📌 Categories of Test-time Compute

inference models (e.g. DeepSeek-R1 and OpenAI o1)'s success demonstrates that there are more optimization techniques than simply "thinking longer" during inference.

In this article we will explore Test-time Compute Various implementation methods, including:

• Chain-of-Thought
• Revising Answers
• Backtracking
• Diversity Sampling
• Other methods

Overall, inference time calculations can be summarized as follows two categories：

1. Search against Verifiers

   • Pass Sample multiple answers And Choose the best answer to optimize reasoning.

2. Modifying Proposal Distribution

   • through training "Thinking" process to improve reasoning skills. Proposal Distribution refers to adjusting the probability distribution of different possible answers when the model generates answers.

Essentially:

• Validator-based search pay more attention to Output quality（Output-focused）。
• Modify proposal distribution Follow input structure（Input-focused）。

🔍 Two main validator types

To better screen and evaluate reasoning answers, we introduce two Verifiers：

1. Outcome Reward Models (ORM)

   • Only the final answer is graded, not the reasoning process.

2. Process Reward Models (PRM)

   • Both the final answer and the reasoning process are scored.

In the next section, we will explore in detail How to apply ORM and PRM to different validation methods！

As the name suggests,Outcome Reward Model (ORM) Only the final answer quality is evaluated, without focusing on the reasoning behind the answer:

• The ORM only looks at the final output and doesn't care how the model arrived at this answer.

In contrast,Process Reward Model (PRM) then the reasoning process itself is evaluated:

• PRM evaluates both the correctness of answers and the plausibility of reasoning paths.

🧐 How does PRM assess the reasoning process?

To illustrate the importance of the inference step more clearly, let's look at an example:

1
2
3
4
5

问题：某个方程的解是多少？

推理步骤 1：首先展开方程，得到 x = 3。
推理步骤 2：错误地将 x = 3 改写为 x = 5。
推理步骤 3：最终输出 x = 5。

In the above example, although the final answer (x = 5) is wrong, the ORM only evaluates the final output and does not pay attention to the incorrect reasoning in the middle.

Or in this case, PRM would find Reasoning step 2 is wrong, and give this step a low score to avoid wrong answers.

🔍 ORM vs. PRM application in reasoning

now you have it Outcome Reward Model (ORM) and Process Reward Model (PRM) differences between them, we next explore how to apply them to various Verification Techniques。

📌Search against Verifiers

The first major category of calculations during inference is Validator-based search, which usually consists of two steps:

1. Generate multiple reasoning procedures and answer samples
2. Score the generated output using a validator (reward model)

🤖The role of validator

A validator is typically a large language model (LLM) fine-tuned to evaluate an outcome (ORM) or process (PRM). A major advantage of using a validator is that there is no need to retrain or fine-tune the large language model (LLM) used to answer the question, only the best answer is selected through a scoring mechanism.

✅ Majority Voting

The simplest way is Does not use reward models or validators, but execute Majority Voting。

📌 Method: Let LLM generate multiple answers and select the answer that appears most often as the final answer.

📌 Example:

1
2
3
4
5

Q: 15 × 3 = ?
A1: 45
A2: 42
A3: 45
最终答案: 45（因其出现频率最高）

This method is also called Self-Consistency, emphasize Generate multiple answers and reasoning steps importance.

🔢 Best-of-N Samples

Best-of-N sampling is the first method involving a verifier. Its basic idea is to generate N sample answers, then use a reward model (RM) to score these answers and select the answer with the highest score.

📌 Steps:

1. Generate multiple answers(Generate N samples using higher or different temperature parameters).
2. Outcome Reward Model (ORM, Outcome Reward Model), each answer is scored via the ORM. The answer with the highest score is selected as the final output. 📌 Example:

1
2
3
4
5

Q: 8 + 5 = ?
A1: 12 (得分 0.2)
A2: 13 (得分 0.9)
A3: 14 (得分 0.4)
最终选择: A2（因其得分最高）

📌 Further optimization:
- If used PRM, then not only the answer is evaluated, but the entire reasoning process is evaluated. Unlike the Outcome Reward Model (ORM), which only evaluates the final answer, the Process Reward Model (PRM) evaluates the quality of the reasoning process. PRM focuses on each step of reasoning, ensuring that the reasoning process is reasonable and coherent, and ultimately selects the candidate answer with the highest overall score.

- Weighted Best-of-N samples: Combining the two verification methods of ORM and PRM, we can weighted scores for all candidate answers and select the answer with the highest total weight. This method is called Weighted Best-of-N samples:.

🚀 Beam Search using Process Reward Model (PRM)

In the process of generating the answer and its intermediate reasoning steps, we can use Beam Search Further optimize the reasoning path.

📌 The core idea of beam search:

• During the reasoning process, multiple possible reasoning paths (called "bundles") are generated.
• Use Process Reward Model (PRM, Process Reward Model) Score each path.
• Similar to Tree of Thought method, always retaining the highest scoring Top 3 reasoning paths, and continue to track these paths during inference.
• If a path has a low score (low PRM score), the inference path is stopped early to avoid unnecessary computational overhead.

📌 Optimized answer filtering method:
Ultimately, all answers generated will use Best-of-N sampling The method performs weighted scoring to ensure that the final answer with the best reasoning path is selected.

🚀 Advantages:

• Avoid wasting computing resources and quickly eliminate low-quality reasoning paths.
• Combined with PRM, it can ensure that the model's reasoning process is more coherent and logical.
• The answer quality is further optimized through the Best-of-N method, making the final answer more reliable.

🎲 Monte Carlo Tree Search (MCTS)

Monte Carlo Tree Search, MCTS) is an algorithm commonly used for decision tree search, and this method can also be used in inference optimization of LLM. MCTS optimizes the inference path through four steps:
📌 Main steps:

1. Selection: Select a leaf node from the current search tree to expand according to a predefined formula.
2. Expand: Create new child nodes based on the selected leaf nodes to explore more possible reasoning paths.
3. Simulation (Rollouts): By randomly generating new reasoning paths, the nodes continue to be expanded until the end point is reached (that is, the final answer is obtained).
4. Backpropagation: Update the score of the parent node based on the final output result to optimize future search decisions.

During the inference process of large language models (LLM), we usually hope to find the best inference path so that the final answer generated is optimal. But in this process, it is necessary to Exploration and Exploitation Strike a balance between:

• Exploitation: Select the path that currently appears to be optimal, taking advantage of known high-quality inference steps.
• Exploration: Select paths with fewer visits to discover potentially better reasoning steps.

Selection Score

When choosing an inference path, we use Selection Score Calculate the priority of each inference step (that is, the node of the tree), the formula is as follows:

$$
\text{Selection Score} = \frac{\text{Total Node Reward}}{\text{Number of Node Visits}} + C \times \sqrt{\frac{\text{Number of Parent Node Visits}}{\text{Number of Node Visits}}}
$$

Among them:

• first item：$$\frac{\text{Total Node Reward}}{\text{Number of Node Visits}}$$ (Exploitation Term)

  • Total Node Reward: The cumulative reward value obtained by this node (indicating its historical performance).
  • Number of Node Visits: The number of times this node has been visited.
  • This calculation is the node's average reward value, nodes with high rewards will be selected first.

• Second item：$$C \times \sqrt{\frac{\text{Number of Parent Node Visits}}{\text{Number of Node Visits}}}$$ (Exploration Term)

  • # of Parent Node Visits: The number of times the parent node has been visited.
  • # of Node Visits: The number of times the current node has been visited.
  • C: A hyperparameter that controls the balance between exploration and exploitation.
  • This encourages exploration of less frequently visited nodes to prevent premature trapping in local optimal solutions.

Summary:

• The first item (Exploitation Term) Let the algorithm favor the choice Paths with better historical performance。
• Second item (Exploration Term) Let the algorithm tend to Explore less visited paths, to avoid falling into local optimality.
• Parameter C Control the balance between the two.

2. Selection and Expand

At this stage, we use Select score To decide which reasoning path is worth continuing to expand:

(1) Selection

• Input: Question
• LLM generates multiple reasoning steps (Reasoning Steps)
  • For example, in the picture, LLM generates 3 inference steps:
    • Thought 1(Rating 0.4)
    • Thought 2(Rating 0.2)
    • Thought 3(Rating 0.1)
• Select the optimal path using Selection Score(random initialization)
  • In the example, the highest rated Thought 1（0.4） is selected.

(2)Expand

• Generate new reasoning steps on the selected reasoning path
• The initial value of these new inference steps is set to 0, indicating that they have not been evaluated yet.

This process is similar to Expansion stage of MCTS, that is:

1. Select the current optimal path (using Select score）。
2. Under this path, new inference steps (unscored child nodes) are expanded.

3. Rollouts (simulation) and Backpropagation (backpropagation)

Once we have extended the inference step, we need to continue exploring and exploit Simulation (Rollouts) and Backpropagation to optimize the entire search process.

(3) Rollouts (simulation)

• After selecting the path, we continue to expand the inference steps until generate final answer。
• This process is similar to Randomly simulate the game to the end in MCTS：
  • We start from the current node and perform a series of inferences until the model generates the final answer.
  • In the picture, we continue the reasoning steps along Thought 1 (0.4).
  • These reasoning steps will ultimately Generate multiple answers(Purple box in the picture).

(4) Backpropagation

• by pairing final answer To score, we can update the scores of all previous nodes participating in inference:
  • PRM（Process Reward Model）: Score the reasoning step itself to measure its reasonableness.
  • ORM（Output Reward Model）: Score the final answer to measure its correctness.
  • These ratings spread upward, update All nodes passed through reward value.
• For example:
  • In the picture, the rating of the final answer results in Thought 1 's rating improved from 0.4 to 0.8。
  • propagates further upward, making The selection score of the parent node is also updated.。

This process ensures:

• Better reasoning paths will gradually earn higher scores, increase the probability of being selected.
• Poorer reasoning paths will be gradually eliminated, to avoid wasting computing resources.

📌 Modifying Proposal Distribution

Modifying Proposal Distribution

During the inference process of a large language model (LLM), we can optimize the model's inference capability by modifying the proposal distribution (Modifying Proposal Distribution). The core idea of this method is:

• No longer rely solely on models to search for correct inference steps(output-based optimization),
• Instead, the model actively generates better inference steps.(input-based optimization).

In other words, instead of testing the results after outputting them, we directly modify how the model selects tokens during the inference process, making it more inclined to choose tokens that can guide the inference rather than immediately outputting the final answer. Modified the probability distribution used for sampling completions, thoughts or tokens. This approach can make the answers generated by the model more accurate, interpretable, and more robust in the face of complex problems.

1. Directly select the Token with the highest probability (Greedy selection)

By default, LLM generates multiple possible tokens as output candidates, sorts them according to their probabilities, and finally selects the token with the highest probability for output. This method is calledGreedy Selection。

As you can imagine, we have a question and a probability distribution for sampling tokens. A common strategy is to choose the token with the highest score.

• For example, given the question What is 3 + 2?, LLM may generate the following candidate tokens:
  • 5(highest probability)
  • 3
  • Adding
  • 4
  • If
• Under the greedy strategy, the model will directly choose 5 as a final answer without reasoning.

Although this method is fast, it has the following problems:

• lack of reasoning ability: The model may directly output the wrong answer because it does not perform inference.
• Poor interpretability: For complex problems, users cannot understand how the model arrives at the answer.

2. Improve answer quality through Reasoning Before Answering

However, please note that there are sometokensMarked red. These tokens are more likely to guide the model into a reasonable reasoning process. Although it is not necessarily wrong to choose the token with the highest score under the greedy strategy, choosing those tokens that can guide the model into the inference process will usually lead to better answers.
Let LLM Make inferences first, then give answers, that is:

• Choose an inference token (e.g. Adding）
• Generate reasoning process step by step, such as:
  • Adding → 3 and 2 gives → 5
  • If → 3 + 1 = 4, 4 + 1 = 5 → 5
  • The total is → 5
• Deduced step by step through the chain of reasoning 5, compared to directly selecting 5, this method is more interpretable and can perform better on complex problems.

3. Guide the reasoning process by modifying the proposal distribution (Re-Ranking Token Probabilities)

when weModify the proposal distribution (probability distribution of token)when, actuallyre-rankThis distribution makes "inference-related" tokens more likely to be selected.
Under this approach, we adjust the proposal distribution of the LLM so that it is more inclined to select inference tokens rather than directly selecting answers:

• By default,5 has the highest probability, and Adding、If The probability of waiting for inference token is low.
• By modifying the proposal distribution, we improve Adding、If The probability makes the model tend to make inferences.

4. How to modify the proposal distribution?

There are two main ways:

1. By Prompt Engineering
   • Modify Prompt to guide the model to generate inference steps.
   • For example:

     1 2	Q: What is 3 + 2? A: Let's think step by step.

2. The training model is more inclined to inference
   • During the fine-tuning process, more training data with inference chains are provided to allow the model to get used to generating inference tokens.

Summary

• Greedy Selection: Fast, but lacks reasoning and poor interpretability.
• Reasoning Before Answering: Improve answer quality and interpretability.
• Modifying Proposal Distribution: Adjust the probability of token selection so that the model is more inclined to select inference tokens and improve the rationality of the overall answer.

This method is inMathematical calculations, logical reasoning, legal reasoning and other tasksis particularly important, making LLM Not only can you "answer correctly", but you can also "explain clearly"。

Prompting

As we use prompt engineering(prompt project) to improve the output, we will try to improve the performance of the model by updating the prompt (prompt). This process may also push the model to show some of the things we saw earlierreasoning(reasoning) process.

1. Change Proposal Distribution

changing proposal distribution, we can provide the model with examples (also called in-context learning), allowing it to mimic a similar reasoning style when generating answers. The figure below shows an example situation:

• Illustrated content: On the left is a simple question “What is 3 + 2?”. “Thoughts” is used inside the model to represent the hidden thinking process, such as:
  1. First, 3 and 1 gives 4.
  2. Then, 4 and 1 gives 5.
  3. I believe the answer is 5.
• Answer(Answer): 5
• On the right side, bars or squares of different colors such as red and blue represent different parts of the reasoning process, indicating that some of them belong to the hidden reasoning process (red), as well as the output results or several intermediate steps (blue).

Through similar examples, the model may imitate similar formats when inferringreasoningand give the final answer.

2. The impact of “Let’s think step-by-step”

We can also simplify the above process by using “Let’s think step-by-step” directly in the prompt. This will change the model proposal distribution, let LLM(Large Language Model) tends to think in steps before answering. As shown below:

• Illustrated content: Change the prompt here to “Let’s think step-by-step”, and the question is still “What is 3 + 2?”.
• The model generates a more explicit reasoning process (illustrated by the red block) and then outputs the correct answer 5.
• The whole idea is similar to Figure 1, but it highlights the impact of "step-by-step thinking" on the generation of the final answer.

However, this does not mean that the model itself has internalized this reasoning ability—itDid not learn fundamentallyTo "reflect" or "correct" mistakes. If the model's initial reasoning process is wrong, then in this static and linear process, it will tend to continue this error instead of correcting its own reasoning.

STaR（Self-Taught Reasoner）

except through prompting(Tip) Let the model temporarily demonstrate inference steps. We can also let the model be rewarded for "generating correct inference steps" during training, so that it can truly "learn" to infer. This usually requiresLarge amounts of data with inference processestraining on and combined with reinforcement learning(reinforcement learning) to reward specific behaviors.

A very controversial ("much-debated") technique is STaR, that is Self-Taught Reasoner. It is to let LLM Generate your own inference data and use this data to train the modelFine tuning（fine-tuning) process.

1. STaR process overview

• This diagram summarizes how STaR works:
  1. Generate reasoning + answer: The model first generates a segment for the input question reasoning(reasoning) and a answer(answer);
     2a. If the answer is correct (Correct answer), then Question, Reasoning, Answer Added to the triplet dataset as training samples (3a);
     3b. Use these triplet data to perform supervised fine-tuning(Supervised fine-tuning), allowing the model to learn to produce correct reasoning and answers in similar situations.

If the model gives a wrong answer, another path is triggered:

• When the model answer to (2b) is wrong, we provide the correct answer as hint(hint) and let the model think about "why this answer is correct";
• That is Generate reasoning only (why this answer is correct?)；
• The new reasoning obtained will still be added to the triplet data, and then performed supervised fine-tuning。

The key takeaway here is that with this approach we canexplicitTrain the model “how it should reason” rather than just letting it temporarily imitate the reasoning process. We need to carry out the reasoning method of the modelsupervise（supervised fine-tuning), thereby "imbuing" the model with the reasoning pattern we want.

2. Automatically generate synthetic training samples

The whole process of STaR is very interesting because itAutomatically generate synthetic training samples（synthetic training examples). These samples not only contain questions and answers, but also a series of reasoning steps that can help the model better learn how to "think." In other studies (e.g. DeepSeek R-1), we can use these synthetic samples todistillation（distill, meaning "refining and retaining key information") reasoning process to other models. That is, a model that has mastered reasoning can help another model learn similar reasoning faster.

Key points:

• Prompting(Tip) It can affect the output style and thinking process of the model. For example, using “Let’s think step-by-step” allows the model to explicitly give inference steps, but it does not guarantee that the model automatically corrects errors.
• STaR（Self-Taught Reasoner) and other methods passGenerate inference data, supervised fine-tuning and reward mechanisms, helping the model truly learn to think and answer questions according to the specified reasoning method.
• No matter which method is used, it can be regarded as proposal distribution Adjustment: either temporary when promptednudge(Guidance), or training from the root of the training to let the model internalize this reasoning process.
• take advantage of in-context learning Providing examples enables the model to imitate the reasoning style.
• use reinforcement learning orSupervise fine-tuning（supervised fine-tuning) can enable the model to gradually master the reasoning pattern we expect.
• STaR The method automatically collects "correct reasoning" data and conducts training, making the model more likely to produce correct and required reasoning steps in subsequent answers.

DeepSeek-R1

1. Introduction

DeepSeek-R1 It is a major version in the field of reasoning models, and its weights have been open sourced. It works directly with OpenAI’s o1 Inference models compete and have a significant impact in this field.

The DeepSeek project is elegantly integrating inference capabilities into its base model (DeepSeek-V3-Base), using a variety of techniques to accomplish this goal.

Interestingly, the project does not rely on additional verifiers during the training process, and does not rely solely on supervised fine-tuning to refine reasoning behavior. On the contrary,Reinforcement Learning (RL) played an important role in it.

Below we will explore together how they train reasoning behavior in the model!

2. DeepSeek-R1 Zero: Critical Exploration for Inference

on the way to DeepSeek-R1 On the road, there is a place called DeepSeek-R1 Zero The experimental model laid the foundation for this breakthrough. it starts from DeepSeek-V3-Base Starting from this, we do not use large-scale supervised fine-tuning to add inference data at all, but only rely on reinforcement learning to gain reasoning abilities.

Training process and system prompts (Prompt)

In this process, they first prepared a very direct prompt (prompt), which was similar in form to the system prompt (system prompt), and was used as part of the inference pipeline. Relevant tips are shown below. Please note that it is explicitly stated that the reasoning process is to be written in <think> Within the label, the answer should be written in <answer> within the label, but does not further specify how the reasoning process should be specifically presented or organized.

In the picture above, you can see a simplified version of the conversation example (System prompt and User prompt) and how the model combinesreasoning(reasoning) put <think> Within the label, placeanswer(answer) put <answer> within the label. This diagram highlights the constraints on the model in the prompt:

• “The assistant first thinks about the reasoning process in the mind and then provides the user with the answer.”
• Request to use <think> make inferences, use <answer> to answer.

No other examples or templates of the "inference process" format are provided here - it is entirely up to the model itself to figure out how to output "Chain-of-Thought" style reasoning text during training.

Reinforcement Learning Rewards

In training, two rule-based reward mechanisms are used:

1. Accuracy rewards
   Rewards are given by testing whether the answers given are correct. If the answer output by the model is correct, the reward will be increased.
2. Format rewards
   reward model pair <think> and <answer> Proper use of labels.

The reinforcement learning algorithm they used is called Group Relative Policy Optimization（GRPO）. The intuitive idea of ​​this algorithm is to make all decisions that lead to a right or wrong answer easier or harder to reoccur. These decisions may include certain sequences of tokens generated by the model, or they may include the reasoning steps themselves (i.e., the thinking process). A schematic diagram of this training phase is given below.

In the figure, the two types of rewards used in the RL (reinforcement learning) process are highlighted:

• “is <think> used?" - for use <think> Label and score.
• “is <answer> used?" - for use <answer> Label and score.

In addition, there are answerscorrectnessAccuracy reward. The loop shown by the arrow in the figure represents the continuous iterative updating of the model during training, making it more and more inclined to the correct inference method and comply with the format requirements.

The emergence of spontaneous reasoning behavior

It's worth mentioning that the researchers didn't provide the model with any examples to tell it <think> How the content in the label should be written or expanded. They simply tell the model:

“It should use <think> tags, and nothing more!”

By conducting actions related to “Chain-of-Thought”indirect reward(That is, as long as the reasoning is correct and the correct format is used, more complete reasoning content is encouraged to be output). The model spontaneously learns to write longer and longer reasoning processes during training, and it is easier to produce correct answers.

The above figure shows the trend that the inference length output by the model during the training process gradually becomes longer as the number of training steps increases. The vertical axis is the average length of each response, and the horizontal axis is the number of training steps. It can be seen that the curve is climbing upward as a whole, which shows that the model continues to tend to output longer and more detailed thinking content (Chain-of-Thought), and therefore obtains higher rewards. This approach shifts most of the computing consumption from the training phase (train-time compute) to the inference phase (test-time compute), that is, a longer thinking process is generated only during inference.

According to research, they found that through this training strategy, the model can discover the optimal Chain-of-Thought style of thinking on its own and demonstrate advanced reasoning capabilities, such as:self-reflection and self-verification。

However, DeepSeek-R1 Zero's model output still has some problems, such as poor readability and sometimes mixed languages. For further refinement at the production or release level, the researchers proposed another option, which is used in the official version DeepSeek R1。

3. Learn more about DeepSeek-R1

to build DeepSeek-R1, the author carried out the following five key steps:

1. Cold Start
2. Reasoning-oriented Reinforcement Learning
3. Rejection Sampling
4. Supervised Fine-Tuning
5. Reinforcement Learning for all Scenarios

Next we will explain in turn.

Step One: Cold Start

In the first step, the researchers first used a high-quality inference data set of about 5,000 tokens. DeepSeek-V3-Base Fine-tune to avoid producing poorly readablecold start problem. This fine-tuning step can make the model's output more readable and avoid confusing inference text at the beginning. A schematic diagram of this process is shown below.

In the picture you can see:

• “DeepSeek-V3-Base” passedSupervised Fine-Tuningmethod, introducing about 5000 high-quality reasoning samples.
• These samples includeReasoning(reasoning) andAnswer(Answer) Two kinds of parts.
• The purpose of this step is to "prevent cold starts", that is, to allow the model to master basic readability reasoning at the beginning.

Step 2: Inference-guided reinforcement learning

After obtaining a preliminary fine-tuned model (the result of the previous step), the author uses DeepSeek-V3-Zero A similar reinforcement learning process is used to train the model, but with the addition oftarget language consistencyincentives to ensure that the model does not mix multiple languages when reasoning and answering.

In addition to the previously mentioned accuracy (accuracy reward) and format (format reward), etc., there are alsolanguage rewardTo ensure that the generated language style or language type is consistent, so as to avoid the phenomenon of "mixed Chinese and English" or "unstable style".

• Format reward: Still paying attention <think> and <answer> of use.
• Accuracy reward: Check whether the answer is correct and whether it can pass the corresponding "unit test".
• Language reward: Check whether the language is consistent, fluent and meets the target language requirements.

These rewards are combined to progressively optimize the model's reasoning and answers for readability, accuracy, and language style through reinforcement learning (RL) loops.

Step 3: Reject sampling

At this stage, the author usesStep 2The model obtained after reinforcement learning is used to generate large-scaleSynthetic inference data, and cooperate with DeepSeek-V3-Base The model is used for "evaluation" and "rule filtering", ultimately generating approximately 600,000 high-quality reasoning samples that can be used for subsequent supervised fine-tuning. At the same time, they also generated about 200,000 additionalnon-inference sample, including data on various tasks such as writing, simple questions and answers, self-awareness, and translation. This process is summarized below.

• Shown on the leftDeepSeek-V3-2 How to sample a large amountReasoning(reasoning) andAnswer(answer), reusing rule-based filtering and DeepSeek-V3-Base judgment (judging the quality of generated content) and retaining better quality inference data (about 600,000 items).
• Shown on the rightnon-inferentialThe (non-reasoning) data sampling process comes from a part of the data used by DeepSeek-V3-Base, with a total of about 200,000 items. This data mainly involves writing, factual QA, self-awareness, translation, etc.

As a result, the researchers obtained "mixed" data with a size of about 800,000 items, including both inference samples and non-inference samples.

Step 4: Supervise and fine-tune

After obtaining the above 800,000 pieces of data, the researchers again DeepSeek-V3-Base Supervise and fine-tune. The specific process is shown in the figure below.

• In the picture, we see that "DeepSeek-V3-Base" is used to executeSupervised Fine-Tuning, using the 800,000 items mentioned aboveHigh-quality inference and non-inference samples。
• This stage allows the model to learn broader and more diverse reasoning forms and task forms based on larger-scale data.

Step 5: Reinforcement Learning in All Scenarios

After the supervised fine-tuning was complete, the researchers continued with something like DeepSeek-R1-Zero method RL (reinforcement learning) training. However, in order to make the model more consistent with human preferences, they introduced more “helpfulness and harmlessness” Reward signals are used to constrain the model's answers.

At the same time, the model is also requiredSummarize the reasoning process, to prevent long, hard-to-read inference text from being displayed in the final output. This step solves the readability issues mentioned earlier.

1. Format reward
   • Is it used correctly? <think> Label writing reasoning content
   • Is it used correctly? <answer> label output answer
2. Accuracy reward
   • Test whether the output compiles (“does it compile?”)
   • Does it pass unit tests? (“does it pass unit tests?”)
3. Preference rewards
   • Follow Helpfulness、Harmlessness、Human preference Wait
   • These preference indicators are evaluated by the RM (Reward Model) module

As can be seen in the picture,Reasoning(reasoning) stage and Answer(Answer) stages need to be used separately <think> and <answer> labels for clear distinction. At the same time, in order to output more concise and readable content, the model may also generate a Summary(Summary) Snippet. The iterative process of reinforcement learning considers multiple reward signals simultaneously to continuously update the model and obtain the final version. DeepSeek-R1。

In the picture above, "RM" is the Reward Model, which is used to score preferences (such as whether the conversation is friendly, whether it meets ethical requirements, etc.), and then feeds the results back to the model.

“And that’s it!"This means DeepSeek-R1 actually DeepSeek-V3-Base It is further optimized through Supervised Fine-Tuning and reinforcement learning (RL). A lot of work goes into ensuringhigh quality dataGeneration and use, and then train such a model with powerful reasoning capabilities.

Distill inference knowledge from DeepSeek-R1 to other models

DeepSeek-R1 own 671 billion (671B) parameters. It is difficult to run a model of this scale on ordinary consumer-grade hardware. For practical reasons, the authors studied how to DeepSeek-R1 The inference capabilities are “distilled” into smaller models (e.g. Qwen-32B) so that it can be deployed and used on consumer-grade hardware.

Distillation Process: Teacher-Student Framework

During the distillation process,DeepSeek-R1 As a teacher model (Teacher), and smaller models (such as Qwen-32B) as a student model (Student). When both face the same prompt (prompt), they will output a set oftoken probability distribution. During training, the student model will try to learn and be close to the output distribution of the teacher model.

• The teacher (DeepSeek-R1) gives his own "proposal distribution". For example, when answering "What is 3 + 2?", the teacher model may tend to output tags such as "Adding", "If", "5", "3" and "4", and assign different probabilities to each.
• The student (Qwen-32B) will continuously update its probability distribution during training to make it closer to the teacher's distribution.

additional explanation：

1. Probability distribution: When the language model generates the next token, it will output probability estimates for all possible tokens.
2. distillation: By comparing the distribution differences between teachers and students, students will gradually adjust their own parameters to make their output closer to the style and reasoning tendency of the teacher model.

The data used for training is exactly what was mentioned before 800,000 high-quality samples——Contains about 600,000 inference samples and 200,000 non-inference samples. The diagram below illustrates this data flow:

• left side Reasoning(reasoning) and Answer(Answer) data, totaling 800,000 items.
• by DeepSeek-R1(Teacher) generate or evaluate to obtain the corresponding probability distribution.
• student model Qwen-32B Then learn according to the distribution of teachers, and finally get a distilled version DeepSeek-R1-Distill-Qwen-32B。

additional explanation：

• The student model not only learned the input-output pattern of the 800,000 samples themselves, but also learned DeepSeek-R1 The inference strategies and probability distributions that are "tended" to be used when facing these data, so as to reproduce similar reasoning capabilities on smaller models.
• “Distilled” models tend to find a better balance between inference quality and computing resources: although they may be slightly inferior to the teacher model in performance, they can still achieve satisfactory results on most common tasks and require lower resources.

Other unsuccessful attempts

During the research process, the DeepSeek team also tried Process Reward Models（PRMs） and Monte Carlo Tree Search（MCTS） and other methods to inject reasoning capabilities, but the results are not ideal:

1. Use MCTS

   • The main problem faced is that the search space is too large, and node expansion can only be strictly restricted. In this way, the effect will be greatly reduced.
   • In addition, it is also quite difficult to refine the Reward Model.

2. Best-of-N strategy using PRMs

   • If you continuously retrain the Reward Model to prevent the model from "reward hacking" behavior, it will bring high computational overhead.

These results do not mean that these technologies are ineffective, but that there are still many limitations and difficulties in their practice on current large-scale language models.DeepSeek-R1 The reason for success depends more onReinforcement learning + supervised fine-tuningcombination, as well as the mining and utilization of large-scale high-quality data.

Summary and Outlook

So far, we have roughly reviewed DeepSeek-R1 The journey of reasoning training. I hope the above can give you a better understanding:

• Test-time compute (calculation during inference) Better results can be achieved by outputting a longer and more sophisticated thinking process (Chain-of-Thought) through the model.
• large scale"Supervise fine-tuning first, then reinforce learning” training process, anddistillationThe technical route to smaller models also shows how to strike a balance between hardware resources and inference performance.

As mentioned before,DeepSeek-R1 Various reward mechanisms are introduced, especially for formatting and human preference, to ensure that answers are both correct and readable. The method of "summarizing the reasoning process" (Summary) has also greatly improved the readability problems caused by the excessive length of the plain text Chain-of-Thought.

More resources

if you are right Large Language Models（LLMs） If you are interested in reasoning topics in , the following resources are worth referring to:

1. The Illustrated DeepSeek-R1
   • A high-quality visual guide by Jay Alammar that details the principles and implementation details behind the DeepSeek-R1 model.
2. A blog post by Hugging Face
   • The focus is on how to expand the amount of calculation in the inference phase, and interesting experiments are given.
3. Video “Speculations on Test-Time Scaling”
   • Dives into the technical details commonly used for various computational extensions during the inference phase.

In addition, the author also mentioned a book on large-scale language models in the article, which contains more visualizations and experimental results. It is a good resource for friends who want to further study inference LLMs to read in depth.

• Official Website of the Book: llm-book.com
• Amazon purchase link: Hands-On Large Language Models: Understanding, Building, and Optimizing LLMs
• GitHub code repository: handsOnLLM/Hands-On-Large-Language-Models

Acknowledgments

Thank you for reading this article about DeepSeek-R1 introduction document. By interpreting all the pictures and text content in sequence, and providing more explanations of the key technologies involved in each link, we hope to make you more aware of DeepSeek-R1 A more comprehensive understanding of the training process, distillation methods and unsuccessful attempts.

In the future, with the improvement of hardware performance and the emergence of more mature training technology,deep reasoningwithmodel distillationIt will surely play a huge role in more practical application scenarios. Let’s wait and see!