How to calculate the perplexity (PPL) of a fixed-length model

Perplexity Level (PPL) is one of the most commonly used metrics for evaluating language models. Before we delve deeper, we should note that this metric applies specifically to traditional language models (sometimes called autoregressive or causal language models), and is not clearly defined for masked language models like BERT (seeModel summary）。

Perplexity is defined as the exponential average negative log-likelihood of a sequence. If we have a tokenized sequence $X = (x_0, x_1, \dots, x_t)$, then the perplexity of $X$ is,

$$
\text{PPL}(X) = \exp \left{ -\frac{1}{t}\sum*{i=1}^t \log p\theta (xi|x*{<i}) \right}
$$

where $\log p\theta (x_i|x{<i})$ is the log-likelihood of the i-th token, conditional on the previous token $x_{<i}$ according to our model. Intuitively, it can be thought of as evaluating the model's ability to predict uniformity over a specified set of tokens in the corpus. Importantly, this means that the tokenization procedure directly affects the perplexity of the model, which should always be considered when comparing different models.

This is also equivalent to the exponentialization of the cross-entropy between data and model predictions. For more intuition about perplexity and its relationship to bits per character (BPC) and data compression, check out this article on The GradientGreat blog post。

Calculating PPL with fixed-length models

If we were not limited by the size of the model context, we would evaluate the model's perplexity by autoregressively decomposing the sequence and conditioning on the entire preorder subsequence at each step, as shown in the figure below.

However, when dealing with approximate models, we are often limited by the number of tokens the model can handle. For example,GPT-2The largest version of has a fixed length of 1024 tokens, so when $t$ is larger than 1024, we cannot directly calculate $p\theta(x_t|x{<t})$。

Instead, sequences are typically broken down into subsequences equal to the model's maximum input size. If the maximum input size of the model is $k$, then we approximate the likelihood of a token $x_t$ by conditioning on only the first $k-1$ tokens (instead of the entire context). When evaluating the perplexity of a model sequence, a tempting but suboptimal approach is to decompose the sequence into disjoint chunks and accumulate the decomposed log-likelihood of each segment independently.

This calculation is fast because the perplexity for each segment can be calculated in a single forward pass, but it is a poor approximation of fully decomposed perplexity and generally yields a higher (worse) PPL because the model has less context during most prediction steps.

Instead, a sliding window strategy should be used to evaluate the PPL of fixed-length models. This involves repeatedly sliding the context window so that the model has more context with each prediction it makes.

1. Infinite context decomposition: If there were no constraints on the model input length, we could use the entire preorder subsequence at each step to predict the next token. This provides the most accurate assessment of the model's performance because each prediction takes into account all previous information.

2. Fixed length limit: In practice, most models such as GPT-2 have a fixed input length limit (e.g. 1024 tokens). When the sequence length exceeds this limit, the conditional probability of each token cannot be directly calculated because the entire sequence cannot be used as a condition.

3. Block approximation: One way to deal with long sequences is to break the sequence into multiple subsequences equal to the maximum input length of the model. Each subsequence is evaluated individually, but this approach may lead to higher perplexity since the full context is not used.

4. Sliding window strategy: To better utilize the available context, a sliding window strategy can be used. This approach attempts to provide the model with more contextual information at each prediction by continuously moving the context window, thereby getting closer to the ideal situation of using full context.

5. Stepped sliding window: A practical compromise is to use a strided sliding window, which can provide sufficient context for each model prediction while ensuring a certain efficiency, thereby improving the calculation of perplexity and the accuracy of model predictions.

These methods are all designed to solve the problem of not being able to directly evaluate the entire sequence due to model input length limitations, trying to make the evaluation more accurate through different techniques, while taking into account the efficient use of computing resources.