Detailed explanation of offline evaluation indicators for recommendation systems

This article will introduce in detail the offline evaluation indicators commonly used in recommendation systems, including precision, recall, accuracy, F1-Score, NDCG, hit rate, AUC, GAUC and log loss. These metrics are crucial for evaluating the performance and effectiveness of recommendation systems. These metrics are crucial for evaluating the performance and effectiveness of recommendation systems.

In a recommendation system, the precision rate measures the proportion of items in the recommendation list that truly match the user's interests, and the recall rate measures the proportion of all items that match the user's interests that are successfully recommended. Accuracy measures the proportion of model predictions that are consistent with actual results. F1-Score takes precision and recall into consideration to comprehensively evaluate the model. NDCG is used to evaluate the ranking quality of recommendation systems, and is especially suitable for recommendation systems that consider element correlation ranking.

We will provide detailed explanations and calculation formulas for each indicator, and give sample code for Python implementation. The suitability of these metrics will be evaluated based on the needs of the recommendation system to help you choose the metrics that are suitable for your own system evaluation.

By gaining an in-depth understanding of these offline evaluation metrics, you will be able to better evaluate and improve your recommendation system and provide more accurate and personalized recommendation services.

1. Precision

Function

Precision is used to measure the accuracy of a classification model, that is, the proportion of samples predicted as positive by the model that are actually positive. In recommendation systems, accuracy can be understood as the proportion of items that truly match the user's interests among the recommended items received by the user.

Calculation formula

In the two-classification problem, the calculation formula of accuracy is:

$$ Precision = \frac{TP}{TP+FP} $$

Among them, TP stands for True Positives and FP stands for False Positives.

Example

Suppose a recommendation system recommends 10 products, 5 of which are of real interest to the user. Then the accuracy is: 5 / 10 = 0.5.

Python implementation

1
2
3
4
5
6
7
8

from sklearn.metrics import precision_score

# y_true 表示真实标签，y_pred 表示预测标签
y_true = [0, 1, 1, 0, 1, 1]
y_pred = [0, 1, 0, 1, 1, 1]

precision = precision_score(y_true, y_pred)
print(f'Precision: {precision}')

Applicability

⭐️⭐️⭐️⭐️⭐️

Precision is very suitable for the evaluation of recommender systems. The main goal of a recommendation system is to accurately recommend products or services that may be of interest to users. If most of the recommended products are of interest to the user, the accuracy will be high.

2. Recall

Function

Recall is used to measure the ability of a classification model to cover positive class samples, that is, among all samples that are actually positive class, the model predicts the proportion of positive class. In recommendation systems, recall rate can be understood as the proportion of items that are successfully recommended by the model among all items that match the user's interests.

Calculation formula

In the binary classification problem, the calculation formula of recall rate is:

$$ Recall = \frac{TP}{TP+FN} $$

Among them, TP stands for True Positives and FN stands for False Negatives.

Example

Assume that there are 20 products that a user is really interested in, and the recommendation system successfully recommends 10 of them. Then the recall rate is: 10 / 20 = 0.5.

Python implementation

1
2
3
4
5
6
7
8

from sklearn.metrics import recall_score

# y_true 表示真实标签，y_pred 表示预测标签
y_true = [0, 1, 1, 0, 1, 1]
y_pred = [0, 1, 0, 1, 1, 1]

recall = recall_score(y_true, y_pred)
print(f'Recall: {recall}')

Applicability

⭐️⭐️⭐️⭐️⭐️

Recall is also very suitable for the evaluation of recommender systems. One of the goals of a recommendation system is to cover as many products or services that users are interested in as possible. If most of the products that users are interested in can be recommended, the recall rate will be high.

3. Accuracy

Function

Accuracy is used to measure the proportion of the prediction results of the classification model that are consistent with the actual results, that is, the proportion of the model predictions that are correct among all samples.

Calculation formula

In the two-classification problem, the calculation formula of accuracy is:

$$ Accuracy = \frac{TP+TN}{TP+FP+TN+FN} $$

Among them, TP stands for True Positives, TN stands for True Negatives, FP stands for False Positives, and FN stands for False Negatives.

Example

Suppose a recommendation system makes predictions for 100 items, and 70 of the predictions are correct, then the accuracy rate is: 70 / 100 = 0.7.

Python implementation

1
2
3
4
5
6
7
8

from sklearn.metrics import accuracy_score

# y_true 表示真实标签，y_pred 表示预测标签
y_true = [0, 1, 1, 0, 1, 1]
y_pred = [0, 1, 0, 1, 1, 1]

accuracy = accuracy_score(y_true, y_pred)
print(f'Accuracy: {accuracy}')

Applicability

⭐️⭐️⭐️

Accuracy has low applicability in recommender systems because recommender systems often face the problem of unbalanced labels. For example, in a product recommendation system, users may not be interested in most products, so the accuracy cannot well reflect the performance of the model.

4. F1-Score

Function

F1-Score is the harmonic average of precision and recall, which is used to comprehensively evaluate the model by considering precision and recall at the same time.

Calculation formula

The calculation formula of F1-Score is:

$$ F1-Score = 2 \times \frac{Precision \times Recall}{Precision + Recall} $$

Example

Assuming that the precision rate of a recommendation system is 0.5 and the recall rate is 0.7, then the F1-Score is: 2 * (0.5 * 0.7) / (0.5 + 0.7) = 0.583.

Python implementation

1
2
3
4
5
6
7
8

from sklearn.metrics import f1_score

# y_true 表示真实标签，y_pred 表示预测标签
y_true = [0, 1, 1, 0, 1, 1]
y_pred = [0, 1, 0, 1, 1, 1]

f1 = f1_score(y_true, y_pred)
print(f'F1-Score: {f1}')

Applicability

⭐️⭐️⭐️⭐️⭐️

F1-Score is very suitable for the evaluation of recommender systems. The recommendation system needs to balance precision and recall. A high precision indicates the accuracy of the recommendation, and a high recall indicates the comprehensiveness of the recommendation. F1-Score is an evaluation index that takes into account both.

5. NDCG (Normalized Discounted Cumulative Gain)

Function

NDCG is an index used to evaluate the ranking quality of recommendation systems, especially for those that consider element relevance ranking. It measures how similar the model's predicted ranked list is to the true ranked list.

Calculation formula

The calculation formula of NDCG is:

$$ NDCG = \frac{DCG}{IDCG} $$

Among them, DCG represents the Discounted Cumulative Gain of the recommendation list, and the calculation formula is:

$$ DCG = \sum_{i=1}^{N} \frac{2^{rel_i} - 1}{log_2(i + 1)} $$

IDCG represents the maximum DCG under ideal conditions, that is, all relevant products are ranked first. The calculation formula is the same as DCG, except that the products are sorted from large to small according to their relevance.

Example

Suppose a recommendation system predicts the relevance of 5 items as [3, 2, 3, 0, 1], then DCG = (2^3 - 1)/log2(1+1) + (2^2 - 1)/log2(2+1) + (2^3 - 1)/log2(3+1) + (2^0 - 1)/log2(4+1) + (2^1 - 1)/log2(5+1). If the ideal ordering is [3, 3, 2, 1, 0], then IDCG can be calculated in the same way, NDCG = DCG / IDCG.

Python implementation

1
2
3
4
5
6
7
8
9

from sklearn.metrics import ndcg_score
import numpy as np

# y_true 表示真实的相关性，y_score 表示预测的相关性
y_true = np.array([[3, 2, 3, 0, 1]])
y_score = np.array([[3, 2, 1, 0, 3]])

ndcg = ndcg_score(y_true, y_score)
print(f'NDCG: {ndcg}')

Applicability

⭐️⭐️⭐️⭐️⭐️

NDCG is well suited for the evaluation of recommender systems, especially for systems where recommendation ranking needs to be considered. NDCG can measure the similarity between the ranked list predicted by the recommendation system and the real ranked list, thereby evaluating the performance of the model.

Due to space limitations, I will discuss the following indicators (Hit Rate, AUC, GAUC, LogLoss) next time

Continue the introduction in the reply.

6. Hit Rate

Function

The hit rate is an indicator to evaluate whether the recommendation system can recommend items that the user is interested in. It can be understood as whether the system "hits" the items that the user is interested in.

Calculation formula

The formula for calculating hit rate is:

$$ HitRate = \frac{Number\ of\ Hits}{Total\ Number\ of\ Tests} $$

Example

Suppose a recommendation system makes predictions for 10 products, and 3 of them are hit, then the hit rate is: 3 / 10 = 0.3.

Python implementation

1
2
3
4
5
6
7
8
9

def hit_rate(recommended_items, true_items):
    return int(any(item in true_items for item in recommended_items))

# 推荐的商品和真正感兴趣的商品
recommended_items = [1, 2, 3, 4, 5]
true_items = [2, 6, 7, 8, 9]

hit_rate = hit_rate(recommended_items, true_items)
print(f'Hit Rate: {hit_rate}')

Applicability

⭐️⭐️⭐️⭐️⭐️

Hit rate is very important for the evaluation of recommendation systems. Because the main goal of the recommendation system is to recommend items that the user is interested in, if the recommended items include items that the user is really interested in, the hit rate will be high.

Thank you very much for your suggestions. Below is an introduction to the modified AUC and GAUC based on your suggestions.

7. AUC (Area Under Curve)

Function

AUC is a commonly used performance evaluation index for classification problems, especially for recommendation systems. It is an index that measures the model's discrimination between positive and negative samples. For each positive sample, calculate the proportion of negative samples whose prediction score is higher than that of the positive sample, that is, the "positive sample rate" of the positive sample. AUC is the average positive sample rate of all positive samples.

Calculation formula

The calculation of AUC can be described as:

1. For each user, the recommendation system’s prediction scores for positive and negative samples are calculated.
2. For each positive sample, calculate the proportion of negative samples its prediction score is higher than, that is, the "positive sample rate" of the positive sample.
3. Calculate the average positive sample rate of all positive samples, that is, AUC.

The specific formula can be expressed as:

$$ AUC = \frac{1}{M}\sum_{i=1}^{M} \frac{1}{P_iN_i} \sum_{j=1}^{P_i} \sum_{k=1}^{N_i} I(s_{ij} > s_{ik}) $$

Among them, $M$ is the number of users, $P_i$ and $N_i$ are the number of positive samples and negative samples of user $i$ respectively, $s_{ij}$ and $s_{ik}$ are the prediction scores of positive sample $j$ and negative sample $k$ of user $i$ respectively, $I(\cdot)$ is the indicator function.

Python implementation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

from sklearn.metrics import roc_auc_score

# 对每个用户计算 AUC，然后取平均
def calculate_auc(users, y_true, y_score):
    auc_list = []
    for user in users:
        auc = roc_auc_score(y_true[user], y_score[user])
        auc_list.append(auc)
    return np.mean(auc_list)

# 用户列表、真实标签、预测得分
users = ['user1', 'user2']
y_true = {'user1': [1, 1, 0, 0], 'user2': [1, 1, 0, 0]}
y_score = {'user1': [0.9, 0.8, 0.5, 0.4], 'user2': [0.7, 0.6, 0.3, 0.2]}

auc = calculate_auc(users, y_true, y_score)
print(f'AUC: {auc}')

In particular, it is important to note that interview questions that require handwritten AUC calculations in Python are often encountered during interviews. An example is shown below, which mainly considers two-classification problems.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

def calculate_auc_manual(y_true, y_score):
    # 首先，获取所有正样本和负样本的索引
    pos_indices = [i for i, x in enumerate(y_true) if x == 1]
    neg_indices = [i for i, x in enumerate(y_true) if x == 0]

    # 计算所有正样本的分数高于多少比例的负样本
    pos_count = len(pos_indices)
    neg_count = len(neg_indices)
    count = 0
    for i in pos_indices:
        for j in neg_indices:
            if y_score[i] > y_score[j]:
                count += 1

    # 正样本率就是所有正样本的分数高于负样本的数量比上总的正负样本对的数量
    pos_rate = count / (pos_count * neg_count)
    
    return pos_rate

# 真实标签
y_true = [1, 1, 0, 0, 1, 0]
# 预测得分
y_score = [0.9, 0.8, 0.1, 0.4, 0.95, 0.6]

auc = calculate_auc_manual(y_true, y_score)
print(f'AUC: {auc}')
``````

在上述代码中，我们首先获取所有正样本和负样本的索引。然后，对于每一个正样本，我们计算其预测分数高于多少比例的负样本。最后，所有正样本的平均正样本率即为 AUC。

注意：在真实环境中，上述实现可能会非常慢，因为它需要对所有正负样本对进行比较。在实际应用中，我们通常会使用更有效的算法来计算 AUC，例如使用排序和计数方法。

### 适用性

⭐️⭐️⭐️⭐️⭐️

AUC 非常适用于推荐系统的评估。AUC 可以衡量模型对正负样本的区分能力，对于推荐系统来说，就是能否准确地找出用户感兴趣的物品。因此，AUC 是一个很好的评价指标。

------

## 8. GAUC (Group Area Under Curve)

###

功能

GAUC 是 AUC 的扩展，主要应用于推荐系统等场景。它的计算方法和 AUC 类似，只是在计算正样本率时，需要按照用户组进行计算，而不是按照单个样本进行计算。

### 计算公式

GAUC 的计算步骤如下：

1. 对于每个用户，计算推荐系统对于正样本和负样本的预测分数。
2. 对于每个用户，计算其正样本的预测分数高于多少比例的负样本，即用户的“正样本率”。
3. 计算所有用户的平均正样本率，即 GAUC。

具体的公式可以表示为：

$$ GAUC = \frac{1}{M}\sum_{i=1}^{M} \left( \frac{1}{P_iN_i} \sum_{j=1}^{P_i} \sum_{k=1}^{N_i} I(s_{ij} > s_{ik}) \right) $$

其中，$M$ 是用户数，$P_i$ 和 $N_i$ 分别是用户 $i$ 的正样本数和负样本数，$s_{ij}$ 和 $s_{ik}$ 分别是用户 $i$ 的正样本 $j$ 和负样本 $k$ 的预测分数，$I(\cdot)$ 是指示函数。

### Python 实现

Python 实现需要根据具体的数据情况进行，以下是一个基本的示例。

```python
from sklearn.metrics import roc_auc_score

# 对每个用户计算 AUC，然后取平均
def calculate_gauc(users, y_true, y_score):
    auc_list = []
    for user in users:
        auc = roc_auc_score(y_true[user], y_score[user])
        auc_list.append(auc)
    return np.mean(auc_list)

# 用户列表、真实标签、预测得分
users = ['user1', 'user2']
y_true = {'user1': [1, 1, 0, 0], 'user2': [1, 1, 0, 0]}
y_score = {'user1': [0.9, 0.8, 0.5, 0.4], 'user2': [0.7, 0.6, 0.3, 0.2]}

gauc = calculate_gauc(users, y_true, y_score)
print(f'GAUC: {gauc}')

Applicability

⭐️⭐️⭐️⭐️⭐️

GAUC is also well suited for the evaluation of recommender systems, especially those where individual differences need to be taken into account. By calculating each user's AUC and averaging it, GAUC can more fully reflect the performance of the recommendation system.

Applicability

⭐️⭐️⭐️⭐️⭐️

GAUC is also well suited for the evaluation of recommender systems, especially those where individual differences need to be taken into account. By calculating each user's AUC and averaging it, GAUC can more fully reflect the performance of the recommendation system.

9. LogLoss (Logarithmic Loss)

Function

LogLoss is a measure of the loss function of a classification model that takes into account the probability value predicted by the model. For binary classification problems, the smaller the value, the better the performance of the model.

Calculation formula

The calculation formula of LogLoss is:

$$ LogLoss = -\frac{1}{n}\sum_{i=1}^{n}[y_i\log(p_i) + (1 - y_i)\log(1 - p_i)] $$

Example

Assume that the predicted probabilities of a recommendation system for three samples are [0.8, 0.6, 0.3], and the real labels of these three samples are [1, 1, 0], then LogLoss can be calculated by substituting the formula.

Python implementation

1
2
3
4
5
6
7
8

from sklearn.metrics import log_loss

# y_true 表示真实标签，y_pred 表示预测的概率
y_true = [1, 1, 0]
y_pred = [0.8, 0.6, 0.3]

logloss = log_loss(y_true, y_pred)
print(f'LogLoss: {logloss}')

Applicability

⭐️⭐️⭐️⭐️

LogLoss is suitable in the evaluation of recommendation systems, but it may not be the most important metric. LogLoss pays more attention to whether the probability value predicted by the model is accurate, while the recommendation system also needs to consider other factors, such as coverage, novelty, etc., in addition to the accuracy of prediction. Therefore, LogLoss can be used as an auxiliary indicator to measure recommendation systems.