mrr_score

ampligraph.evaluation.mrr_score(ranks)

Mean Reciprocal Rank (MRR)

The function computes the mean of the reciprocal of elements of a vector of rankings ranks.

It is used in conjunction with the learning to rank evaluation protocol of ampligraph.evaluation.evaluate_performance().

It is formally defined as follows:

\[MRR = \frac{1}{|Q|}\sum_{i = 1}^{|Q|}\frac{1}{rank_{(s, p, o)_i}}\]

where \(Q\) is a set of triples and \((s, p, o)\) is a triple \(\in Q\).

Note

This metric is similar to mean rank (MR) ampligraph.evaluation.mr_score(). Instead of averaging ranks, it averages their reciprocals. This is done to obtain a metric which is more robust to outliers.

Consider the following example. Each of the two positive triples identified by * are ranked against four corruptions each. When scored by an embedding model, the first triple ranks 2nd, and the other triple ranks first. The resulting MRR is:

s        p         o            score   rank
Jack   born_in   Ireland        0.789      1
Jack   born_in   Italy          0.753      2  *
Jack   born_in   Germany        0.695      3
Jack   born_in   China          0.456      4
Jack   born_in   Thomas         0.234      5

s        p         o            score   rank
Jack   friend_with   Thomas     0.901      1  *
Jack   friend_with   China      0.345      2
Jack   friend_with   Italy      0.293      3
Jack   friend_with   Ireland    0.201      4
Jack   friend_with   Germany    0.156      5

MRR=0.75
Parameters

ranks (ndarray or list, shape [n] or [n,2]) – Input ranks of n test statements.

Returns

mrr_score – The MRR score

Return type

float

Examples

>>> import numpy as np
>>> from ampligraph.evaluation.metrics import mrr_score
>>> rankings = np.array([1, 12, 6, 2])
>>> mrr_score(rankings)
0.4375