mrr_score¶
-
ampligraph.evaluation.
mrr_score
(ranks)¶ Mean Reciprocal Rank (MRR)
The function computes the mean of the reciprocal of elements of a vector of rankings
ranks
.It is used in conjunction with the learning to rank evaluation protocol of
ampligraph.evaluation.evaluate_performance()
.It is formally defined as follows:
\[MRR = \frac{1}{|Q|}\sum_{i = 1}^{|Q|}\frac{1}{rank_{(s, p, o)_i}}\]where \(Q\) is a set of triples and \((s, p, o)\) is a triple \(\in Q\).
Note
This metric is similar to mean rank (MR)
ampligraph.evaluation.mr_score()
. Instead of averaging ranks, it averages their reciprocals. This is done to obtain a metric which is more robust to outliers.Consider the following example. Each of the two positive triples identified by
*
are ranked against four corruptions each. When scored by an embedding model, the first triple ranks 2nd, and the other triple ranks first. The resulting MRR is:s p o score rank Jack born_in Ireland 0.789 1 Jack born_in Italy 0.753 2 * Jack born_in Germany 0.695 3 Jack born_in China 0.456 4 Jack born_in Thomas 0.234 5 s p o score rank Jack friend_with Thomas 0.901 1 * Jack friend_with China 0.345 2 Jack friend_with Italy 0.293 3 Jack friend_with Ireland 0.201 4 Jack friend_with Germany 0.156 5 MRR=0.75
Parameters: ranks (ndarray or list, shape [n] or [n,2]) – Input ranks of n test statements. Returns: mrr_score – The MRR score Return type: float Examples
>>> import numpy as np >>> from ampligraph.evaluation.metrics import mrr_score >>> rankings = np.array([1, 12, 6, 2]) >>> mrr_score(rankings) 0.4375