add precision@k and recall@k #58

graytowne · 2017-09-07T04:34:53Z

I add two new ranking metrics: precision@k and recall@k.

There still remain some works to do:

When evaluation recommender models, one should ignore all the rated item(from training data), as they won't appear again in the future (test data).
More metrics like MAP@k and AUC still need to implement.

maciejkula · 2017-09-09T11:40:10Z

spotlight/evaluation.py

@@ -96,6 +96,60 @@ def sequence_mrr_score(model, test):
    return np.array(mrrs)


+def precision_recall_at_k(model, test, k):


Could we make k an optional argument (with a default value)?

If yes, I guess the only natural default would be None and return the P@k/R@k for the whole ranking (although the function name would become a little misleading).

This is why I usually call the function just precision/recall, and put the "at"s only in the variable names. But I guess the current name is what most people expect to find in the library because the literature chose to put the k in the metric name...

I think one name that would be consistent with the existing naming convention would be precision_recall_score.

maciejkula · 2017-09-09T11:41:03Z

spotlight/evaluation.py

+        The model to evaluate.
+    test: :class:`spotlight.interactions.Interactions`
+        Test interactions.
+    k: int or array of int,


I'm not sure where I sit on making this a list:

yes, it's more efficient;

but it is also more complex.

Could we maybe make it so that if a scalar is passed a scalar is also returned?

I think having two options make it more confuse. How about change it to only scalar or only list? I design it for list before, as in research, people usually plot experiment figures to see how performance changes when varying k.

I think doing both list and scalar version is a nice compromise.

We should set k=10 as the default value. Then

if a list is passed, return tuple of lists of arrays

if a scalar is passed (or the default value is kepy), we return tuple of arrays.

OK, got it. I'll make the changes soon.

Having said that, I would also be happy for this to be only scalar.

maciejkula · 2017-09-09T11:41:45Z

spotlight/evaluation.py

+
+    test = test.tocsr()
+
+    if not isinstance(k, list):


This should check for more possible iterables: tuples, numpy arrays and so on.

maciejkula · 2017-09-09T11:41:56Z

spotlight/evaluation.py

+        targets = row.indices
+
+        for _k in k:
+            pred = np.asarray(predictions)[:_k]


Minor: predictions is already a numpy array.

Yes, you are correct.

maciejkula · 2017-09-09T11:42:47Z

spotlight/evaluation.py

+
+        for _k in k:
+            pred = np.asarray(predictions)[:_k]
+            num_hit = len(set(pred).intersection(set(targets)))


np.in1d(targets, pred) may be faster?

I tested in my laptop(2012 Macbook) and find out that the 'set intersection' version is slightly faster than 'np in1d' version (1s vs. 1.2s) on ML100k dataset.

Here is the code:

num_hit = sum(np.in1d(pred, targets))

Sounds good.set it is.

maciejkula · 2017-09-09T11:43:27Z

tests/factorization/test_implicit.py

@@ -10,7 +10,7 @@
 from spotlight.factorization.implicit import ImplicitFactorizationModel
 from spotlight.factorization.representations import BilinearNet
 from spotlight.layers import BloomEmbedding
-
+from spotlight.evaluation import precision_recall_at_k


Would you mind adding a new separate test file for evaluation metrics?

maciejkula · 2017-09-09T11:44:39Z

spotlight/evaluation.py

+    Parameters
+    ----------
+
+    model: fitted instance of a recommender model


We should make this accept train interactions as well in the same way other metrics do.

This is so that we can disregard the model giving high scores to known interactions. This reflects real-world system implementations.

maciejkula · 2017-09-13T20:28:21Z

spotlight/evaluation.py

+    -------
+
+    (Precision@k, Recall@k): numpy array of shape (num_users,)
+        A tuple of Precisions@k and Recalls@k for each user in test.


We should be clearer about what this returns. I suggest tuple of lists of arrays (or tuple of arrays in the case of scalar k), so that the i-th element of list is the (num_users ,) array of precision/recall at k[i].

maciejkula · 2017-09-18T21:34:16Z

Thanks for the improvements! There are still one or two things I'd like changed. I hope it's OK if I make a PR to your branch in the next couple of days?

graytowne · 2017-09-18T21:39:01Z

Not a problem! Take your time, and I can add more metrics (MAP, AUC) after that.

This commit does the following: - simplify the code - improve the docs - some efficiency improvements in eliminating train interactions

Simplify and improve tests.

maciejkula · 2017-10-14T15:01:49Z

🚀

graytowne added 4 commits September 6, 2017 21:24

add precision and recall

366ecaf

add precision and recall

47b48b5

clean code

3c44b2d

fix print for python3

75bf9d5

maciejkula requested changes Sep 9, 2017

View reviewed changes

maciejkula reviewed Sep 9, 2017

View reviewed changes

Merge branch 'master' of github.com:maciejkula/spotlight

1d290ac

maciejkula reviewed Sep 13, 2017

View reviewed changes

changes for requested changes, add test file

d022c05

maciejkula and others added 2 commits October 13, 2017 22:01

Simplify and improve tests.

b14ddcd

This commit does the following: - simplify the code - improve the docs - some efficiency improvements in eliminating train interactions

Merge pull request #1 from maciejkula/precision-recall

79c43f1

Simplify and improve tests.

maciejkula approved these changes Oct 14, 2017

View reviewed changes

maciejkula merged commit 2db4845 into maciejkula:master Oct 14, 2017

beeva-ramiromanso mentioned this pull request Nov 13, 2017

Map metric #71

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add precision@k and recall@k #58

add precision@k and recall@k #58

graytowne commented Sep 7, 2017

maciejkula Sep 9, 2017

villasv Sep 13, 2017

maciejkula Sep 13, 2017

maciejkula Sep 9, 2017

graytowne Sep 13, 2017

maciejkula Sep 13, 2017

graytowne Sep 13, 2017

maciejkula Sep 13, 2017

maciejkula Sep 9, 2017

maciejkula Sep 9, 2017

graytowne Sep 11, 2017

maciejkula Sep 9, 2017

graytowne Sep 11, 2017

graytowne Sep 11, 2017

maciejkula Sep 12, 2017

maciejkula Sep 9, 2017

maciejkula Sep 9, 2017

maciejkula Sep 13, 2017

maciejkula Sep 13, 2017

maciejkula commented Sep 18, 2017

graytowne commented Sep 18, 2017

maciejkula commented Oct 14, 2017

		@@ -96,6 +96,60 @@ def sequence_mrr_score(model, test):
		return np.array(mrrs)


		def precision_recall_at_k(model, test, k):

add precision@k and recall@k #58

add precision@k and recall@k #58

Conversation

graytowne commented Sep 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maciejkula commented Sep 18, 2017

graytowne commented Sep 18, 2017

maciejkula commented Oct 14, 2017