Skip to main content

Learners : Matchbox recommender : Learner API : Evaluation

In order to evaluate finding related items, the predicted set of related items needs to be restricted to such items which have been rated by some users in common with the query item. Here we cover how to make such predictions and how to evaluate them.

Related items for evaluation can be found using the FindRelatedItemsRatedBySameUsers method of the evaluator:

var evaluator = new  RecommenderEvaluator<Dataset, User, Item, int, int, Discrete>(  
var relatedItemsForEvaluation = evaluator.FindRelatedItemsRatedBySameUsers(  

In this example trainedRecommender is a trained recommender and testDataset is the instance source of the test set. Related items will be found for each unique item in the dataset.

The user in each instance will also be queried by the data mapping. This allows a list of users to be constructed who have rated this item which can then be examined to see which other items have been rated by these users. This gives a list of “potentially related” items, and then predictions are made from this list. Ratings will not be queried by the data mapping at this stage. They will only be needed during evaluation. The parameter maxRelatedItemCount specifies the number of related items to find for each item. But if the number of possible related items for a given item is less than the value of minRelatedItemPoolSize, then the item is skipped. The last parameter allows for easy removal of items from the predictions for which there is not sufficient information for later evaluation. Another parameter which is used to control this is minCommonRatingCount - it is guaranteed that all related items have been rated by at least that many users in common with the query item.

Once the restricted related item predictions are produced, they can be evaluated using the RelatedItemMetric method of the evaluator:

var l1SimNdcg = evaluator.RelatedItemsMetric( testDataset,  
    minCommonRatingCount, Metrics.Ndcg, Metrics.NormalizedManhattanSimilarity));

The ranking metric used to evaluate related items is one of the following:

The fifth argument to the RelatedItemsMetric method is the rating similarity function, the value of which used as gain in the computation of the metrics above. It takes in two vectors and returns a real number. Pre-defined similarity functions include:

The way evaluation works is the following. First, the sets of users who rated each item are extracted from the input data. These sets are then reduced to the users who rated both items. Then, for each user in these sets the rating given to the corresponding item is taken. This forms two item rating vectors, which are used as inputs to the functions listed above.