RecSys11: OrdRec: an ordinal model for predicting personalized item rating distributions
Recommendation system paper challenge (9/50)
Why this paper?
RecSys’11 Best Paper.
What problem do they solve?
A top-n recommender system with rating feedback
What others solve this problem?
Most researchers utilized rating as numerical view. And predict the rating considering weighted (temporal effect, similar users). However, in several common scenarios, there is no direct link between the user feedback and numerical values, even though the feedback is richer than a binary “like-vs-dislike” indication.
purchasing the product > bookmarking or wish-listing > search and browse
and rating does not mean too much as we expect.
Another scenario is when users are asked to enter their feedback by a comparative ranking of a set of products.
They argue that user rating depends on user internal scale.
What is their model?
They viewing user feedback on products as ordinal as OrdRec model, a point-wise ordinal approach, letting it scale linearly with data size.
An important property of OrdRec is an ability to output a full probability distribution of the scores.
They introduce S − 1 ordered thresholds, associated with each of the rating values besides the last.
First a random score z_ui is generated from a normal distribution centered at the internal score
They replace accumulated normal distribution to logistic function.
What is the Data?
RMSE (root mean squared error):
Decent solutions in RMSE terms can contain no personalization power ranking-wise. For example, on the Netflix dataset a predictor explaining only rating biases could get much better RMSE than others.
Yet, All user-dependent biases play no role when ranking items for a single user, while the item-related biases are not personalized. Thus, it will yield the same item ranking for all users.
FCP (Fraction of Concordant Pairs):
It indicates that the ratio of correctly order. A measure that generalizes the known AUC metric into non-binary ordered outcomes
ESTIMATION OF RECOMMENDATION CONFIDENCE
- It can improve user trust in the system and altering user behavior by adding confidence
- When the system pick among several items with the same expected rating, it can favor the item with higher confidence
In OrdRec, since the model output the distribution, we can easily employ standard deviation, entropy, or Gini for that distribution.
They formulate the binary classification to evaluate the level of confidence in the predictions of the model. For example, in the Netflix dataset, if the model’s prediction is 3.5 stars and the true rating is 4 stars, then the model is within 1 rating level, whereas if the true rating is 5 stars, then it is not.
By adding the feature from OrdRec, the AUC becomes much better.
What is their contribution?
- A CF framework treating user ratings as ordinal rather than numerical, thereby being directly applicable to a wider variety of systems.
- Flexibly associating different semantics to the available scores, depending on the user.
- Predicting the full probability distribution of the scores rather than a single score.
- Enhancing and integrating with many known CF methods.
- New methods and evaluation metrics for assessing confidence in recommendations
Other related blogs:
Best paper in RecSys: