Engineer Blog Note: Contextual relevance in ads ranking

Engineer blog link

I learn many things in that blog.

But here, I do not summarize the blog. I just take some notes I want to remember.

Using the plot to measure the model performance

Usually I prefer to only measure log loss, but it looks like it does not give me more detail. And I think how about by more detail categories, their log loss. Kind of things.

I did not notice maybe we can use this simple plot to understand the performance in different intervals. Very interesting!

Using human label relevance

Yeah, I know it is strange.

But recently I read some papers, they talk about some disadvantage of only rely on model (click, log loss). It looks like only human can clearly and true measure the relevance and if we only rely on clicks, conversion, they still have noise.

However, it takes much time and money to run it. It looks like a trade-off.

Yet, it is worth in Image Relevance case. In image case, it is not hard to measure relevance. Only reply on clicks leading more noise because most of time they are not demanding request, which means non-click does not mean irrelevant.

Tracking the relevant each stage

It is a really good plot.

Because they have higher quality of measurement (relevance from human) and it is different the output from the model. It is good to plot.

Without it, if we only have pCTR, it does not give benefit to plot it because model control pCTR, surely, after model rank, higher pCTR will be in higher rank.