https://www.pinterest.com/pin/647744358911454390/

KDD 21':Understanding and Improving Fairness-Accuracy Trade-offs in Multi-Task Learning (google)

🤗 Recommendation system paper challenge (31/50)

🤔 What problem do they solve?

所以這篇paper在於解決 fairness on MTL的問題

😎 Contribution

定義新的metric:

  • ARFG: 對每個task去計算MTL中的FPRGap跟STL的FPRGap的比值,在平均起來
  • ARE: 對每個task去計算MTL中的error rate跟STL的error rate的比值,在平均起來

baseline 方法: Per-Task Fairness Treatment

how to measure fairness loss?

  • minimizing the correlation between group membership and the predictions over negative examples (讓每個task中的negative sample的正確性之間的相關性最小)
  • kernel-based distribution matching through Maximum Mean Discrepancy (MMD) over negative examples (讓每個task的MMD最大)
  • minimizing FPR gap directly in the loss (goal)

直覺想法,讓每個task的正確性預測越相近,他們的FPR就越接近

提出方法: Multi-Task-Aware Fairness Treatment (MTA-F)

baseline 跟MTA-F比較

可以發現只要只是Fairness的F改了,對於shared layer我們想用大家共同的data去train,對於task-specific layer,我們只想用task-specific data去跑

舉例來說, 從下表中,如果我們針對task1去找shared-negative-data跟specific-negative-data

  • specific-negative-data: 只有他negative,其他task是positive,所以是 E
  • shared-negative-data: task1 negative而且扣掉specific-negative-data,所以是 (EFGH — E) = FGH

用了以上的data,我們就可以跑training

以下是詳細的algorithm

😮 Background:

  • 業務(given a image,我們不只想知道這是男生女生,還想知道他快樂與否)
  • data (我們有task A, taskB, 也有一個大data可能有些沒有label for task A, or for task B), 因此想到我們也許可以把這個data直接餵給task A, task B一起去optimize

Fairness metric

  • all subgroups receive the same proportion of positive outcomes: 通常在Demographic parity的case
  • equal opportunity and equalized odds: equal TPR (true positive rates) and FPR (false positive rates) across different subgroups, 更加務實,此篇論文用這metric

Fairer representation learning

  • fairer representation learning

Fairness mitigation

  • single-task learning setting (pre-processing the data embeddings for downstream job, post-processing model’s prediction)
  • intervening the model training process has also been popular, including adding fairness constraints or regularization

Multi-task learning

pros:

  • exploits task relatedness with inductive bias learning (learning a shared representation across related tasks is beneficial for harder tasks or tasks with limited training examples.) 對難的問題跟限制data很有幫助
  • forcing tasks to share model capacity (regularization) ->避免overfitting,更general
  • proving compact and efficient form of modeling which enables training and serving multiple prediction quantities for large-scale systems -> serving time更scalable system

cons:

  • 可能讓task1好但task2不好
  • 現存方法是reducing the task training conflicts 還有improving the Pareto frontier

Fairness in multi-task learning

  • fairness in multi-task regression models and uses a rank-based non-parametric independence test to improve fairness in ranking
  • multi-task learning enhanced with fairness constraints to jointly learn classifiers that leverage information between sensitive groups

Fairness-accuracy trade-off and Pareto fairness

  • fairness跟accuracy有 a trade-off 關係 for single task
  • 各個accuracy trade-off cross tasks

幾個key questions

  • How does inductive transfer in multi-task learning implicitly impacts fairness?
  • How do we measure the fairness-accuracy trade-off for multi-task learning
  • Are we able to achieve a better Pareto efficiency in fairness across multiple tasks, by exploiting task relatedness and shared architecture that’s specific to multi-task learning?

心得

而fairness也越來越受到重視,希望model盡可能學習不同subgroup的metric讓我們的model不會太bias在某些group上

這兩種trade-off都是很具有挑戰性的,作者試著把這兩個trade-off去結合成一個新的問題,再提出metric跟方法來解決,屬於modeling paper (比較沒有涉及serving, production的部分),對於學習modeling來說,是個不錯的範本

結語

🙃 Other related blogs:

KDD 19': Sampling-bias-corrected neural modeling for large corpus item recommendations

KDD 18': Notification Volume Control and Optimization System at Pinterest

CVPR19' Complete the Look: Scene-based Complementary Product Recommendation

NAACL’19: Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence

NIPS’2017: Attention Is All You Need (Transformer)

KDD’19: Learning a Unified Embedding for Visual Search at Pinterest

BMVC19' Classification is a Strong Baseline for Deep Metric Learning

KDD’18: Graph Convolutional Neural Networks for Web-Scale Recommender Systems

🤩 Conference

http://iccv2019.thecvf.com/submission/timeline

CVPR: Conference on Computer Vision and Pattern Recognition

http://cvpr2019.thecvf.com/

KDD 2020

https://www.kdd.org/kdd2020/

Top Conference Paper Challenge:

https://medium.com/@arthurlee_73761/top-conference-paper-challenge-2d7ca24115c6

My Website:

https://light0617.github.io/#/

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store