# KDD 19': Heterogeneous Graph Neural Network

# 🤗 Recommendation system paper challenge (28/50)

Github: https://github.com/chuxuzhang/KDD2019_HetGNN

# 🤔 What problem do they solve?

**They would like to generate Heterogeneous Graph embedding consisting of graph structure information and node content information.**

# 😮 What are the challenges?

Few of them can jointly consider heterogeneous structural (graph) information as well as heterogeneous contents information of each node effectively.

- many nodes could not connect to all types of neighbors
- A node could carry unstructured content
- Different types of neighbors contributes differently to node embedding

# 😎 Overview of the models: HetGNN

They propose a heterogeneous graph neural network model to resolve this issue.

Specifically, they first introduce a **random walk** with restart strategy to **sample** a fixed size of strongly correlated heterogeneous neighbors for each node and group them based upon node types.

Next, we design a neural network architecture with two modules to aggregate feature information of those sampled neighboring nodes. The first module encodes “deep” feature interactions of heterogeneous contents and generates content embedding for each node.

The second module aggregates content (attribute) embeddings of different neighboring groups (types) and further combines them by considering the impacts of different groups to obtain the ultimate node embedding.

Finally, we leverage a graph context loss and a mini-batch gradient descent procedure to train the model in an end-to-end manner.

## Sampling Heterogeneous Neighbors (C1)

Most of other GNNs models have some issues:

- They cannot capture feature information from different types of neighbors.
- Besides that, They are weakened by various neighbor sizes.
- They are not suitable for aggregating heterogeneous neighbors which have different content features.

They propose a heterogeneous neighbors sampling strategy based on **random walk with restart (RWR)**.

- RWR collects all types of neighbors for each node
- the sampled neighbor size of each node is fixed and the most frequently visited neighbors are selected;
- neighbors of the same type (having the same content features) are grouped such that type-based aggregation can be deployed.

## Encoding Heterogeneous Contents (C2)

Given a node, it has different type neighbors.

Input: 1 neighboring node

output: 1 embedding

For each neighboring node, we would like to get its encoding but different type of node has different contents. How do we aggregate the information together? **Bi-LSTM!**

We can apply pre-trained model to get embedding of each information and feed into **Bi-LSTM (to capture deep interactions) **and then** Mean Pooling **to aggregate it.

Note that the Bi-LSTM operates on an unordered content set.

The advantages:

(1) it has concise structures with relative **low complexity** (less parameters), making the model implementation and tuning relatively easy

(2) it is capable to **fuse** the heterogeneous contents information, leading to a strong expression capability;

(3) it is flexible to add extra content features, making the model extension convenient.

## Aggregating Heterogeneous Neighbors (C3)

As C1, given 1 node, we have neighboring nodes. Thanks to C2, each neighboring node can encode to 1 embedding.

But how do we aggregate these neighboring nodes together?

Group by Type and for each type run **Bi-LSTM**!

**Same Type Neighbors Aggregation**

After grouping neighboring nodes, in each group, we have several nodes in same type. So we can run **Same Type Neighbors Aggregation.**

We employ Bi-LSTM to aggregate content embeddings of all t-type neighbors and use the average over all hidden states to represent the general aggregated embedding.

We use different Bi-LSTMs to distinguish different node types for neighbors aggregation. Note that the Bi-LSTM operates on an unordered neighbors set, which is inspired by GraphSAGE.

**Types Combination**

Again, given a node, we have neighboring nodes and we group them into several group by types. So each type, we have 1 embedding.

How to aggregate these embedding?

Attention layer

To combine these type-based neighbor embeddings with v’s content embedding, we employ the attention mechanism.

The motivation is that different types of neighbors will make different contributions to the final representation of v.

## Objective and Model Training

A **graph context loss **and a **mini-batch gradient descent**.

They applied negative sampling (1 by 1) and similar toDeepWalk, they apply random walk to get negative samples.

Finally, we have this whole picture.

# 🥴 What else in this paper?

In this paper, they also discuss several experiments.

## (RQ1) How does HetGNN perform vs. state-of-the-art baselines for various graph mining tasks, such as link prediction (RQ1–1), personalized recommendation (RQ1–2), and node classification & clustering (RQ1–3)?

Better

## (RQ2) How does HetGNN perform vs. state-of-the-art baselines for inductive graph mining tasks, such as inductive node classification & clustering?

Better

## (RQ3) How do different components, e.д., node heterogeneous contents encoder or heterogeneous neighbors aggregator, affect the model performance?

- HetGNN has better performance than No-Neigh in most cases, demonstrating that aggregating neighbors information is effective for generating better node embeddings.
- HetGNN outperforms Content-FC, indicating that the Bi-LSTM based content encoding is better than “shallow” encoding like FC for capturing “deep” content feature interactions.
- HetGNN achieves better results than Type-FC, showing that selfattention is better than FC for capturing node type impact

## (RQ4) How do various hyper-parameters, e.д., embedding dimension or the size of sampled heterogeneous neighbors set, impact the model performance?

# 🙃 Other related blogs:

KDD 19': Applying Deep Learning To Airbnb Search

KDD 17': Visual Search at eBay

KDD 18': Real-time Personalization using Embeddings for Search Ranking at Airbnb

KDD 18': Notification Volume Control and Optimization System at Pinterest

KDD 19': PinText: A Multitask Text Embedding System in Pinterest

CVPR19' Complete the Look: Scene-based Complementary Product Recommendation

COLING’14: Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts

NAACL’19: Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence

NIPS’2017: Attention Is All You Need (Transformer)

KDD’19: Learning a Unified Embedding for Visual Search at Pinterest

BMVC19' Classification is a Strong Baseline for Deep Metric Learning

KDD’18: Graph Convolutional Neural Networks for Web-Scale Recommender Systems

WWW’17: Visual Discovery at Pinterest

# 🤩 Conference

**ICCV**: *International Conference on Computer Vision*

http://iccv2019.thecvf.com/submission/timeline

**CVPR**: *Conference on Computer Vision and Pattern Recognition*

**KDD 2020**

Top Conference Paper Challenge:

https://medium.com/@arthurlee_73761/top-conference-paper-challenge-2d7ca24115c6

My Website: