Graph Attention Networks (GAT)

In a GCN, every neighbor is treated equally (or scaled only by their degree). However, in the real world, some neighbors are more important than others. Graph Attention Networks (GAT) introduce the Attention Mechanism to graphs, allowing nodes to dynamically "prioritize" which neighbors to listen to.

---

1. Why Attention on Graphs?

Consider a social network where you are a node. You might have 500 friends, but only 5 of them are your family members. In a GCN, your "family" information is diluted by the other 495 friends. GAT learns that the connections to your family should have higher attention coefficients, giving them more weight in the message-passing step.

The Key Advantages

• Dynamic Weighting: Unlike GCN, the weights are not fixed by the graph structure; they are learned based on node features.
• Anisotropy: GAT can treat different neighbors differently, even if they have the same degree.
• Inductive: Like GraphSAGE, GAT is naturally inductive because it learns an attention function that can be applied to new nodes.

---

2. How it Works: The Attention Mechanism

For every edge $(u, v)$ in the graph, GAT computes an attention coefficient $e_{uv}$:

Step 1: Compute Energy

We first compute the "energy" between two nodes using a shared linear transformation $W$ and an attention vector $\vec{a}$:

$e_{uv} = \text{LeakyReLU} \left( \vec{a}^T [W h_u \parallel W h_v] \right)$

where $\parallel$ denotes concatenation.

Step 2: Normalize (Softmax)

To make the coefficients comparable across different neighborhoods, we normalize them using a softmax function:

$\alpha_{uv} = \frac{\exp(e_{uv})}{\sum_{k \in \mathcal{N}(u)} \exp(e_{uk})}$

Step 3: Multi-Head Attention

To stabilize the learning process, GAT often uses multi-head attention. We calculate $K$ independent attention heads and either average or concatenate their results. This is similar to how Transformers work.

---

3. Implementation in PyTorch

A simplified GAT layer implementation:

gat_layer.py

import torch
import torch.nn as nn
import torch.nn.functional as F

class GATLayer(nn.Module):
    def __init__(self, in_features, out_features, alpha=0.2):
        super().__init__()
        self.W = nn.Parameter(torch.zeros(size=(in_features, out_features)))
        nn.init.xavier_uniform_(self.W.data, gain=1.414)
        
        self.a = nn.Parameter(torch.zeros(size=(2 * out_features, 1)))
        nn.init.xavier_uniform_(self.a.data, gain=1.414)

        self.leakyrelu = nn.LeakyReLU(alpha)

    def forward(self, h, adj):
        # h: [N, In_Feats]
        # adj: Adjacency matrix [N, N]
        
        # 1. Linear Transformation
        Wh = torch.mm(h, self.W) # [N, Out_Feats]
        N = Wh.size()[0]

        # 2. Compute Attention Energy (e)
        # Combine every node with every other node
        Wh_repeat = Wh.repeat_interleave(N, dim=0)
        Wh_tile = Wh.repeat(N, 1)
        combined = torch.cat([Wh_repeat, Wh_tile], dim=1) # [N*N, 2*Out_Feats]
        
        e = self.leakyrelu(torch.matmul(combined, self.a).view(N, N))

        # 3. Masked Softmax (Only consider actual neighbors)
        zero_vec = -9e15 * torch.ones_like(e)
        attention = torch.where(adj > 0, e, zero_vec)
        attention = F.softmax(attention, dim=1)

        # 4. Aggregate
        h_prime = torch.matmul(attention, Wh)
        return F.elu(h_prime)

---

4. GAT vs. GCN

Property	GCN	GAT
Edge Weights	Static ($1/\sqrt{d_i d_j}$)	Dynamic (Learned Attention)
Compute Cost	Low (Matrix Mult)	Higher (Attention per edge)
Flexibility	Limited by Topology	Highly Adaptive

Pro Tip: Attention Heads

In production, always use at least 8 attention heads. Just like in NLP, different heads tend to learn different types of relationships (e.g., one head might focus on local community structure, while another focuses on long-range semantic bridges).