Manifold Attention with MAtt#

This tutorial demonstrates how to use MAtt (Manifold Attention Network) for EEG classification. MAtt applies attention mechanisms on the SPD manifold to weight temporal segments by their discriminative importance.

Introduction#

MAtt [Pan et al., 2022] introduces attention mechanisms on the SPD manifold:

  1. Patch-based Processing: Segments the signal into temporal patches

  2. Covariance per Patch: Computes SPD matrices for each segment

  3. Manifold Attention: Weights patches using Log-Euclidean distances

  4. Aggregation: Combines weighted SPD matrices for classification

This allows the model to focus on the most discriminative time periods within each trial, improving classification and interpretability.

Setup and Imports#

import warnings

import matplotlib.pyplot as plt
import torch

from braindecode import EEGClassifier
from moabb.datasets import BNCI2014_001
from moabb.paradigms import MotorImagery
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import LabelEncoder
from skorch.callbacks import EpochScoring, GradientNormClipping
from skorch.dataset import ValidSplit

from spd_learn.models import MAtt


warnings.filterwarnings("ignore")

Loading the Dataset#

dataset = BNCI2014_001()
paradigm = MotorImagery(n_classes=4)

print(f"Dataset: {dataset.code}")
print("Paradigm: 4-class motor imagery")
Choosing from all possible events
Dataset: BNCI2014-001
Paradigm: 4-class motor imagery

Creating the MAtt Model#

MAtt architecture:

  1. Spatial Conv: Learns spatial filters

  2. Temporal Conv: Extracts temporal features

  3. Patch Embedding: Segments into n_patches temporal windows

  4. Covariance + TraceNorm: SPD matrix per patch

  5. AttentionManifold: Queries, keys, values on SPD manifold

  6. ReEig + LogEig: Project to tangent space

  7. Linear: Classification

The attention mechanism computes:

\[\text{attention}(Q, K) = \text{softmax}\left(\frac{1}{1 + \log(1 + d_{LE}(Q, K))}\right)\]

where \(d_{LE}\) is the Log-Euclidean distance.

n_chans = 22
n_outputs = 4

model = MAtt(
    n_chans=n_chans,
    n_outputs=n_outputs,
    n_patches=6,  # Number of temporal segments
    temporal_out_channels=32,  # Temporal feature dimension
    temporal_kernel_size=25,  # ~100ms at 250Hz
    temporal_padding=12,  # Keep time dimension
    attention_in_features=32,  # Input to attention (must match temporal_out_channels)
    attention_out_features=24,  # Output from attention
)

print("MAtt Architecture:")
print(model)
MAtt Architecture:
MAtt(
  (add_extra_dim): Rearrange('batch chan time -> batch 1 chan time')
  (spatial_filter): Conv2d(1, 22, kernel_size=(22, 1), stride=(1, 1))
  (spatial_batch_norm): BatchNorm2d(22, eps=1e-05, momentum=0.1, affine=True, bias=True, track_running_stats=True)
  (temporal_feature_extractor): Conv2d(22, 32, kernel_size=(1, 25), stride=(1, 1), padding=(0, 12))
  (temporal_batch_norm): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, bias=True, track_running_stats=True)
  (squeeze_channel): Rearrange('batch kernel 1 time -> batch kernel time')
  (patch_cov_layer): Sequential(
    (0): PatchEmbeddingLayer()
    (1): CovLayer()
    (2): TraceNorm()
  )
  (manifold_attention): AttentionManifold(
    (q_trans): ParametrizedBiMap(
      (parametrizations): ModuleDict(
        (weight): ParametrizationList(
          (0): _Orthogonal()
        )
      )
    )
    (k_trans): ParametrizedBiMap(
      (parametrizations): ModuleDict(
        (weight): ParametrizationList(
          (0): _Orthogonal()
        )
      )
    )
    (v_trans): ParametrizedBiMap(
      (parametrizations): ModuleDict(
        (weight): ParametrizationList(
          (0): _Orthogonal()
        )
      )
    )
  )
  (re_eig): ReEig()
  (tangent): LogEig()
  (flatten): Flatten(start_dim=1, end_dim=-1)
  (linear): Linear(in_features=1800, out_features=4, bias=True)
)

Training the Model#

subject_id = 1
batch_size = 32
max_epochs = 100
learning_rate = 1e-4  # Low learning rate for stable SPD learning

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"\nUsing device: {device}")

# Cache configuration
cache_config = dict(
    save_raw=True,
    save_epochs=True,
    save_array=True,
    use=True,
    overwrite_raw=False,
    overwrite_epochs=False,
    overwrite_array=False,
)

# Load data
X, labels, meta = paradigm.get_data(
    dataset=dataset, subjects=[subject_id], cache_config=cache_config
)

# Encode labels
le = LabelEncoder()
y = le.fit_transform(labels)

# Split by session
train_idx = meta.query("session == '0train'").index.to_numpy()
test_idx = meta.query("session == '1test'").index.to_numpy()

print(f"\nData shape: {X.shape}")
print(f"Training samples: {len(train_idx)}")
print(f"Test samples: {len(test_idx)}")

# Create classifier
# Note: SPD networks benefit from gradient clipping to prevent
# divergence during training on the Riemannian manifold.
clf = EEGClassifier(
    model,
    criterion=torch.nn.CrossEntropyLoss,
    optimizer=torch.optim.Adam,
    optimizer__lr=learning_rate,
    train_split=ValidSplit(0.1, stratified=True, random_state=42),
    batch_size=batch_size,
    max_epochs=max_epochs,
    callbacks=[
        (
            "train_acc",
            EpochScoring(
                "accuracy", lower_is_better=False, on_train=True, name="train_acc"
            ),
        ),
        ("gradient_clip", GradientNormClipping(gradient_clip_value=1.0)),
    ],
    device=device,
    verbose=1,
)

# Train
clf.fit(X[train_idx], y[train_idx])

# Evaluate
y_pred_train = clf.predict(X[train_idx])
y_pred_test = clf.predict(X[test_idx])

train_acc = accuracy_score(y[train_idx], y_pred_train)
test_acc = accuracy_score(y[test_idx], y_pred_test)

print(f"\n{'=' * 50}")
print(f"Results for Subject {subject_id}")
print(f"{'=' * 50}")
print(f"Train Accuracy: {train_acc * 100:.2f}%")
print(f"Test Accuracy:  {test_acc * 100:.2f}%")
Using device: cpu

Data shape: (528, 22, 1001)
Training samples: 240
Test samples: 288
  epoch    train_acc    train_loss    valid_acc    valid_loss     dur
-------  -----------  ------------  -----------  ------------  ------
      1       0.1823        1.4130       0.3333        1.3765  1.8364
      2       0.3646        1.3572       0.2917        1.3431  1.7292
      3       0.3438        1.3349       0.4583        1.3215  1.7119
      4       0.4792        1.2978       0.3750        1.3080  1.7123
      5       0.4792        1.2778       0.4167        1.2904  1.7114
      6       0.6198        1.2457       0.5833        1.2687  1.7124
      7       0.6615        1.2247       0.6250        1.2532  1.7094
      8       0.6771        1.2025       0.5417        1.2324  1.7124
      9       0.6354        1.1692       0.5000        1.2162  1.7236
     10       0.7708        1.1387       0.5417        1.1971  1.7252
     11       0.6458        1.1264       0.5417        1.1815  1.7147
     12       0.6875        1.1035       0.5833        1.1569  1.7199
     13       0.7135        1.0997       0.5417        1.1545  1.7167
     14       0.6302        1.0643       0.5000        1.1254  1.7264
     15       0.7604        1.0256       0.6250        1.1141  1.7363
     16       0.7812        1.0179       0.5833        1.0962  1.3429
     17       0.7812        1.0006       0.5000        1.0752  1.6212
     18       0.7656        0.9795       0.6667        1.0580  1.7695
     19       0.8281        0.9521       0.7500        1.0440  1.6395
     20       0.8073        0.9387       0.5000        1.0308  1.3474
     21       0.7708        0.9288       0.6667        1.0141  1.2814
     22       0.7917        0.9018       0.5833        1.0058  1.5314
     23       0.8021        0.8833       0.6667        0.9954  1.3073
     24       0.8438        0.8722       0.7083        0.9774  1.3294
     25       0.8281        0.8655       0.7500        0.9703  1.2836
     26       0.8438        0.8409       0.7500        0.9581  1.3081
     27       0.8385        0.8221       0.6250        0.9420  1.3199
     28       0.8281        0.8091       0.6667        0.9272  1.3150
     29       0.8542        0.7969       0.7083        0.9195  1.3050
     30       0.8854        0.7630       0.7083        0.9053  1.3410
     31       0.8125        0.7802       0.6667        0.8976  1.3214
     32       0.8750        0.7517       0.7917        0.8861  1.3083
     33       0.8073        0.7444       0.6250        0.8921  1.3008
     34       0.8333        0.7283       0.7500        0.8691  1.3436
     35       0.9062        0.7115       0.7500        0.8652  1.2766
     36       0.8594        0.7110       0.7083        0.8576  1.3200
     37       0.8802        0.6956       0.7917        0.8457  1.3086
     38       0.8750        0.6931       0.7917        0.8424  1.2797
     39       0.9010        0.6685       0.7917        0.8317  1.3222
     40       0.8750        0.6563       0.7500        0.8220  1.3302
     41       0.8542        0.6427       0.7083        0.8108  1.2964
     42       0.8646        0.6510       0.7083        0.8195  1.3113
     43       0.8698        0.6314       0.7083        0.8157  1.3383
     44       0.8542        0.6321       0.6667        0.8033  1.4101
     45       0.8750        0.6148       0.7917        0.7846  1.7817
     46       0.8802        0.6024       0.7500        0.7868  1.7502
     47       0.8802        0.6089       0.7500        0.7801  1.7561
     48       0.8958        0.5988       0.6667        0.7879  1.7551
     49       0.8958        0.5831       0.7917        0.7657  1.6958
     50       0.8854        0.5884       0.7083        0.7717  1.3227
     51       0.9167        0.5683       0.7500        0.7699  1.3318
     52       0.9115        0.5518       0.7083        0.7537  1.4628
     53       0.9010        0.5493       0.7917        0.7474  1.7344
     54       0.9010        0.5391       0.7083        0.7425  1.7076
     55       0.9115        0.5540       0.7500        0.7491  1.7364
     56       0.8906        0.5351       0.7917        0.7412  1.7437
     57       0.8958        0.5361       0.7083        0.7357  1.7854
     58       0.9323        0.5170       0.7500        0.7304  1.4609
     59       0.8906        0.5087       0.7917        0.7313  1.3340
     60       0.8958        0.5038       0.7917        0.7237  1.3459
     61       0.9219        0.4977       0.7500        0.7214  1.4262
     62       0.9062        0.5143       0.7083        0.7225  1.2904
     63       0.9219        0.4886       0.7917        0.7072  1.3171
     64       0.9271        0.4750       0.7917        0.7139  1.2977
     65       0.9219        0.4824       0.7083        0.7249  1.3074
     66       0.9115        0.4759       0.7917        0.6987  1.3662
     67       0.9167        0.4805       0.7917        0.7027  1.3270
     68       0.9323        0.4587       0.7083        0.7043  1.4115
     69       0.9323        0.4506       0.7917        0.7032  1.7320
     70       0.9375        0.4480       0.7083        0.6950  1.7812
     71       0.9271        0.4481       0.7500        0.6901  1.9428
     72       0.9375        0.4389       0.7500        0.6826  1.9415
     73       0.9375        0.4182       0.7917        0.6758  1.9045
     74       0.9531        0.4368       0.7500        0.6803  1.9185
     75       0.9427        0.4200       0.7917        0.6741  1.8609
     76       0.9062        0.4367       0.7917        0.6778  1.7720
     77       0.9271        0.4246       0.7500        0.6749  1.7737
     78       0.9479        0.4119       0.7083        0.6816  1.7928
     79       0.9323        0.4072       0.7917        0.6685  1.7795
     80       0.9479        0.3899       0.7500        0.6707  1.7779
     81       0.9479        0.3926       0.7500        0.6732  1.7769
     82       0.9375        0.3874       0.7917        0.6646  1.7759
     83       0.9427        0.3937       0.7083        0.6638  1.7738
     84       0.9479        0.3923       0.7500        0.6614  1.7811
     85       0.9479        0.3769       0.7500        0.6599  1.7779
     86       0.9427        0.3716       0.7500        0.6578  1.7855
     87       0.9583        0.3730       0.7500        0.6498  1.8521
     88       0.9583        0.3689       0.7917        0.6382  1.7913
     89       0.9688        0.3616       0.7500        0.6583  1.7634
     90       0.9427        0.3671       0.7500        0.6550  1.7853
     91       0.9427        0.3626       0.7083        0.6467  1.8268
     92       0.9479        0.3508       0.7083        0.6624  1.5766
     93       0.9583        0.3502       0.7500        0.6418  1.4841
     94       0.9688        0.3454       0.7917        0.6398  1.8455
     95       0.9583        0.3370       0.7500        0.6444  1.7683
     96       0.9583        0.3316       0.7500        0.6428  1.8484
     97       0.9688        0.3255       0.7500        0.6325  2.0322
     98       0.9688        0.3272       0.7917        0.6288  1.5719
     99       0.9583        0.3286       0.7083        0.6479  1.6443
    100       0.9792        0.3158       0.7500        0.6247  1.2109

==================================================
Results for Subject 1
==================================================
Train Accuracy: 95.83%
Test Accuracy:  76.39%

Visualizing Results#

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Training history
history = clf.history
epochs = range(1, len(history) + 1)

ax1 = axes[0]
ax1.plot(epochs, history[:, "train_loss"], "b-", label="Train Loss", linewidth=2)
ax1.plot(epochs, history[:, "valid_loss"], "r--", label="Valid Loss", linewidth=2)
ax1.set_xlabel("Epoch", fontsize=12)
ax1.set_ylabel("Loss", fontsize=12)
ax1.set_title("Training and Validation Loss", fontsize=14)
ax1.legend(fontsize=10)
ax1.grid(True, alpha=0.3)

ax2 = axes[1]
ax2.plot(epochs, history[:, "train_acc"], "b-", label="Train Acc", linewidth=2)
ax2.plot(epochs, history[:, "valid_acc"], "r--", label="Valid Acc", linewidth=2)
ax2.set_xlabel("Epoch", fontsize=12)
ax2.set_ylabel("Accuracy", fontsize=12)
ax2.set_title("Training and Validation Accuracy", fontsize=14)
ax2.legend(fontsize=10)
ax2.grid(True, alpha=0.3)
ax2.set_ylim([0, 1])

plt.tight_layout()
plt.show()
Training and Validation Loss, Training and Validation Accuracy

Understanding Manifold Attention#

The attention mechanism in MAtt operates differently from standard attention:

Standard Attention (Euclidean):

\[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right) V\]

Manifold Attention (Log-Euclidean):

\[\text{energy}_{ij} = d_{LE}(Q_i, K_j) = \|\log(Q_i) - \log(K_j)\|_F\]
\[\text{weights}_{ij} = \frac{1}{1 + \log(1 + \text{energy}_{ij})}\]
\[\text{output}_i = \sum_j \text{softmax}(\text{weights})_{ij} \odot V_j\]

This respects the Riemannian geometry of SPD matrices, computing meaningful distances on the manifold rather than in Euclidean space.

Summary#

In this tutorial, we demonstrated:

  1. Creating a MAtt model with patch-based temporal segmentation

  2. Training for motor imagery classification

  3. Understanding the manifold attention mechanism

MAtt is particularly useful when:

  • Different time segments have varying discriminative power

  • You want interpretable attention weights

  • The data has complex temporal dynamics

Total running time of the script: (2 minutes 45.694 seconds)