Stream-aware Side Adaptation for Large Pre-trained Multimodal Embedding Models in Sequential Recommendation
Accepted as a regular paper at ACM MM 2026 (10–14 November 2026).
This repository is a demonstration of Stresa (stream-aware side adaptation): a side-adaptation framework that injects lightweight adapters into frozen multimodal embedding backbones for sequential recommendation, without updating the large pre-trained model itself.
How Stresa works (high level). Item representations from a frozen multimodal encoder (e.g. Qwen3-VL) are treated as modality streams. Lightweight side adapters adapt these streams for the recommendation objective. An optional ReSA (residual stream adapter) replaces a fixed residual with learned per-stream gates, and SHAF (stream-aware hidden-adapter fusion) blends successive hidden states when stacking adapters on the text tower (Sinkhorn routing in the paper configuration). A lightweight Transformer sequence encoder then consumes the adapted item embeddings for next-item prediction.
This repo ships demonstration code: reference model/training wiring, MicroLens data paths, scripts to cache frozen backbone embeddings, and a distributed training entry point (run.py). It is intended to illustrate the method and let you try the pipeline end-to-end—not as a complete reproduction package yet.
By ACM MM 2026 (10–14 November 2026), we will open-source assets needed to fully reproduce the paper results, including:
- curated datasets used in the experiments
- precomputed backbone hidden states / embedding caches
- configs and scripts aligned with the reported tables
Watch this repository for updates.
Python 3.10+ is recommended. Install dependencies with pip install -r requirements.txt and install a PyTorch build that matches your CUDA environment (versions are not pinned in this repository).
| Path | Purpose |
|---|---|
data/microlens/ |
Interaction pairs and title TSVs (small copies included). Optional LMDB from data/microlens/build_lmdb.py if you use image-conditioned preprocessing. |
stored_vectors_microlens/qwen3vl_embeddings/ |
Cached Qwen3-VL per-item tensors or a consolidated file from tools/consolidate_vectors.py. Not shipped (large). |
From the repository root:
python embeddings/preprocess_qwen3_vl_micro.py --image-source noneUse embeddings/preprocess_qwen3_vl_2b_micro.py for the 2B variant. To keep only last-layer vectors: python embeddings/extract_last_layer.py --directory stored_vectors_microlens/qwen3vl_embeddings --prefix qwen3vl.
Requires populated embedding caches under stored_vectors_microlens/ as above (and LMDB if your config expects it).
python scripts/run_microlens_stresa_best.pyExample: CUDA_VISIBLE_DEVICES=0 MASTER_PORT=29500 python scripts/run_microlens_stresa_best.py.
The script passes hyperparameters consistent with a paper-style configuration (including identity mixing on the adapter streams and coupled fusion on the text adapter stack). Further options are documented via python run.py --help and parameters.py.
run.py,parameters.py— training loop and argument definitions.model/— encoders and multimodal adapter stack.data_utils/— datasets and evaluation helpers.embeddings/— scripts to build frozen backbone caches.tools/— optional vector consolidation for faster I/O.