Large Multi-modal Models Can Interpret Features in Large Multi-modal Models

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

• QARM: Quantitative Alignment Multi-Modal Recommendation at Kuaishou (2024)
• Debias your Large Multi-Modal Model at Test-Time with Non-Contrastive Visual Attribute Steering (2024)
• On the Comparison between Multi-modal and Single-modal Contrastive Learning (2024)
• Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities (2024)
• Can sparse autoencoders make sense of latent representations? (2024)
• EMMA: Efficient Visual Alignment in Multi-Modal LLMs (2024)
• Cross-Modal Consistency in Multimodal Large Language Models (2024)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Hey HF community 😊,

I am preparing a presentation on this paper and I have a question regarding the analogy to sparse coding used to explain monosemantic features in section 2.1. Sparse Auto-encoders for Disentanglement.

The text states the following:

In this context, $W_{2}W\_2$
acts as an overcomplete dictionary [1] for the input data,
with its rows forming the dictionary vectors, and $zz$ serv-
ing as the sparse coefficients corresponding to these vec-
tors.

In the traditional sparse coding literature referenced ([10, 30]), the signal $xx$ is typically reconstructed as $x\approx D\alpha x\; \backslash approx\; D\; \backslash alpha$, where the columns of the dictionary $DD$ are the dictionary vectors (atoms), and the sparse coefficients $\alpha \backslash alpha$ scale these columns.

Looking at the SAE decoder equation $\widehat{x}=W_2\cdot z\backslash hat\{x\}\; =\; W\_2\; \backslash cdot\; z$ (Eq. 2), the reconstruction $\widehat{x}\backslash hat\{x\}$ is also formed as a linear combination of the columns of $W_{2}W\_2$, weighted by the sparse coefficients $zz$ (i.e., $\widehat{x}={\sum }_i{z}_i\cdot W_2[:,i]\backslash hat\{x\}\; =\; \backslash sum\_i\; z\_i\; \backslash cdot\; W\_2[:,i]$).

This seems to suggest that, for the synthesis analogy $\widehat{x}=W_2\cdot z\backslash hat\{x\}\; =\; W\_2\; \backslash cdot\; z$ to hold, the columns of $W_{2}W\_2$ (the decoder weight matrix) should be considered the dictionary vectors, rather than the rows.

Could someone please clarify this point? Was the intention for the columns of $W_{2}W\_2$ to be the dictionary vectors in this analogy, or is there perhaps another interpretation (maybe involving the encoder or matrix transposes) that leads to the 'rows' terminology here?

Any help would be greatly appreciated!