Can Transformers Smell Like Humans?

Summary

This paper mainly focus on the question "Can transformers smell like humans". By using a chemical structure transformer called MoLFormer to encode the odorant molecules, this paper proposes that the MoLFormer pre-trained representation can classify the odors of variety of molecules without careful fine-tuning. The method builds a relationship between odorant chemical strutures and their odors, and the odors are labeled by human-beings.

Strengths

1. This paper tries to use pre-trained transformer to build the relationship between chemical structures and odors, and to some extent, shows that transformer has ability to do odorant classification. 2. This paper applies multiple kinds of databases to support the proposed argumentation, and the databases cover variety of labeling methods to label the odors. Also, experiments are included for every applied labeling methods.

Weaknesses

1. The most important part of the odors classification system is the network, unfortunately, the paper only directly apply a well-designed transformer (MoLFormer) without any modification. 2. It looks like this paper dose not finetune the MoLFormer using the applied databases. The paper argues that the model can achieve good performance without fine-tuning, however, it is better to include experiments to support this argument.

Questions

1. How many models (beside MoLFormer) does the research group consider to generate odorant representations? 2. Do the authors consider to fine-tune the MoLFormer for better performance? 3. Besides transformer, do the authers consider more kinds of models such CNN and graph network?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

1. It is better to do more analyses about the fine-tuning of the network. 2. It is better to consider to include some innovation in the existing models.

Summary

This paper took a transformer model that was pre-trained on general chemical structures and tested whether the resulting model representations aligned with human olfactory perception. Specifically, the authors used a transformer for chemical structures called MoL-Former. MoL-Former was trained via masked token prediction loss. The representations from MoL-Former can predict odor labels from experts and odor ratings from non-experts (although the correlations between the model and rating scores is quite low). The representations are also highly correlated with physiochemical descriptors that are related to human olfactory perception. The authors also compare the MoL-Former self-supervised transformer model to other baseline models (Open-POM, which is supervised with odor labels; and DAM, which predicts similarities between pairs of odors). MoL-Former generally performs comparably to these models, even though the baseline models are supervised and MoL-Former is self-supervised.

Strengths

Transformer models have been revolutionary in modeling sensory modalities of vision and audition, but olfaction is comparatively understudied. If olfaction can also be explained with transformer models, it indicates that the domain-general learning mechanisms of transformers can explain sensory processing in the brain more broadly. Put more simply, the results support a universal, domain-general learning mechanism for sensory processing across modalities. This, in and of itself, is an interest finding worthy of publication. Another strength of the paper is that it relies on self-supervision to pre-train the transformer. This is important both practically and theoretically. From a practical standpoint, acquiring labelled datasets is challenging, whereas acquiring large-scale unlabelled datasets is becoming increasing more feasible. From a theoretical standpoint, this provides a better comparison to human olfaction than supervised models. Most human olfactory learning (and sensory learning more generally) is unsupervised. We are rarely given labels for odors, and animals are able to learn about odors without any verbal labels. Finally, the inclusion of baseline models for comparison to the unsupervised transformer is important for interpreting their results. By comparing to OpenPOM and DAM, the author demonstrate that a self-supervised transformer can perform comparably to previous supervised models.

Weaknesses

Major - Section 4.1 Expert labels classification was difficult to follow. How was dimensionality reduced to 20, through PCA perhaps? How was the same procedure applied to DAM, given that DAM already has less than 20 features? Why wasn't the visualization in Figure 3 also performed on DAM? - Sections 4.2 & 4.3 need noise ceilings in order to be interpretable. The authors analyzed each model's predicted average rating with the actual average rating from participants. However, it is unreasonable to expect a model to be more correlated with average ratings than a typical individual human participant. The noise ceiling computes the average correlation between the human participants and mean performance to estimate a reasonable "ceiling" performance that could be expected from models. When the data are very noisy, the noise ceiling will be low (and the human benchmark will generally be less useful because of its low precision), and when the data are not noisy, the noise ceiling will approach 1. Minor - The first sentence of the paper references Damasio, 1989 and Meyer & Damasio, 2009. These are both fantastic papers, but they don't really fit the sentence where they are referenced. References like DiCarlo & Cox, 2007 and Olshausen & Fields, 2004 strike me as more relevant. - A representational similarity matrix (RSM) would be a great visualization for Section 4.3. It would be really interesting to visualize how the patterns of similarity between different odorants compared across models and humans. You don't need to add it, but it would be helpful for readers to visualize how the models and humans are representing the different odors. - The description of SMILES is a bit hard to follow. Could you provide an example and explain how (or if) the inputs to Open-POM, DAM, and MoL-Former differ?

Questions

Does the input to MoL-Former (and the baseline comparison models) correspond to olfactory receptors? If the input data from olfactory receptors does not match the input data to MoL-Former, then it's not actually completing the same task, and it becomes much harder to argue that the model can "smell like humans."

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See weaknesses above.

Summary

This paper investigates the ability of MOLformer, a model trained on SMILES strings in a BERT-like fashion, to predict the human assessment of odors. The assessments were based on natural language descriptors of odors by human experts, ratings on different NL descriptors by naive subjects, and finally, on similarity ratings between odors. It compares MOLformer, which was trained in an unsupervised manner, to an engineered model (DAM) and a supervised model Open-POM. Despite being trained in an unsupervised manner to represent molecules, MOLformer holds its own against the supervised models. In addition, MOLFormer shines when evaluated using an RSA technique, suggesting that MOLformer captures the relationships between molecular odors better than the other models.

Strengths

+ Assessing models representations of odors is an understudied area. I believe this is the first work to do this. + quite a few datasets were used, providing binary, rating scale, and similarity judgment data. These might be the only ones available + The results are relatively convincing

Weaknesses

+ Despite a rather systematic reporting of models and datasets, some things were left out of the descriptions. For example: - Apparently the GS-LF datasets features are binary. I assume this means that that particular descriptor was used by at least 1 judge. So is it the case that if multiple judges used the same descriptor, the results were still recorded as binary? - how dimensionality was reduced. I assume PCA. How much variance was captured in 20 PCs? - There were apparently descriptors in the similarity dataset as well (line 123). These weren’t described. + the references in the introduction were weird. See below. Typos, comments: The references you use in the introduction seem rather bizarre. Two Damasio references for encoding sensory input into a high-dimensional space? Reference #4 seems out of place in this context. References 8-10 - you have no primary references here; for example, most would reference DiCarlo’s lab here. Reference #12 is weird. Why not reference the original transformer model? And while it’s true that for language, they are self-supervised, the original use of them for vision was supervised. It is only later that self-supervised vision models (e.g., the Masked Autoencoder) came about. line 53: Olfactory is mis-spelled! Figure 2: You might try using UMAP or t-SNE here instead of (or in addition to) PCA. In your results section, you never say what the train-test split was (i.e., 80/20?) I was thinking - they should use RSA - and then you did! This is a very interesting result because, rather than just being able to predict human results, the model is better at both of the others in capturing the *relationships* between the odors, just from their chemical descriptions. - Figure 6: Explain why didn’t you evaluate DAM here. - You didn’t describe how the physicochemical descriptors were obtained. Line 284: In section 4.4, it wasn’t clear that for this analysis, you were trying to discover the reasons for the alignment. You should say something about that when introducing that analysis. lines 289-292: It makes sense to me that the low-level features of the molecules would be best encoded in earlier layers and not in later layers, where more abstract features would be encoded. This seems to fit well with the way vision models work.

Questions

See my questions above: 1. variance captured by PCA 2. How are the physicochemical descriptors derived? 3. Try UMAP or t-SNE

Rating

7

Confidence

5

Soundness

4

Presentation

2

Contribution

3

Limitations

These are discussed.

thanks for the response

Thanks for your response to my criticisms. Since I already gave the paper what I consider a relatively high score, I will leave it at that. I think it's a very interesting paper, showing that a self-supervised transformer trained on molecular structures (without anything specific to odors), can reasonably well predict odor ratings by humans. Good job, and I hope the paper is accepted.