• Corpus ID: 235458110

Do Large Scale Molecular Language Representations Capture Important Structural Information?

  title={Do Large Scale Molecular Language Representations Capture Important Structural Information?},
  author={Jerret Ross and Brian M. Belgodere and Vijil Chenthamarakshan and Inkit Padhi and Youssef Mroueh and Payel Das},
Predicting chemical properties from the structure of a molecule is of great importance in many applications including drug discovery and material design. Machine learning based molecular property prediction holds the promise of enabling accurate predictions at much less complexity, when compared to, for example Density Functional Theory (DFT) calculations. Features extracted from molecular graphs, using graph neural nets in a supervised manner, have emerged as strong baselines for such tasks… 
Stepping Back to SMILES Transformers for Fast Molecular Representation Inference
ST-KD is proposed, an end-to-end SMILES Transformer for molecular representation learning boosted by Knowledge Distillation, which shows competitive results on latest standard molecular datasets PCQM4M-LSC and QM9, with 3-14× inference speed compared with existing graph models.
Chemical transformer compression for accelerating both training and inference of molecular modeling
DeLiCaTe is introduced, a new generative transformer model that achieves comparable performance in QSAR and VS modeling and it is anticipated that the model compression strategy provides a pathway to the creation of effective generative transformers for organic drug and material design.
Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast
iMolCLR, improvement of Molecular Contrastive Learning of Representations with graph neural networks (GNNs) in two aspects: mitigating faulty negative contrastive instances via considering cheminformatics similarities between molecule pairs and fragment-level contrasting between intramolecule and intermolecule substructures decomposed from molecules are proposed.
Chemformer: a pre-trained transformer for computational chemistry
The Chemformer model is presented—a Transformer-based model which can be quickly applied to both sequence-to-sequence and discriminative cheminformatics tasks and it is shown that self-supervised pre-training can improve performance and significantly speed up convergence on downstream tasks.
Vision Xformers: Efficient Attention for Image Classification
This work modifications the ViT architecture to work on longer sequence data by replacing the quadratic attention with efficient transformers, such as Performer, Linformer and Nyströmformer of linear complexity creating Vision X-formers (ViX), and shows that all three versions of ViX may be more accurate than ViT for image classification while using far fewer parameters and computational resources.
Resource-efficient Hybrid X-formers for Vision
  • Pranav Jeevan, A. Sethi
  • Computer Science
    2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
  • 2022
Three modifications to vision transformer (ViT) architectures are tested that address the quadratic bottleneck by using linear attention mechanisms, called X-formers, and replaced the learnable 1D position embeddings in ViT with Rotary Position Embedding (RoPE), which increases the classification accuracy for the same model size.
Explaining Artificial Intelligence Generation and Creativity: Human interpretability for novel ideas and artifacts
Various motivations, algorithms, and methods for explaining either the workings of generative/creative AI algorithms or the generative /creative artifacts they produce are reviewed.


The Message Passing Neural Networks for Chemical Property Prediction on SMILES.
Molecular Property Prediction: A Multilevel Quantum Interactions Modeling Perspective
A generalizable and transferable Multilevel Graph Convolutional neural Network (MGCN) for molecular property prediction, which represents each molecule as a graph to preserve its internal structure and directly extracts features from the conformation and spatial information followed by the multilevel interactions.
MolCLR: Molecular Contrastive Learning of Representations via Graph Neural Networks
This work presents MolCLR: Molecular Contrastive Learning of Representations via Graph Neural Networks (GNNs), a self-supervised learning framework for large unlabeled molecule datasets and proposes three novel molecule graph augmentations: atom masking, bond deletion, and subgraph removal.
CheMixNet: Mixed DNN Architectures for Predicting Chemical Properties using Multiple Molecular Representations
The proposed CheMixNet models not only outperforms the candidate neural architectures such as contemporary fully connected networks that uses molecular fingerprints and 1-D CNN and RNN models trained SMILES sequences, but also other state-of-the-art architecture such as Chemception and Molecular Graph Convolutions.
Analyzing Learned Molecular Representations for Property Prediction
A graph convolutional model is introduced that consistently matches or outperforms models using fixed molecular descriptors as well as previous graph neural architectures on both public and proprietary data sets.
Self-Supervised Graph Transformer on Large-Scale Molecular Data
A novel framework, GROVER, which stands for Graph Representation frOm self-supervised mEssage passing tRansformer, which allows it to be trained efficiently on large-scale molecular dataset without requiring any supervision, thus being immunized to the two issues mentioned above.
Convolutional Embedding of Attributed Molecular Graphs for Physical Property Prediction.
A convolutional neural network is employed for the embedding task of learning an expressive molecular representation by treating molecules as undirected graphs with attributed nodes and edges, and preserves molecule-level spatial information that significantly enhances model performance.
N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules
The N-gram graph is introduced, a simple unsupervised representation for molecules that is equivalent to a simple graph neural network that needs no training and is complemented by theoretical analysis showing its strong representation and prediction power.
SMILES2Vec: An Interpretable General-Purpose Deep Neural Network for Predicting Chemical Properties
SMILES2vec is developed, a deep RNN that automatically learns features from SMILES to predict chemical properties, without the need for additional explicit feature engineering, and demonstrates that neural networks can learn technically accurate chemical concept and provide state-of-the-art accuracy, making interpretable deep neural networks a useful tool of relevance to the chemical industry.
Pushing the boundaries of molecular representation for drug discovery with graph attention mechanism.
A new graph neural network architecture called Attentive FP for molecular representation that uses a graph attention mechanism to learn from relevant drug discovery datasets and achieves state-of-the-art predictive performances on a variety of datasets and that what it learns is interpretable.