Linformer: Self-Attention with Linear Complexity
- Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, Hao Ma
- Computer ScienceArXiv
- 8 June 2020
This paper demonstrates that the self-attention mechanism of the Transformer can be approximated by a low-rank matrix, and proposes a new self-Attention mechanism, which reduces the overall self-ATTention complexity from $O(n^2)$ to $O (n)$ in both time and space.
CLEAR: Contrastive Learning for Sentence Representation
- Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, Hao Ma
- Computer ScienceArXiv
- 31 December 2020
This paper proposes Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies in order to learn a noise-invariant sentence representation and investigates the key reasons that make contrastive learning effective through numerous experiments.
The Number of Scholarly Documents on the Public Web
- Madian Khabsa, C. Lee Giles
- Computer SciencePLoS ONE
- 9 May 2014
The number of scholarly documents available on the web is estimated using capture/recapture methods by studying the coverage of two major academic search engines: Google Scholar and Microsoft Academic Search, and shows that among these fields the percentage of documents defined as freely available varies significantly.
The CHEMDNER corpus of chemicals and drugs and its annotation principles
- Martin Krallinger, O. Rabal, A. Valencia
- Computer ScienceJournal of Cheminformatics
- 19 January 2015
The CHEMDNER corpus is presented, a collection of 10,000 PubMed abstracts that contain a total of 84,355 chemical entity mentions labeled manually by expert chemistry literature curators, following annotation guidelines specifically defined for this task.
Entailment as Few-Shot Learner
- Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, Hao Ma
- Computer ScienceArXiv
- 29 April 2021
A new approach is proposed, named as EFL, that can turn small LMs into better few-shot learners, and improves the various existing SOTA few-shots learning methods by 12%, and yields competitive few- shot performance with 500 times larger models, such as GPT-3.
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning
- Yuning Mao, Lambert Mathias, Madian Khabsa
- Computer ScienceAnnual Meeting of the Association for…
- 14 October 2021
A unified framework, UniPELT, is proposed, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism, indicating that a mixture of multiple P ELT methods may be inherently more effective than single methods.
Building Natural Language Interfaces to Web APIs
- Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, P. Pantel, Michael Gamon, Mark J. Encarnación
- Computer ScienceInternational Conference on Information and…
- 6 November 2017
This work proposes the first end-to-end framework to build an NL2API for a given web API, and applies it to real-world APIs, and shows that it can collect high-quality training data at a low cost, and build NL2APIs with good performance from scratch.
Online Person Name Disambiguation with Constraints
- Madian Khabsa, Pucktada Treeratpituk, C. Lee Giles
- Computer ScienceACM/IEEE Joint Conference on Digital Libraries
- 21 June 2015
An extension to the density-based clustering algorithm (DBSCAN) to handle online clustering so that the disambiguation process can be done iteratively as new data points are added, and implements two types of clustering constraints to demonstrate the concept.
CiteSeerX: AI in a Digital Library Search Engine
- Jian Wu, Kyle Williams, C. L. Giles
- Computer ScienceThe AI Magazine
- 27 July 2014
This work presents key AI technologies used in the following components of CiteSeerX: document classification and deduplication, document and citation clustering, automatic metadata extraction and indexing, and author disambiguation.
Learning to identify relevant studies for systematic reviews using random forest and external information
- Madian Khabsa, A. Elmagarmid, I. Ilyas, Hossam M. Hammady, M. Ouzzani
- Computer ScienceMachine-mediated learning
- 1 March 2016
This work introduces a novel method for representing systematic reviews based not only on lexical features, but also utilizing word clustering and citation features that is shown to outperform previously used features in representing systematic Reviews, regardless of the classifier.
...
...