SimpleQuestions Nearly Solved: A New Upperbound and Baseline Approach

@inproceedings{Petrochuk2018SimpleQuestionsNS,
  title={SimpleQuestions Nearly Solved: A New Upperbound and Baseline Approach},
  author={Michael Petrochuk and Luke S. Zettlemoyer},
  booktitle={EMNLP},
  year={2018}
}
The SimpleQuestions dataset is one of the most commonly used benchmarks for studying single-relation factoid questions. In this paper, we present new evidence that this benchmark can be nearly solved by standard methods. First we show that ambiguity in the data bounds performance on this benchmark at 83.4%; there are often multiple answers that cannot be disambiguated from the linguistic signal alone. Second we introduce a baseline that sets a new state-of-the-art performance level at 78.1… CONTINUE READING
10
Twitter Mentions

Tables, Results, and Topics from this paper.

Key Quantitative Results

  • Second we introduce a baseline that sets a new state-of-the-art performance level at 78.1% accuracy, despite using standard methods.
  • We introduce a method for automatically identifying many such ambiguities in the data, for both the entities and relations, and show that performance is upperbounded at 83.4%.
  • Despite its simplicity, this approach achieves 78.1% accuracy for predicting Freebase subject-relation queries, surpassing all previous models.
  • Our model achieves 78.1% accuracy on the SimpleQuestions test set, a new state-of-the-art without ensembling or data augmentation (Table 3).

Citations

Publications citing this paper.
SHOWING 1-10 OF 10 CITATIONS

Pretrained Transformers for Simple Question Answering

  • 2019
VIEW 8 EXCERPTS
CITES BACKGROUND, METHODS & RESULTS
HIGHLY INFLUENCED

Message Passing for Complex Question Answering over Knowledge Graphs

Svitlana Vakulenko, Javier David Fernández García, Axel Polleres, Maarten de Rijke, Michael Cochez
  • ArXiv
  • 2019
VIEW 1 EXCERPT
CITES BACKGROUND

References

Publications referenced by this paper.