Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

@inproceedings{Reimers2017ReportingSD,
  title={Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging},
  author={Nils Reimers and Iryna Gurevych},
  booktitle={EMNLP},
  year={2017}
}
  • Nils Reimers, Iryna Gurevych
  • Published in EMNLP 2017
  • Computer Science, Mathematics
  • In this paper we show that reporting a single performance score is insufficient to compare non-deterministic approaches. We demonstrate for common sequence tagging tasks that the seed value for the random number generator can result in statistically significant (p < 10^-4) differences for state-of-the-art systems. For two recent systems for NER, we observe an absolute difference of one percentage point F1-score depending on the selected seed value, making these systems perceived either as state… CONTINUE READING

    Create an AI-powered research feed to stay up to date with new papers like this posted to ArXiv

    Citations

    Publications citing this paper.
    SHOWING 1-10 OF 207 CITATIONS

    Low-Rank RNN Adaptation for Context-Aware Language Modeling

    VIEW 10 EXCERPTS
    CITES BACKGROUND
    HIGHLY INFLUENCED

    Universal Machine Learning Methods for Detecting and Temporal Anchoring of Events

    VIEW 5 EXCERPTS
    CITES BACKGROUND & METHODS
    HIGHLY INFLUENCED

    Neural Semantic Role Labeling using Verb Sense Disambiguation

    VIEW 2 EXCERPTS
    CITES BACKGROUND
    HIGHLY INFLUENCED

    There and Back Again: Cross-Lingual Transfer Learning for Event Detection

    VIEW 3 EXCERPTS
    CITES METHODS
    HIGHLY INFLUENCED

    An Experimental Analysis of Multi-Perspective Convolutional Neural Networks

    VIEW 3 EXCERPTS
    CITES METHODS & BACKGROUND
    HIGHLY INFLUENCED

    Exploring Segment Representations for Neural Semi-Markov Conditional Random Fields

    FILTER CITATIONS BY YEAR

    2016
    2020

    CITATION STATISTICS

    • 7 Highly Influenced Citations

    • Averaged 67 Citations per year from 2017 through 2019

    • 11% Increase in citations per year in 2019 over 2018

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 36 REFERENCES

    Incorporating Nesterov Momentum into Adam

    VIEW 7 EXCERPTS
    HIGHLY INFLUENTIAL

    Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

    VIEW 5 EXCERPTS
    HIGHLY INFLUENTIAL

    Long Short-Term Memory

    VIEW 5 EXCERPTS
    HIGHLY INFLUENTIAL

    Backpropagation Applied to Handwritten Zip Code Recognition

    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL

    Adam: A Method for Stochastic Optimization

    VIEW 4 EXCERPTS
    HIGHLY INFLUENTIAL

    Why Does Unsupervised Pre-training Help Deep Learning?

    VIEW 5 EXCERPTS
    HIGHLY INFLUENTIAL

    Neural Networks for Machine Learning - Lecture 6a - Overview of mini-batch gradient descent

    • Geoffrey Hinton
    • 2012
    VIEW 4 EXCERPTS
    HIGHLY INFLUENTIAL

    Enriching Word Vectors with Subword Information

    VIEW 1 EXCERPT