Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs

@article{Ballesteros2015ImprovedTP,
  title={Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs},
  author={Miguel Ballesteros and Chris Dyer and Noah A. Smith},
  journal={ArXiv},
  year={2015},
  volume={abs/1508.00657}
}
We present extensions to a continuousstate dependency parsing method that makes it applicable to morphologically rich languages. [] Key Method Starting with a highperformance transition-based parser that uses long short-term memory (LSTM) recurrent neural networks to learn representations of the parser state, we replace lookup-based word representations with representations constructed from the orthographic representations of the words, also using LSTMs.

Figures and Tables from this paper

A Morphology-Based Representation Model for LSTM-Based Dependency Parsing of Agglutinative Languages
TLDR
This work proposes two word representation models for agglutinative languages that better capture the similarities between words which have similar tasks in sentences and shows that their morphology-based embedding models improve the parsing performance for most of the agglutanative languages.
Character Composition Model with Convolutional Neural Networks for Dependency Parsing on Morphologically Rich Languages
TLDR
A transition-based dependency parser that uses a convolutional neural network to compose word representations from characters that shows great improvement over the word-lookup model, especially for parsing agglutinative languages.
Neural Joint Model for Transition-based Chinese Syntactic Analysis
TLDR
These models are the first neural approaches for fully joint Chinese analysis that is known to prevent the error propagation problem of pipeline models and also explore bi-LSTM models with fewer features.
Read, Tag, and Parse All at Once, or Fully-neural Dependency Parsing
TLDR
A dependency parser implemented as a single deep neural network that reads orthographic representations of words and directly generates dependencies and their labels that reaches state-of-the-art performance on Slavic languages from the Universal Dependencies treebank.
Transition-Based Dependency Parsing using Perceptron Learner
TLDR
The proposed model, which adds more relevant features to the Perceptron Learner, outperforms a baseline arc-standard parser and beat the UAS of the MALT and LSTM parsers.
A Stack LSTM Transition-Based Dependency Parser with Context Enhancement and K-best Decoding
TLDR
This paper proposes the following two enhancements based on a transition-based dependency parser with stack long short-term memory: using the context of a word in a sentence, and applying K-best decoding to expand the searching space.
Greedy Transition-Based Dependency Parsing with Stack LSTMs
TLDR
A greedy transition-based parser that learns to represent parser states using recurrent neural networks using a new control structure for sequential neural networks—the stack long short-term memory unit (LSTM).
Stack-propagation: Improved Representation Learning for Syntax
TLDR
This work proposes a simple method for learning a stacked pipeline of models which it calls “stack-propagation”, and applies this to dependency parsing and tagging, where the hidden layer of the tagger network is used as a representation of the input tokens for the parser.
Explicitly modeling case improves neural dependency parsing
TLDR
The experiments on Czech, German and Russian suggest that adding explicit morphological case—either oracle or predicted—improves neural dependency parsing, indicating that the learned representations in these models do not fully encode the morphological knowledge that they need, and can still benefit from targeted forms of explicit linguistic modeling.
Character-level Representations Still Improve Semantic Parsing in the Age of BERT
TLDR
A new method of analysis based on semantic tags demonstrates that the character-level representations improve performance across a subset of selected semantic phenomena.
...
...

References

SHOWING 1-10 OF 67 REFERENCES
Structured Training for Neural Network Transition-Based Parsing
TLDR
This work presents structured perceptron training for neural network transition-based dependency parsing, and provides indepth ablative analysis to determine which aspects of this model provide the largest gains in accuracy.
Transition-based Dependency Parsing Using Recursive Neural Networks
TLDR
A general compositional vector framework for transitionbased dependency parsing that captures semantic relatedness between phrases similarly to a constituency-based counterpart from the literature, and introduces the concept of a Transition Directed Acyclic Graph that allows us to apply Recursive Neural Networks for parsing with existing transition-based algorithms.
A Neural Probabilistic Structured-Prediction Model for Transition-Based Dependency Parsing
TLDR
A neural probabilistic structured-prediction model for transition-based dependency parsing, which integrates search and learning is proposed, giving performance comparable to the best linear parser.
Joint Morphological and Syntactic Analysis for Richly Inflected Languages
TLDR
Evaluation on five morphologically rich languages shows consistent improvements in both morphological and syntactic accuracy for joint prediction over a pipeline model, with further improvements thanks to lexical constraints and word clusters.
Chinese Parsing Exploiting Characters
TLDR
This work investigates Chinese parsing from the character-level, extending the notion of phrase-structure trees by annotating internal structures of words by building a joint segmentation, part-of-speech tagging and phrase-Structure parsing system that integrates character-st structure features.
Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network
TLDR
A new part-of-speech tagger is presented that demonstrates the following ideas: explicit use of both preceding and following tag contexts via a dependency network representation, broad use of lexical features, and effective use of priors in conditional loglinear models.
Learning Character-level Representations for Part-of-Speech Tagging
TLDR
A deep neural network is proposed that learns character-level representation of words and associate them with usual word representations to perform POS tagging and produces state-of-the-art POS taggers for two languages.
A Single Generative Model for Joint Morphological Segmentation and Syntactic Parsing
TLDR
A single joint model for performing both morphological segmentation and syntactic disambiguation which bypasses the associated circularity is proposed.
Compositional Morphology for Word Representations and Language Modelling
TLDR
This paper presents a scalable method for integrating compositional morphological representations into a vector-based probabilistic language model, and performs both intrinsic and extrinsic evaluations, presenting results on a range of languages which demonstrate that the model learns morphological representation that both perform well on word similarity tasks and lead to substantial reductions in perplexity.
Non-Projective Dependency Parsing in Expected Linear Time
TLDR
A novel transition system for dependency parsing, which constructs arcs only between adjacent words but can parse arbitrary non-projective trees by swapping the order of words in the input, shows state-of-the-art accuracy.
...
...