Unsupervised Learning of Syntactic Structure with Invertible Neural Projections

@inproceedings{He2018UnsupervisedLO,
  title={Unsupervised Learning of Syntactic Structure with Invertible Neural Projections},
  author={Junxian He and Graham Neubig and Taylor Berg-Kirkpatrick},
  booktitle={EMNLP},
  year={2018}
}
Unsupervised learning of syntactic structure is typically performed using generative models with discrete latent variables and multinomial parameters. In most cases, these models have not leveraged continuous word representations. In this work, we propose a novel generative model that jointly learns discrete syntactic structure and continuous word representations in an unsupervised fashion by cascading an invertible neural network with a structured generative prior. We show that the… Expand
Cross-Lingual Syntactic Transfer through Unsupervised Adaptation of Invertible Projections
TLDR
This paper proposes to learn a generative model with a structured prior that utilizes labeled source data and unlabeled target data jointly and yields an average of 5.2% absolute improvement on POS tagging and 8.3%absolute improvement on dependency parsing over a direct transfer method using state-of-the-art discriminative models. Expand
Unsupervised Learning of PCFGs with Normalizing Flow
TLDR
A neural PCFG inducer which employs context embeddings (Peters et al., 2018) in a normalizing flow model to extend PCFG induction to use semantic and morphological information. Expand
Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads
TLDR
A novel fully unsupervised parsing approach that extracts constituency trees from PLM attention heads that is adaptable to low-resource languages, as it does not rely on development sets, which can be expensive to annotate. Expand
STRUCTFORMER: JOINT UNSUPERVISED INDUCTION
There are two major classes of natural language grammars — the dependency grammar that models one-to-one correspondences between words and the constituency grammar that models the assembly of one orExpand
Adapting Unsupervised Syntactic Parsing Methodology for Discourse Dependency Parsing
One of the main bottlenecks in developing discourse dependency parsers is the lack of annotated training data. A potential solution is to utilize abundant unlabeled data by using unsupervisedExpand
Compound Probabilistic Context-Free Grammars for Grammar Induction
TLDR
A formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context free grammar, which is modulated by a per-sentence continuous latent variable, which induces marginal dependencies beyond the traditional context-free assumptions. Expand
PCFGs Can Do Better: Inducing Probabilistic Context-Free Grammars with Many Symbols
TLDR
A new parameterization form of PCFGs based on tensor decomposition is presented, which has at most quadratic computational complexity in the symbol number and therefore allows us to use a much larger number of symbols. Expand
The Return of Lexical Dependencies: Neural Lexicalized PCFGs
TLDR
Novel neural models of lexicalized PCFGs are presented that allow us to overcome sparsity problems and effectively induce both constituents and dependencies within a single model and results in stronger results on both representations than achieved when modeling either formalism alone. Expand
FlowPrior: Learning Expressive Priors for Latent Variable Sentence Models
TLDR
This work proposes adding the importance-sampled log marginal likelihood as a second term to the standard VAE objective to help when learning the prior, and demonstrates that the model, which is called FlowPrior, shows a substantial improvement in language modeling tasks compared to strong baselines. Expand
Effective Estimation of Deep Generative Language Models
TLDR
A sober view of the problem, a survey of techniques to address it, novel techniques, and extensions to the model are surveyed, and a systematic comparison using Bayesian optimisation finds that many techniques perform reasonably similar, given enough resources. Expand
...
1
2
3
4
...

References

SHOWING 1-10 OF 52 REFERENCES
Unsupervised Neural Dependency Parsing
TLDR
A novel approach to unsupervised dependency parsing that uses a neural model to predict grammar rule probabilities based on distributed representation of POS tags that outperforms previous approaches utilizing POS correlations and is competitive with recent state-of-the-art approaches on nine different languages. Expand
CRF Autoencoder for Unsupervised Dependency Parsing
TLDR
An unsupervised dependency parsing model based on the CRF autoencoder that is discriminative and globally normalized which allows us to use rich features as well as universal linguistic priors and evaluated the performance of the model on eight multilingual treebanks. Expand
StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing
TLDR
StructVAE is introduced, a variational auto-encoding model for semi-supervised semantic parsing, which learns both from limited amounts of parallel data, and readily-available unlabeled NL utterances, and outperforms strong supervised models. Expand
Unambiguity Regularization for Unsupervised Learning of Probabilistic Grammars
TLDR
It is shown that unambiguity regularization is beneficial to learning, and in combination with annealing (of the regularization strength) and sparsity priors it leads to improvement over the current state of the art. Expand
Unsupervised Dependency Parsing without Gold Part-of-Speech Tags
TLDR
It is shown that categories induced by unsupervised word clustering can surpass the performance of gold part-of-speech tags in dependency grammar induction, and a sequence modeling technique is introduced that combines the output of aword clustering algorithm with context-colored noise, to allow words to be tagged differently in different contexts. Expand
Grammar induction from (lots of) words alone
TLDR
It is shown that grammar induction from words alone is in fact feasible when the model is provided with sufficient training data, and two new streaming or mini-batch algorithms for PCFG inference that can learn from millions of words of training data are presented. Expand
Head-Driven Statistical Models for Natural Language Parsing
  • M. Collins
  • Computer Science
  • Computational Linguistics
  • 2003
TLDR
Three statistical models for natural language parsing are described, leading to approaches in which a parse tree is represented as the sequence of decisions corresponding to a head-centered, top-down derivation of the tree. Expand
Painless Unsupervised Learning with Features
TLDR
This work shows how features can easily be added to standard generative models for unsupervised learning, without requiring complex new training methods, and applies this technique to part-of-speech induction, grammar induction, word alignment, and word segmentation. Expand
Improving Unsupervised Dependency Parsing with Richer Contexts and Smoothing
TLDR
This paper introduces basic valence frames and lexical information into an unsupervised dependency grammar inducer and shows how this additional information can be leveraged via smoothing to produce state-of-the-art results. Expand
Unsupervised Dependency Parsing with Acoustic Cues
TLDR
Results show that using word duration can improve parse quality relative to words-only baselines, and support the idea that acoustic cues provide useful evidence about syntactic structure for language-learning infants, and motivate the use of word duration cues in NLP tasks with speech. Expand
...
1
2
3
4
5
...