Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer

@article{Cao2019BetterLD,
  title={Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer},
  author={Yanshuai Cao and Peng Xu},
  journal={ArXiv},
  year={2019},
  volume={abs/1905.11978}
}
In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex long-range dependency. We show how the `next sentence prediction (classification)' heuristic can be derived in a principled way from our mutual information estimation framework, and… CONTINUE READING

References

Publications referenced by this paper.
SHOWING 1-10 OF 35 REFERENCES

Similar Papers

Loading similar papers…