Modeling Order in Neural Word Embeddings at Scale

Andrew Trask, David Gilmore, Matthew Russell
Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:2266-2275, 2015.

Abstract

Natural Language Processing (NLP) systems commonly leverage bag-of-words co-occurrence techniques to capture semantic and syntactic word relationships. The resulting word-level distributed representations often ignore morphological information, though character-level embeddings have proven valuable to NLP tasks. We propose a new neural language model incorporating both word order and character order in its embedding. The model produces several vector spaces with meaningful substructure, as evidenced by its performance of 85.8% on a recent word-analogy task, exceeding best published syntactic word-analogy scores by a 58% error margin. Furthermore, the model includes several parallel training methods, most notably allowing a skip-gram network with 160 billion parameters to be trained overnight on 3 multi-core CPUs, 14x larger than the previous largest neural network.

Cite this Paper

BibTeX
@InProceedings{pmlr-v37-trask15, title = {Modeling Order in Neural Word Embeddings at Scale}, author = {Trask, Andrew and Gilmore, David and Russell, Matthew}, booktitle = {Proceedings of the 32nd International Conference on Machine Learning}, pages = {2266--2275}, year = {2015}, editor = {Bach, Francis and Blei, David}, volume = {37}, series = {Proceedings of Machine Learning Research}, address = {Lille, France}, month = {07--09 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v37/trask15.pdf}, url = {https://proceedings.mlr.press/v37/trask15.html}, abstract = {Natural Language Processing (NLP) systems commonly leverage bag-of-words co-occurrence techniques to capture semantic and syntactic word relationships. The resulting word-level distributed representations often ignore morphological information, though character-level embeddings have proven valuable to NLP tasks. We propose a new neural language model incorporating both word order and character order in its embedding. The model produces several vector spaces with meaningful substructure, as evidenced by its performance of 85.8% on a recent word-analogy task, exceeding best published syntactic word-analogy scores by a 58% error margin. Furthermore, the model includes several parallel training methods, most notably allowing a skip-gram network with 160 billion parameters to be trained overnight on 3 multi-core CPUs, 14x larger than the previous largest neural network.} }
Endnote
%0 Conference Paper %T Modeling Order in Neural Word Embeddings at Scale %A Andrew Trask %A David Gilmore %A Matthew Russell %B Proceedings of the 32nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2015 %E Francis Bach %E David Blei %F pmlr-v37-trask15 %I PMLR %P 2266--2275 %U https://proceedings.mlr.press/v37/trask15.html %V 37 %X Natural Language Processing (NLP) systems commonly leverage bag-of-words co-occurrence techniques to capture semantic and syntactic word relationships. The resulting word-level distributed representations often ignore morphological information, though character-level embeddings have proven valuable to NLP tasks. We propose a new neural language model incorporating both word order and character order in its embedding. The model produces several vector spaces with meaningful substructure, as evidenced by its performance of 85.8% on a recent word-analogy task, exceeding best published syntactic word-analogy scores by a 58% error margin. Furthermore, the model includes several parallel training methods, most notably allowing a skip-gram network with 160 billion parameters to be trained overnight on 3 multi-core CPUs, 14x larger than the previous largest neural network.
RIS
TY - CPAPER TI - Modeling Order in Neural Word Embeddings at Scale AU - Andrew Trask AU - David Gilmore AU - Matthew Russell BT - Proceedings of the 32nd International Conference on Machine Learning DA - 2015/06/01 ED - Francis Bach ED - David Blei ID - pmlr-v37-trask15 PB - PMLR DP - Proceedings of Machine Learning Research VL - 37 SP - 2266 EP - 2275 L1 - http://proceedings.mlr.press/v37/trask15.pdf UR - https://proceedings.mlr.press/v37/trask15.html AB - Natural Language Processing (NLP) systems commonly leverage bag-of-words co-occurrence techniques to capture semantic and syntactic word relationships. The resulting word-level distributed representations often ignore morphological information, though character-level embeddings have proven valuable to NLP tasks. We propose a new neural language model incorporating both word order and character order in its embedding. The model produces several vector spaces with meaningful substructure, as evidenced by its performance of 85.8% on a recent word-analogy task, exceeding best published syntactic word-analogy scores by a 58% error margin. Furthermore, the model includes several parallel training methods, most notably allowing a skip-gram network with 160 billion parameters to be trained overnight on 3 multi-core CPUs, 14x larger than the previous largest neural network. ER -
APA
Trask, A., Gilmore, D. & Russell, M.. (2015). Modeling Order in Neural Word Embeddings at Scale. Proceedings of the 32nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 37:2266-2275 Available from https://proceedings.mlr.press/v37/trask15.html.

Related Material