Natural Language Understanding (NLU) is a field of artificial intelligence that aims to make computers understand human language. In NLU, word embedding is an important technique that converts words into vector forms, allowing computers to better handle natural language.
The development of word vectors can be traced back to the 50s of the last century, when people used one-hot encoding to convert words into vector forms. One-hot encoding represents each word as a vector, the dimension of the vector is equal to the size of the vocabulary, only one element in the vector is 1, the rest of the elements are 0, the position of this element corresponds to the position of the word in the vocabulary. For example, for a vocabulary of 10,000 words, the word "apple" can be represented as a vector of length 10,000, where the 1,000th element is 1 and the rest of the elements are 0.
However, there are some problems with one-hot encoding. First of all, the dimension of the vector is large, resulting in high computational costs. Second, the distance between vectors does not reflect the semantic similarity between words because they are orthogonal. For example, the distance between the vector "apple" and "orange" is the same as the distance between the vector "apple" and "banana", but "apple" and "orange" are closer to semantically similar.
To solve these problems, people began to study how to represent words as low-dimensional vectors so that these vectors can capture the semantic similarity between words. In 2003, Bengio et al. proposed a method called the neural probabilistic language model, which uses a neural network to learn word vectors. Specifically, the model uses a three-layer neural network to predict the next word, where the first layer is the input layer, the second layer is the hidden layer, and the third layer is the output layer. The input layer receives the first n words, adds their word vectors, then obtains a new vector representation through the hidden layer, and finally predicts the probability of the next word through the output layer. In this process, the model learns the word vector for each word by updating the weights between the input layer and the hidden layer through a backpropagation algorithm.
The introduction of neural probabilistic language models marks a new era of word vectors. Over the next decade, many researchers began using neural networks to learn word vectors. In 2013, Mikolov et al. proposed a model called word2vec that can efficiently learn word vectors in large-scale corpora. The word2vec model is trained in two ways: the Continuous Bag of Words (CBOW) model and the Skip-gram model. The goal of the CBOW model is to predict the current word based on the contextual word, while the goal of the skip-gram model is to predict the contextual word based on the current word. Both models use neural networks to update word vectors through backpropagation algorithms.
In addition to the word2vec model, there are some other word vector models, such as GloVe, FastText, etc. These models achieve good results in different corpus and tasks. For example, the GloVe model uses global lexical statistics to learn word vectors, which can better handle rare words and polysemy words. The FastText model decomposes words into subwords and adds the vectors of subwords to obtain the vector of words, which can better deal with unknown words and word misspellings.
In general, the development of word vectors has gone from one-hot coding to neural probabilistic language models to models such as word2vec, GloVe, and FastText. These models all use neural networks to learn word vectors and are able to capture semantic similarities between words. In the field of natural language processing, word vectors have become a fundamental technology, providing important support for many tasks.
