The General Ideas of Word Embeddings

The building stones are therefore characters instead of words.The word embeddings outputted by FastText look very similar to the ones provided by Word2Vec..However, they are not calculated directly..Instead, they are a combination of lower-level embeddings.There are two main advantages to this approach..First, generalization is possible as long as new words have the same characters as known ones..Second, less training data is needed since much more information can be extracted from each piece of text..That is why there are pre-trained FastText models for way more languages than for every other embedding algorithm.Photo by Eugenio Mazzone on UnsplashThe Main Take-AwaysI chose these three algorithms because they represent three general ideas on how to calculate word embeddings:Word2Vec takes texts as training data for a neural network..The resulting embedding captures whether words appear in similar contexts.GloVe focuses on words co-occurrences over the whole corpus..Its embeddings relate to the probabilities that two words appear together.FastText improves on Word2Vec by taking word parts into account, too..This trick enables training of embeddings on smaller datasets and generalization to unknown words.Let me know in the comments or on Twitter if it helped you or if you want to add something..I’m also happy to connect on LinkedIn..Thanks for reading!Additional MaterialWord2VecIf you look for a deep-dive, here are two excellent blog posts by Manish Chablani:Word2Vec (skip-gram model): PART 1 — Intuition.Most of the content here is from Chris’s blog..I have condensed it and made minor adaptations.towardsdatascience.comIf you are looking for an example of how Word2Vec found its way into other areas of machine learning, have a look at this blog post from Ramzi Karam:Using Word2vec for Music RecommendationsHow we use neural networks to transform billions of streams into better recommendations.towardsdatascience.comGloVeBrendan Whitaker wrote a five-part series on GloVe which I highly recommend.. More details

Leave a Reply