Owing to the fact that the logarithm of a ratio equals the difference of logarithms, this objective associates (the logarithm of) ratios of co-occurrence probabilities with vector differences in the word vector space. The training objective of GloVe is to learn word vectors such that their dot product equals the logarithm of the words' probability of co-occurrence. In this way, the ratio of probabilities encodes some crude form of meaning associated with the abstract concept of thermodynamic phase. Only in the ratio of probabilities does noise from non-discriminative words like water and fashionĬancel out, so that large values (much greater than 1) correlate well with properties specific to ice, and small values (much less than 1) correlate well with properties specific of steam.

Both words co-occur with their shared property water frequently, and both co-occur with the unrelated word fashion infrequently. Here are some actual probabilities from a 6 billion word corpus:Īs one might expect, ice co-occurs more frequently with solid than it does with gas, whereas steam co-occurs more frequently with gas than it does with solid. For example, consider the co-occurrence probabilities for target words ice and steam with various probe words from the vocabulary. The main intuition underlying the model is the simple observation that ratios of word-word co-occurrence probabilities have the potential for encoding some form of meaning.

GloVe is essentially a log-bilinear model with a weighted least-squares objective.

Download fun workouts near me