For example the sentence “The government did not introduce the tax bill” could be represented by “S government S S introduce S tax bill” with ‘S’ standing for a stop word. As a result the amount of data that has to be processed is reduced with a simple matching and removing/replacing of stop words with no or minimal impact on the information contained. There are several lists freely available.

Find Catalan, Czech, Danish, Dutch, French, English, German, Hungarian, Italian, Norwegian, Polish, Portugese, Spanish, and a Turkish stop word list at Ranks.nl.

Arabic, Bulgarian, Czech, French, English, Finish, German, Hungarian, Italian, Roumanian, Russian, Spanish, Swedish, Polish and Portuguese stop word lists are available from Jacques Savoy’s page.

The snowball project offers English, French, Spanish, German, Portuguese, Italian, Dutch, Swedish, Norwegian, Danish, Russian, Finnish and Hungarian stop lists. As it is part of a stemmer project the lists are not in one place and have to be downloaded from each language page.

In my research I intensely worked with it and found it to be useful with large corpora. It parses a corpus of a or several documents and generates the word vectors in a HAL matrix excluding words contained in a stop list. A stop list is a collection of words and letters that have ambiguous features and are semantically expressionless, for example I, you, are, a, b, c, d and so on. This allows to use a simple parser and despite the lack of stemming reduces the amount of word significantly. To limit the number of columns and rows (which can be set by a parameter) frequency is used. So only the x most frequent words are used for columns and y most frequent ones for the rows. The columns have an additional gap feature where by default the 50 most frequent words are ignored. According to INFOMAP these words often are ambiguous because of their high frequency. The HAL matrix is not a simple count of occurrences of words but uses a Term Frequency – Inverse Document Frequency (TF-IDF) measure to weight a word and uses this value when parsing the text and incrementing the matrix.

Once the corpus is parsed and the HAL matrix computed a SVD based on a Lanczos algorithm is performed on it. The resulting left matrix U is then truncated to the columns pre-set in the parameters (default 100) or less if the Lanczos algorithm converges earlier.

Documents are mapped into the space once the SVD and dimensional reduction is completed. Each document vector is a summation of the word( vector)s contained in a document. The query engine of the software allows to query for terms or combination of terms and finds the closest and documents. A nice extra feature is the implementation of a logical NOT to the query engine. So one could query suite NOT clothes to remove possible clothing meaning from the query and focus on alternative meanings like a lawsuit. This is done by creating the query vector of the first part of the query and making it orthogonal to the second part, the NOT, of the query. As a result the final query vector will be orthogonal (unrelated) to the NOT part but retain all other information of the positive part of the query. This simple but brilliant approach was developed by Dominic Widdows and published in Orthogonal Negation in Vector Spaces for Modelling Word-Meanings and Document Retrieval.

While the results of INFOMAP are good and intuitively right, my research has revealed some shortcomings and subsequently lead me to develop my own implementation with several improvements. I will not discuss the details of the problems as its part of my thesis and maybe part of publications to come. Nevertheless, it is a great starting point for anyone interested in playing around with semantic vector spaces. I certainly gained a great insight by using it.

Schütze introduces Context Vectors which are second order co-occurrences while Word Vectors are first order. Word vectors are created in form of a HAL matrix. Context vectors are a summation of word vectors close to the a single occurrence of the word under investigation. As a result word vectors a general representation of a word while a context vector is representation of a context of a single occurrence of a word. Latter are more focused and also only valid for the word in the particular context.

Context vectors are then clustered to identify areas of meaning which is a collection of close context vectors. The centre of such a cluster according to Schütze is a Sense. An example he makes illustrates this. The context vectors of suite might be attribute to different senses. If suite has a legal context and appears with words like judge and law it would be attributed to a sense vector (topic) representing legal meanings. Another time the word might be encountered surrounded by word like tailor and shirt resulting in an attribution of the context vector to a clothing sense.

To reduce the dimensionality of his matrix and take advantage of its positive characteristics Schütze employs Singular Value Decomposition (SVD). He assumes it helps to uncover latent meaning. I would tend to attribute SVD’s positive influence to a combination of amplification and noise filtering of the matrix. If I understood his paper SVD is only employed on the initial word matrix and not the context matrix. This would make sense as latter should have been much less sparse than former.

To test his work Schütze uses pseudo-words. To construct them he picks two word( vector)s which have very little in common, e.g. door and banana, and conflates them into one pseudo-word. Once he parses his text and clusters it, it allows him to easily identify if a context of the pseudo-word was related to door or banana and as such attributed to the right sense. His results show that these artificial ambiguous words are identified to a higher degree than the natural ambiguous words like suite. Furthermore, abstract senses that are encountered in words like space or pairs like wide range are harder to attribute. This is likely due to their appearance in more contexts than other words and as a result higher ambiguity.

The two other points he makes are latent meaning being embedded in the columns of the matrix as well as high order co-occurences. Latter appears to be disputed by Landauer and as explained by Apperceptual appears to have little influence. I am not certain how much difference there is between latent meaning and high order co-occurrences. It might very well be that these two are closely linked if not the same. If one thinks about Hinrich Schütze’s Automatic word sense discrimination it seems to make a similar point with context and second order co-occurences. Assuming the context of a word is similar to the mentioned columns and the contained latent meanings then one could argue that they are nothing more than high order co-occurrences. To be honest it is not a completely accurate comparison as Schütze bases his work on a HAL matrix and not LSA.

The basic premise the work relies on is that words with similar meaning repeatedly occur closely (also known as co-occurrence). As an example in a large corpus of text one could expect to see the words mountain, valley and river appear often close to each other. The same might be true for mouse, cat and dog.

One could now create a square matrix of a text where all unique words n are represented as a row and column. Now we can read the text and every time we read a new word we look its row vector up in the matrix. Then we take x words on the right and on the left and increment the corresponding column in the row vector for each word. This is a simple sliding window parsing. We can also account for the closeness of a word by incrementing by a larger number for a word closer to the centre word, e.g. a word next to the centre could result in an increment of 5 in its column and a word 4 words away could result in a 2 increment.

Naive HAL Matrix

As a result words co-occurring have similar rows. If we look at the simplified example in the above matrix we can see that mountain, valley and river have similar rows and so do mouse, cat and dog. These rows can be interpreted as vectors with n dimensions. The “distance” between vectors then becomes a proxy for the similarity of meanings of the words represented by the vectors. The “distance” often is measured as the cosine of the angle between two vectors. As a result identical vectors, pointing in the same direction, have an angle of 0 degrees and a cosine value of 1. Unrelated vectors would be orthogonal with an angle of 90 degrees and a cosine value of 0. To ease the cosine calculation matrices are often normalised along the rows to the unit length of 1 of the row vectors.

Following the example it also shows that even words not directly co-occurring can share meaning. Dog for example does not appear close to mouse but through its shared meaning with cat also shares meaning with mouse. As a result one can easily group words by their meaning even if they share it only indirectly.

While similar experiments had been done before Lund and Burgess published their work it still was a great breakthrough. Their approach is completely automated and opposite to earlier work does not rely on humans selecting dimensions and training semantic vector spaces. Only the information in a corpus is used to create the matrix and the resulting vector space and thus has no external bias through influence by human actors.