For example the sentence “The government did not introduce the tax bill” could be represented by “S government S S introduce S tax bill” with ‘S’ standing for a stop word. As a result the amount of data that has to be processed is reduced with a simple matching and removing/replacing of stop words with no or minimal impact on the information contained. There are several lists freely available.

Find Catalan, Czech, Danish, Dutch, French, English, German, Hungarian, Italian, Norwegian, Polish, Portugese, Spanish, and a Turkish stop word list at Ranks.nl.

Arabic, Bulgarian, Czech, French, English, Finish, German, Hungarian, Italian, Roumanian, Russian, Spanish, Swedish, Polish and Portuguese stop word lists are available from Jacques Savoy’s page.

The snowball project offers English, French, Spanish, German, Portuguese, Italian, Dutch, Swedish, Norwegian, Danish, Russian, Finnish and Hungarian stop lists. As it is part of a stemmer project the lists are not in one place and have to be downloaded from each language page.

For starters you would need an Integrated Development Environment (IDE). It allows you to write code, save, organise, build, compile and debug it. I will not bore you too much because very likely you have heard already about IDEs. One that is hard to beat especially if you are into C#, C++, ASP.NET or VB is Microsoft’s Visual Studio. The good news is that there is a free version, Visual Studio 2008 Express, out there. The bad news is that it is stripped of many features. Some of them quite useful to most of us. Even if you ‘only’ develop your student project, thesis or start up self-employed work. Luckily, there are nice people out there who share their solutions to their/our problems.

One of the most useful tools in software development are version or revision control systems. They keep track of changes in files and if necessary help you organise several people work on several versions of the same files. Even if you are working alone the benefits are tremendous. Assume you have two computers one at home/desktop and one at university/work/laptop. With a version control system and internet access, you can install a version control system and synchronise you your work from wherever you are. If you suddenly share your work with a friend then (s)he can contribute under a new user name and you can exchange your work and stay updated. Most importantly you have a central repository of your work which works automatically as double backup. Firstly it will be on a different machine and secondly it will allow you to reverse to any version of any file which is the ultimate ‘Undo’ function.

• So what do you use? Well there are several out there but I am very happy with using Subversion.
• Looks complicated? Some web site providers throw in free subversion (or cvs) repositories which can be a good way to start.
• How do you use it on your machine without getting into the dark world of command line tools? On your Windows machine use TortoiseSVN to connect with your repository through simple right click context menu on any folder/file.

You could go ahead and start your work now but I would recommend at least one more tool. Independent of your programming style a unit test is always a good idea. You might be the Test-Driven Development (TDD) or the write first test later type. Even if latter you should and will need to do a unit test if you want to keep your code stable and free of as many bugs as possible. Use an automated unit test tool like NUnit (if you are using C# or csUnit for any .NET language) to do the hard work for you once you have written your tests. One downside is that the IDE integration does not work because Visual Studio Express does not allow plugins to my knowledge. All that means is that you have to start the unit test program yourself but what is a click every now and then when it is for free!

My last recommendation, a profiler, is not a necessity but can be very useful for some and teaches you a lot at the same time. If you have a performance sensitive application a profiler will be essential to work out where you waste time and where it is worth to optimise. A very good and (currently until released in version 1.0) free one is IJW Profiler which can deal with Java and C# applications. I hope they keep it free for private and academic use. Just run your application from the profiler and it will keep a track of how long what in your application took allowing you to track down the parts that take the most time and optimise them.

The Voronoy (or Voronoi) Tessellation (Voronoy 1908) is a technique that enables the division of a such multi-dimensional spaces into subspaces. Its application defines geometric areas equivalent to subspaces by defining several vectors as centres of subspaces. Any other vector in space can then be attributed to the closest centre vector effectively dividing the whole space in subspaces. Thus an excellent choice to divide semantic vector spaces.

Voronoi diagrams and Delaunay tessellations are one of a few truly interdisciplinary concepts with relevant material to be found in, but not limited to, anthropology, archaeology, astronomy, biology, cartography, chemistry, computational geometry, crystallography, ecology, forestry, geography, geology, linguistics, marketing, metallography, meteorology, operations research, physics, physiology, remote sensing, statistics, and urban and regional planning. (Okabe, Boots, Sugihara and Chiu, 2000)

2D Voronoy Tessellation

Aurenhammer (1991) describes Voronoy Tessellation as “one of the most fundamental data structures in computational geometry” which are used in modelling natural phenomena, to investigate “their mathematical, in particular, geometrical, combinatorial, and stochastic properties” and their computational representation. It also offers various methods for clustering of multi-dimensional data.

Georgy Feodosevich Voronoy/Вороной Георгий Феодосьевич

In my research I intensely worked with it and found it to be useful with large corpora. It parses a corpus of a or several documents and generates the word vectors in a HAL matrix excluding words contained in a stop list. A stop list is a collection of words and letters that have ambiguous features and are semantically expressionless, for example I, you, are, a, b, c, d and so on. This allows to use a simple parser and despite the lack of stemming reduces the amount of word significantly. To limit the number of columns and rows (which can be set by a parameter) frequency is used. So only the x most frequent words are used for columns and y most frequent ones for the rows. The columns have an additional gap feature where by default the 50 most frequent words are ignored. According to INFOMAP these words often are ambiguous because of their high frequency. The HAL matrix is not a simple count of occurrences of words but uses a Term Frequency - Inverse Document Frequency (TF-IDF) measure to weight a word and uses this value when parsing the text and incrementing the matrix.

Once the corpus is parsed and the HAL matrix computed a SVD based on a Lanczos algorithm is performed on it. The resulting left matrix U is then truncated to the columns pre-set in the parameters (default 100) or less if the Lanczos algorithm converges earlier.

Documents are mapped into the space once the SVD and dimensional reduction is completed. Each document vector is a summation of the word( vector)s contained in a document. The query engine of the software allows to query for terms or combination of terms and finds the closest and documents. A nice extra feature is the implementation of a logical NOT to the query engine. So one could query suite NOT clothes to remove possible clothing meaning from the query and focus on alternative meanings like a lawsuit. This is done by creating the query vector of the first part of the query and making it orthogonal to the second part, the NOT, of the query. As a result the final query vector will be orthogonal (unrelated) to the NOT part but retain all other information of the positive part of the query. This simple but brilliant approach was developed by Dominic Widdows and published in Orthogonal Negation in Vector Spaces for Modelling Word-Meanings and Document Retrieval.

While the results of INFOMAP are good and intuitively right, my research has revealed some shortcomings and subsequently lead me to develop my own implementation with several improvements. I will not discuss the details of the problems as its part of my thesis and maybe part of publications to come. Nevertheless, it is a great starting point for anyone interested in playing around with semantic vector spaces. I certainly gained a great insight by using it.

Schütze introduces Context Vectors which are second order co-occurrences while Word Vectors are first order. Word vectors are created in form of a HAL matrix. Context vectors are a summation of word vectors close to the a single occurrence of the word under investigation. As a result word vectors a general representation of a word while a context vector is representation of a context of a single occurrence of a word. Latter are more focused and also only valid for the word in the particular context.

Context vectors are then clustered to identify areas of meaning which is a collection of close context vectors. The centre of such a cluster according to Schütze is a Sense. An example he makes illustrates this. The context vectors of suite might be attribute to different senses. If suite has a legal context and appears with words like judge and law it would be attributed to a sense vector (topic) representing legal meanings. Another time the word might be encountered surrounded by word like tailor and shirt resulting in an attribution of the context vector to a clothing sense.

To reduce the dimensionality of his matrix and take advantage of its positive characteristics Schütze employs Singular Value Decomposition (SVD). He assumes it helps to uncover latent meaning. I would tend to attribute SVD’s positive influence to a combination of amplification and noise filtering of the matrix. If I understood his paper SVD is only employed on the initial word matrix and not the context matrix. This would make sense as latter should have been much less sparse than former.

To test his work Schütze uses pseudo-words. To construct them he picks two word( vector)s which have very little in common, e.g. door and banana, and conflates them into one pseudo-word. Once he parses his text and clusters it, it allows him to easily identify if a context of the pseudo-word was related to door or banana and as such attributed to the right sense. His results show that these artificial ambiguous words are identified to a higher degree than the natural ambiguous words like suite. Furthermore, abstract senses that are encountered in words like space or pairs like wide range are harder to attribute. This is likely due to their appearance in more contexts than other words and as a result higher ambiguity.

The two other points he makes are latent meaning being embedded in the columns of the matrix as well as high order co-occurences. Latter appears to be disputed by Landauer and as explained by Apperceptual appears to have little influence. I am not certain how much difference there is between latent meaning and high order co-occurrences. It might very well be that these two are closely linked if not the same. If one thinks about Hinrich Schütze’s Automatic word sense discrimination it seems to make a similar point with context and second order co-occurences. Assuming the context of a word is similar to the mentioned columns and the contained latent meanings then one could argue that they are nothing more than high order co-occurrences. To be honest it is not a completely accurate comparison as Schütze bases his work on a HAL matrix and not LSA.

The basic premise the work relies on is that words with similar meaning repeatedly occur closely (also known as co-occurrence). As an example in a large corpus of text one could expect to see the words mountain, valley and river appear often close to each other. The same might be true for mouse, cat and dog.

One could now create a square matrix of a text where all unique words n are represented as a row and column. Now we can read the text and every time we read a new word we look its row vector up in the matrix. Then we take x words on the right and on the left and increment the corresponding column in the row vector for each word. This is a simple sliding window parsing. We can also account for the closeness of a word by incrementing by a larger number for a word closer to the centre word, e.g. a word next to the centre could result in an increment of 5 in its column and a word 4 words away could result in a 2 increment.

Naive HAL Matrix

As a result words co-occurring have similar rows. If we look at the simplified example in the above matrix we can see that mountain, valley and river have similar rows and so do mouse, cat and dog. These rows can be interpreted as vectors with n dimensions. The “distance” between vectors then becomes a proxy for the similarity of meanings of the words represented by the vectors. The “distance” often is measured as the cosine of the angle between two vectors. As a result identical vectors, pointing in the same direction, have an angle of 0 degrees and a cosine value of 1. Unrelated vectors would be orthogonal with an angle of 90 degrees and a cosine value of 0. To ease the cosine calculation matrices are often normalised along the rows to the unit length of 1 of the row vectors.

Following the example it also shows that even words not directly co-occurring can share meaning. Dog for example does not appear close to mouse but through its shared meaning with cat also shares meaning with mouse. As a result one can easily group words by their meaning even if they share it only indirectly.

While similar experiments had been done before Lund and Burgess published their work it still was a great breakthrough. Their approach is completely automated and opposite to earlier work does not rely on humans selecting dimensions and training semantic vector spaces. Only the information in a corpus is used to create the matrix and the resulting vector space and thus has no external bias through influence by human actors.