The LSA_ToolkitTM

Release Version 0.2

Introduction

The LSA_Toolkit is a robust scalable efficient library that provides the necessary capabilities for performing Latent Semantic Analysis tasks on a collection of data. The toolkit is implemented as a library with a C++ API allowing it to be easily integrated into your existing applications or front ended with software customized to your specific needs. Ultimately the LSA_Toolkit will span all of the LSA processing phases from tokenization to final query processing and analysis. It is designed to be open ended and configurable as to resource usage. Future release versions will support parallel grid computation, running on 1 to n processors as requested.

Instructions on this page cover:

• Installation
• How to use the LSA_Toolkit
• How to obtain support

This version of the LSA_Toolkit performs the core computations for Latent Semantic Analysis - taking the contents of a sparse matrix and performing the truncated singular value decomposition to produce an LSA Space which represents the semantic clustering of the information in the input sparse matrix. Use of the LSA_Toolkit at this point still requires knowledge of the necessary precursor steps and weighting calculations that must be done to produce the input matrix, as well as the ability to use the produced LSA Space for analysis. Low level query processing functions have been introduced in this version, as well as storage and loading of SparseMatrix and LSASpace objects. Other aspects of the toolkit have been improved that are not externally visible to the end user, but move the entire implementation forward toward the planned functionality for future release versions.

Installation

The LSA_Toolkit is packaged as a library to be utilized by your specific application code. The package consists of the following items:

• For Linux 64bit platform:
  • libLSA_Toolkit.a - static library file
  • libLSA_Toolkit.so - shared library file
  • LSA_Toolkit-include - a directory containing include files for C++
  • example.cpp - an example of how to use the LSA_Toolkit library
  • example.make - a simple makefile for example.cpp that shows how to link to the LSA_Toolkit library at compile time.
  • rlm - the license manager program
  • rlmutils - license manager utilities

You will be issued a site specific license file - LSA_Toolkit.lic

Installing on the Linux 64bit platform:

• First, copy the include files from LSA_Toolkit-include directory to your local include directory (typically /usr/local/include), or you may simply add the path for the LSA_Toolkit-include directory to your include path for compilation.
• Next, copy the libLSA_Toolkit.a and libLSA_Toolkit.so files to the location where you keep local libraries for your system (typically /usr/local/lib), or you may add the path for the libraries to the link path during compilation.
• If you copied the libraries to a lib directory you must then run ldconfig so that the new libraries will be recognized.

After these instructions are completed, you should be able to build against the LSA_Toolkit library. See the example.make file for a simple build using the library.

What to do with the LSA_Toolkit.lic file

• Evaluation License - In order to execute a program using the LSA_Toolkit library you must have a valid license key. For evaluation purposes, the LSA_Toolkit.lic is not locked to a specific machine. A copy of your LSA_Toolkit.lic file containing your key may be placed in the same directory as the executable you build, or if you choose to run the rlm license server you must have a .lic file indicating where the license server may be found on your network. This .lic file must be in the same directory as your executable. For instructions on setting up the license server, see below.

• Version 0.2 license management - In order to execute a program using the LSA_Toolkit library you must have a valid license key. License keys may be issued as floating licenses utlizing the rlm license server, or may be node-locked to a specific machine and used without the rlm license server. If you choose to use a floating license and run the rlm license server you must have a .lic file located in the same directory as your executable indicating where the license server may be found on your network. If you are using a node-locked license, a copy of your LSA_Toolkit.lic file containing your key may be placed in the same directory as the executable you build.

Setting up the license server:

We are using the RLM license manager from Reprise software for license key management.

If you will be using the license server to support floating licenses your license key in LSA_Toolkit.lic will be locked to the machine which will run the license server. The LSA_Toolkit.lic file may be placed in the same directory that you start the rlm license server from or you may place it in another location defined in the rlm license environment (See the RLM End User Manual for more information).

You must also put a .lic file (like serverloc.lic) in the same directory as the executable you build which will identify where the license server may be found on your network. This file can consist of a single line:

    HOST hostname hostid [port]

An example serverloc.lic file would be:

    HOST colorado 00a0cc3c9ba5 5053

You may obtain the hostid for your machine by running the rlmutil program:

        rlmutil rlmhostid

The port number defaults to 5053 unless you configure it differently in the HOST or ISV line of the LSA_Toolkit.lic file the rlm license server is using.

How to use the LSA_Toolkit Library

In version 0.2 typical usage would be:

• Construct an Environment.

• Construct a SparseMatrix to use with the Environment context.

• Load information into the SparseMatrix
  • SparseMatrix::put
  • SparseMatrix::incr
  • SparseMatrix::textLoad

• Save or load SparseMatrix objects to/from binary files
  • SparseMatrix::binDump
  • SparseMatrix::binLoad

• Construct an LSASpace using the SparseMatrix with the Environment context

• Save or load LSASpace objects to/from binary files
  • LSASpace::binDump
  • LSASpace::binLoad

• Optimize the LSASpace for query processing using the LSASpace::qOptimize method

• Perform basic query processing to examine the semantic proximity of terms and documents using the following methods:
  • LSASpace::qRxR
  • LSASpace::qRxC
  • LSASpace::qCxR
  • LSASpace::qCxC

See the example.cpp file in the download package for an example program using the LSA_Toolkit.

Support

Support is available from Small Bear Technologies, Inc. Please contact us for support options.