Best of both worlds:
Integrating search and browse

September, 2001

What if we could combine:

•  the comprehensiveness of full text keyword searching;

•  the versatility of database searching;

•  the convenience of browsing an A - Z index or table of contents? 

What if we could do it by leveraging our existing content infrastructure using a customized search engine and a meta tag program? This article describes a recent pilot project designed to reduce the hidden costs of Internet information retrieval by customizing the search function and integrating it with a browse function.

Pilot objectives
To see what we were trying to achieve, consider the following example. Suppose you are looking for new management techniques that have been used successfully in other companies. If you use Google to search the Montague site for "best practices," you'll get a list of seven documents (see below). If you ask Google to search the entire Internet, you'll get over a million, most of which are advertisements.

Google allows web sites to use its search free of charge, but webmasters can't customize it. And, because it's not installed on our server, it can only find documents on our public site. Documents on the password-protected Society site are excluded from the search results. 

Now go to the Index and browse the A - Z list to find "best practices." Select "B" at the top of the page, then click on "best practices" in the left frame. A list of documents for that term will be displayed in the right frame. 

The Index is powered by a relational database (Knowledge Base), also used in our taeching Lab. When you click on a term, you're using the database's built-in search engine and special web pages that have been designed to display database values. 

Database search vs. advanced search
We can refine our strategy by searching specific combinations of fields in our database or by using Google's advanced search.

The importance of complete and accurate metadata
The Knowledge Base search and the Google advanced search show the importance of structure and data. You can't search on publication date if your database or document doesn't contain that kind of information. And your search will fail if the value entered for publication date is inaccurate or missing.

This kind of descriptive information, called "metadata," consists of two parts -- a label and a value. In a database, metadata resides in fields. In a web page or Microsoft Office document, metadata resides in meta tags. The example below illustrates how some of the field values in the Knowledge Base record for the "Knowledge Transfer" article were incorporated into the corresponding web page as meta tags during our pilot project.

Best of both worlds
The problem with normal keyword searching is that, unless you are already familiar with the content, you don't know which words to enter in the search box. If you guess wrong, you'll either get no documents (even though some relevant documents may exist) or too many (many of which are irrelevant). Databases and advanced searches provide more precision and flexibility, but they require carefully structured, accurate data.

Browsing, using an A - Z index or list of topics, solves these problems by presenting a list of choices within a circumscribed, standardized universe. Because good indexes require human input, they are considered to be expensive to create and maintain. But, if you consider the amount of time a good index can save as well as the efficiencies possible with categorization technology, the cost becomes more reasonable.

How do you get the best of both worlds -- searching and browsing -- without inconveniencing users and making a large investment in software and personnel? You do it in the same way that you build a house. Instead of private wells, septic tanks, and generators for each dwelling, you connect to public utilities. Instead of building windows from scratch, you install pre-fabricated modules. And you use technology to reduce costs and improve quality.

Taxonomy: an information "public utility"
In a retrieval system, a taxonomy is like a public utility. A complete taxonomy includes a thesaurus (system of cross references), a controlled vocabulary (list of standard terms), and classification scheme (categories arranged in a hierarchy), and a set of descriptive attributes (e.g. publication date). Like a public utility, the same taxonomy can be used by more than one author, publisher, or even organization.

We began building our taxonomy in 1998 because we needed to save time in research and publishing. By the summer of 2001, we had created Knowledge Base records for all the articles in the Montague Institute Review. Each article record in the Knowledge Base includes descriptive attributes (e.g. author, title, publication date) along with links to taxonomy records (e.g. vocabulary terms and categories). Each taxonomy record includes a definition (if available) along with links to broader terms, narrower terms, related terms, and articles.

Enhancing content creation
Shortly after we began building the taxonomy, we realized that we could save time and add value for web site visitors by enhancing our content creation process as well. We incorporated indexing and classification tasks into the research, composition, and editing work flows. The effect was twofold:

1. "Richer" documents. Normally, a web page or other document consists of the text, title, file name, and file date. Each of the documents in our Knowledge Base is a node in an information network that includes links to sources, source material annotations, author bios, ownership and security codes, metadata links, glossaries, and usage statistics.

2. From collection to "corpus." Instead of single documents with no apparent relationship, our collection became more like a book or anthology with references to related articles and a unified set of indexes.

By incorporating much of the indexing work into the content creation process and using pre-defined categories and terms from the taxonomy, we reduced the cost of maintaining the index browse function and set the stage for customizing the search engine.  

Using technology
With the taxonomy and the enhanced content creation process in place, it was relatively easy to customize search and integrate the search and browse functions. The key was selecting a search engine that could make use of the descriptive material (metadata) already in our Knowledge Base. We selected Ultraseek because it can read metadata in an HTML page (other options are Verity and NextPage).

But first we had to get the metadata (certain field labels and field values) from our Knowledge Base into each HTML page on our Web site. 

To make this easier, we tested a software tool called the Watchfire Metadata Management System, previously known as Metabot (another option is HiSoftware's Metadata Server). Metabot works like a spreadsheet, allowing you to add metadata tags and make global changes to the metadata values in all the documents on your web site. An example of a global value would be "publisher" (the "publisher" meta tag value would be "Montague Institute" for all our documents). It also speeds up the process of adding metadata values manually. An example would be the Knowledge Base ID number, which is unique to each article.

Benefits
The completed pilot has given web site visitors another retrieval option that can save time and enrich the searching experience.

Now try the Ultraseek browse feature. Click on the "performance improvement" link (between the search box and the list of matching documents).

Visitors can see related categories and definitions without having to access the index in a separate operation. On the same screen, they can get a list of all documents containing the search term ("performance improvement") as well as a list of articles recommended by our editors (even though the recommended documents might not contain the actual phrase "performance improvement"). Because the Ultraseek search engine can access metadata in the documents, it displays a more meaningful document summary that includes a short description and the date of publication.

But better document summaries aren't the only benefit of Ultraseek's ability to read meta tags. This feature also improves the advanced search feature. If you go to the Ultraseek Search and click on "Advanced Search." Search the body for the word "taxonomy" and select dates between today's date and 1 January 2001. You should get about fifteen documents -- all published in 2001. Click on the "Sort by date" link, and Ultraseek will sort the results by publication date.

Reducing the hidden costs of information retrieval
We now know that making large collections of unedited and unstructured documents available on the Internet has a hidden cost. The cost is so high that some employees revert to using conventional sources -- trusted colleagues and fee-based services. To reduce the hidden costs of web-based information services, we need a small list of highly relevant results along with accurate and meaningful summaries. To account for variations in language and culture, we need cross references to synonymous terms. To help us discover new or related concepts, we need a topical outline of a collection's contents. And we need it in a convenient, easy to use format. The new features available through our pilot -- topic browse, search-within-results, Best Bets, a more accurate advanced search, and better document summaries -- bring us one step closer to meeting these needs.  

Leveraging the human investment
As the hidden costs of information retrieval go up, so does the return on investments in human editing and classification. The return is higher if the intellectual capital invested at each stage of the process can be captured and used again. For example, some employees use "personal knowledge management" tools to store and share information about web sites, articles, and other sources. Many corporate publications -- e.g. user manuals, white papers, reports and presentations -- often contain indexes, glossaries, and bibliographies. Often these by-products of the research and editorial processes are not captured and applied to reduce the cost of intranet and internet information retrieval. Without them, search engines and classification tools cannot live up to their potential, and the investment necessary to bring order to the intranet/internet chaos is greater than necessary.

Scaling up
Our pilot was designed to support an upstream publishing model, where local editors capture knowledge, maintain the knowledge base, and edit both print and electronic publications. Some Society members, especially in consulting firms. 

Scaling up the model to serve the entire company means replicating the local system, each specialized collection supported by its own Knowledge Base, customized search engine, and taxonomy. Integrating all of them would be an information "public utility" consisting of a corporate thesaurus and a search engine, configured to search all the collections.

Other alternatives
Companies that have large document collections but lack our Knowledge Base and editorial infrastructure, may find that auto-categorization programs such as Semio are a cost effective way to create a topical browse function. But auto-categorization software doesn't actually insert index terms (or other metadata) into the documents themselves. The terms, topics, and relationships that the software creates can't be exported or searched, as they can in a relational database. Without metadata in the documents, it's harder to customize an accurate advanced search or make search results more meaningful.

Conclusion
Our Ultraseek and Metabot pilot helped us reduce the hidden costs of information retrieval by enhancing the search function, integrating search and browse, and incorporating editorial input in the search results.

The return on investment was relatively high because we could:

• leverage our infrastructure investment in a document catalog, database of index terms (controlled vocabulary) and a thesaurus;
• amortize the ongoing costs over multiple functions -- web site navigation, research, and content creation;
• use the Knowledge Base is an insurance policy that allows us to migrate our metadata to newer technologies as they become available.