Are there any industry benchmarks, frameworks or methodologies for comparing quality of search engines? Does any reference data on relevancy of leading enterprise search engines (e.g. FAST, SharePoint 2007, Google Appliance) exist?
The short answer is no. If you're trying to decide which search product to buy, there are no benchmarks or reference data that allow you to compare them. One reason is that the vendors have no incentive to participate in such an exercise. Another is that apples-to-apples comparisons are difficult because each organization has its own unique requirements — even if they are in the same industry. Finally, search engine performance depends to some extent on how it is customized and how the content is structured. In other words, you may be better off testing and tweaking your current search engine instead of shopping for a new one.
That said, we did find some valuable tools and techniques that can be used to evaluate search engine options:
a government-sponsored benchmarking test bed for new retrieval technologies;
information technology analyst reports;
a metric called MRR that can be applied to search engine results;
an analytics service for Google searches, both public and intranet;
