Use of RealNames

As an un-anticipated consequence of our choice of the type of queries used in this study, any engine that uses the RealNames navigation service [2] will have an edge in this test. RealNames links queries of this type to the true home-pages for the corresponding organization. On inspection of the results, we found that the best two engines in Figure 1 (Engine 4 and Engine 3) both use the RealNames service. The other two might be using it but there is no way for us to verify that by just looking at the results page. The best engine in Figure 1, Engine 4, used RealNames for 28 out of the 100 queries whereas the second-best engine, Engine 3, used RealNames for 36 queries.

This definitely gives an edge to these two engines. Unfortunately, we can't find out how the relevant pages would have been ranked by these engines if they were not using the RealNames service. Also, since our main objective is to compare these engines to the TREC algorithms, even if we remove the queries for which RealNames was used, the commercial engines still have a very large lead over the TREC algorithms, and the results from our experiments will stand. In all, it is quite safe to say that the commercial engines are using algorithms that are more effective for the type of queries used in this study.