WebKhoj: Indian language IR from Multiple Character Encodings

• Prasad Pingali, International Institute of Information Technology, Hyderabad, India
• Jagadeesh Jagarlamudi, Language Technologies Research Centre, IIIT, India
• Vasudeva Varma, International Institute of Information Technology, Hyderabad, India

Full text:

Track: Technology for Developing Regions

Slot: 16:00-17:30, Wednesday 24th May

Today web search engines provide the easiest way to reach information on the web. In this scenario, more than 95% of Indian language content on the web is not searchable due to multiple encodings of web pages. Most of these encodings are proprietary and hence need some kind of standardization for making the content accessible via a search engine. In this paper we present a search engine called WebKhoj which is capable of searching multi-script and multiencoded Indian language content on the web. We describe a language focused crawler and the transcoding processes involved to achieve accessibility of Indian langauge content. In the end we report some of the experiments that were conducted along with results on Indian language web content.

Other items being presented by these speakers

• Location Based Web Search on GSM/GPRS Mobile Phones (Developers Track)