Consulting - Vertical Search

What is vertical search?

Vertical search is an app built around a specific industry vertical, such as jobs, real estate, classifieds, automobiles, etc.

Examples of vertical search sites are:

Vertical search lifecycle

Practically every vertical search engine which aggregates data goes through the following steps:

  1. Crawl - Crawl primary data sources
  2. Extraction - Extract data into a database
  3. Index - Index the data for searching
  4. Search - Present data to user in a human-friendly format

Challenges

Some challenges in building and running a vertical search engine include:

  • Data sources - not all web sites lend themselves to harvesting
  • Developer expertise - broad knowledge of crawling, searching and webapps required
  • Maintenance - sites change, and some change often, making maintenance of extraction rules time-consuming

How I can help

I've built a couple of vertical search sites from zero to launch, using either Nutch or a hand-rolled crawler.

I'm intimately familiar with the challenges and pitfalls one usually runs into, and can generally save the team quite abit of time by pre-empting problems.

I've also designed an xml-based site definition fileformat (and a corresponding GUI) which makes it easy to outsource the creation/maintenance of rules which control crawling behavior and data extraction.
07/04/08 | Kelvin Tan | Lucene Vertical Search Consultant

NOTE: The content of this website is accessible with any browser. The graphical design though relies completely on CSS2 styles. If you see this text, this means that your browser does not support CSS2. Consider upgrading to a standard conformant browser like Mozilla Firefox or Opera but also Apple's Safari or KDE's Konqueror for example. It may also be that you are looking at a mirror page which did not copy the CSS for this page. Or if some pictu res are missing, then the mirror may not have picked up the contents of the inc directory.