Vertical Search Consultant

What is vertical search?

Vertical search is an app built around a specific industry vertical, such as jobs, real estate, classifieds, automobiles, etc.

Examples of vertical search sites are:

Homethinking.com (real estate)
Trulia.com (real estate)
Indeed.com (jobs)
SimplyHired.com (jobs)
ModernSearch.com.au (real estate,jobs)

Vertical search lifecycle

Practically every vertical search engine which aggregates data goes through the following steps:

Crawl - Crawl primary data sources
Extraction - Extract data into a database
Index - Index the data for searching
Search - Present data to user in a human-friendly format

Challenges

Some challenges in building and running a vertical search engine include:

Data sources - not all web sites lend themselves to harvesting
Developer expertise - broad knowledge of crawling, searching and webapps required
Maintenance - sites change, and some change often, making maintenance of extraction rules time-consuming

Flexile

I'm the creator and maintainer of Flexile, a vertical search toolkit built specifically for powering aggregation-type vertical search engines. If you're looking to launch a vertical search engine, I recommend you check it out first!

How I can help

I've built a couple of vertical search sites from zero to launch, using either Nutch or a hand-rolled crawler.

I'm intimately familiar with the challenges and pitfalls one usually runs into, and can generally save the team quite abit of time by pre-empting problems.

I've also designed an xml-based site definition fileformat (and a corresponding GUI) which makes it easy to outsource the creation/maintenance of rules which control crawling behavior and data extraction.