Custom Crawlers

What's a crawler?

Crawlers, or bots, fetch and downloads web pages/URLs in an automated fashion.

Crawlers range in scale, from small ones that only fetch a couple hundred URLs, to web-scale beasts like the GoogleBot that indexes a sizable portion of the entire Internet.

My crawler experience

Building crawlers is my passion and expertise. I've crawled and extracted data from websites ranging from the mundane to the insanely difficult.

I've built a number of custom crawlers from scratch for companies such as Homethinking.com and Enormo.com.

I'm also the creator and maintainer of Flexile, a vertical search crawler and extraction engine built specifically for powering vertical search engines.

How I can help

If you're looking to launch a vertical search engine, do check out my Flexile vertical search crawler first.

Otherwise, if you need some data extracted or some websites crawled but don't have the expertise or experience, I'm pretty sure I can be of assistance.

We could work it one of 2 ways:

Outsourced model

In the outsourced model, I develop and run the crawls, then extract the data and present it to you as a deliverable.

Pros:
No infrastructure costs required on your end, and no crawl or programming expertise required. You need only MS Excel or a MySQL database to read the data. Lower costs.

Cons:
No access to source-code.

Consulting model

In the consulting model, I function as a consultant to your company and you own the intellectual property I create in the form of the crawler. I develop the crawler and install it on your own servers.

Pros:
You own the source code and have total control over the crawls.

Cons:
More costly. Some technical expertise required to run crawls.