The following piece is taken from http://www.roboshopper.com/ and will discuss the first of two methods in setting up Comparison Shopping engines, called "crawling". This is the more complex form of developing a Comparison Shopping Engine, but understanding this step may help readers understand how comparison shopping engines actually work.

"If you set out to build a comparison shopping engine, there are basically two ways you can approach the problem. The first way is to "crawl" merchant (online store) websites in much the same way that a search engine like Google examines and indexes the information in web pages. This means that you build a software program to visit the merchant websites, analyze the HTML pages and find the information and prices for the products they offer.

While crawling works great for general web search engines, it much harder to build a comparison shopping engine this way. That's because the information on and the format of merchant sites varies widely. The product description and even the product title can vary among merchant websites, so that it can be difficult for the shopping engine to even be sure that it is comparing identical products. Also, any time that a merchant changes the format of its website, the crawler may stop working properly. Of course, a crawler needs to continuously visit the merchant websites in order to insure that information is up to date."

http://www.roboshopper.com/comparison-shopping-engines.htm