The first Search engine was created by Alan Emtage, a student McGill university. His search engine was Called “Archie”. Archie worked by downloading directory listings from files that were located on anonymous FTP sites; Thus creating a searchable database of filenames. As Archie was limited to filenames only, it was a year later, in 1991, when Mark McCahill at the University of Minnesota created “Gopher”, which was able to index plain documents. Many of the sites that were first indexed by Gopher are now websites on the www.
Early search technology was very crude and innacurate. Many search engines started catching onto linking schemes and using them to provide accurate results. For the longest time search engines like altavista dominated the search engine market along with yahoo and others. Google rose to prominence in about 2001. For Google to have the impact that it did was justified with their revolutionary technology and concepts including PageRank, and link popularity. How many remote web sites and web pages that link to a given page is taken into consideration with PageRank, on the premise that good or desirable pages are linked to more than others. The PageRank of linking pages and the number of links on these pages contribute to the PageRank of the linked page. This makes it possible for Google to order its results by how many web sites link to each found page. Google’s technology was leading, as well as their interface which has always been very minamalist.
The rate at which the web grows everyday produces large challenges for search engines, in that they cannot possible index all of the pages, that are updating all of the time. This is largely due to dynamically generated pages, and creates the condition that we know as “the invisible web”.
With the huge money involved in search engines and the internet, rest assured that companies like Google and Yahoo are working hard to ensure that they can cover as much as the web as possible, and bring it all back to you in the form of accurate search results.
