Understanding Search Engines
There are broadly two types of search engine/directory services on the WEB. The term “search engine” is often used to describe both types. But crawler-based search engines and human-powered directories gather their listings in significantly different ways.
Crawler-based search engines such as HotBot.com, Google.com MSN and nowadays Yahoo, crawl the web by means of a “web spider”. The spider is a powerful computer program that access thousands of Web pages and records what it finds. It possesses a level of intelligence, allowing it to make fairly sophisticated judgments on relevancy and store the information in its database. When you search one of these systems, you’re actually searching the records in the spider’s database.
The spiders operate an a sort of milk round. Once your site is on a spider’s list it will recieve periodic re-cisits, and so changes to your site get recorded on the search engine. Page titles, body copy and other elements all play a role in how the page is given relevancy.
Crawler-based search engines have three major elements. We’ve already mentioned the spider. it visits a web page, reads it, and then follows links to other pages within the site. It then returns to the site on a regular basis to identify site changes.
The information the spider finds goes into the index. The Index is a huge electronic list containing a copy of every web page that the spider finds. the time between being spidered and indexed can be several months. until a site is indexed it’s not available to searchers; this is why you can submit a site to a search engine and see no apparent results for an extended period.
The third part of a search engine is the search program, it sifts through the millions of pages in the index to identify relevant matches to a search. There are three main crawler based engines Google, MSN and YAHOO between them having approximately 80-90%.
A human-powered directory, such DMOZ and YAhoo directory, depends on human editors for its listing. During submission you submit a short description to the directory for youe entire site, the editors may take this description or write one for the sites under review. A search looks for matches only in the descriptions submitted.
Changing your web pages has no effect on your listing. Things that are useful for improving a listing with a crawler-based search engine have no impact on improving a listing in a directory. However a good site, with good content, is more likely to get reviewed for three that a poor site.
Times have changed, it used to be that a search engine either presented crawler-based results or human-powered listings. It is now extremely common for both types of results to be used for Website listing. A hybrid search engine will usually favour one type of listing over the other. For example, Yahoo is a human-powered search engine, however it also presents crawler-based results provided by its own engine spider.
Comments
Leave a Reply
You must be logged in to post a comment.
