7.22 Search Engines and Directories

Some 85% of accessible web pages are found through search engines and directories. Directories list hierarchically by topic, and submissions are vetted by humans. By contrast, search engines are non-personal devices using complex (secret and changing) algorithms to rank sites, displaying them according to the keywords typed in by search engine users. Search engines are also divided into 'organic' (where ranking is free) and the commercial or pay-per-click variety (where site owners pay for top rankings).

Much material is not reached by the search engines. Even by 2004, a study by NEC Research Institute suggested that total search engine coverage had fallen from 60% to 42%.{10}

Search engines perform three operations. They firstly use a 'web crawler' or 'spider' to regularly search the web, finding sites and following links to collect information on all or most pages. Then they index that information, considering text, text headings, meta tags, graphics labels and links from and to the pages concerned. Lastly, they sort and store that information in ways that can be readily accessed by users of the search engine. Google stores whole pages, inbound link information and action taken on any AdWord links. Altavista stores every word.

When a user enters search words (also called keywords) into a search engine, the program searches its index and selects the best matches, ranking them by 1. relevance, 2. authority, 3. number and quality of incoming links (Google), 4. semantic clustering of keywords, and 5. statistical analysis of the keywords on the page (distribution, density, etc.). Some search engines (e.g. Ask) also allow searches by whole sentence queries rather than simply by key words. Search engines earn revenues from advertising and by incorporating pay-per-click services.

Web directories rely on sites being recommended to them (submissions) and do not generally search the web. They are created and maintained by humans rather than search algorithms, and list sites by category and subcategory. The best known are Yahoo! Directory and the Open Directory Project. The first has a paid submission service. The second is the most prestigious on the web, and is free. In some directories the paid-for-inclusions are ranked according to their bid amount.

Directories also supply information to search engines.

Search Engine Popularity

There are many search engines, but only five are now important to the English-speaking world. {4}

 

USA

UK

 

% Volume

% Visits

% Volume

% Visits

Google

65.55

63.40

90.16

83.68

Yahoo

15.46

12.21

2.94

3.46

Bing

13.97

14.47

4.19

4.94

Ask

2.74

2.15

1.36

1.89

AOL

1.58

1.29

-

0.27

Total

99.30

93.52

98.65

94.24

Search engine use varies by country.

 

National Search

Engine(s)

Google

Yahoo

Bing

Brazil

-

97%

1%

1%

Czech Republic

47.7%

40.8%

-

-

China

64.7%

30.9%

-

-

Denmark

15%

80%

1%

1%

France

-

89.5%

2.5%

2.8%

Germany

2%

89%

2%

-

India

c2%

94%

c2%

c2%

Italy

c3%

91%

c3%

c3%

Japan

3.8%

31.3%

56.2%

-

Mexico

-

c90%

c5%

c5%

Netherlands

1%

95%

1%

3%

Norway

-

95%

2%

2%

Portugal

10%

90%

-

-

Russia

c8%

68%

23%

1%

Slovakia

-

98%

1%

1%

South Korea

93%

4%

3%

-

Spain

4%

73.4%

17.4%

-

Sweden

-

71%

14%

c8%

UK

-

c90%

c5%

c4%

USA

 

c70%

14.4%

9.9%


Some search engines use or incorporate other search engine results. Examples:

      Yippy: also searches the 'deep web', i.e. beyond the range of conventional engines.
      Dogpile: returns results from leading search engines, including Google, Yahoo! and Bing.

Search engines specialize, or are particularly useful for some types of information. Examples:

      Webopedia: simple Internet and computer information.
      SearchEngineGuide. Search engines and guides listed under nearly 100 categories.
      ResourceShelf. Results of web searches by librarians and others.

History

The first tool created to search the Internet was the 1990 'Archie', followed a year later by the better-known 'Gopher'. In 1994 came 'WebCrawler', the first 'full text' crawler-based search engine, which was followed over the next ten years by search engines attracting considerable interest and investment: Magellan, Excite, Infoseek (now Go), Inktomi, Northern Light, AltaVista and Yahoo! It was a period of intense competition, with search engines incorporating results from other search engines, and sometimes acquiring the search engine company itself. Overture, for example, owned AlltheWeb and AltaVista. Yahoo! acquired both Inktomi (2002) and Overture (2003). Microsoft launched MSN Search in 1998 using Inktomi search results, then displayed Looksmart listings, blended results in from Inktomi, switched to Google technology until 2004, developed its own search technology thereafter, launched Bing in 2009, and finally leased that technology to Yahoo Search. Google provided a ranking method based on number and quality of incoming links, and rose rapidly to prominence after 2000.

The search engine market is now dominated by a few big players. Netmarketshare {5} gave the market share in April 2011 as: Google 84.64%, Yahoo, 5.15%, Baidu 4.30%, Bing 3.91%, Ask, 0.53%, AOL 0.38%, Excite 0.02%, Lycos 0.01% and AltaVista 0.01%. MSN, Microsoft Live Search and All the Web were shown as 0.00% (presumably less than 0.01%). All markets were global except Baidu (China), Bing and Microsoft Live Search.

Questions

1. What three operations do search engines perform?
2. Provide a short history of search engine development.
3. How do search engines specialize? Give some examples.
4. How do search engines differ from search directories?
5. What are the five most popular search engines in the US? What are the more striking differences in other countries?

Sources and Further Reading

Need the references and resources for further study? Consider our affordable (US $ 4.95)  pdf ebook. It includes extensive (3,000) references, plus text, tables and illustrations you can copy, and is formatted to provide comfortable sequential reading on screens as small as 7 inches.

   Get your eBook here.