Search engines – a great research tool

The web is a big place. With an estimated 250 million websites, searching for the information you want can be intimidating.

The best way of finding a particular piece of information is by using an online search service or tool that can help you find specific web pages by using key word searches or drilling down through lists of information categories.

There are three basic types of internet search tools: search engines, subject directories, and metasearch engines.

Standard search engines – These are best used for finding a specific item (e.g. Brother MFC Printer 3870CN). They can be a good place for looking up a specific website. The results, however, don’t have sites grouped by subject and may include non-relevant sites.

Subject directories – If you are after general subject information (e.g. MFC printers) these are a good place to begin especially if you don't have many specifics. Information in these directories is selected and grouped by a human intermediary. You can also see what other people most often choose in your particular subject area.

Metasearch engines – These engines search several sites at the same time. Although convenient, they allow less control than a standard search engine. They can be a good option if you want to go to a specific site but you don't know the web address. Some allow you to actively choose the specific engines you wish to search, which can be useful if you want to limit the results.

Most search engines (but not subject directories) rely on automated agents called robots or spiders that crawl through the web and find information from individual sites. They pick up words from the title, URL, or the full text of web pages – depending on the ranking algorithms of the specific search engine.

Although all engines operate in generally the same way, they each have their specialties. A closer look will reveal differences in the number of websites contained in an engine's database, the thoroughness in covering particular topics, the speed of the search, and the relevancy of the responses returned as a result of a search. Different search engines tend to catalogue their information in different ways.

Many search engines function both as search engines and subject directories. Most people tend to use their favourite engine, and sometimes they may try another just out of interest. There is no perfect search engine.

Getting started

The best way to learn about search engines is to start using one.

Don't be surprised if you find that the same keyword generates widely different responses on various search engines, as they all use different methods of locating matches to your query.

Google is the most widely used of all search engines, its name now almost synonymous with web searching. Google is a metasearch engine that conducts the search across many different search engines at once. It ranks its results based on the amount of hits (and links from other sites) per URL. Google also has a separate subject directory, a news portal that lists news by the most recently updated articles, a toolbar you can use so you don't need to visit its site to search, a directory of listservs and groups, and an image gallery.

The Open Directory Project is a volunteer-edited directory that aims to be the most comprehensive directory on the web. Experts volunteer to be editors of certain content areas. The Open Directory has listings by order of importance.

Excite.com is a portal and comprehensive search engine that updates its entire database frequently. Because it attempts to index only the most popular pages on the web, Excite is a good choice if you're looking for large, popular sites, but not so good if you want to find more obscure sites. It has a collection of searching tools and tips.

HotBot enables the researcher to perform power searches through its database which includes the databases of Google, Lycos, and Ask.com, so the results are more comprehensive. Its benefits include a selection of ‘skins’ to customise the look and feel of the user interface, and advanced filtering options to hone and refine your search.

LookSmart functions both as a subject directory and a search engine and can search its directory, the entire web, or its own database of articles. You can search for a topic and then pull down a menu to choose the type of magazine within which you would like the database to search. Its drawback is that it doesn't have any academic journals in its database.

WiseNut is another LookSmart-based engine. The WiseGuide Feature automatically creates categories for you. It therefore can function simultaneously as a subject directory and a search engine. Wisenut consistently yields many relevant results and their ‘Sneak-a-peek’ feature allows you to preview a site before taking the time to click in.

Mamma searches a variety of engines, directories, and content sites. Its special feature is that it eliminates duplicates and ranks them according to relevance. It also will tell you which search engine was used to find a particular result.

SurfWax is a metasearch engine whose results are based on meaning, rather than keywords. It allows you to manage your results in several different views and has a ‘site snaps’ feature to give you a snapshot of a site before clicking into it. It also provides a breakdown of the statistics for each search by displaying how many results came from each particular database.

Yahoo, best known as a search engine, is actually a hierarchical directory. Users can search the web through Yahoo, but it works best for surfers who want to pick their way through organised lists of topic-specific resources. You can search the entire Yahoo site, or you can search within each category.

Other popular search engines you might want to try are AltaVista, GoTo.com, Northern Light, Lycos, Ask.com and Kartoo.

A comprehensive listing with reviews of search engines is available at search engine watch.

Searching strategies

The most effective strategy is to be specific. Don't be afraid to tell a search engine exactly what you are looking for. Many real language search engines like Ask.com will facilitate this by allowing you to enter the exact phase you want.

Some sites like Alta Vista will try to match capital letters in your search and this will limit your results to words that are spelled with all capital letters. So avoid capital letters when searching.

Most search engines provide a link to an advanced search page where you can use more robust and specific search parameters. The options will vary with different search engines. Advanced search pages generally allow you to restrict the following.

• Search to a particular country or domain (e.g. .org sites or to a specific web address).
• Date to search for pages dependent on a particular date or within a specified time frame.
• File format – so you can search for pages that contain only certain types of media, such as music (MP3) or pictures (JPEG or GIF).
• Language – you can limit or expand the results to include languages other than English.
• Key word location – to limit searches to where the key word is in relation to the document i.e. in the title, text, or links of a page.
• Phrase search – to find pages that contain a specific phrase, done by enclosing the phrase within quotation marks, such as ‘respite care’.

If you still find searching a problem, try looking at the online Help guides available on each engine. Alternatively there are often free sessions on how to search effectively on the internet at many public libraries, educational institutions and community centres. See also some possible training resources.

Finding the needle in the haystack

Searching can be daunting at first because of the thousands of results that are often thrown up. There are several trillion pages on the web – some are great, some are terrible and most are pretty mediocre. How do you know which results are credible and useful and which are not? One of the features of the web is that anyone can post material online. There is no peer review, no vetting, no one to remove or edit material that is wrong or misleading if posted onto a private site.

Don’t feel you need to look through all 302 000 results that your search throws up. If you can’t find the information you want in the first two pages of results start again with a different or narrower search. In general a big search engine will follow some procedure to rank results by ‘significance’ and ‘relevance’ to your search keywords. These terms can mean different things to different search engines, but Google, for example, will rank pages partly by the number of other sites linking to that page and the number of hits the page receives. This at least tells you that the pages near the top of the results list are considered important enough to link to by a lot of other people, or popular enough to be looked at by a lot of people.

Some tips to help you sieve through the results:

Does the URL of the result page look credible? Does it come from a ‘reliable brand’ – a site belonging to a government department (the site will have <.gov> as part of its address) or educational institution <.edu> or look like it might belong to respected or known institution or company?

Does the size and professionalism of a site suggest credibility? – usually but not always a reliable indicator.

Does the information on the site look current and kept up to date?

Remain skeptical: don’t even click on a result unless the short text that goes with it seems meaningful and useful. Always remember that the web is a wonderfully anarchic place: the information you want will most likely be there but you will often have to avoid the crackpots and shysters to find it.

What to do when you’re stumped

Although there are many ways to find information on the web, sometimes you simply cannot find the information that you need in the amount of time that you have. When you find that it is too time-consuming to weed through all the hits that your search yields, try posting a question to a listserv or a message board like, ‘Does anyone know what is a good site for Adelaide web developers?’

Don’t forget that even in the internet age, word of mouth and peer recommendation are still a major way that people find out about things. Ask around – other staff, people in related organisations, friends. Chances are that someone has already wanted to find out the same thing as you.

Keep track of useful sites

Once you have found a website of particular interest, you will probably want to visit frequently. You can save a record of special pages by adding them to a section of your internet browser called Favourites (if you're using Internet Explorer) or Bookmarks (Netscape Navigator).