Archive
Index For
the most recent changes go to What's
NewReturn to
Internet Search FAQ
or URLs
for a rainy day Past additions made month by month to the Internet Search FAQ.
Archive
Index For
the most recent changes go to What's
New
Return to
Internet Search FAQ
or URLs
for a rainy day
Most recent at the top. Click on the bouncing
to
return here.
March
2000
1. We're delighted to receive a Britannica Internet Guide Award "as one of the best on the Internet for quality, accuracy of content, presentation and usability." Those nice Encyclopedia people (who themselves run an exceedingly good Internet Guide) go on to say that the Internet Search FAQ is a site that "Britannica editors have rated one of the most valuable and reliable, in the company of an elite group of Web sites."
Excuse my blushes, but (clears throat) let's get on with the work...
2. A new development coming over the Internet horizon is the weblog, or Blog to its fans. Blogs vary enormously, but essentially they are a blend of on-line newsletter and discussion group - sites where individuals and/or subscribers can post news stories, links, discussion points or the detailed minutiae of their life. Some are riveting, some are less so, and the range of interests is idiosyncratic. The form is still in its infancy as far as providing useful search material is concerned, but the infant shows promise.
To poke around among the bloggers try looking at the Blogs listed at:
Eatonweb Blog
Portal
Blogger
Groksoup
Look for the option to list by categories, or search on a particular search term. In addition, should you feel the urge to set up your own Blog, the last two sites listed above will help you do it for free.
December
1999
1. Can you rely on the validity of what you find on the Net?
As the Internet grows, this question is becoming increasingly important, so I've added this month a totally new section devoted to questions of reliability. It looks at the main issues involved and suggests some ways to judge how reliable your source of information may be:
8.1 How reliable is the Net?
Things are clearly changing all the time. The Internet is growing bigger - and as many have discovered a web page or Usenet posting can look the same whether created by an world authority or a student. The key issues are:
Accuracy
Most traditional media have standards of fact-checking, which need not be followed by the creator of a web page. The same applies to discussion groups. In misc.writing, for example, a writer accused 90% of the advice posted about copyright law of being wrong.
However, we shouldn't overstate the case. Mistakes also occur in venerable legal textbooks. The problem is that we grow up learning to judge the validity of traditional media. Often this comes from the context in which it appears - we value information in a medical journal, for example, over a teen magazine. On the Net, that context is often missing or severely limited.
Authenticity
How authentic is a website or posting? Many health sites, for example, are created by drugs companies, but don't reveal that fact on the site itself. There are important-sounding history sites which are run by right-wing extremists.
Even the domain name is no proof of authenticity, as constant legal wrangles continue to prove. Although the Net is subject to laws of libel, misrepresentation and advertising standards codes (despite suggestions to the contrary) these laws are not always easy to enforce, and take time.
Ageing and Fluidity
Books and magazines have the grace to look their age. There's no dust on a website that hasn't been updated for over two years. Many sites don't even show a date.
Or you may find the reverse problem. The web is so fluid that the solid site you relied on for information on a regular basis may simply disappear overnight. And every search engine is filled with out-of-date links to sites that no longer exist.
Accessibility
The information you want may exist, but may be buried under a load of dross. Top of my personal hate-list are sites that offer to rig search engines so that your personal URL appears "near the top of any lists." Using techniques which fool search engines into thinking a page is more relevant than it is, they turn the usefulness of the web on its head.
8.2 What can I do about it?
8.2.1 Double-check
Journalists always double-check information unless it comes from a totally secure source. The same should apply for any information you need to verify from the Net. Either find another reliable Internet source, or use traditional means - books, telephone, etc.
8.2.2 Look for "branded" sites
"Branded" sites from organisations you know and trust are likely to be among the more reliable - although even they should still be treated with care, and you should not take the domain name as proof on its own. Many authentic-sounding domain names have been bought up by others.
Government sites generally provide reliable statistics, as reliable as government statistics ever are! Electronic versions of publications such as established broadcasters, newspapers and specialised journals are likely to have been prepared as carefully as their traditional counterparts. Academic departments of universities can be good, too, but check to see if you're reading the work of a professor or student...
8.2.3 Use "filtered" directories
Search engines use automatic web-crawling "spiders" to trawl for pages, and generally make no attempt to judge the value of the sites they find. However, the better search engines have developed a range of strategies for excluding sites that try unfairly to raise their profile, with a variable degree of success.
By contrast, directories are selected by humans, giving a greater reassurance that the sites will at least be relevant to your query. Better still are the directories which give star ratings to valued sites such as Magellan and Excite,and especially the value-added guides at Britannica, About and Clearinghouse.
Most promising of all is the growth of specialised high-quality subject-based services, if there's one that covers your area. ELib is dedicated to developing such subject-based review lists, including a medical resource at OMNI. Another medical resource can be found at the appropriately named Achoo.
8.2.4 Look at internal evidence
In all cases, you should check websites for internal evidence of quality: writing style, language, range of content, level of detail, clarity of design - all can give important clues as to the expertise of the provider.
Writers with depth of knowledge and experience tend to be precise, rather than vague in their use of language, and will normally include detailed material and evidence to back up what they say. And while slickness is no guarantee of quality, a badly organised site suggests that the content may be sloppy in other ways too.
In addition, look for references and citations, clear identification of who "owns" the site, the date of the last update, contact information and email links. Lack of any or all of these should make you increasingly suspicious of the validity of what you find.
8.2.5 Ask who's paying
Good content takes time and effort, and while some people are happy to provide this for nothing, there's no incentive quite like hard cash to ensure the website is kept accurate and up-to-date.
In some cases, it may be that the site relies on advertising, and therefore has a built-in incentive to keep visitors happy and supplied with good content. However, in others the paymaster may be a drugs company, a political organisation or an individual with an axe to grind.
In the long run, if you can afford it, you may feel it's safer to pay yourself, and use one of the subscription-based services. However, even if you do pay, all the above considerations still apply.
8.3 How should Internet sources be cited?
Standards for citing the Internet resources are not yet established. However, you can find some useful suggestions at A Brief Citation Guide for Internet Sources.
2. One of the most difficult things to duplicate on the Net is the ability to browse around a subject, slowly but thoroughly building up a solid base of knowledge. Somehow it's a great deal easier in a physical library where you can find lengthy books on specific topics. Web pages have a tendency to be lighter in content than most books, and following links can be a remarkably hit and miss affair.
Enter Etour.com. When you sign up with Etour you give them your preferred topics out of a list of hundreds, and every time you log in, they'll present you with a different but specially prepared random tour that will take you to web sites that fit your personal interests. There are also extra functions, built into a window at the bottom of the screen - such as the ability to rate each site with a thumbs-up or a thumbs-down. The ratings are themselves displayed, so that you can see how many people liked or disliked each site in turn.
Along with WebRings, this could in time develop into a remarkably useful way to deepen the possibilities of research on the Net.
3. Further help for finding good information on Usenet is Remarq - a directory/search engine for discussion groups which gives star ratings to newsgroups and discussion topics. Top forums are determined by a mathematical algorithm which takes into account various factors such as how many times web users visit the forum and posts made in a forum. Each discussion topic is also assigned a score in the same way.
4. Housekeeping: I've had a New Millennium tidy up, added a new, faster-loading Welcome page with headlines of what's new in the FAQ and simplified the FAQ name to "Internet Search FAQ" - to reflect the range of people visiting the page, which no longer by any means is limited to writers and academic researchers, nor should be.
New Urls
November
1999
1. Dejanews have changed their URL http://www.deja.com along with their format. There are new discussion features, but not everyone is so pleased.
"The new improved format is IMHO anything but an improvement," writes Jim Burgess. "The Power Search in particular is virtually unusable, but I've heard that this is so only if you are using older software (like my Windows V3.1 with Netscape V3). However, I've discovered that the old format Dejanews is still available from http://www.deja.com/=dnc/"
2. Encyclopedia Brittanica http://www.britannica.com/ has abandoned its subscription and is now available free on the Net. Immediately after the announcement the Brittanica server went down, swamped with searches, but should be back in order by now.
3. If you still desperately need that obscure or out-of-print book in real, paper, form, then Bibliofind may be able to help you. Bibliofind is a union of 500 booksellers around the world.
New and Changed URLs
October
1999
We're back. The film has been filmed and is being editing and I finally have time to get back to a bulging in-box of suggested additions to this FAQ. Thank you everyone for being so patient.
1. I've added a new section on finding people. It's still very embryonic, so any comments, suggestions, criticisms, etc would be gratefully accepted. (Usual house rules: only recommend sites which you know from experience are of general use to a fair number of people. If recommending your own site, try to back up the suggestion with some evidence from an unbiased satisfied customer or two).
4.1.4 How can I find specific people?
There are many resources on the Net that can help you locate and even make contact with specific people - famous or not, individuals or companies. Whether they'll be of any use to you will depend on a number of factors, not least geographical.
As with so much on the Internet, the vast majority of resources are devoted to the USA. So there's little difficulty in finding directories and databases with look-up or even reverse look-up facilities covering just about every member of the US population, alive or dead.
(Particularly intriguing, in passing, is Ancestry.com which among its useful resources for genealogical research allows you to find the social security number and other details of any dead American.... and then offers a facility to write a letter! Do they know of some postal service that we don't?)
More wide-ranging are the directories of email addresses. However these are far from all-inclusive, even assuming your target has an email address. Some Internet Service Providers - such as CompuServe and AOL used to provide a look-up service which included all subscribers (and probably still do) but only for other subscribers, as I understand.
For the rest, directories such as BigFoot rely on finding email addresses of those who have web-pages or post regularly to newsgroups. By no means does this include everybody. Expect to have to try a number of sites before you find a lead.
In Urls For A Rainy Day - there are numerous search facilities giving a number of meta-search engines, people searchers and reference sites which offer specific people-finding databases. Particularly useful are those such as All-In-One or Langenberg which have links to many different "people" sites on one page.
There are also databases devoted to certain types, eg: politicians.
Organisations are generally easier to find through a search engine. But even then it is not always easy - especially if the organisation doesn't have a web page of its own. However, David Brager tells of one very useful site. If you know an organisation or individual's domain name (ie: the bit of the web address before .com, .co.it or whatever) you can use it to find all kinds of details, from contact e-mail and snail-mail addresses to phone numbers at http://www.vservers.com/before/dnscheck/.
Whether looking for people or organisations, in difficult cases you may need to try the more refined methods for finding information by using Search Engines, or posting questions on Newsgroups or Mailing Lists (as described in the FAQ section 4.2)
2. I must say I'm fairly jaundiced when it comes to metasearch engines, as I seem to get notice of a new site almost every month, and most seem fairly similar, so I didn't hold out much hope when I heard from the makers of theinfo.com.
More fool me. Theinfo.com is not actually a metasearch engine at all, but I suppose you'd call it a "multi-search" engine. When you enter your search terms, it opens a window for each of the search sites individually. Beware, this can be a bit overwhelming if you've selected all possible sites!
Theinfo.com makes no attempt to combine search results, so you have to be prepared for a fair bit of sifting, but that can be an advantage in some cases, as different search engines rate sites in very different ways. So this approach is useful for those more difficult searches, where your search terms may be less easy to narrow down.
For example, I tried to find a site relating to the feature film "Go". Now being both a verb and a game the word "Go" is likely to appear on a million pages, even capitalised, so I wasn't surprised to find zilch on the first attempt. I closed the myriad new search windows that theinfo.com had opened (and selected a few less sites!) and tried "Go AND cinema". Some metasearch engines have difficulties with search terms that use expressions such as AND, OR, +, - etc. In this case, some of the sites used by theinfo.com still came up with nothing useful, but others put the film "Go" at the very top.
(Additional Note: If I'd looked carefully enough I'd also have noticed they have a set of specialist search sites, including a category for "Movies". There are ten specialist categories - good, but not as wide-ranging as the 23 in Search Spaniel.)
Theinfo.com doesn't pretend to be the only search engine you'll ever use. Like any search source, you need to have the right kind of query. But most of us tend to be lazy and stick with just a couple of search engines that we are used to. One bonus that comes with using a multi-search engine like theinfo.com or Search Spaniel is that you get a chance to discover new engines you may never have seen before, and to catch up on the latest advances of those you may not have touched for months (or even years!) Thus it can be a quick way to test the strengths and weaknesses of different search engines against each other.
There is even a page on the site that explains when it's useful to use theinfo.com as opposed to other sites, an excellent feature I wish more search engines would adopt: http://www.theinfo.com/about/whyuse.html
3. Liszt the directory of mailing lists have been bought by Topica an even larger list of mailing lists - and one which also helps people put mailing lists together, subscribe, unsubscribe, read on-line and of course search for topics.
Both directory and search engine, it will not only locate useful mailing lists for you, but allow you to search them for relevant messages and discussions.
It's attractively laid-out, easy to use and covers a myriad subjects, with a list of links to other directories of lists, in case you can't find what you want on-site. They'll even help you start a mailing list of your own to cover your favourite search subject if you so wish!
4. If you're looking for specific pictures or sound, I hear more good things about the search engine HotBot http://www.hotbot.com which provides tick boxes to allow your search to include still images, video or audio sound clips, or even shockwave animations. Said to be one of the best MP3 search engines at the moment.
5. Finally: want to find out more about urban legends and/or computer viruses? Here's a couple more sites to debunk those myths and leave you sleeping easier at night - with thanks to Khem Caigan who supplied them.
Datafellows:
one of the best sites for the latest on virus myths
MiningCo
Urban Legends and Folklore
New and changed URLs
May
1999
1. HumanSearch has evolved into KnowPost. It still runs by getting real human beings to answer the questions you pose, but you no longer have to pay money for the service. Instead, for each new question you pose, you have to answer a question for someone else. Excellent idea.
Added Sites:
April
1999
1. A few weeks ago the Internet Research FAQ was proud to receive an accolade from webzine The Control Voice. At least, I think it was the FAQ (it may just be an accolade for me personally) - and it may not actually be an accolade.
To quote: If you're getting this mail, you made some outstanding contribution to The Control Voice, whether you know it or not. You may have written something for us, you may have had your web site praised by us or, frankly, you may have been made fun of by us.
Whatever your part in making our webzine great, we decided to come up with a special award for people just like you: the "You're Neat!" award.
After such fulsome praise, what more can one say?
2. Nick Tompkins adds more details about new search engine Google (currently still under test).
I am currently training employees at a major TV company, and some of them have asked for internet training so I have been showing them Google along with a few other search engines running searches for the same subject matter to see how much relevant material is found. Google came out tops and a producer found his own name on a site selling videos of a film he made a few years ago. He went straight to the legal dept to see if they owed him any money.....
I also typed my girlfriend's name in and it came up with a page of info about her at the Univ she went to. Scary really.
3. I've stumbled upon another picture search engine (if you remember last month we looked at Yahoo's ImageSurfer). Anyway, this one is ImageFinder. It gives you a number of different databases to search for a variety of types of image - eg: the Smithsonian Photographic Collection or Colombia University Image and Video Catalog.
4. Two new sites which are more useful as aids to dealing with the material you find, than finding the material in the first place. Babelfish will translate a word, chunk of text, or even a whole web page to and from a variety of languages automatically. Quite a wheeze for impressing your friends. (Have multilingual versions of your own web-site for example).
Meanwhile the horribly named Voycabulary offers foreign language as well as specialised dictionaries, along with a thesaurus. You type in the URL of a webpage, and then click on any word on that page for its definition, translation or an alternative.
And for wider word searching, there's Onelook Dictionaries - search 1,696,778 words across 258 dictionaries.
5. As many parents are concerned about what children might find on the Internet, this month sees a new category in the FAQ of resources that are especially suitable for children. In truth, most of the resources listed elsewhere in the FAQ are very useful for children, and similarly, many of the so-called "children's" links will be equally useful to adults.
However, please note that the links are not "guarded" in any way, and indeed nothing will stop the Net-savvy child from roaming more widely if s/he wishes. I'm not going to get into the debate about "Nanny" software. However, I personally believe the best and most satisfying way to help your child research safely is to be close at hand, either assisting or just ensuring that (if possible) the computer used for surfing is in the family living space. Enjoy.
Other New Links
March
1999
Nick Tompkins writes to tell us about a new search engine (currently still under test) - Google.
The page is strikingly simple and functional, and seems to work very fast. It gives brief details, a coloured bar indicating how relevant the page is to your search terms, and the size of the web page - to help estimate how fast it might download.
In many cases, you can also choose to use a cached version of the selected web page, which should arrive faster (suitable for web pages whose content won't have changed recently).
Look in particular for the "search within results" link at the foot of the results page, which gives you a chance to narrow down your search, if your first search pulled out too many sites, or too many that weren't relevant.
February
1999
1. It's not often we see a totally new departure in searching the web, but Yahoo's Image Surfer looks like one. Image Surfer is a search engine for images which gives you thumbnails of the images it finds, rather than just text descriptions. It works simply and efficiently and you can search by keywords or by categories, as with the standard Yahoo directory.
The only criticisms at this stage are that the database is still rather limited and that even at this early stage there were a number of links which didn't work. But even this may not be a problem if a thumbnail image is good enough for your purposes. Oh, and for some reason Yahoo don't seem to want to publicise Image Surfer on their main page. Try it out at http://isurf.yahoo.com.
2. Sal Towse continues our ongoing discussion of the best metasearch engines. She writes: 'I've used Dogpile and MetaCrawler for a long time, but these days my metasearch engine of choice is Profusion which uses Alta Vista, InfoSeek, Snap, Excite, Lycos, WebCrawler, Magellan, Yahoo and GoTo.
'What I like about Profusion is that it's quick and it lets you choose "all" "three best" "three quickest" or "manually." What I really like about it is the results are combined and then presented so I don't (like Dogpile) get the same results in AltaVista re-presented with Lycos and with WebCrawler and with...'
In addition, Profusion can check if the links it finds are still live. Straight into the "favourite searchers" slot.
3. Meanwhile, for Mac owners only, Mike Shields tells us about Sherlock, 'another method of research, for those of us using a Mac with OS 8.5. Sherlock, performs the same feats as Metacrawler. Allows fuzzy searches as well.
'What makes this really powerful, is that you can create a Sherlock plugin for your site, so that your site will be included in the searches.
'You only search the plugins that you yourself load. So, for instance, if Charles Deemer creates a plugin for his site, and you check it off, then you search his site, if you've loaded his plugin before hand. You can also choose which of the search engines to utilize.
'It's a very fuzzy search. I can ask a question like, "Who is Charlie Harris?" And set it off and running, and it should come back with a few interesting things.'
More information from http://www.apple.com/sherlock/
4. Savvysearch has changed address and upgraded many of its features. In particular it now handles not 25 but 200 search engines. Give it another look at http://www.savvysearch.com/.
You can also subscribe to receive occasional updates when savvysearch makes changes, by sending a blank email message to: savvysearch-announce-subscribe@savvysearch.com.
FEBRUARY'S NEW URL
January
1999
Lots of new links and sites for the New Year.
1. Lloyd Colston points me in the direction of Dogpile saying, "I use MetaCrawler first, then Dogpile, because one search lets me hit the big boys. This results in faster more meaningful research time. The reason I use two is that sometimes Metacrawler goes down (like call good computers should <g>)."
Metacrawler has long been on our list. Dogpile is yet another a metasearch engine, but it's impressively fast, and you can set a time limit to the search, eg 20 secs, to help ensure that speed.
Unlike some other metasearch engines, it doesn't delay things by searching every source at once, but gives you the option to continue your search through the search engines, a few at a time. Well worth adding to the list.
2. A different kind of Multi-Search facility has been created by Chuck Langenberg at www.langenberg.com/ covering a vast multitude of categories, from government to finding people, maps to cooking, encyclopedia to translation. When I first saw the Langenberg site, I felt it was very US-centred - a potential problem for some kinds of searches - but Chuck has worked hard on this, and there are now tags by each resource marking which searches are US-only, which are International, and so on. Even the most experienced researcher will find some new ideas for searches here.
3. And then there's MiningCo recommended by researcher Michelle McIntyre. This is not so much a search engine as a wide-ranging subject-based resource, boasting "500 expert guides mining the net on thousands of topics." The range is impressive, although again the more modern, social and political areas tend to be rather too American for International users.
Users should also be aware that MiningCo tends to put everything inside its own frame, so that outside resources (even this very FAQ) tend to appear as if they are created by MiningCo. TIP: If you want to break out of the MiningCo embrace, right-click on the link you want, rather than left-click (Mac users: hold the mouse button down over the link until the context menu opens) and choose "open in new window".
Such niggles aside, it's a very useful resource - good for general browsing when you want to gather a wide range of knowledge about a subject, rather than detailed information on a very specific name or idea.
4. Continuing our irregular discussion of Intelligent Agents, I have added another site - BotSpot. Here you'll find a number of downloadable "bots" which claim to help you search the net in more efficient ways. Try, for example, clicking on "Best of the Bots." Once more, anyone who finds that these live up to their claims is asked to email me so that I can pass on the good news.
New Urls
Finally, you can download a 10-day free trial of NameBase's entire index from http://www.pir.org/nameinf2.html. The database includes individuals, corporations and groups compiled from 600 investigative books published since 1962, and thousands of pages from periodicals since 1973 - covering the international intelligence community, political elites Right and Left, assassinations, scandals, Latin America, big business, and organized crime.
(They also tell you how to reset the 10-day timer if you cannot afford the $79 sale price NameBase are asking to keep this nonprofit project going.)
© Charlie Harris 1996-2000
(This FAQ may be copied in whole or in part for non-profit making purposes only, provided you tell me you're doing it, adequate credit is given to those who helped towards it and the home address is given http://www.search-faq.com. In addition, if anyone wishes to post any part anywhere on the Net, you must commit to keeping the posting up-to-date on a monthly basis - there's too much out-of-date stuff on the Net already. Our Disclaimer which disclaims responsibility for everything in the entire universe can be found in the main FAQ)