More on finding, or not finding, your special Web site (InfoWorld)

November 10, 1997
More on finding, or not finding, your special Web site
I wrote last week about a service provided by Danny Sullivan of Calafia Consulting, called Search Engine Watch. Sullivan maintains a vast database on the strengths and weaknesses of all major search engines on his own Web site, located at http://www.searchenginewatch.com.
In this week's conclusion of this topic, I'll take you a little further toward solving the mystery of why top search engines might not list a particular Web site you're looking for -- or your own site.
In addition to its wealth of free information, Search Engine Watch also provides more detailed analysis to subscribers who contribute $25 per year. (You can send a check payable to Danny Sullivan, 2836 Judah St., San Francisco, CA 94122, or use a credit card at searchenginewatch.com/subscribe.htm.)
I delved into the subscriber-only area with my own user name and found coverage of a serious topic.
A variation of spamming -- sending commercial, junk e-mail to thousands of random addresses -- is "search-engine spamming" or "spam-dexing." This is an attempt to get a Web page to appear in the top-ten list of a particular keyword search. Page authors send a search engine numerous Web pages containing various tricks to boost the ratings.
Sullivan cites one case of a commercial Web site that included several hidden child-care tags to make the site jump to the top when parents searched for this topic. But the site had nothing to do with child care. The site sold children's clothing, but the author assumed parents would search on child care more often than it might occur to them to buy children's clothing sight unseen.
At first glance, this practice seems relatively harmless. But when repeated by thousands of Web sites, it can seriously corrupt the meaningfulness of the search results that Web users get. It isn't long before any given top-ten list doesn't contain a single site that really is about the subject you would expect.
This problem has become so serious that several search engines have made it difficult for Web-page creators at some locations to submit legitimate information for indexing. It's common for creative Web authors to submit new pages to several search engines. The indexes may send "spiders" to that site later for additional material. But if your site is hosted at a location that has been heavily used by spammers, you may find the submission process no longer works automatically.
This is particularly true if you use a hosted Web service that provides free, advertising-supported Web pages to members. One service, GeoCities.com, boasts more than 500,000 members with home pages. But free pages allow spammers to operate with little risk. As a result, at least one search engine, InfoSeek, eliminated its automatic URL submission procedure for pages hosted at GeoCities.com and other providers. Pages could still be submitted via e-mail.
CompuServe and America Online members' pages are largely filtered out by indexers such as AltaVista and HotBot. InfoSeek indexes more than 103,000 pages that bear the domain name ourworld.compuserve.com, but the other two indexes report less than 8,000.
To determine whether this problem affects your Web site (or sites for which you're searching) use the CheckURLStatus page of major indexes. Or check Sullivan's site for full details on this regrettable situation.

Brian Livingston is the co-author of several best-selling Windows books, including the most recent Windows 95 Secrets (IDG Books). Send comments to brian_livingston@infoworld.com. Unfortunately, he cannot answer individual questions.
Missed a column? Go back for more.

Copyright © 1997 InfoWorld Publishing Company

InfoWorld.com complies with the ASME guidelines with IDG extensions For New media.