Back at the start of the Internet craze, the revenue model seemed simple: Get as many people to visit your site as possible — accumulate eyeballs. Sell advertising to set in front of those eyeballs (at what nowadays seems like exorbitant prices). Repeat, while raking in the cash with both hands. 

But the Internet "shook out," with the larger sites getting more and more eyeballs. The economy went sour. The Internet advertising market collapsed, with too much inventory for too few buyers. 

After the implosion of the Internet advertising market, search engines and searchable subject indexes found themselves in a difficult position. Maintaining an index for millions (now billions) of Web pages is a costly task, and users were getting more impatient with ad banners and pop-up advertising windows, while the ads themselves brought in less and less revenue. Search engines faced the ultimate ultimatum: Generate revenue or vanish. 

Beyond advertising, search engines have basically two potential revenue streams: the visitor, who wants to find sites or use products and services made available by the search engine, or the Web site owner, who wants to make sure that their site is properly indexed in the search engine. 

Some search engines, like Yahoo!, have gone with both streams. Their consumer services, like Yahoo! Mail premium versions, serve the many visitors to their site. But Yahoo! also requires business sites to pay a $299 annual fee to be considered for inclusion in the Yahoo! directory. Other search engines sell both pay-for-placement opportunities and URL inclusion opportunities for site owners. This column examines the URL inclusion programs offered by various search engines. 

A distinction should be made between URL inclusion programs and pay-for-placement programs. URL inclusion programs guarantee (within certain guidelines) placement in a search engine's index within a certain period of time. These programs also usually guarantee that the page, once in the index, will have its content refreshed periodically (once every 48 hours, for example). However, the programs do not change the rank of the indexed page. For example, if the relevancy ranking algorithms placed a particular URL as #10 for a certain keyword search, the fact that the URL's owner had joined a URL inclusion program to make sure that its content was re-indexed very 24 hours, this fact would not change the site's ranking; it would remain #10 for that keyword search. 

On the other hand, pay-for-placement programs do change the ranking of URLs. Probably the most famous pay-for-placement program is at Overture, formerly GoTo. If you go to http://www.overture.com and do a search for the word "flowers," you'll see that each of the first 211 listings have a note by them: "Cost to advertiser" and an amount. 

This amount ranges from 1 cent for listing #211 all the way up to, at this writing, $1.87 for listing number one. It's easy to see that, in this case, the URL owner's payment is affecting the ranking of the search result. 

URL inclusion programs have been around for a couple of years, though only recently have a majority of search engines adopted them. One of the first ones to do so was Inktomi. 

Inktomi

Inktomi's program is called Inktomi Search Submit [http://www.inktomi.com/products/web_search/submit.html]. If you're squinting up at the ceiling trying to remember where you've heard of a search engine called Inktomi, quit squinting. Inktomi provides search services to other sites and portals, including iWon, MSN, and HotBot. Joining its program (first URL $39 a year, additional URLs $25 a year, special pricing available for more than 1,000 URLs) puts your URL in the index provided to these sites and refreshes it every 48 hours. 

Note that URLs must meet Inktomi's editorial guidelines. Guidelines warn against practices considered deceptive to search engine indexing (like font-matching, deceptive content, improper use of competitor's trademarks, etc.) as well as more subjective practices (such as "excessive" use of pop-ups). 

Inktomi's program is fairly simple: Pay your money and submit your URL. But AltaVista's program is a bit of a hybrid. In addition to offering URL inclusion it also offers URL enhancement, the ability to add elements to a URL (again, without changing the ranking of the URL). 

AltaVista

AltaVista has two programs for site builders. The Express Inclusion program adds URLs to the AltaVista index and refreshes them weekly. Members of the Expression Inclusion program can also join the Listing Enhancements program, which allows members to add logo, text links, and other custom enhancements to URL listings. 

Before the implementation of the inclusion programs, AltaVista received between tens of thousands to hundreds of thousands of URL submissions per day to its 22 global search sites. Even after implementation of the new programs, the free submissions remain relatively consistent. Public reaction to the inclusion programs has been "very positive," reports Chris Kermoian, director of Product Marketing at AltaVista. "In fact, Webmasters were the first people to request a method to pay to have their URLs added to search engines quickly. They tell us that they see the program as a valuable tool." 

A recent survey of the more than 20,000 members of AltaVista's paid inclusion program indicated that more than 75 percent of the members felt that fast inclusion into AltaVista's index was valuable to them, and more than 70 percent found the weekly refreshes valuable. 

This value doesn't come without price, however. AltaVista's Express Inclusion subscription period lasts for 6 months (unlike a year for Inktomi) and costs $39 for the first URL, $29 for each additional nine URLs, and $19 for each additional 490 URLs after that. (If you have more than 500 URLs, AltaVista offers a different program for which you can get a quote.) URLs with adult content cost $79 each. 

Enhanced listings have their own price set depending on what you get. Logos included in the listings, for example, cost $50 for one URL and $36 for each additional URL (again, for a 6-month subscription period). The inclusion of a custom tagline (which can be updated up to once a week) costs $25 for one URL and $18 for each additional URL. The enhancements do not change URL rankings, nor are they indexed. So if you were thinking about buying a URL enhancement and then stuffing it full of search engine-friendly keywords, think again. 

Despite the cost of the inclusion programs, Kermoian doesn't think they'll either change the makeup of AltaVista's index or force out the small and noncommercial sites that can't afford URL inclusion. "The hobby and small money sites will continue to be fully represented. We continue to crawl the Web and process freely submitted URLs on a regular basis." In fact, AltaVista has developed a system for monitoring the Web for noncommercial sites four times a day. "We want to do this for the quality of our Web search," said Krista Thomas, directing of Marketing Communications for AltaVista. "We think this is going to add a lot of value to our index." 

AltaVista is a full-text search engine; that is, it indexes every word of every page added to its index. That's very different from a searchable subject index like Yahoo!, which lists only page URLs and descriptions without indexing every word of every page. But like AltaVista and Inktomi, Yahoo! has its own URL inclusion program. 

Yahoo!

A first glance at Yahoo!'s search engine may lead one to believe it has only a pay-for-placement program. The first results from the search engine are "Sponsor Matches," but those are actually generated and provided by Overture (discussed earlier in this article) and not Yahoo!. Yahoo! has an in-listing advertising program, but these results are separated from regular listings by an orange box. 

Yahoo! was one of the first searchable subject indexes, and, for a long time, anyone could submit to their site for free. In the last couple of years Yahoo! finally instituted a one-time fee of $299 fee ($600 for adult sites) for commercial sites that wanted to submit to the Yahoo! index. Then at the end of 2001 Yahoo! changed the one-time fee into a yearly cost for inclusion in the directory. (Noncommercial sites which just want fast consideration can also use Yahoo! Express.) And just because the fee is paid doesn't mean that the submitted URL will get included in the index; fee payment guarantees only that Yahoo! will review the site within 7 days and get back to you, letting you know whether your site has been accepted or not. If it declines your site, Yahoo! will tell you why. 

Yahoo! also reserves the right to edit your description and to place the site wherever it considers most appropriate. Additionally, Yahoo! has a laundry list of guidelines for sites submitting under Yahoo! Express, including that the site must support multiple browsers, all links on the site must work, and that the site must be in English or have an English-language version available. Furthermore, once listed via Yahoo! Express, the sites are treated the same as any other site in the Yahoo! index. For example, if you want to change your site description, you'll have to use the same queue as the other millions of sites in Yahoo!'s index. 

Despite the restrictions and the pricing of Yahoo! Express, Yahoo! thinks the pricing will ultimately benefit everyone. "We have added more human and technical resources as a result of the annual fee," said a Yahoo! spokesperson. "We feel like there is a need for this service." Especially if, as the spokesperson noted, consumers are using Yahoo! to search for more products and services now. 

Earlier in this article, we noted that Yahoo! sells services both to visitors to its site (with services like Yahoo! Mail) and to site builders who want to be added to Yahoo!'s searchable subject index. LookSmart, however, provides far fewer services to visitors and provides its searchable subject index to many online partners. 

LookSmart

LookSmart's listings get viewed at a variety of sites around the Web, including MSN and AltaVista. And while comprising a good part of LookSmart's revenue, the listings are not a large part of the index. "At the end of fourth quarter 2001, listings represented 62 percent of LookSmart's total revenues. We have more than 3 million URLs in the directory and just 6.5 percent are paid listings," said Liz Haggerty, director of Investor Relations for LookSmart. This may change. 

LookSmart recently launched a new program that differs from other URL inclusion programs; it charges per click for traffic from its site. If you want to join its URL inclusion programs, two fee levels are offered. First is the editor review fee of $49, in which the URL and description are evaluated for appropriateness and accuracy. "If a submitting site follows the editorial guidelines they should be fine," said Robert Goldberg, senior VP of Sales, Marketing, and International Opportunity. 

Once a site is listed in the index, it will be charged 15 cents for each click-through to the site, with a minimum charge of $15 a month (100 clicks.) There's also a maximum charge of $2,500 a month. If the idea of paying $2,500 a month for a search engine listing makes your head hurt, you can set a spending cap. After LookSmart usage meets the cap you set, your site will not appear in search results again until the next month. (LookSmart will send you an e-mail and alert you to see if you want to extend your cap.) 

LookSmart and Yahoo! are both searchable subject indexes; AltaVista is a full-text search engine. But Ask Jeeves offers both, a full-text search engine and a searchable subject index. 

Ask Jeeves

Ask Jeeves has been known for their natural language query engine, Ask Jeeves, for several years. But with the recent acquisition of the search engine Teoma, it provides both a searchable subject index (Ask Jeeves) and a full-text search engine (Teoma). And both sites offer URL inclusion programs. 

Ask Jeeve's Site Submit is still, at this writing, in beta testing, but it's still fully functional. Submitted URLs are included in the site databases within 7 days and are refreshed every 7 days. 

Site URLs cost $30 for the first URL and $18 for each additional URL (up to 1,000 URLs total). Strangely enough the inclusion period is 15 months, not 1 year or 6 months. 

Ask Jeeves has a very clear usage policy (no misleading text, no submitted pages with duplicate content, no sites that misuse affiliate or referral programs) and submitted URLs that violate spam policies can get kicked out of the index without a refund. 

All the sites mentioned so far are fairly famous in the search world — what Internet user hasn't heard of Yahoo!? But there's at least one search engine that keeps a low profile, even as it builds an index of over 800 million pages. And it has a URL inclusion program too. 

FAST

If you haven't heard of FAST, maybe you've heard of AllTheWeb, a full-text search engine at http://www.alltheweb.com . And if you haven't heard of them you've probably heard of the portal Terral Lycos, which is powered by FAST. 

Before FAST launched its URL inclusion program, the search engine got over 1 million submissions a day through its free submission. (FAST still has a free submission program, but it takes 6-8 weeks for submitted URLs to be added to the database.) It's still getting that many, though public reaction to the new program has been good. "Takeoff's been excellent," said Stephen Baker, director of Business Development and Marketing for FAST. "We expected hundreds of subscriptions and we ended up with thousands." Most of the subscribers to the service have been small customers. 

Though the reaction has been beyond what FAST expected, Baker doesn't expect the program to change the index very much. "We've got 800 million pages in the search engine now and we expect to have 2 billion pages at the end of the second quarter. We don't expect more than 3-5 percent of the index to be paid URL inclusion." 

FAST sells its URL inclusion services through partner sites, including Lycos. Lycos calls the program Lycos InSite Select. Submitted URLs are added to the index within 48 hours and refreshed every 48 hours. Inclusion lasts for a year. 

Unlike the other programs mentioned in this article, InSite Select charges a membership fee on top of URL inclusion fees. Membership costs $18 a year, with a $12 per-URL charge. 

Conclusion

Though the economy seems to be getting better and tech companies are making the first tentative steps toward recovery, Web-based businesses are still in a tough spot. URL inclusion seems on the surface like a fair way to generate revenue — it provides value-added services for those site builders who want them, but doesn't change site rankings to the detriment of search engine users. But such programs will have to be in place for a while before it becomes clear whether so many small businesses and online stores using inclusion programs are changing the tenor of search engine indexes. 

WHERE'S GOOGLE? 

Almost every major searchable subject index and full-text search engine has a URL inclusion program. Except one: Where's Google? Google, which has presented itself as a user-oriented search engine almost from the very beginning, doesn't feel URL inclusion programs would be appropriate. "We do not have plans to offer URL inclusion in the future," said Google spokesman David Krane. "Our goal is to provide our users with objective, trusted search results. The introduction of a URL inclusion program would upset this balance."