the beginning, there were eyeballs.
Back at the start
of the Internet craze, the revenue model seemed simple: Get as many people
to visit your site as possible — accumulate eyeballs. Sell advertising
to set in front of those eyeballs (at what nowadays seems like exorbitant
prices). Repeat, while raking in the cash with both hands.
But the Internet
"shook out," with the larger sites getting more and more eyeballs. The
economy went sour. The Internet advertising market collapsed, with too
much inventory for too few buyers.
After the implosion
of the Internet advertising market, search engines and searchable subject
indexes found themselves in a difficult position. Maintaining an index
for millions (now billions) of Web pages is a costly task, and users were
getting more impatient with ad banners and pop-up advertising windows,
while the ads themselves brought in less and less revenue. Search engines
faced the ultimate ultimatum: Generate revenue or vanish.
search engines have basically two potential revenue streams: the visitor,
who wants to find sites or use products and services made available by
the search engine, or the Web site owner, who wants to make sure that their
site is properly indexed in the search engine.
Some search engines,
like Yahoo!, have gone with both streams. Their consumer services, like
Yahoo! Mail premium versions, serve the many visitors to their site. But
Yahoo! also requires business sites to pay a $299 annual fee to be considered
for inclusion in the Yahoo! directory. Other search engines sell both pay-for-placement
opportunities and URL inclusion opportunities for site owners. This column
examines the URL inclusion programs offered by various search engines.
A distinction should
be made between URL inclusion programs and pay-for-placement programs.
URL inclusion programs guarantee (within certain guidelines) placement
in a search engine's index within a certain period of time. These programs
also usually guarantee that the page, once in the index, will have its
content refreshed periodically (once every 48 hours, for example). However,
the programs do not change the rank of the indexed page. For example, if
the relevancy ranking algorithms placed a particular URL as #10 for a certain
keyword search, the fact that the URL's owner had joined a URL inclusion
program to make sure that its content was re-indexed very 24 hours, this
fact would not change the site's ranking; it would remain #10 for that
On the other hand,
pay-for-placement programs do change the ranking of URLs. Probably
the most famous pay-for-placement program is at Overture, formerly GoTo.
If you go to http://www.overture.com
and do a search for the word "flowers," you'll see that each of the first
211 listings have a note by them: "Cost to advertiser" and an amount.
This amount ranges
from 1 cent for listing #211 all the way up to, at this writing, $1.87
for listing number one. It's easy to see that, in this case, the URL owner's
payment is affecting the ranking of the search result.
URL inclusion programs
have been around for a couple of years, though only recently have a majority
of search engines adopted them. One of the first ones to do so was Inktomi.
is called Inktomi Search Submit [http://www.inktomi.com/products/web_search/submit.html].
If you're squinting up at the ceiling trying to remember where you've heard
of a search engine called Inktomi, quit squinting. Inktomi provides search
services to other sites and portals, including iWon, MSN, and HotBot. Joining
its program (first URL $39 a year, additional URLs $25 a year, special
pricing available for more than 1,000 URLs) puts your URL in the index
provided to these sites and refreshes it every 48 hours.
Note that URLs
must meet Inktomi's editorial guidelines. Guidelines warn against practices
considered deceptive to search engine indexing (like font-matching, deceptive
content, improper use of competitor's trademarks, etc.) as well as more
subjective practices (such as "excessive" use of pop-ups).
is fairly simple: Pay your money and submit your URL. But AltaVista's program
is a bit of a hybrid. In addition to offering URL inclusion it also offers
URL enhancement, the ability to add elements to a URL (again, without changing
the ranking of the URL).
AltaVista has two
programs for site builders. The Express Inclusion program adds URLs to
the AltaVista index and refreshes them weekly. Members of the Expression
Inclusion program can also join the Listing Enhancements program, which
allows members to add logo, text links, and other custom enhancements to
Before the implementation
of the inclusion programs, AltaVista received between tens of thousands
to hundreds of thousands of URL submissions per day to its 22 global search
sites. Even after implementation of the new programs, the free submissions
remain relatively consistent. Public reaction to the inclusion programs
has been "very positive," reports Chris Kermoian, director of Product Marketing
at AltaVista. "In fact, Webmasters were the first people to request a method
to pay to have their URLs added to search engines quickly. They tell us
that they see the program as a valuable tool."
A recent survey
of the more than 20,000 members of AltaVista's paid inclusion program indicated
that more than 75 percent of the members felt that fast inclusion into
AltaVista's index was valuable to them, and more than 70 percent found
the weekly refreshes valuable.
This value doesn't
come without price, however. AltaVista's Express Inclusion subscription
period lasts for 6 months (unlike a year for Inktomi) and costs $39 for
the first URL, $29 for each additional nine URLs, and $19 for each additional
490 URLs after that. (If you have more than 500 URLs, AltaVista offers
a different program for which you can get a quote.) URLs with adult content
cost $79 each.
have their own price set depending on what you get. Logos included in the
listings, for example, cost $50 for one URL and $36 for each additional
URL (again, for a 6-month subscription period). The inclusion of a custom
tagline (which can be updated up to once a week) costs $25 for one URL
and $18 for each additional URL. The enhancements do not change URL rankings,
nor are they indexed. So if you were thinking about buying a URL enhancement
and then stuffing it full of search engine-friendly keywords, think again.
Despite the cost
of the inclusion programs, Kermoian doesn't think they'll either change
the makeup of AltaVista's index or force out the small and noncommercial
sites that can't afford URL inclusion. "The hobby and small money sites
will continue to be fully represented. We continue to crawl the Web and
process freely submitted URLs on a regular basis." In fact, AltaVista has
developed a system for monitoring the Web for noncommercial sites four
times a day. "We want to do this for the quality of our Web search," said
Krista Thomas, directing of Marketing Communications for AltaVista. "We
think this is going to add a lot of value to our index."
AltaVista is a
full-text search engine; that is, it indexes every word of every page added
to its index. That's very different from a searchable subject index like
Yahoo!, which lists only page URLs and descriptions without indexing every
word of every page. But like AltaVista and Inktomi, Yahoo! has its own
URL inclusion program.
A first glance
at Yahoo!'s search engine may lead one to believe it has only a pay-for-placement
program. The first results from the search engine are "Sponsor Matches,"
but those are actually generated and provided by Overture (discussed earlier
in this article) and not Yahoo!. Yahoo! has an in-listing advertising program,
but these results are separated from regular listings by an orange box.
Yahoo! was one
of the first searchable subject indexes, and, for a long time, anyone could
submit to their site for free. In the last couple of years Yahoo! finally
instituted a one-time fee of $299 fee ($600 for adult sites) for commercial
sites that wanted to submit to the Yahoo! index. Then at the end of 2001
Yahoo! changed the one-time fee into a yearly cost for inclusion in the
directory. (Noncommercial sites which just want fast consideration can
also use Yahoo! Express.) And just because the fee is paid doesn't mean
that the submitted URL will get included in the index; fee payment guarantees
only that Yahoo! will review the site within 7 days and get back to you,
letting you know whether your site has been accepted or not. If it declines
your site, Yahoo! will tell you why.
Yahoo! also reserves
the right to edit your description and to place the site wherever it considers
most appropriate. Additionally, Yahoo! has a laundry list of guidelines
for sites submitting under Yahoo! Express, including that the site must
support multiple browsers, all links on the site must work, and that the
site must be in English or have an English-language version available.
Furthermore, once listed via Yahoo! Express, the sites are treated the
same as any other site in the Yahoo! index. For example, if you want to
change your site description, you'll have to use the same queue as the
other millions of sites in Yahoo!'s index.
Despite the restrictions
and the pricing of Yahoo! Express, Yahoo! thinks the pricing will ultimately
benefit everyone. "We have added more human and technical resources as
a result of the annual fee," said a Yahoo! spokesperson. "We feel like
there is a need for this service." Especially if, as the spokesperson noted,
consumers are using Yahoo! to search for more products and services now.
Earlier in this
article, we noted that Yahoo! sells services both to visitors to its site
(with services like Yahoo! Mail) and to site builders who want to be added
to Yahoo!'s searchable subject index. LookSmart, however, provides far
fewer services to visitors and provides its searchable subject index to
many online partners.
get viewed at a variety of sites around the Web, including MSN and AltaVista.
And while comprising a good part of LookSmart's revenue, the listings are
not a large part of the index. "At the end of fourth quarter 2001, listings
represented 62 percent of LookSmart's total revenues. We have more than
3 million URLs in the directory and just 6.5 percent are paid listings,"
said Liz Haggerty, director of Investor Relations for LookSmart. This may
launched a new program that differs from other URL inclusion programs;
it charges per click for traffic from its site. If you want to join its
URL inclusion programs, two fee levels are offered. First is the editor
review fee of $49, in which the URL and description are evaluated for appropriateness
and accuracy. "If a submitting site follows the editorial guidelines they
should be fine," said Robert Goldberg, senior VP of Sales, Marketing, and
Once a site is
listed in the index, it will be charged 15 cents for each click-through
to the site, with a minimum charge of $15 a month (100 clicks.) There's
also a maximum charge of $2,500 a month. If the idea of paying $2,500 a
month for a search engine listing makes your head hurt, you can set a spending
cap. After LookSmart usage meets the cap you set, your site will not appear
in search results again until the next month. (LookSmart will send you
an e-mail and alert you to see if you want to extend your cap.)
LookSmart and Yahoo!
are both searchable subject indexes; AltaVista is a full-text search engine.
But Ask Jeeves offers both, a full-text search engine and a searchable
Ask Jeeves has
been known for their natural language query engine, Ask Jeeves, for several
years. But with the recent acquisition of the search engine Teoma, it provides
both a searchable subject index (Ask Jeeves) and a full-text search engine
(Teoma). And both sites offer URL inclusion programs.
Ask Jeeve's Site
Submit is still, at this writing, in beta testing, but it's still fully
functional. Submitted URLs are included in the site databases within 7
days and are refreshed every 7 days.
Site URLs cost
$30 for the first URL and $18 for each additional URL (up to 1,000 URLs
total). Strangely enough the inclusion period is 15 months, not 1 year
or 6 months.
Ask Jeeves has
a very clear usage policy (no misleading text, no submitted pages with
duplicate content, no sites that misuse affiliate or referral programs)
and submitted URLs that violate spam policies can get kicked out of the
index without a refund.
All the sites mentioned
so far are fairly famous in the search world — what Internet user hasn't
heard of Yahoo!? But there's at least one search engine that keeps a low
profile, even as it builds an index of over 800 million pages. And it has
a URL inclusion program too.
If you haven't
heard of FAST, maybe you've heard of AllTheWeb, a full-text search engine
. And if you haven't heard of them you've probably heard of the portal
Terral Lycos, which is powered by FAST.
Before FAST launched
its URL inclusion program, the search engine got over 1 million submissions
a day through its free submission. (FAST still has a free submission program,
but it takes 6-8 weeks for submitted URLs to be added to the database.)
It's still getting that many, though public reaction to the new program
has been good. "Takeoff's been excellent," said Stephen Baker, director
of Business Development and Marketing for FAST. "We expected hundreds of
subscriptions and we ended up with thousands." Most of the subscribers
to the service have been small customers.
Though the reaction
has been beyond what FAST expected, Baker doesn't expect the program to
change the index very much. "We've got 800 million pages in the search
engine now and we expect to have 2 billion pages at the end of the second
quarter. We don't expect more than 3-5 percent of the index to be paid
FAST sells its
URL inclusion services through partner sites, including Lycos. Lycos calls
the program Lycos InSite Select. Submitted URLs are added to the index
within 48 hours and refreshed every 48 hours. Inclusion lasts for a year.
Unlike the other
programs mentioned in this article, InSite Select charges a membership
fee on top of URL inclusion fees. Membership costs $18 a year, with a $12
Though the economy
seems to be getting better and tech companies are making the first tentative
steps toward recovery, Web-based businesses are still in a tough spot.
URL inclusion seems on the surface like a fair way to generate revenue
— it provides value-added services for those site builders who want them,
but doesn't change site rankings to the detriment of search engine users.
But such programs will have to be in place for a while before it becomes
clear whether so many small businesses and online stores using inclusion
programs are changing the tenor of search engine indexes.
Almost every major
searchable subject index and full-text search engine has a URL inclusion
program. Except one: Where's Google? Google, which has presented itself
as a user-oriented search engine almost from the very beginning, doesn't
feel URL inclusion programs would be appropriate. "We do not have plans
to offer URL inclusion in the future," said Google spokesman David Krane.
"Our goal is to provide our users with objective, trusted search results.
The introduction of a URL inclusion program would upset this balance."