Launched in mid-1996 by Wired Digital, HotBot is the least-trafficked mainstream search site on the Net. Wired Digital would not disclose the number of hits the site receives, but according to Relevant-Knowledge, HotBot had 4.1 million visitors in May, compared with league-leading Yahoo at 30.6 million and Excite's 15.9 million tally.

But as Wired Digital enters a new phase without its namesake print publication — which was bought last month by Advance Publications- HotBot stands out as the flagship property among sites that include Hot Wired and Wired News. Among HotBot's comparatively modest user base is a following of technically savvy Web users, and the search engine has earned acclaim from several industry publications for the relevance of its results.

To boost its appeal, the company has recently made changes both to the site's interface, with a fresh redesign dubbed Release 5.0, and to its server infrastructure to improve performance and functionality.

As director of software development for Wired Digital, Bowen Dwelle oversees HotBot's infrastructure. Dwelle just completed the process of refitting HotBot's back-end architecture as a completely Windows NT Server environment for Web serving.

HotBot moved to NT from Unix in order to better work with the next generation of Inktomi Corp.'s search engine technology, which provides the back-end searching for HotBot. Changing platforms also enabled HotBot's staff to be able to better control the site's look and feel, Dwelle said. Most of HotBot's previous interface elements were designed and implemented by Inktomi.

"NT certainly isn't as mature as Unix for Web serving," Dwelle said. "But the trade-off there is a little more work in terms of scalability and stability for being on the NT technology bandwagon."

Another major goal of the refit was to shorten user response time, Dwelle said. Part of that was accomplished by reworking some elements of the site's design.

"We did a significant amount of analysis of the way the pages were being rendered by the browsers and delivered," he said. "Once we did that, we took a nice jump downward in the amount of bandwidth we were using, but we'll reclaim that with growth."

Before the redesign, HotBot's average daily bandwidth usage was at about 33 Mbps. The site is now running at about 25 Mbps, according to Dwelle.

Wired Digital last month moved HotBot from its San Francisco headquarters to a colocation facility run by Exodus Communications in Santa Clara, Calif.-the same data center where Inktomi colocates its own search servers-which provides UI) to 50 Mbps of Internet bandwidth. A direct T-1 line connects Wired Digital's offices to Exodus. Eventually, all of Wired Digital's sites will be moved to Exodus, Dwelle said.

HotBot's site runs on 15 Windows NT 4.0 servers, with a Cisco LocalDirector device handling load balancing. A cluster of five Sun Ultra 2300s running NetGravity 3.0 handles ad-serving duties.

"We've found that in terms of hardware scalability, we'll eventually get into bigger machines, but in general we've been happier with a larger number of smaller machines," Dwelle said. "It gives us more flexibility in terms of load balancing and redundancy."

CONNECTED AT THE SOURCE

HotBot's Web servers are connected to Inktomi's 166 Sun Ultra 2 servers over Exodus' 100-Mbps network. HotBot communicates to the search servers via the Inktomi Data Protocol (IDP), a proprietary query interface provided to Inktomi's partners. which in addition to Hot Bot include Yahoo. Cnet's Snap, and Canada.com.

When a user enters a search into the HotBot interface, HotBot's NT servers pass the information on the query to the Inktomi servers via the IDP protocol. One of Inktomi's 166 servers receives the query, then passes it on to the others in the cluster. Each server then looks through its part oft be database to find the pages that best match the query search terms, and sorts them according to Inktomi's relevance algorithm. Once sorted, the search results are passed in parsed text format back to the HotBot NT servers, which present the results.

The Inktomi database itself is everchanging as its crawlers constantly prowl the Web for new and updated Web pages. A cluster of 20 Intel-based machines split the crawling and indexing duties. The database has grown to include about 110 million individual documents, more than double the 54 million it held last June, according to an Inktomi spokesman.

As with most search engines, traffic spikes can be driven by whatever is on people's minds on a given day. HotBot has had its "Clinton-Lewinsky" spike, its "Seinfeld" spike, and its "Pakistan nuclear bomb" spike, Dwelle said. But the sheer size of the service and the Inktomi database tends to make those spikes less dramatic.

"As a content site, you might see your traffic go up five times or ten times on a particular day if a story breaks that is relevant to its content," Dwelle said. "For us, we might see traffic jump two times."

Demographically, HotBot tends to be the search engine of choice among technology enthusiasts, something that HotBot sees as a strength and a testament to the power of its technology. But industry analysts said that as major media companies — like NBC and Disney-muscle into the Web portal business, strong search engine technology may not be enough to help HotBot compete.

"Hot Bot is going through some of the similar experiences that AltaVista is going through," said Chris Charron, senior analyst with Forrester Research. "They're having a tough time parlaying a strong search engine into something more.

While HotBot has added a new shopping directory feature and other content through partnerships with such sites as Bell Atlantic's BigYellow business directory site, its core mission continues to he the search function.

Dwelle said HotBot is planning to launch new features, currently under development in cooperation with lnktomi, that will begin to appear on the HotBot site in the coming months. However, he declined to provide details on what the new elements might include.

He did indicate what future services will probably not include-HotBot will never seek to become a portal site, Dwelle said, a message that is consistent with the direction for the site that other Wired Digital executives have described.

"We've never been a 'me-too' kind of site," Dwelle said. "There are features of the so-called portals that we're interested in. But we're certainly not interested in building just another portal."