Published by: Technology Review
Written by: Erica Naone
A real-time search engine bases its results on users' browsing habits.
Later this week, a new "real-time" and "social" search engine called Wowd will open a beta version of its service to the public. The company says that its search results include only pages that have actually been visited by its users, and that its ranking algorithms offer information based on its freshness and popularity.
According to CEO Mark Drummond, Wowd is trying to strike a balance between the up-to-date but chaotic results produced by a site like Twitter and the slower-to-change results that come from traditional search engines such as Google. He expects Wowd to be particularly valuable to users who want to know what content is currently popular, and who see search as an additional feature.
"If what you're interested in is reference search, Google is great, and we don't imagine that we or anyone else is going to displace that," Drummond says. "If, instead, you want to find out what the planet is talking about, that's where we think Wowd can win."
Anyone will be able to visit Wowd's website and perform basic searches, but the company hopes that many users will agree to install Wowd's software on their machines and have their browsing tracked anonymously. This is how the company hopes to build its search index. Unlike traditional search engines, which use crawlers to build up a database of pages on the Web and store that information in vast data centers, Wowd's users will be both the crawler and the storage.
Drummond says users have an incentive to install the code because it also unlocks additional features, including a recommendation engine and advanced search features such as the option to search through one's own Web history.
Users who agree to install the software will feed back to Wowd a list of the sites they visit. Drummond says the company views a visit to a Web page as evidence that the page contains some interesting content. The idea is similar to that used by recommendation sites such as Digg or StumbleUpon, only Wowd offers users a more passive way to participate. "Voting for something [on these sites] requires an order of magnitude more emotional commitment than just looking at something," Drummond says.
Wowd's system will also store bits of its search index on each user's computer. The company uses an algorithm called Kademlia, which is more commonly employed in peer-to-peer file-sharing systems, to keep its index available even when some users shut their computers down. This is done by storing each bit of information on several machines and by using statistical analysis to determine how to share it in order to keep it available.
Drummond stresses that the system is designed to safeguard privacy. For example, when a user visits a page--nominating it for Wowd's index in the process--that page is actually accessed and indexed by a different user's computer. This process protects against accidentally collecting information that is protected by passwords or cookies. The site also does not index any pages protected by encrypted Internet protocols.
Wowd doesn't simply report on what its users are doing in real time, as some Twitter search engines do, for example. Instead it uses its own algorithms that balance a page's freshness against its apparent popularity. When searching in Wowd, a user can choose to see results ranked by popularity or by freshness. In either case, the page will automatically update and adjust its results over time if the user leaves it up.
Wowd contains several ways for users to discover new content, including a "hot list" on its front page that simply shows the top sites being visited by users. Later, Drummond says, the site will add more ways for people to discover what's popular in particular interest areas.
Other search companies, for example the German company Faroo, have also tried to use a distributed architecture to obtain more real-time search results. Faroo's CEO, Wolf Garbe, says that the approach significantly reduces the costs for a search startup. He also believes that any real-time search engine will have to depend on users in some way, since even a company with resources as large as Google's can't crawl the Web anew every five minutes.
Petar Maymounkov, a researcher at MIT who helped design the Kademlia algorithm, says that Kademlia is suited to Wowd's architecture. However, he notes that Kademlia isn't well-protected against malicious users who may want to game the system. Whether any distributed architecture can be secure when there's no central control over who participates remains an open question, he says.
Maymounkov adds that it will be interesting to see how well Kademlia works as the foundation for a search engine. In the 10 years that the algorithm has been used, he explains, it's mainly powered file sharing. And since file sharing is most often used to obtain free movies and music, he says that users of such software tend to be willing to donate a lot of computing power and to put up with some hiccups in availability to retrieve that information. User expectations for a general search engine may be different, he says.
Wowd "is a nice idea, but they've got some work to do," says Daniel Tunkelang, cofounder of a search company called Endeca, based in Cambridge, MA. The preview version of the site, in his opinion, doesn't provide enough high-quality search results to make it stand out from other attempts at real-time search. As more users sign up after the site's launch, this may provide better-quality data and improve the results, Tunkelang says, adding that he's reserving judgment until then.
