What will it take to topple Google?

Google's international dominance in the search market is finally being challenged by new start-up engines and the rise of real-time search. Media Week finds out if anyone is capable of shaking Google's foundations.

What will it take to topple Google?

For a company that handled a decisive 90.58% of all UK search volumes for the four weeks to 27 June (source: Hitwise), Google has a remarkable number of competitors.

From recent market entrants such as Bing, Cuil and Wolfram Alpha to the increasingly credible threat from a flood of real-time search engines, any number of would-be predators circle the search giant, waiting for a gap to emerge in its defences.

In fact, the level of activity in search has been so high in recent months, that many have concluded we must be on the brink of a genuine shift. Well, are we? Quite possibly, although it's unlikely Google will be left behind in all the excitement.

For some years, new concepts in search have been routinely dressed up as Google killers, from the ongoing evolution of the semantic web and mobile search to the rise of Twitter and real-time search. Somewhere among these developments, search experts suspect, is the kernel of a concept that could turn the whole game around.

That notion, combined with a spate of high-profile search engine launches in the past few months and a rare revenue dip for Google in Q1 2009, has fuelled a faint conviction that Google might somehow be slipping.

The launch of Wolfram Alpha on 15 May started the chattering in the wider marketplace. Even The Sun dubbed the fledgling answer engine "Google's search engine rival",  before it emerged that Wolfram Alpha was essentially a scientific knowledge bank, rather than an index of the web.

A fortnight later, Bing, Microsoft's so-called decision engine, launched in the UK in beta form, receiving favourable reviews and posting reasonable, although hardly Google-threatening, numbers. Once again, Google's continued dominance was briefly questioned, but the swing from Google to Bing was not dramatic.

Nevertheless, the search giant was allegedly so shaken by Bing that co-founder Sergey Brin put himself in charge of a task force to get to the bottom of Microsoft's new algorithmic secrets. Google hasn't confirmed that piece of news, although its executives are now comfortable enough to make jokes about "Binging" for information online.

Paul Stoddart, head of search at Microsoft UK, says the very well-funded Bing is driven by the belief that, while Microsoft has a long way to go in search, Google's solutions aren't perfect either.

"We have been working on search for about four-and-a-half years and, right from the start, all our research has shown that people aren't happy with many of their search results. About 50% of search results go unanswered in the way people expect them to. When people actually think about it, they realise ‘this wasn't the search I wanted'."

Microsoft's belief is shared by many other upstart search engines, from Hunch, the decision engine launched in June by Flickr co-founder Caterina Fake and others, to Twitter-inspired real-time engines such as OneRiot, Twazzup and Scoopler.

Andrew Girdwood, head of search at Bigmouthmedia, feels that, while Bing in its US incarnation is "pretty damn good", real-time search is the area that offers most potential. He says: "Twitter is showing people there is another type of internet use out there, another type of search that Google isn't really tapping into."

Real time or otherwise, not every new search engine is necessarily aiming to take on the big boys on its own. A company such as semantic search start-up Powerset illustrates another model: sell yourself to the big boys (to Microsoft for about $100m in June 2008) rather than wage an expensive and fruitless campaign to compete. Powerset's technology is now an important component of Bing.

"Something like Powerset is very inspirational if you are an entrepreneur," says Girdwood. "They basically said: ‘We are doing search differently. We are not telling you how - we will show little snippets to influential bloggers and then sell to Microsoft.' That gives other people the motivation to try."

Of course, Google also continues to innovate. In the past eight months, it has launched SearchWiki, Google Squared and the new Rich Snippets format.

But there is a strong sense in all corners of the market that Google needs competition. This feeling is shared - certainly while the competition lags so far behind in terms of share - by Google itself.

Google spokesman Anthony House says: "We have been saying for a long time that search is not a solved problem, but no one seems to believe us. It is really good that services such as Twitter, Wolfram Alpha and Bing have come along, as they show there are many different ways to look for information."

Because it all adds to the sum of human knowledge? Well, that's certainly part of it. But also, as House points out, because "it shows  the barriers to entry aren't so high that we are in a sinecure".

So there is the challenge, search start-ups. Launch against the market leader and strengthen its regulatory position. In these changing times, there is arguably more opportunity than ever to help shuffle search forward. But don't be altogether surprised if you end up owned by Google or Microsoft.

New entrants to the market
Despite the fact many people seem to know only one search engine, the web is increasingly teeming with them. And they have all had fun coming up with their names, with US start-up Duck Duck Go perhaps the winner on that score.

Among the more conventional runners, Wolfram Alpha, a project created by British physicist and developer Stephen Wolfram, was not the most obvious candidate to secure the level of coverage it did. Designed with factual, academic queries in mind, the engine was described by one website as "like plugging into an electric brain", but its aims are very different to those of Google or most other comparable sites.

While Wolfram Alpha uses its own semantic technology to interpret searches, it doesn't comb the web for its answers - rather it draws them from its own vast database. Speaking at the launch of the site, Wolfram said: "I've wanted to make the knowledge we've accumulated in our civilisation computable. I was not sure it was possible, but I'm a little surprised it worked out so well."

Of the more commercially minded participants in the great race for second place, Bing is by far the most aggressive and apparently the most credible. It plans to spend $100m on marketing in the US and, although it is still in trial mode here, the UK is next up for the big push, according to Microsoft UK's head of search, Paul Stoddart. "We want to make sure Bing in Britain is the best Bing in Britain we could possibly have," he says.

With a new algorithm, some colourful features and a claim to have a deeper understanding of search context, Bing is making many of the right noises, but Microsoft chief executive Steve Ballmer has conceded long-term development is key. The performance of general search in Bing's US version has already impressed many, including iCrossing chief technical officer Paul Doleman.

He says: "I came across a site called BlindSearch that allows you to compare the performance of several engines and I really couldn't tell which search results were Bing and which were Google. In fact, when I was absolutely sure the best results were Google, they invariably turned out to be Bing."

While Bing and Wolfram Alpha have taken the lion's share of the headlines this year, it is easy to forget Cuil, one of the great hopes of 2008.

Founded a year ago by a team including several former Google staff, and loudly boasting of the world's largest search index, its share of search traffic has so far been very small, although it continues to develop its methods.

In common with Indian real-time start-up Yauba, Cuil makes a promise not to store users' IP addresses or search activity. Yauba puts it best, with possibly the world's shortest privacy policy: "We do not keep any personally identifiable information. Period."

Mobile search
The internet was traditionally accessed through conventional computers, as was search.

But with the mobile internet finally gaining genuine traction and offering new possibilities for many different varieties of functional search, the traditional search community is having to re-examine its fundamental principles.

iCrossing's Doleman believes mobile gives leading companies the clout to create their own world of search. He says: "Mobile is an area where Google will face competition from hardware vendors such as Nokia, Samsung and Sony Ericsson, and from operators, who for years have dominated with a walled-garden approach. And what's to stop Apple having its own iPhone search experience that people use, rather than Google?"

Onlookers have high hopes that Bing will provide search tailored to local areas. Ciao and Multimap are already in the Microsoft stable and further tie-ups with Yell are apparently a possibility.

Steak co-founder Duncan Parry is among those who believe local search could be an avenue for Microsoft.

He says: "I always feel that, although Google has made some strides forward in recent months, local search in the UK is still not brilliant. Thanks to the partnerships Microsoft has made, it might be able to do something interesting."

Of course, these are sectors yet to be fully built and yet to be won or lost.

But in a search market where even the most ambitious rival lags so profoundly behind the market leader by any measurement, you can't blame anyone for working towards an alternative future.

Competition from the old guard
The recent reappearance of Jeeves the butler on our TV screens, and indeed of Ask.com, the search engine for whom he works, was a reminder of the fact that the old guard of search has not given up the fight against Google.

Nonetheless, the numbers remain emphatic: as well as handling more than 90% of UK search volumes, Google took 80.9% of search engine visits in June, compared to Bing's 3%, Yahoo's 2.6% and Ask.com's 2%.

The UK is admittedly one of Google's strongest markets and it finds far less competition here than it does in the US where, on the day of Bing's launch, Google received visits from 73.2% of all searchers, compared to Yahoo's 22.1%, Microsoft's 11.8%, Ask's 4.6% and AOL's 1.9% (source: Compete*).

All the same, as the UK and US relaunch of Ask.com proves, the established rivals still see something to play for. Bigmouthmedia's Girdwood points out: "Even a tiny percentage share is quite a lot in a multibillion dollar market."

Ask's refocused proposition is a Q&A "answer farm" of 300 million pieces of data drawn from sources across the web, which are tailored to queries using semantic search and then mingled with regular search results.

Yahoo's announcement in May that it is moving beyond Google's "10 blue links" to "a web of objects" demonstrates that it is prepared to contribute its own share of high-concept search philosophy. Its goal, inspired by its experience of mobile search, is to reflect users' real-world intent in its search results and to offer up fewer documents and more shortcuts and non-textual content.

AOL has not relaxed its grip on life either. In June, it announced an enhanced search service for its Truveo video-sharing subsidiary.

*Figures do not add up to 100% because some users searched in more than one place.

Real-time search
The Google model has reigned supreme and in some comfort for a decade, but it hasn't taken long for the real-time world of Twitter to position itself as the most likely step-change in our search habits.

In a world suddenly flooded with constant updates, it seems incredible that search engines once fretted about how to keep up with bloggers.

These days, anyone with a mobile is theoretically a potential source of world-changing news and these bulletins flood in by the second.

There is a mild irony in categorising Twitter as the future of search, since Twitter's own search function currently exhibits none of the sophistication of even the most basic conventional internet search engine.

Instead, it lists results purely chronologically and the fact it only indexes itself makes its proposition still more one-dimensional.

Nevertheless, as a real-time search engine attached to a social phenomenon, Twitter Search has real significance, even if Twitter's founders are not yet close to developing a mature version.

As co-founder Biz Stone wryly told Media Week in May: "We have a hunch there's an opportunity in search, as Google has proved."

Other people have the same hunch, which is why a tribe of new engines have rapidly built themselves around the concept of real-time search, all with a slightly different modus operandi.

These include Topsy, which serves up the hyperlinks it finds embedded in Twitter messages that appear to be discussing your search term; OneRiot, which does a similar thing, and also searches Digg, YouTube and others; Twazzup, which is much like Twitter Search, but with an algorithm that recognises factors other than recency; and Scoopler, which offers updates from Twitter, Flickr, Digg and Delicious, ranked chronologically and according to popularity.

Meanwhile, earlier this month, Bing started integrating more Twitter-based, real-time data into its search results, and Google is reportedly looking to create a microblogging search engine of its own.

Some rumours claim Google plans to actually swallow Twitter whole, with talks to acquire the business in their final stages.