The search for better search

Search engine technology is having to make a great leap forward thanks to the social web.

The technology behind searching the internet has been refined and accelerated over the past decade by successive generations of software engineers. However, the fundamentals of the technology – web crawling, indexing and searching to build results (see “Search terms”) still formed the basis of most search engines’ findings.

That is until the advent of social networking. Websites such as Facebook have attracted 800 million users and social networking sites have become the place where consumers are making decisions about brands and products.

This means search engines have been relegated to a tool for searching for a product once a consumer’s mind has been made up and marketing spend that once went to search engines is slowly leaking to social networking sites.

So now search engines pay closer attention to the way humans interact with their technology and look for ways to capture our interests, likes and dislikes, based on the way we interact through social media and how that influences search terms.

In the tradition of anthropology, Google is paying researchers to observe its users “in the wild”.

It wants to codify more about what influences search terms and whether this knowledge can be used to produce better results. “We’re watching the way people use the internet in their natural habitat,” explains Google “search research” scientist Dan Russell.

“We’re trying to discover more about the way people write based on what they inherently know about an area.”

Meanwhile, experienced searchers are getting better at creating the right combination of words to find what they’re after.

Search terms have been getting longer, Russell observes, and average four words compared with just 2.5 words in 2007.

“Experiences searchers are very good at grouping and ordering terms to find exactly what they are looking for,” Russell says.

“Google’s personalisation provides results based on location, time of day and previous searches but we’re not sure how to tell how experienced a searcher is. That’s more complicated.”

Over the past year, Bing, Yahoo! and Google have tried to help searchers refine their techniques by introducing software that automatically generates popular search threads based on what’s has already been typed.

There was some concern that this software would reduce the diversity of queries. But according to Russell, it has had the opposite effect, increasing the length and diversity of combinations searchers use when attempting to locate information.

“If someone started out searching for Sydney Opera House, for example, they might come across a search thread which more accurately captures what they were looking for, or something they hadn’t considered,” Russell says.

But with 800 million people posting content to social media sites such as Twitter, LinkedIn and Facebook, search engines are seeking ways to capture the sentiment we express and how it influences the information we seek.

Yahoo! and Facebook have been collaborating to this end. Meanwhile, Google started its own social network, Google+, to try to capture our recommendations but its affect on the way search results are indexed and delivered remains a work in progress.

“There’s a lot of energy now around Google+ and a lot of interest in the social space,” Russell says. “But it’s like constantly changing the engine in mid flight, because there’s also all sorts of new material appearing on the internet every day.”

BRW

Comments (0)