If you answered 20 you're absolutely correct!
Unfortunately for Google, what they proved is NOT their accusation that Bing is copying Google's search results.
For the three of you that don't already know what I'm talking about, allow me to fill you in.
Google manually ranked sites for 100 different gibberish search terms that had no results in Bing or Google. 20 engineers then installed the Bing toolbar and performed searches for those gibberish terms using Internet Explorer, clicking on the page they'd previously manually ranked at the top of their own results.
In 9 of the 100 tests, Google claims they started seeing the site they'd manually ranked show up ranked #1 over at Bing.
This, Google says, is smoking gun proof Bing is "copying" Google's search engine rankings. Google then served the hit piece story up on a silver platter to Danny Sullivan of Search Engine Land and it just happened to "go to print" the morning of Bing's highly publicized search event.
Google's Matt Cutts and Bing's Harry Shum then duked it out on the Future of Search panel with Google again claiming Bing was copying their results, Bing denying it, and for good measure throwing in an allegation that Google makes billions helping to monetize spam.
Since the internet barely slows down long enough to read headlines, that's where most of the coverage of this ongoing battle between the search engines stops.
Unfortunately, that means the jury of public perception won't notice the plethora of problems with Google's test, accusations, and conclusions.
When reading the kind of bombastic accusation Google leveled today, it's easy to look past the details. And when Google is able to provide an example or two that seems to support their accusations, the evidence seems "pretty convincing" to quote Dave Winer.
However, in 91 of the 100 queries, Bing's results did NOT mirror the Google fabrications. Of course, you don't hear about those 91 results from Google in their official blog post on the issue, or from Matt Cutts when he was on the panel. Instead they prop up the 7 or 9 (Danny Sullivan's article wasn't very clear what the precise number was) as proof that this is happening across a large number of Bing's search results.
Remember, these 100 queries were complete gibberish and had few if any results before Google performed the test. Yet they only saw a success rate of 9%! If click data from Google were being used heavily within Bing's ranking algorithm, one would expect to have seen a MUCH higher success rate in Google's test.
To make the leap from a 9% success rate on search queries manufactured to confirm Google's suspicions to assume this is impacting enough queries to create a "statistical pattern... too striking to ignore" takes a leap of faith or a preconceived confirmation bias.
Bing Admits Using User Data
Another fact Google seems to gloss over is that they instructed their engineers to click on the Google listings and provide Bing click data.
This is important because it's absolutely no secret Bing uses click data from it's users to influence their rankings. It's disclosed in the agreement for the Bing toolbar, and Bing made it quite clear that this kind of data was being used in response to Danny Sullivan's article.
Despite apparently being beyond Matt Cutt's comprehension (see below), Bing's stance is fairly simple. They use click data collected from users as one of what they claim is over a thousand factors influencing their rankings.
By choosing gibberish search terms that produced no natural results, Google effectively eliminated all other factors that could have influenced Bing's rankings. The terms were found on none of the pages in the index and certainly aren't used in any links pointing to any sites. So when Google engineers suddenly provided a data point (the click data collected via the Bing toolbar) for Bing to latch onto, it's no surprise that even a minor factor was able to influence 9% of the rankings.
There are a couple of points that are very important to establish at this point. First, ranking the same site #1 for any given query doesn't mean Bing is "copying" Google's results.
Second, the data being collected does NOT belong to Google. Yes, the data is being collected when the user is visiting Google, however, the users opted to provide their use data to Bing. Bing didn't somehow hack into Google's data, and Bing didn't scrape Google's search result page. Bing used data provided by their users to influence their rankings, which is a practice they've readily admitted is in use.
Bing SHOULD Use User Data
To take it one step further, given the complete lack of any other ranking factors, Bing owes it to their users to use any and all available data to provide the best experience possible. Would it be better for users if Bing refused to serve a result that click data suggests is relevant simply because the data was gathered on a Google property?
Of course not.
Proof NOT in the Pudding?
Had Google manually altered their rankings, and NOT provided Bing with corresponding click data (or perhaps even clicked on the site ranking 10th instead of first) Google's accusations would have been supported if Bing's rankings mimicked Google's.
Unfortunately for Google, they DID provide user click data. As a result, the only thing their "sting" proved was that click data influenced Bing's rankings in 9 of the 100 search queries Google manufactured, NOT that Google's rankings were copied in any way, shape, or form.
If you enjoyed this article you might want to read our follow up piece, Bing Sting Snares Google. Enjoy!