Has Google been hacked?
At some point this morning, every single Google search started bringing up linkjacked results with each result flagged like this:
Seems that every single site has now been Net Nannied into oblivion - doesn't matter what you search for, EVERYTHING is flagged with "This site may harm your computer".
No news out of Google as at 9:56am Eastern, nothing on the Google blog, and no response yet from the handful of people I know at Google who I've sent email to - but then, it is Saturday morning. Have to believe someone at the Google HQ is on this though. It seems pretty clear they've been hacked in some way - and it's a hack on a huge scale.
Meanwhile, in the absence of regular media coverage, the Twitter stream is on fire. Search for #googmayharm or #googmeltdown on Twitter and follow the story as it unfolds in real time there.
This is destined to be yet another example of Twitter's emerging importance - denied their Google lifeline, people are turning to Twitter in droves to find out what's going on, ask questions, swap stories. It's the global digital heartbeat of our time.
UPDATE: 10:17 est - I thought at first it was fixed. The same innocuous search for "disney" I ran above now comes up clean. Tested this - it's still broken with other searches, but the second time you run a search it comes through OK.
UPDATE: 10:19 est - Now looks like it's really getting fixed. I think they're rolling the cleanup through servers and datacentres. Some searches still bust, but most are clean. Depends on which server cluster your search hits. Now just waiting to see what Google's PR people are going to say about this. Certainly not the catastrophic digital alzheimer's story some tweets seemed to suggest, but made for an interesting and exciting little half an hour there while we contemplated the death of our groupmind.
UPDATE: 10:27 est - Interesting... I wonder if this global Net nanny hack swept across more than just Google's search servers. I have my blog set to auto-forward all of my new posts to my Gmail account (paranoid belt-and-braces backup). This post got flagged by Gmail as spam. That's certainly never happened before. Was the Google hack wider than just search?
UPDATE 10:32 est - a good point made by
John Minnihan (@jbminn on Twitter): I've been carelessly throwing around the word "hacked", but there's no real evidence yet to say whether this was a hack or just a cockup in updating something at Google's servers. This could have been something like an accidental tweak of their malware filters that then rolled out through their entire back end. Curious to see what Google says.
UPDATE 10:40 est - I've seen a suggestion floating around Twitter that the source of the meltdown may have been server failure at StopBadware.org, described as "Google's outsourced malware partner". Perhaps, but that seems a little unlikely. Would Google's infrastructure really be so ill-designed as to allow a single point of failure to knacker their entire search operation like this? More likely, I think, that the flood of click-through traffic to stopbadware.org (linked to from every broken search result this morning) caused the Stop Badware servers to grind to a halt after the fact.
FINAL UPDATE:Feb 2, 11:44 est - It's a couple of days later and this Google brownout is old news now. For the sake of completeness, though, I wanted to just add one final update. As this post on the Official Google Blog states, it turns out that the source of the problem was actually a maddeningly simple human error. Looks like there's some shared responsibility between Google and StopBadware.org (here's the post from StopBadware about the issue), and a little unsurprising finger-pointing going on.
Now that the dust has settled and all is once more right with the world, it's worth noting that Google's response was genuinely impressive here. Problems are bound to happen. Sometimes, even relatively small errors can have catastrophic results - it was a single-character coding error, for example, that ultimately led to NASA's emergency "destructive abort" of the Mariner 1 spacecraft at a cost of many millions of dollars. The test of any individual or organization's mettle is how they respond when things go pear-shaped.
In this case, Google caught the issue fast, diagnosed and rolled out a fix, and then owned up to the problem on their blog and in media interviews, providing full information about how they goofed. Good job. Even better, Marissa Mayer, Google's Vice President of Search Products & User Experience, put her name to the post on the Google blog - not some junior communications staffer or anonymous spokesdrone.
The only thing I'd like to have seen them add to this would be to open that blog to comments. There was an enormous amount of online conversation about this issue, it would be great to see Google fully joining that conversation, as opposed to this uni-directional broadcast approach.
They are maintaining a list of all trackbacks to their blog post, so that all sides of the discussion get some airtime. But for an issue as big as this, I'd like to see them diving into a comment thread and addressing people's questions and concerns in an open dialogue.
Still, a pretty solid crisis response, and one which should help mitigate any damage to their reputation from this short-lived but very high-profile issue.
Seems that every single site has now been Net Nannied into oblivion - doesn't matter what you search for, EVERYTHING is flagged with "This site may harm your computer".
No news out of Google as at 9:56am Eastern, nothing on the Google blog, and no response yet from the handful of people I know at Google who I've sent email to - but then, it is Saturday morning. Have to believe someone at the Google HQ is on this though. It seems pretty clear they've been hacked in some way - and it's a hack on a huge scale.
Meanwhile, in the absence of regular media coverage, the Twitter stream is on fire. Search for #googmayharm or #googmeltdown on Twitter and follow the story as it unfolds in real time there.
This is destined to be yet another example of Twitter's emerging importance - denied their Google lifeline, people are turning to Twitter in droves to find out what's going on, ask questions, swap stories. It's the global digital heartbeat of our time.
UPDATE: 10:17 est - I thought at first it was fixed. The same innocuous search for "disney" I ran above now comes up clean. Tested this - it's still broken with other searches, but the second time you run a search it comes through OK.
UPDATE: 10:19 est - Now looks like it's really getting fixed. I think they're rolling the cleanup through servers and datacentres. Some searches still bust, but most are clean. Depends on which server cluster your search hits. Now just waiting to see what Google's PR people are going to say about this. Certainly not the catastrophic digital alzheimer's story some tweets seemed to suggest, but made for an interesting and exciting little half an hour there while we contemplated the death of our groupmind.
UPDATE: 10:27 est - Interesting... I wonder if this global Net nanny hack swept across more than just Google's search servers. I have my blog set to auto-forward all of my new posts to my Gmail account (paranoid belt-and-braces backup). This post got flagged by Gmail as spam. That's certainly never happened before. Was the Google hack wider than just search?
UPDATE 10:32 est - a good point made by
John Minnihan (@jbminn on Twitter): I've been carelessly throwing around the word "hacked", but there's no real evidence yet to say whether this was a hack or just a cockup in updating something at Google's servers. This could have been something like an accidental tweak of their malware filters that then rolled out through their entire back end. Curious to see what Google says.UPDATE 10:40 est - I've seen a suggestion floating around Twitter that the source of the meltdown may have been server failure at StopBadware.org, described as "Google's outsourced malware partner". Perhaps, but that seems a little unlikely. Would Google's infrastructure really be so ill-designed as to allow a single point of failure to knacker their entire search operation like this? More likely, I think, that the flood of click-through traffic to stopbadware.org (linked to from every broken search result this morning) caused the Stop Badware servers to grind to a halt after the fact.
FINAL UPDATE:Feb 2, 11:44 est - It's a couple of days later and this Google brownout is old news now. For the sake of completeness, though, I wanted to just add one final update. As this post on the Official Google Blog states, it turns out that the source of the problem was actually a maddeningly simple human error. Looks like there's some shared responsibility between Google and StopBadware.org (here's the post from StopBadware about the issue), and a little unsurprising finger-pointing going on.
Now that the dust has settled and all is once more right with the world, it's worth noting that Google's response was genuinely impressive here. Problems are bound to happen. Sometimes, even relatively small errors can have catastrophic results - it was a single-character coding error, for example, that ultimately led to NASA's emergency "destructive abort" of the Mariner 1 spacecraft at a cost of many millions of dollars. The test of any individual or organization's mettle is how they respond when things go pear-shaped.
In this case, Google caught the issue fast, diagnosed and rolled out a fix, and then owned up to the problem on their blog and in media interviews, providing full information about how they goofed. Good job. Even better, Marissa Mayer, Google's Vice President of Search Products & User Experience, put her name to the post on the Google blog - not some junior communications staffer or anonymous spokesdrone.
The only thing I'd like to have seen them add to this would be to open that blog to comments. There was an enormous amount of online conversation about this issue, it would be great to see Google fully joining that conversation, as opposed to this uni-directional broadcast approach.
They are maintaining a list of all trackbacks to their blog post, so that all sides of the discussion get some airtime. But for an issue as big as this, I'd like to see them diving into a comment thread and addressing people's questions and concerns in an open dialogue.
Still, a pretty solid crisis response, and one which should help mitigate any damage to their reputation from this short-lived but very high-profile issue.
