I've written about this in the past, but this morning's short-lived global Google meltdown seems an appropriate time to repeat the thought.

For years now, I’ve been bringing up E.M. Forster’s extraordinary short story “The Machine Stops” in the context of discussions about Vannevar Bush, Ted Nelson or any conversation touching on our society’s increasing dependence on, and faith in, technology.

It seems hardly anyone has ever heard of this story. People know Forster, of course, for the obvious novels (Passage to India, Howards End, etc.) and the Merchant-Ivory movies of his work. But he was also an exceptionally gifted short story writer, on a par with O. Henry in his mastery of the concise art.

I first read The Machine Stops as part of a collection of stories that were issued as required reading in my fourth year of secondary school in England. Many years later, I found an online copy to download and re-read on my old Palm Vx. This morning's events make me want to go back and read it once again.

In the story, Forster paints a bleak picture of a post-apocalyptic dystopia in which humanity has become so utterly dependent on technology as to be rendered completely helpless when, as the title suggests, the “Machine” that runs the world and all forms of life support, simply stops working.

Forster's Machine has grown over time to become so big and complex that no one living person or group is able to fully grok the complex workings of the thing to start fixing it.

It would be wrong to over-dramatise this morning's very brief Google outage as anything remotely as catastrophic, of course. But for about 20 entertaining minutes there, it seemed like people worldwide had a tiny glimpse into the fearful abyss of a world without Google (and yes, my tongue's more than a little way into my cheek).

Being deprived of our groupmind, even for such a short time, caused an extraordinary flood of messages on Twitter. The search for Twitter hashtag #googmayharm reached 100 pages of posts (about 1500 individual tweets) in under an hour and fast overtook the Super Bowl as the hottest rising story.

“As technology advances, our relative understanding decreases, and our helplessness and confusion increases,” as that Weinberger bloke once said.

Indeed. The curious thing for me is that I'm left more reassured than worried about all this.


When one key part (in this case Google) completely fails - however briefly - we may have a moment of panic, but we quickly learn to route around the damage. There are lots of other search engines out there still; many alternative ways to complete the synaptic connections we've grown accustomed to outsourcing to the great gods of Google.

We should worry less, perhaps, about what happens when a dominant provider such as Google fails, and more about what might happen if the Net ever reaches the point of working too well.

At some point this morning, every single Google search started bringing up linkjacked results with each result flagged like this:


Seems that every single site has now been Net Nannied into oblivion - doesn't matter what you search for, EVERYTHING is flagged with "This site may harm your computer".

No news out of Google as at 9:56am Eastern, nothing on the Google blog, and no response yet from the handful of people I know at Google who I've sent email to - but then, it is Saturday morning. Have to believe someone at the Google HQ is on this though. It seems pretty clear they've been hacked in some way - and it's a hack on a huge scale.

Meanwhile, in the absence of regular media coverage, the Twitter stream is on fire. Search for #googmayharm or #googmeltdown on Twitter and follow the story as it unfolds in real time there.

This is destined to be yet another example of Twitter's emerging importance - denied their Google lifeline, people are turning to Twitter in droves to find out what's going on, ask questions, swap stories. It's the global digital heartbeat of our time.

UPDATE: 10:17 est - I thought at first it was fixed. The same innocuous search for "disney" I ran above now comes up clean. Tested this - it's still broken with other searches, but the second time you run a search it comes through OK.

UPDATE: 10:19 est - Now looks like it's really getting fixed. I think they're rolling the cleanup through servers and datacentres. Some searches still bust, but most are clean. Depends on which server cluster your search hits. Now just waiting to see what Google's PR people are going to say about this. Certainly not the catastrophic digital alzheimer's story some tweets seemed to suggest, but made for an interesting and exciting little half an hour there while we contemplated the death of our groupmind.

UPDATE: 10:27 est - Interesting... I wonder if this global Net nanny hack swept across more than just Google's search servers. I have my blog set to auto-forward all of my new posts to my Gmail account (paranoid belt-and-braces backup). This post got flagged by Gmail as spam. That's certainly never happened before. Was the Google hack wider than just search?

UPDATE 10:32 est - a good point made by John Minnihan (@jbminn on Twitter): I've been carelessly throwing around the word "hacked", but there's no real evidence yet to say whether this was a hack or just a cockup in updating something at Google's servers. This could have been something like an accidental tweak of their malware filters that then rolled out through their entire back end. Curious to see what Google says.

UPDATE 10:40 est - I've seen a suggestion floating around Twitter that the source of the meltdown may have been server failure at StopBadware.org, described as "Google's outsourced malware partner". Perhaps, but that seems a little unlikely. Would Google's infrastructure really be so ill-designed as to allow a single point of failure to knacker their entire search operation like this? More likely, I think, that the flood of click-through traffic to stopbadware.org (linked to from every broken search result this morning) caused the Stop Badware servers to grind to a halt after the fact.

FINAL UPDATE:Feb 2, 11:44 est - It's a couple of days later and this Google brownout is old news now. For the sake of completeness, though, I wanted to just add one final update. As this post on the Official Google Blog states, it turns out that the source of the problem was actually a maddeningly simple human error. Looks like there's some shared responsibility between Google and StopBadware.org (here's the post from StopBadware about the issue), and a little unsurprising finger-pointing going on.

Now that the dust has settled and all is once more right with the world, it's worth noting that Google's response was genuinely impressive here. Problems are bound to happen. Sometimes, even relatively small errors can have catastrophic results - it was a single-character coding error, for example, that ultimately led to NASA's emergency "destructive abort" of the Mariner 1 spacecraft at a cost of many millions of dollars. The test of any individual or organization's mettle is how they respond when things go pear-shaped.

In this case, Google caught the issue fast, diagnosed and rolled out a fix, and then owned up to the problem on their blog and in media interviews, providing full information about how they goofed. Good job. Even better, Marissa Mayer, Google's Vice President of Search Products & User Experience, put her name to the post on the Google blog - not some junior communications staffer or anonymous spokesdrone.

The only thing I'd like to have seen them add to this would be to open that blog to comments. There was an enormous amount of online conversation about this issue, it would be great to see Google fully joining that conversation, as opposed to this uni-directional broadcast approach.

They are maintaining a list of all trackbacks to their blog post, so that all sides of the discussion get some airtime. But for an issue as big as this, I'd like to see them diving into a comment thread and addressing people's questions and concerns in an open dialogue.

Still, a pretty solid crisis response, and one which should help mitigate any damage to their reputation from this short-lived but very high-profile issue.

I'm back from chairing an enjoyable, lively and (I thought) really interesting panel session at tonight's IABC professional development event, Social Media and the Modern Communicator. Many thanks to the IABC Toronto Chapter for organizing and promoting this sold-out event, and to the terrific panelists for giving generously of their time and knowledge - shout out to Mathew Ingram, Jen Evans and Boyd Neil.

Too tired to blog at length, but a quick observation and some links I promised...

First, probably the most startling moment of the evening, for me, was very early on just as we were getting warmed up. I've been speaking about social media at conferences, seminars and other events for nearly seven years, and I figured we must be getting way beyond the 101 level by now.

We had an audience of just over 200 professional communicators at the event tonight. In an effort to gauge the general awareness and knowledge level of the audience, I asked a couple of quick qualifying questions. Here they are, with my rough assessment of the results based on a show of hands:

1. How many people here are actively blogging?
- Approximately a dozen people, perhaps 20 at most (out of the 200)

2. How many people here are on Twitter?
- Close to 60% of the room!

This blows my mind. I know that Twitter is a heck of a lot quicker and easier to get started with than full-on blogging, and I guess it requires less commitment and close to zero tech skills, but I'm still delighted and amazed at just how many people in Toronto have caught the bug.

Hey! Shel Israel! - we got your Twitterville right here!

Is this what it's like elsewhere? Has the growth of Twitter been as fast in other cities, or is the T.O. really as special as we like to think it is? With so many Twitter apps out there, has anyone worked up a Google Maps mashup that shows the concentration of tweets per capita in various parts of the world?

Fascinating stuff (for a complete nerd like me). I'm going to have to do some more digging around to see if this is anomalous or if it just seems that way from inside the bubble.

Meanwhile, my esteemed panelists and I dropped a number of links and tips during tonight's session, which I promised I'd try to catalogue here. I don't know that I captured all of them, but here are the ones I remember.

Social Media Policies
A few good examples were mentioned, including those used at Dell, IBM, and elsewhere. I've been collecting and bookmarking something of a list of interesting social media and corporate blogging policies for a couple of years using the Delicious social bookmarking tool. You'll find all of these (including the Dell and IBM examples) here: http://delicious.com/michaelocc/policy

I also mentioned (with my tongue only half in my cheek) the shortest (and one of the best) HR policy manuals ever written ("Rule 1: Use Good Judgement," etc.). I blogged about this a while back in the context of policies for corporate Twitter use, here.

Thirdly, you might be interested in a sample of one of the "online interaction" policies we've helped develop for our clients. You can find one in the privacy policy at the foot of the Herbal Magic site, here.

Tools for Internal Social Media
There was a good question at one point about "Twitter behind the firewall". Our panelists rattled off a bunch of examples, probably too fast for many people to note down. Here are some applications worth checking out.

Yammer (Twitter-like internal micro-blogging, as used by Boyd's firm)

Present.ly (think: Yammer, but with better admin controls and UI options. My colleague, Dave Fleet, has a great review of Present.ly here)

For the technically adept, there's also Laconica - a DIY platform to build your own Twitter-like apps (of which the best known implementation so far is at Identi.ca)

It's also worth mentioning, that if you want to add full-fledged blogging inside the firewall, it's very easy to set up a Wordpress installation for internal communications purposes. Works well, easy to administer, and there are a bunch of good people around who can help you get things working how you want them (including, I'm cheesily obliged to point out, a certain great firm that can offer both the design & build work and the strategic consulting help).

Recommended Reading
I asked the panelists for book recommendations and think they offered some terrific ideas. In no particular order, here are the ones I can recall us mentioning (and a couple of bonus titles we didn't, but perhaps should have):

Here Comes Everybody - Clay Shirky

What Would Google Do? - Jeff Jarvis

Groundswell - Charlene Li and Josh Bernoff

Web Analytics: An Hour a Day - Avinash Kaushik

The Cluetrain Manifesto - Levine, Locke, Searls & Weinberger

Small Pieces Loosely Joined - David Weinberger

Everything is Miscellaneous - David Weinberger

(You notice a theme here, btw? Basically, you should just read everything UofT alum David Weinberger has ever written, including his splendid blog. Yes, he's an old friend. Yes, I'm completely biased - but the man is a certified genius and a funny, wonderful writer)

Gonzo Marketing - Chris Locke

Wikinomics - Don Tapscott and Anthony Williams

And finally, a personal favourite I think a lot more people ought to read:

Ambient Findability - Peter Morville

And the last little housekeeping link in this now-not-so-short post - we mentioned the US Air Force's "decision tree" used to determine how and when they will respond to online discussion. Dave Fleet (yes, him again) has a post on the topic here and Toronto's favourite accordion-playing supergeek social media pioneering thriller from Manila, Joey deVilla, has a bigger, updated version of the chart, here. (Hey, Joey! I think I just made you sound like @Mike_56)

That's all I can think of for now. Thanks again to all that attended, to the boss for some great live tweeting, and to everyone following on Twitter for splendid questions and discussion during and after.

(Already said this on Twitter earlier tonight, but I'm rather inappropriately pleased with it, so couldn't resist the crosspost...)


It's pretty obvious why Loblaws' "No Name" brand cookies are not as tasty as the fancy ones. They're sans nom.

That is all.

This is a tale of two disc deaths - and two happy endings.

Over the Christmas break, the 100GB hard drive on my work laptop (let's call it Dell #1) crashed. I couldn't boot, couldn't get through Vista's Startup Repair, couldn't rebuild Vista from the original discs - it was dead, dead, dead.

Then just last Saturday, Sausage accidentally knocked the home laptop (Dell #2 - 40GB) off the dining table, while it was running - effectively killing its hard drive too.

Ouch.

The good news with Dell #1, my work machine, is that there weren't any valuable client documents or other irreplaceable files on the C: drive. All that stuff was safely stored on the servers (having worked in the document management business for a big chunk of my career, I'm pretty careful about that stuff).

There were, however, a big mess of family photos, old documents, and personal files on the drive. Again, these were all files I had copies of elsewhere; the only problem being that the elsewhere, in this case, was our home laptop.

Double ouch.

My work laptop isn't all that old, so it was good to learn that the drive was still under warranty. Once The IT Department and I had done all we could to prove to ourselves that the drive was indeed utterly b0rked, we reported it to Dell who duly sent out a brand new 100GB drive the very next day. Outstanding. I was able to get the work laptop back up and running fairly quickly (although I'm still tweaking and tuning the setup to get it back to the way it was before the crash).

Meanwhile, I'm left with a big hole in my personal document files, and a dead laptop at home.

As a first step, I sent the drive from Dell #1 out to a data recovery lab for a quote. Meanwhile, I spent an unhappy and fruitless evening trying to rebuild Windows on the drive of Dell #2. Diagnosis: that drive was also utterly and completely b0rked.

The price estimate for the drive from Dell #1 came back from the recovery lab the next day. $1,800. Triple ouch. Really, I'm not too surprised - data recovery is difficult work and, as the lab guys will always tell you - you have to think of the value of the data they're restoring for you. If I'd made the mistake of keeping a lot of client work on the local drive, $1,800 to restore it would have been a snip. But for a bunch of personal files I might be able to recover by other means? Hmmm...

Instead, I asked the lab to send the drive back to me and moved on to a two-stage plan B. I hopped over to TigerDirect.ca to browse their cable selection, then paid a visit later that evening to our local cheapo computer shop (the excellent Beach Impressions on Queen Street East) and scored a replacement drive for Dell #2.

Installing and setting up the new drive in the home laptop has (so far) proven entirely painless - and I'm in the process of restoring everything that we lost, thanks to the wonders of Mozy Home backup. I can't recall who first turned me onto Mozy, but I can enthusiastically recommend it to anyone looking for a seamless backup solution.

This little beastie sits in the background, sending backup data up into the cloud whenever your machine is idle. It's just about foolproof. For a reasonable annual fee, you get unlimited online storage and a backup regime you don't even have to think about.

With Dell #1 back in one piece and happy again, and Dell #2 on the road to complete recovery, thanks to the wonders of Mozy's cloud storage, the only thing remaining was to see what I could do with that dead drive from Dell #1.

Again - I knew I'd have most of the family photos and stuff I needed up on Mozy's servers - but what if there was something missing? Something I'd only had on Dell #1 and had forgotten to synch?

Thanks to a tip from a friend, I knew TigerDirect would probably have what I needed. Sure enough, I was able to find a cable to connect the fancy SATA drive from Dell #1 to a USB port on Dell #2. I found this little marvel for a mere twenty bucks.

As I write this, the "dead" drive from Dell #1 is churning away on the table beside me, happily squirting data onto the new drive I installed in Dell #2. Hooking up the USB-to-SATA connection couldn't have been easier and, although a few files seem to be FUBAR, most of the photos and other stuff on the old drive appear to be intact.

So. The final reckoning:

Professional data recovery quote for one dead 100GB drive: $1,800
Estimated cost for professional recovery on 40GB drive: hard to say, but I doubt it would have been any less than $800
Estimated total: $2,600

One USB-to-SATA drive cable: $19.99 plus shipping & tax (about $30 all in)
One replacement 80GB hard drive for Dell #2: $55.57 including tax.

Total data loss: minimal
Net savings: about $2,500

That make me one happy (and smug) geek.

I took a quick snap of Ruairi's journal this morning before he packed it into his school bag. His teacher encourages the class to keep a journal in which they write about the interesting and important events in their daily life. See if you can decode Ruairi's entry from yesterday:


His writing's not the best yet (he's a December baby) but once you tune into his phonetic style it makes perfect sense. It does my heart good to note that the most noteworthy event in my baby boy's life yesterday was that "Broc Oboma beacam the prosudint the presadint." Big day for all of us, Ruairi. One to savour for a long time.