The Web Factory: weathering a 36-hour hardware disaster
Micron21 has maintained a tightknit relationship with The Web Factory for over five years. So, when the web hosting business was struck by a critical server incident with potentially disastrous consequences, there was only one course of action: comprehensive support. Even in the midst of difficult circumstances.
What do you do when a customer is facing a critical hardware problem, with potentially massive consequences for their business?
What if that same customer is in the process of migrating hardware to an alternative, competing provider? As far as the Micron21 team is concerned, the answer remains the same. You provide support.
When reputations are on the line and relationships are there to honour, you simply do whatever it takes.
Here's how our technicians offered comprehensive support for a small hosting business experiencing critical hardware failure over a 36 hour period—even while The Web Factory was in the midst of transferring hardware to a new provider.
SOLUTION
There was no malice or anything, at any point. I was thinking with my wallet. Sometimes you get complacent with the deal you've got. I hadn't had anything serious go wrong in a long time.
The background
Grant, the owner of The Web Factory, a small-to-medium hosting service provider, has maintained a relationship with Micron21 for many years. He's witnessed the evolution of the business and watched our suite of services expand.
But some time last year, he decided it was time to change provider.
There was no malice or anything, at any point. I was thinking with my wallet. Sometimes you get complacent with the deal you've got. I hadn't had anything serious go wrong in a long time.
While travelling to the Micron21 Data Centre from his operations base in Moss Vale, Grant's server began experiencing serious issues.
The original purpose of Grant's trip was to migrate co-located equipment from our tier four facility into the new provider's environment. However, the situation began to unfold differently.
As Grant was en route from New South Wales down to Victoria, things were heading south in more ways than one.
The server went down the night I arrived in Melbourne to move the equipment on to the new environment. It was a critical failure.
The issue
A week before the incident, a hard drive failed on Grant's server. It was replaced quickly under a maintenance agreement, it appeared to rebuild, and everything seemed fine.
When the server crashed on the night of the issue, it was complaining of another failed hard drive. Our operations centre immediately identified the issue and set about clarifying and rectifying the problem.
Upon further investigation, we discovered the RAID array had degraded. As soon as our technicians rebuilt the first drive, a second drive failed the following day.
When the first hard drive failed, it punctured the RAID array, so the error from the original drive had copied to its mirror throughout the backup process.
When the mirror drive was replaced, the same error was copied back to the new hard drive. Due to firmware issues, the problem compounded itself until it basically corrupted everything, forcing multiple drives to fail one after the other.
Through proactive monitoring, we were able to identify the issues as they evolved and keep Grant constantly updated. Full transparency was key in alleviating his concern.
Micron21 are immediately aware when something goes wrong, and they're working on it. They're alert and involved in what's going on: they make your problem, their problem.
Still, Grant's issues were compounding. With clients beginning to ask questions, he began to feel the pressure.
It was very stressful from my point of view, with my customers wondering ‘What's going on?' I fielded probably 400 phone calls. But with the Micron21 team in my corner, I could confidently reassure those customers.
He continues: "we stayed online through borrowed hardware that the Micron21 team actually provided. I think it was about 36 hours duration for the issue, and throughout most of that time the majority of the business was online".
The team were instrumental in keeping The Web Factory online throughout the initial issue, and getting it back into a stable state.
With the business online and operational, Micron21 engineers set about the complicated task of recovering data from impossibly degraded hard drives.
We leveraged relationships with industry-leading vendors Dell and VMWare to accelerate the process.
Says Grant: "The team dealt with all of the third party providers, which was a huge load off. That's a convoluted process when you haven't done it before. They were dealing with people through the day and night; the whole support team working in shifts."
"Micron21 leveraged their sway with third party providers on my behalf, to ensure we got things done quickly."
By the end of the first 24 hours, our team was attempting to recover data from a RAID array that had lost three of eight drives. Eventually, we managed to recover 50% of the data with no loss.
At the end of the 36 hour ordeal, we had successfully managed to minimise the damage through maximum effort from the team.
The team's involvement, seniority, and their expertise is critical. But they also genuinely care. There was never any question about what they were going to do or how they were going to treat me.
The outcomes
After the incident, Grant decided to re-contract with Micron21 and renew his agreement instead of migrating his services to the new provider.
"For me, this experience hammered home that Micron21 is all about partnership. Even though the business is growing and expanding, the team is there when something bad happens."
"You've got to think and say to yourself: in a disaster situation, will this new team offer the same level of service? Will the new provider be in a position to perform for your business when they need to?"
There's a depth of knowledge at Micron21 that is hard to beat.
The experience has cemented the value of The Web Factory's comprehensive backup strategy – a strategy originally established and implemented under the guidance of the Micron21 team.
"The team strongly encouraged me over the years to adopt a decent backup strategy. Most web hosts don't include backups with their hosting at all."
For Grant, things have changed.
"Now, we have two layers of backups at all times, and periodic off-site backups as well. Three layers. At all times."
And his advice for anyone considering a change in provider? Or migrating to Micron21?
"Everyone's a good host when things are running well, but what you really want is a partner and a provider who's going to pull out all the stops when they aren't. That's exactly what this team does."
"I'm glad for the experience. If not for aligning myself with a team like Micron21, I don't know how a business like mine would survive this sort of incident, with the level of diligence and support I received."
The whole experience hammered home the level of personal care. Even though the business has grown and evolved and expanded, there's still that level of care that exists.
“They have a philosophy of doing whatever it takes to solve the problem. I know we're not their biggest customer and at the time I was actually looking to change providers. None of that mattered. They pulled out all the stops.”
— – Grant M, The Web Factory
