When It All Goes Wrong.

As most of you know we deal solely with small business.  We are talking mostly, really small, circa less than 20 users.  However, those users are just as likely to delete an important file, and all the other risks are there, fire flood, server failure etc.  Arguably more so as a lot of small businesses are now located in massive buildings with many other small businesses all running equipment from god knows where, and hardly any of it installed in a professional way.   A great example is that I do not have one customer whose servers are located in a purpose build server room with air conditioning.  I know, I can hear all you engineers from big money businesses rolling your eyes.  The simple fact of the matter is the money is simply not there, consequently we are constantly dealing with make do and mend.

Traditionally small business backups have been a few, in some cases two, in some cases five, external hard disks, using Windows backup tools, which for the most part work perfectly.  We can restore an entire server fairly quickly.  This was a perfect solution for the SBS product line where there was a single server therefore a single image.  However, since Microsoft ditched the product line and made us install Exchange on a separate server this has caused a big issue, all of a sudden there are two servers, which need two lots of Backup devices even if they are installed in one VMWare Environment. 

Now we really did not want to have to retrain our entire customer base in how to check their backups, and we really wanted a way to keep a track of them, as windows no longer has a friendly way to e-mail a backup log to us each day.  Oh, I miss the SBS administrator e-mails.  So, we have had to get creative in finding a solution to this.  I know a lot of you are going say, oh that’s easy just use an online backup service.  Yeah, try and get a director to pay for that!  I was really impressed with the Datto solution, however selling it proved to be really, really hard.  Customers loved all the features, and the fact that it had built in Virtualisation technology so they could fire up a server if theirs failed in a few minutes.  But the E-brake went on as soon as the monthly subscription costs was uttered, bear in mind, for some of them, this was more than half of what they pay me for an entire month’s support.

So we had to get creative.  We have played with many different NAS devices over the years but the one that we really like is the Synology products as they really do offer something that is super affordable for small business.  The product line offers single drive, two drives, four drive and larger arrays and they include iSCSI.  Which gives us a great way to backup maintaining all the Windows incremental backups.  If you use traditional NAS then it has to backup the entire server each time, and deletes the old image as it goes.  iSCSI maintains all the history so you can go back a day, four days, ten days etc.

The way this system works, we place into the network (normally on the second server NIC) a Synology disk station, we provide two iSCSI Luns, one for each server (more if needed) of an appropriate size, which we then connect to directly from within Windows (effectively becoming a local disk) we then use these as the backup locations from within Windows Server.  The backup rolls onto this each night, no worries about users forgetting to plug the drives in, or change them etc.  Now we use the HyperBackup built into the Synology Disk Station to roll the LUN backups onto the external hard disks (which for the most part they already have)

This allows us to roll out really good robust proven backup systems for approximately ¼ the cost of the Datto system, and with no ongoing monthly fee.   Small business loves this.  It gives them a shiny box to point at, and say that is my backup.  They have emergency off-site backups.  Also, the Disk Station monitors and e-mails us to let us know the LUN’s are being backed up, and will let us know the second one of the external disks goes wonky.

The Modern Backup Solution.

Firstly, and lets be absolutely clear on this, Tape Backup is Dead, Dead, Dead.

Secondly, unless you have 3 backup disks, and can guarantee that at least one is off-site at all times, and you have a cloud based backup in place (you will know as you will be paying a cloud provider or IT-MK Limited to provide a cloud based service) then we really need to talk about your backup solution. 

“According to Aveco, 20 percent of companies will suffer fire, flood, power failures, terrorism or hardware or software disaster. Of those without a DRP:
* 80 percent will fail in just over a year
* 43 percent will not even reopen
* 93 percent that experience a significant data loss are out of business within five years.”

All modern systems these days are using disk based backup,  however these can come in a number of “flavours”.  All Disk Systems have a number of advantages over the older tape systems, the most noted is in the following situation, you require the restoration of a small file, say >1Mb, now that file could be located near the end of the backup, a tape system has to read through more or less the entire tape to recover the file, a disk system being random access, can go directly to the file location.  This means that disk systems are invariably faster at recovery in this situation.

Disks sound great but what are the disadvantages?

Disks are a great way of backing up, there is no way around that, the weak link in the chain is, as always human beings, disk’s don’t always get changed, disk’s get left on site, normally next to the servers they intended to be protecting.  In the event of a fire, flood, theft or other catastrophe, they can all to easily be destroyed.  The worst case is that there is not a disk left off-site or you only have one disk!  Which is destroyed and you are unable to recover any data.

The Drawback of Cloud

Cloud based backup systems provide the best level of protection however, in a busy network with a relatively slow internet connection (ADSL for example) there can be some lag between a file changing and being backed up.  This can be exasperated if there are databases in play, as each time the database changes the whole thing has to be uploaded.

Cloud systems are notoriously slow to recover from, as what goes up via ADSL has to come back via ADSL, a whole server of data say 100Gb can take days to come back down the line.

What Are The Ways Forward?

Well there are a couple of ways forward, firstly, ensure that there are enough disks with a good image on, off-site, to recovery at least everything except the last 7 days ideally more.  Ensure that there is NO time AT ALL when ALL the disks are in the same location, this requires a minimum of three disks to achieve.

Disk to Disk to Cloud

This is a newish concept however one that allows you to take advantage of the best of all worlds, firstly there are no tapes.  Secondly the primary backup is still a rotating disk platform with 3 or more disks, thirdly sitting behind the disk backup is a cloud based backup.  In the event of a user deleting data, there is the immediate recovery from the disk, in the event of a fire, flood, theft, then there is primarily the offsite disk that can be used to recover the information, thirdly if for some reason that disk is damaged (I have had this where a server was flooded, the on-site backups were destroyed and the user responsible for the off-site disk dropped it on the way in!) then the cloud is sitting there ready with your data.

Backup Systems

Windows Server 2003 was the last Server Operating system not to provide an Image Based backup system.  An Image of a system means that we (the IT Support team) can rebuild a server super quick, in a disaster, to different hardware.  We get another server (or high powered desktop) and simply recover the image directly onto the new machine, this is about the quickest recovery you could hope for typically >4 hours providing hardware is available.

What if I am using 2003 backup still? 

Then we really have to talk, 2003 backup is littered with inconsistency and we really should be talking now about bringing you up to a newer operating system for your server, one that contains at the least an image based backup system.  However we should really be looking at proper proprietary backup systems and software.

But my data is critical, and we can’t be down ever.

We can cope with this situation, however you really have to decide how critical your data is, and can you cope with a bit of downtime?  To provide a fully fault tolerant system, which would involve special software, geographically distributed servers, high speed leased lines between the server locations, which can prove expensive, and we are talking tens of thousands of pounds to do properly.

Why Now?

IT-MK Limited has up until now allowed customers to drive their strategies when it comes to backup.  Often the primary argument comes down to cost implications.  Over the course of the next 3 months we will be performing a major overhaul of our records on site, as part of this we will be conducting in-depth on-site reporting.  Part of this will be a report on server conditions and backup planning.  There will also be options for bring the systems up to speed.   If you would like to do this sooner rather than later, then simply drop us a line and we will be happy to oblige.  This site visit and report, as it is with a view of updating our own records, will be free of charge, we hope to have this completed by the beginning of September.

What do IT-MK Limited Use and Do You Have  A DRP in Place?

IT-MK Limited take very seriously our customers’ needs and requirements, as such we have the following systems in place:

Firstly our servers are all mirrored pairs, that is for each server we have there is another identical one ready and waiting to take over, this applies to mail, database, and file systems.  We use a system called Distributed File System so that all our data is simultaneously mirrored onto two independent non virtualised servers, these servers are from different vendors and contain disks manufactured by different providers to avoid any implication of a faulty batch of components effecting both servers simultaneously.  These servers are geographically diverse, one being located in Towcester and one being located in Milton Keynes, the link between is FTTC Optic Fibre DSL.   Each server has an APC 3000Ah uniteruptable power supply which ensures continued operation for approximately 40 minutes should the mains power fail.

Secondly each server has an disk based image backup system attached to it for immediate recovery and rapid recovery.

Thirdly we use a cloud based system for off-site backup this provides a last chance saloon recovery should things go that far!

We undertake regular updates and maintenance and perform a full image recovery test once every six months ensuring that our systems are in good, recoverable, condition.