When It All Goes Wrong.

As most of you know we deal solely with small business.  We are talking mostly, really small, circa less than 20 users.  However, those users are just as likely to delete an important file, and all the other risks are there, fire flood, server failure etc.  Arguably more so as a lot of small businesses are now located in massive buildings with many other small businesses all running equipment from god knows where, and hardly any of it installed in a professional way.   A great example is that I do not have one customer whose servers are located in a purpose build server room with air conditioning.  I know, I can hear all you engineers from big money businesses rolling your eyes.  The simple fact of the matter is the money is simply not there, consequently we are constantly dealing with make do and mend.

Traditionally small business backups have been a few, in some cases two, in some cases five, external hard disks, using Windows backup tools, which for the most part work perfectly.  We can restore an entire server fairly quickly.  This was a perfect solution for the SBS product line where there was a single server therefore a single image.  However, since Microsoft ditched the product line and made us install Exchange on a separate server this has caused a big issue, all of a sudden there are two servers, which need two lots of Backup devices even if they are installed in one VMWare Environment. 

Now we really did not want to have to retrain our entire customer base in how to check their backups, and we really wanted a way to keep a track of them, as windows no longer has a friendly way to e-mail a backup log to us each day.  Oh, I miss the SBS administrator e-mails.  So, we have had to get creative in finding a solution to this.  I know a lot of you are going say, oh that’s easy just use an online backup service.  Yeah, try and get a director to pay for that!  I was really impressed with the Datto solution, however selling it proved to be really, really hard.  Customers loved all the features, and the fact that it had built in Virtualisation technology so they could fire up a server if theirs failed in a few minutes.  But the E-brake went on as soon as the monthly subscription costs was uttered, bear in mind, for some of them, this was more than half of what they pay me for an entire month’s support.

So we had to get creative.  We have played with many different NAS devices over the years but the one that we really like is the Synology products as they really do offer something that is super affordable for small business.  The product line offers single drive, two drives, four drive and larger arrays and they include iSCSI.  Which gives us a great way to backup maintaining all the Windows incremental backups.  If you use traditional NAS then it has to backup the entire server each time, and deletes the old image as it goes.  iSCSI maintains all the history so you can go back a day, four days, ten days etc.

The way this system works, we place into the network (normally on the second server NIC) a Synology disk station, we provide two iSCSI Luns, one for each server (more if needed) of an appropriate size, which we then connect to directly from within Windows (effectively becoming a local disk) we then use these as the backup locations from within Windows Server.  The backup rolls onto this each night, no worries about users forgetting to plug the drives in, or change them etc.  Now we use the HyperBackup built into the Synology Disk Station to roll the LUN backups onto the external hard disks (which for the most part they already have)

This allows us to roll out really good robust proven backup systems for approximately ¼ the cost of the Datto system, and with no ongoing monthly fee.   Small business loves this.  It gives them a shiny box to point at, and say that is my backup.  They have emergency off-site backups.  Also, the Disk Station monitors and e-mails us to let us know the LUN’s are being backed up, and will let us know the second one of the external disks goes wonky.

Advertisements

Personal Mobile Phones and Corporate E-Mail

This article is aimed at upper level management decision makers for small and medium sized business.

Exchange has some awesome features, and one thing I had never considered was the following:

An employee has a personal mobile phone, onto which he has asked to have his corporate e-mail account, his manager thinks this is a perfectly reasonable request and calls me in to set it up.  Employee then leaves company, manager wants mail off the phone.  Now Exchange can cope with exactly this.  However, it does it in rather a sledge hammer nut kind of way, when adding the account to the phone, you will see a pop up screen which outlines the powers the phone is allowing the exchange server, when you issue a remote wipe from the exchange server it assumes the phone is a corporate device and factory resets it, including any data stored on it.

So now you have effectively protected your company but trashed the data of the ex-employee.  It is therefore imperative that any such device that is not owned by the company has a waiver signed by the employee to the effect of, on termination of employment the telephony equipment must either be surrendered to the company in order that company data may be erased, or by issuing the company power over all data on the device to be destroyed at will, via wireless sync.  If this is not acceptable then the request for mobile based e-mail on a personal device should be refused or if necessary the company should provide a company owned device for this purpose.

Please lets think about this before we jump in to these things.

Posted in Uncategorized. Leave a Comment »

Security… Going Forward…

Over the last 24 months we have seen so many new virus, trojans, and attack vectors our minds are starting to boggle a bit.  However, without doubt the most destructive of all is a variant called Crypto Locker.  Basically, Crypto Locker gets into your system, it then immediately starts encrypting files, starting on the local machine and progressing to any server shares, NAS device shares, even RDP sessions in some newer cases, and encrypts anything it can find.  Once it has nothing else to encrypt it spits up a smug message proudly stating what it has done and that you need to pay something to someone to get the decryption key.

It’s normally at this point we get the call.  Invariably we go in and confirm that the files are now encrypted, and here is the bit that shocks everyone, there is ABSOLUTLEY NOTHING WE CAN DO, with the sole exception of canning the network of all infected machines and restoring everything from a backup.  Workstations have to be wiped and re-installed, data is invariably lost, and in one case, no-one had bothered changing the backup drive for month, and you got it, it was encrypted.

Why do people do it?  Well the simple answer is money, alone Crypto Locker is suspected of generating millions of pounds in ransom fees.  People pay it because they are scared, especially home users, or people without a backup desperate to save their data.  Don’t be one of them.

To be successful in not losing your data to Crypto Locker you MUST employ a three-way defence strategy, firstly you should ensure every single system that connects to your corporate network has a good quality anti-virus installed and this should include heuristic analysis, and zero hour updates, these are features that you will not get with a “free” antivirus.  Secondly you must ensure you have a reliable rotating backup.  We do perform weekly checks on all contract holder backup systems.  But you are ultimately responsible for your company’s data and strategy.  Thirdly, if you have people bringing in their own equipment such as laptops, phones tablets etc, these can be carriers. 

I have long had the discussion with managers and directors who see no issue with giving people access to the “internet” via their Wi-Fi Connection, which in principal is fine, but that should not be Wi-Fi connected to your organisations central data network or you are opening yourself up to these devices being used as carriers for incoming viri, trojans, they can be used to steal data, most devices can accommodate a vast amount of storage space these days, just think, SD cards are available in 512Gb, which for a small business could be a vast amount of data, if not an entire copy. 

As the director, you are responsible for anything connected to your network, so for example if someone brings in a laptop which is full of “media”, which could cover anything from child exploitation, to pirated mainstream films and music, and they happen to be using some sort of peer-to-peer network sharing on that device, then who is responsible as it’s using your internet connection?  Now just think, peer-to-peer network apps have been available on smartphones for some time.

What’s your point??

Well I am trying to impress on middle to upper management, we are past the point, technologically speaking, where we can trust our company’s data security to the wind.  There are so many things out there that are ready to screw over corporate networks and are specifically designed to take your money and steal your data illegally, and there is, in 99.9% of cases, absolutely no recourse.  You have to take responsibility for your company’s data.  I know, it’s nice for people to be able to use Facebook and facetime at work to stay in touch, but if you wish to provide this service to them, get a £20 per month Internet connection with dynamic IP set up away from your corporate network and let them use that.  Keep the corporate network clean, and prevent opportunity for foreign systems and devices to even connect to it.  We can’t be responsible for a device that comes in to your building from anywhere, connects to your network, and steals data or infects your company computers with a virus. 

Does this scenario ring true in your office, someone shows you and a group of others a questionable video on their mobile phone on some weird website you have never heard of, a few minutes later that phone is plugged into a USB socket on a corporate workstation “to charge”, that connection is also a data connection, that user is now exposing your corporate network to everything it’s picked up from dodgy wifi access points, virus, trojans, exploits etc that it has ever come into contact with.

Pay for a decent reliable, branded, with support Anti-Virus, ensure Firewalls and routers are patched and rules updated on a minimum 6-month schedule.  Educate your staff what is and what is not acceptable usage of the corporate network.  Turn off USB ports where possible, have public access wireless internet access installed away from your core network, which can also provide ports for sales reps etc that come in and want internet access.  We understand the need for people to be connected, we also understand first-hand the dangers this brings to each and every business.

Posted in Uncategorized. Leave a Comment »

Our Internal Systems … Overhaul details…

Over the Christmas period we will be overhauling our internal systems, which is why, this year, we are closing our doors on the 19th!

We will be fully rebuilding our rack with completely new equipment.  The rack will consist of the following:

1 x Dell 1950 Dual Xeon 1U Server – 32Gb RAM – VMWARE1
1 x Dell 1950 Dual Xeon 1U Server – 16Gb RAM – VMWARE2
1 x Dell 1950 Dual Xeon 1U Server – 8GB RAM – VMWARE3
2 X Dell 2950 Dual Xeon 2U Servers – 16Gb RAM – SANARRAY1 – SANARRAY2
1 x HP ML350 G5 – Single Xeon 4U Server – 8Gb RAM – ASTERISK (Phone System)
1 x HP N54L Microserver – Single Xeon 8Gb RAM – ISCSI Backup Target
1 x Watchguard X550 Firewall – Main Internet Connection
1 x Watchguard X20E Firewall – Between Main Network and SAN for management of SAN
1 x SunFIre V210 – Dual SparcIII Server running SQUID Proxy Server
1 x SAN Switch (May be Fibre, To be confirmed)
1 X Managed Gig Switch for Main Network

We estimate this will take us about 4 days to fully configure and install, please bear with us during this time 🙂

Posted in Uncategorized. Leave a Comment »

Replaced Your Server? Don’t Bin It, Re-function it…

About 12 months ago we started playing around with a new product called FreeNAS. This is a piece of Open Source software. You will probably have heard us banging on about Open Source before, but for those of you who have missed it, Open Source is exactly that, anyone can download the code and play with it. Basically compiled by a bunch of volunteers and tossed into the world.

FreeNAS is an operating system that has very small overheads and turns pretty much any box into a NAS (Network Attached Storage) device. However with a little bit of IP from us we have changed this into a system that can be reliably utilized as part of a disaster recovery strategy.

It works like this, once your new shiny server is in place and the final roles have been transferred we wait, normally 6 months, however in the case of our latest server installs which have been from bare metal to VMWare virtualized environments we have scaled this back to 1-2 weeks. As the VM Convert is essentially an image of the original taking with it everything and removing the human factor in copying data. So once the VM is up and running and your server image has been converted, you are left with an old fileserver, that normally is between 3-5 years old and quite capable of becoming a host.

We remove the old server software and install our copy of FreeNAS, setting up the system to present to your new fileserver as ISCSI (This is effectively allowing the server direct access to the disk drives in the FreeNAS), providing a significant overhead advantage to traditional CIFS (Windows Shares). So if your old server had for example 4 x 1Tb disk drives we can present this as either a single 4Tb Drive, or a redundant 2Tb drive to your new server. We can then use the inbuilt windows backup to roll a backup on to this box.

If your new server has two network cards then we can present the ISCSI effectively as a SAN Array, which means that there is no overhead from this clogging up the network fabric.

We then utilize your removable drives (external backup disks) to back up this FreeNAS volume.

Advantages:

  • FreeNAS cost £0 as it is Open Source
  • This gives us a massive library of quick backup recovery options for files, e-mails and indeed the whole server
  • The slow process of copying on to the external device is carried out by the FreeNAS box, and so network performance is not effected by the copy process.
  • Solid redundant disks provide reliable storage for backup history.
Posted in Uncategorized. Leave a Comment »

The Modern Backup Solution.

Firstly, and lets be absolutely clear on this, Tape Backup is Dead, Dead, Dead.

Secondly, unless you have 3 backup disks, and can guarantee that at least one is off-site at all times, and you have a cloud based backup in place (you will know as you will be paying a cloud provider or IT-MK Limited to provide a cloud based service) then we really need to talk about your backup solution. 

“According to Aveco, 20 percent of companies will suffer fire, flood, power failures, terrorism or hardware or software disaster. Of those without a DRP:
* 80 percent will fail in just over a year
* 43 percent will not even reopen
* 93 percent that experience a significant data loss are out of business within five years.”

All modern systems these days are using disk based backup,  however these can come in a number of “flavours”.  All Disk Systems have a number of advantages over the older tape systems, the most noted is in the following situation, you require the restoration of a small file, say >1Mb, now that file could be located near the end of the backup, a tape system has to read through more or less the entire tape to recover the file, a disk system being random access, can go directly to the file location.  This means that disk systems are invariably faster at recovery in this situation.

Disks sound great but what are the disadvantages?

Disks are a great way of backing up, there is no way around that, the weak link in the chain is, as always human beings, disk’s don’t always get changed, disk’s get left on site, normally next to the servers they intended to be protecting.  In the event of a fire, flood, theft or other catastrophe, they can all to easily be destroyed.  The worst case is that there is not a disk left off-site or you only have one disk!  Which is destroyed and you are unable to recover any data.

The Drawback of Cloud

Cloud based backup systems provide the best level of protection however, in a busy network with a relatively slow internet connection (ADSL for example) there can be some lag between a file changing and being backed up.  This can be exasperated if there are databases in play, as each time the database changes the whole thing has to be uploaded.

Cloud systems are notoriously slow to recover from, as what goes up via ADSL has to come back via ADSL, a whole server of data say 100Gb can take days to come back down the line.

What Are The Ways Forward?

Well there are a couple of ways forward, firstly, ensure that there are enough disks with a good image on, off-site, to recovery at least everything except the last 7 days ideally more.  Ensure that there is NO time AT ALL when ALL the disks are in the same location, this requires a minimum of three disks to achieve.

Disk to Disk to Cloud

This is a newish concept however one that allows you to take advantage of the best of all worlds, firstly there are no tapes.  Secondly the primary backup is still a rotating disk platform with 3 or more disks, thirdly sitting behind the disk backup is a cloud based backup.  In the event of a user deleting data, there is the immediate recovery from the disk, in the event of a fire, flood, theft, then there is primarily the offsite disk that can be used to recover the information, thirdly if for some reason that disk is damaged (I have had this where a server was flooded, the on-site backups were destroyed and the user responsible for the off-site disk dropped it on the way in!) then the cloud is sitting there ready with your data.

Backup Systems

Windows Server 2003 was the last Server Operating system not to provide an Image Based backup system.  An Image of a system means that we (the IT Support team) can rebuild a server super quick, in a disaster, to different hardware.  We get another server (or high powered desktop) and simply recover the image directly onto the new machine, this is about the quickest recovery you could hope for typically >4 hours providing hardware is available.

What if I am using 2003 backup still? 

Then we really have to talk, 2003 backup is littered with inconsistency and we really should be talking now about bringing you up to a newer operating system for your server, one that contains at the least an image based backup system.  However we should really be looking at proper proprietary backup systems and software.

But my data is critical, and we can’t be down ever.

We can cope with this situation, however you really have to decide how critical your data is, and can you cope with a bit of downtime?  To provide a fully fault tolerant system, which would involve special software, geographically distributed servers, high speed leased lines between the server locations, which can prove expensive, and we are talking tens of thousands of pounds to do properly.

Why Now?

IT-MK Limited has up until now allowed customers to drive their strategies when it comes to backup.  Often the primary argument comes down to cost implications.  Over the course of the next 3 months we will be performing a major overhaul of our records on site, as part of this we will be conducting in-depth on-site reporting.  Part of this will be a report on server conditions and backup planning.  There will also be options for bring the systems up to speed.   If you would like to do this sooner rather than later, then simply drop us a line and we will be happy to oblige.  This site visit and report, as it is with a view of updating our own records, will be free of charge, we hope to have this completed by the beginning of September.

What do IT-MK Limited Use and Do You Have  A DRP in Place?

IT-MK Limited take very seriously our customers’ needs and requirements, as such we have the following systems in place:

Firstly our servers are all mirrored pairs, that is for each server we have there is another identical one ready and waiting to take over, this applies to mail, database, and file systems.  We use a system called Distributed File System so that all our data is simultaneously mirrored onto two independent non virtualised servers, these servers are from different vendors and contain disks manufactured by different providers to avoid any implication of a faulty batch of components effecting both servers simultaneously.  These servers are geographically diverse, one being located in Towcester and one being located in Milton Keynes, the link between is FTTC Optic Fibre DSL.   Each server has an APC 3000Ah uniteruptable power supply which ensures continued operation for approximately 40 minutes should the mains power fail.

Secondly each server has an disk based image backup system attached to it for immediate recovery and rapid recovery.

Thirdly we use a cloud based system for off-site backup this provides a last chance saloon recovery should things go that far!

We undertake regular updates and maintenance and perform a full image recovery test once every six months ensuring that our systems are in good, recoverable, condition.

IT-MK Limited – Now Available In Central London

We have a new dedicated department dealing with Central London the London centric team can be contacted on 0203 384 8549, and are available now to deal with your requirements.

Posted in Uncategorized. Leave a Comment »