In today’s post I will discuss the design of a data protection environment. I am a firm believer of having a data protection plan which covers backup and disaster recovery but also mitigation against ransomware / disasters and IT security breaches.
2017 gave a lot companies a hard lesson why a good design of your data protection is very important to get your data back and why you should not ignore to spend time to design a good backup infrastructure. (no matter what vendor you are using!!)
So today I will focus on the design of backup and disaster recovery infrastructure.
Traditionally, companies installed a backup server which than was configured for a weekly full, daily incremental backup and monthly backups all to tape. Over the 10 last years or so, this has shifted to backup to disk. While this is great and provides more performance, especially on recovery, it also provides challenges. What disk to use, can I use my decommissioned production hardware, would this NAS do for backup target are a few of the questions we get as pre-sales.
As an example, fictional company X decided to replace their 8-year-old production environment with a brand new all flash environment and to save some pennies they are using the old environment as their “new” backup environment. 1 year in and they got hit by a ransomware attack and need to restore 80% of their environment back. While they are now used to all singing and dancing new flash environment they had to rely on their old environment to get the data back as fast as possible, which for obvious reasons did not go back that fast..
The main issue we see with old hardware is could not be reliable enough and not performant enough. So can you use old hardware?? You can, but don’t expect miracles.. In today’s world we rely on our data and your backup infrastructure is as important as your production infrastructure. Save a few pennies now by using old hardware could cost you dearly. I would use new reliable hardware with maintenance for my backup infrastructure, it does not have to be flashy but it need to be reliable and performant enough for you to restore within your SLA’s.
So where do we start with our design?
First things first… make sure you know your data, what recovery SLA’s does some data need to have, what retention and how many copies of data.
Each company is different and has different needs and therefore one company backup design could not be right for another company.
Personally, I make always a diagram of what the infrastructure looks like today. This gives me an understanding of the environment and can quite often see how a new backup design fits in.
Secondly it is number crunching time, how many servers to protect? How much data to protect, what is the year on year growth over the last few years (industry standard is around 15%), what is your daily change rate (again industry standard is around 2%) what retention on disk would you like, do you want to replicate it to a second site, do you have remote sites to replicate to main, cloud strategy etc etc. A lot of questions to go through but these are important to get the correct design.
At Arcserve we provide our partners and customers an online estimator tool that can help with the design. This is a simplified tool that gives you a quick understanding of what resources you would need.
Click here for the Arcserve UDP Estimator tool
For larger enterprise environments we will use more advanced calculations that the tool above.
Once you calculated the resources needed, it is time to decide what storage to use.. and this is where things can get iffy.. Every backup vendor has a different approach of how to store data, so what is good for one could be different for another one, bear this is mind. The reason is that every vendor uses different algorithms to store data. Arcserve has the ability to use deduplication (like others of course, but each one uses different methods) and deduplication is great to reduce the data size on disk. To improve performance, you can choose to use Memory deduplication or use a dedicated read intensive SSD.
The main datastore sits on spinning disk, and best practise is to use more spindles with smaller size disks rather than use less spindles with larger disks. Why?
Most common in backup infrastructures are a NL-SAS disk or SATA disk, which has a maximum of 75 IOPS, if you have 24 TB RAW using 4x 8TB gives you max 300IOPS and when using 12x2TB disks gives you 900 IOPS which is significant faster.
With the storage done, it is time for the backup server(s). what do you choose for you backup server?
First off, make sure your backup server is not in the production environment. If you choose to a have a virtual backup server make sure it as a different environment to sit in.
With Arcserve UDP, I personally prefer a physical RPS server with storage attached using DAS / iSCSI or FC. The CPU and memory requirements are different per environment, it depends on what roles are you installing on the server, deduplication in memory or not, how many replication jobs going on etc.
In larger environments, often a distributed infrastructure is planned which means we split the roles over multiple servers, for instance a dedicated console server, multiple RPS servers and multiple proxy servers.
The best practise for any large environment is to do a Proof of Concept prior to designing a solution. This will help you to understand how a solution works, if its fits your needs as an organisation. But also, will help the vendor to understand how your environment works and what resources would be needed. At Arcserve, I have done many PoC’s and it will give real time information on performance, deduplication and compression ratio’s etc. Which all helps when creating a design.
Now I mentioned a lot about backup to disk, what about tape?
In the UK I hardly see any tape deployments, that being said, in my other regions (Nordics and Netherlands) I see a lot of tape deployments. Tape is still valid in my honest opinion. However, it gets a different role…
Tape’s role has shifted as a primary backup target to archival and offline copies. With UDP customers use the copy to tape task to set their monthly backup of a plan to tape. Main reason for storing onto tape is regulations, ie data needs to be stored for very long retentions.
If tape is part of your company strategy make sure it is part of the design, what HBA to use, what tape library to use, maybe have tape located at the replica site so copy to tape happens after replication at the secondary site.
Now the design for on-premise is done is time to think about replication / cloud strategy.
When replicating data to another location whether it is cloud or a replica location it is important to think about the network resources. What resources do you need to get the data over the wire? More importantly, if we need to recover the data over a wire how long will it take? Does the vendor have a seeding process in place (yes we have, its called jump start) etc
Another thing to think about is what to replicate? Do you really have to replicate your WSUS (patch server) or your internal IT servers? Going back to an older post in my blog here make yourself a tier list of high priority servers and determine how often to replicate and how-to DR these.
So you can see that it is important to think about the infrastructure design of a backup and DR scenario.
What else do you need to think about?
Once you decided on the physical aspects of the design it is important on how to implement it?
Think about ransomware or virus attack. They can affect your backup infrastructure too..
So here are my tips;
- Never install your backup server inside your production Active directory / LDAP domain Arcserve UDP is Windows based, you don’t want accidentally domain admins to spread ransomware into a backup environment.
- Use dedicated backup service accounts
- Disable SMB v1 if you don’t need it – SMB v1 is still used for older windows 2003 servers and linux kernels 6.6 and older. If you don’t have these you can disable SMB v1 on your servers
- Never use domain admin accounts for daily user tasks i.e. read my email with domain admin privilege etc ect (you’ll be surprised how many admins there are….) use a normal user account for your laptop / workstation and rdp into a management server with a special domain account for you
- Use firewall rules between backup environment and production environment. Secure your backup environment!
I hope you found this post useful and if you have any questions ping me a DM.