Information Systems:Backup Server

History and overview

The Backup server is an IBM System x3550 originally purchased in the summer of 2010. The purpose and intent of the server was to house the Symantec BackupExec software and have a tape library connected to facilitate backups of all the Windows based servers in production. To begin with, the server only had one SAS controller card for the tape library. The controller card then had a "SAS Splitter" that broke out into two connections, one for each tape drive in the library. This proved unworkable and the splitter was traded back to the reseller for a second SAS controller card that was installed inside the server. From then each tape drive had a single SAS cable connected to its own controller. Note that the two controller cards are different models, one being 3GB and the other 6GB.

The Backup server does have two 1GB NICs but only the primary is used even though both are cabled. In some environments (hopefully ours in the future) there is either a separate backup network or a VLAN that allows backups to be streamed without mixing in with other traffic. As of 2016, the backups do stream over the flat network and performance is acceptable.

The Backup server does not have an IMM or RSA card installed but it is connected to the Server Room KVM.

The Backup server was also purchased with four large 300GB hard drives configured in a RAID5 set. The first 100GB was partitioned into the C: drive and the rest into a D:. The purpose and intent of this design was to take advantage of the deduplication technology in the BackupExec software present in 2010. The idea being that the daily backups from all the Windows servers would be streamed into the dedup database on the D: drive and then be written to tape in its deduped state. This design seemed to indicate that the amount of data written to tape would be smaller and allow for more data growth during the lifespan of the hardware. Unfortunately, it never really worked like that. The dedup software was buggy and slow and the BackupExec software wrote "hydrated" data to tape which completely undermined any value from the dedup feature. When the BackupExec license and support contract was renewed in the summer of 2011, the dedup feature was not renewed and the D: on the Backup server went unused. From then on the backup jobs streamed data from client servers directly to tape, bypassing any dedup step.

The operating system on the Backup server is Windows Server 2008 R2. Any newer server OS is not supported by IBM on that machine. The device drivers for the SAS controller cards and the tape drives and the library chassis itself all need to be installed and present and they must be the device drivers provided by IBM. Drivers that come with Symantec BackupExec do exist but they won't be supported by IBM. BackupExec needs the IBM device drivers for the hardware in order to properly understand what drives and slots and capabilities are present in the tape library. BackupExec uses the device drivers to move tapes from the slots into the drives and then back into the slot when the job is complete.

The original version of BackupExec was the 2010 R2 version in the summer of 2010. In December of 2010, the SuperServer was replaced with new hardware (x3650) and during December and January, there were extensive problems with getting reliable backups of the Superserver by BackupExec. The technical problems were due to bugs in how BackupExec handled certain VSS timeouts. The "fix" was to only do one backup job on the Superserver per 24 hour period. The Mon-Fri nightly backup of Superserver starts at 11:59 and goes for about 5 hours. What BackupExec can't handle is doing another backup of Superserver at say 11:00AM the next morning. This limitation is why the monthly backups of the Superserver need to run on a Saturday and not on a weekday even though VSS is perfectly capable of doing backups while the server is being used during working hours. The other reason why the monthly backups need to occur on a Saturday is that BackupExec is not smart enough to substitute the monthly tape on the last night of each month like BRMS can. If the Superserver backup job stalls at some point half way through, the fix is to cancel the job and reboot both the Superserver and the Backup server and allow the next days job to start as scheduled.

BackupExec 2010 R2 was upgraded to R3 in 2011 and then upgraded to the 2015 version in the summer of 2015. The 2015 version has a very different GUI compared to 2010. The 2015 version is a little bit more stable and reliable than 2010 but it still does not handle VSS in a graceful way and the error messages when a client server reboots during a backup for some reason are still cryptic. For this reason WindowsUpdates are setup to only reboot servers for updates on Sunday when there are no backup jobs running. Since there are three desktops also being backed up, those jobs sometimes fail when those desktops reboot for patching. It really depends on the timing. BackupExec will only stream data from one client server or desktop at a time so the entire list of machines to be backed up occurs in serial not in parallel.

Notes and special instructions

Do not install anti-virus software, especially Symantec anti-virus software on the Backup server. Performance will be unusable and there are direct incompatibilities when Symantec Endpoint Protection and BackupExec are installed on the same server. These incompatibilities remain even though Symantec divested the BackupExec product to Veritas in 2015.

Future considerations

If uniPHARM is successful in its implementation of a VMware infrastructure, BackupExec will need to be replaced with some other software package such as Veeam. The low quality of the technical support provided by Veritas does not justify continuing to use BackupExec especially when the hardware it is housed in is 6 years old. The only obstacle to a wholesale change in backup software would be the ability of the replacement package to read tapes created by BackupExec. Veritas has set an end of support date for BackupExec 15 on November 7, 2017.

Oh And By The Way
The technical support contract expired for Veritas BackupExec in August of 2018.  Management decided to not renew.  If the BE16 software is having any difficulties after that date, it will be bad.

Replacement backup/storage server project

For x86 backups, a large storage pool will be needed. As of October 2019, there are several options to consider as to the server that will be used for Veeam backups of the x86 infrastructure:

• Old superserver (x3650 m3); physical
• Old domain controllers (x3250 m5); physical
• Virtual backup machines in hosts (x3650 m5)

ERRORS/FAILED BACKUPS

If a backup fails , at this time using the backupexec software IT must go into the software and push forward the next backup. The backups become hung until the appropriate drive tape is inserted, this is important on holidays and general failures. Always check the schedule to make sure the next backup will occur.