Information Systems:Tapes / LTO Tape Backups
Weekday Morning Tape Procedure
TODO Optimize this section, break into background info and instructions -norwizzle (talk)
Each weekday morning, the tapes that have been used for the previous night's backup need to be removed from both of the tape libraries so that they can be taken offsite by BC Records. Having the tapes taken offsite, protects that data from disaster or theft here at uniPHARM.
The first step in this daily procedure is to check the results of the previous night's backup. For the backup that occurs on the Power8, an emailed report is sent that shows which tape number was used. The email is sent to Jeremy, Darren and Norwin. The email also contains a link to a PDF on the SuperServer which has detailed information on what was backed up to tape and how to restore the data on that tape using BRMS.
For the backups that cover the SuperServer and other Windows servers, there are also emailed reports from BackupExec sent to Jeremy, Darren and Norwin. The BackupExec reports also show which tape number was used and what was successfully copied. The key information is what tape numbers were used so that the correct tapes can be removed from the libraries. Only the tapes used during the previous night's backup need to be removed, the other tapes need to remain in the library. The Power8 backup tape will be in the library attached to the Power8 in the left most rack in the server room. The tapes used for the SuperServer and the other Windows servers are in the library in the middle of the centre rack in the server room.
The single Power8 tape can be removed from the library at any time after the End Of Day process is finished. The two tapes used for the SuperServer and other Windows servers can only be removed from the library between 8AM and 6PM daily. The time restriction is a function of how BackupExec works. The BackupExec library is logically locked between 7PM and 8AM the next morning. Both libraries have a pair of tape magazines that are to the left and right of the centre green LCD screen. Only one magazine, left or right can be opened at any time. You cannot open both magazines at the same time.
Both libraries have a web administration page that can show which tapes are in what slots inside the magazine. It is not necessary to use the web administration page to control the magazine but it can help if someone is not familiar with handling tapes.
http://barttape.unipharm.local
- Username: Admin
- Password: NewVisionIT2051
http://backupexectape.unipharm.local
- Username: Admin
- Password: NewVisionIT2051
Please note that if you use the web administration page to release and unlock a magazine, there is a short time limit where the library relocks the magazine so walk quickly from your workstation to the server room if the web administration page is used to open and unlock the magazines.
The easier method is to use the buttons and LCD display on the front of the libraries in the server room. Below is the button sequence to open the left side magazine:
- Press the left arrow to view the Main Menu
- Press the minus button to move the cursor to the Control Menu
- Press left arrow button to enter the Control Menu
- Press the minus button twice to move the cursor to the Magazine Menu
- Press the left arrow button to enter the Magazine Menu
- Press the left arrow button to instruct the library to release the left side magazine
- Pull the left side magazine out of the library
- Press the X button multiple times to return to the IBM Logo
To open the right side magazine:
- Press the left arrow to view the Main Menu
- Press the minus button to move the cursor to the Control Menu
- Press left arrow button to enter the Control Menu
- Press the minus button twice to move the cursor to the Magazine Menu
- Press the left arrow button to enter the Magazine Menu
- Press the minus button to move the cursor to the Right option
- Press the left arrow button to instruct the library to release the right side magazine
- Pull the right side magazine out of the library
- Press the X button multiple times to return to the IBM Logo
To initiate a strategic missile launch, repeat the first button sequence while reciting the "I am a little teapot" nursery rhyme.
The green LCD screen will indicate when the magazine is unlocked and when it can be pulled out of the library. When ready, use the handle on the front of the magazine and pull smoothly towards yourself. The tapes can be pushed out of the magazine by inserting a finger into a square slot on the outer part of the magazine. Do not tip the magazine over so that the tapes fall out of their slots - that's bad mmkay.
Once the correct tape is removed from the magazine, the magazine can be re-inserted back into the library. Make sure the magazine is all the way inside the library and the front handle is flush with the front panel. After each magazine insertion, the library will use a robotic arm to rescan all the barcodes on the tapes. The library must be given time to do this scan before the other magazine can be opened.
Repeat this process to get all 3 daily tapes. Once you have the 3 daily tapes, fill out the clipboard hanging on the wall and record what tape numbers were used for yesterdays date. Then take the tapes to the small gray safe at the receptionist's desk and put the tapes inside the safe. The key is either in the safe's key hole or in the top drawer of the receptionists desk. BC Records will come between 10AM and noon Monday to Friday to pickup any tapes that are in the safe so it is critical to complete this procedure between 8AM and 10AM each weekday. Failure to remove the previous night's backup tapes does not prevent future scheduled backups from occurring, it just puts the most recent and valuable backup at risk of being damaged in a disaster or stolen.
If at any time the library starts to flash an orange blinking light on the front panel, stop what you are doing and call IBM at 1-800-426-7378. Provide them with the machine type number and serial number of the library and they will come onsite to fix and or repair.
How The Tape Rotations Work
uniPHARM uses 3 different backup tape rotations. The purpose of a tape rotation is to have enough tapes to keep a long series of backup "images" while at the same time saving money versus just using a brand new tape for each days backup.
The first tape rotation is for the Power8 and it is 42 tapes in length. The tape library for the Power8 has 23 slots with 1 slot being reserved for a 1 time use monthly backup and 1 slot for the 999999 tape used as a temporary working tape. On the first Monday of a rotation, 21 tapes are loaded into the library plus the single use monthly tape to be used on the night of the last day of each month. 3 weeks later the next 21 tapes are delivered. On the Power8, the software that controls which tape is used on what day is BRMS. BRMS keeps track of which tape is next to be used in the rotation based on when it was used last. It also controls when the monthly tape is used instead of a daily tape. BRMS has been programmed to do full system saves of the Power8 in a restricted state for both the daily and monthly jobs. The backups done by BRMS are sufficient to do a complete restore onto new hardware. The Power8 library does not have enough slots for a cleaning cartridge so it needs to be cleaned manually when it produces an error message that requests a cleaning. A cleaning cycle happens automatically when the cleaning tape is loaded al the way into the drive - not just the library.
The tapes used by the Power8 are LTO5 tapes using barcode labels that follow the 0000xx format, where xx is 01-42. The monthly tapes are also LTO5 and use a barcode that starts with 2 zeros and then the 2 digit fiscal year and then the 2 digit accounting period.
The second tape rotation is for the SuperServer. It is 20 tapes in length and also employs a single use monthly tape. The library attached to the BackupExec server has the right side magazine reserved for the backup tapes backing up the SuperServer. That library (which is not connected to the Power8 in any way) has 2 separate drives within. It has an LTO6 drive and an LTO4 drive. Only the LTO6 tape drive is used in conjunction with LTO6 tapes, to backup the SuperServer. The 20 tape rotation also starts on a Monday and is split into 2 deliveries of 10 tapes. Tapes 1-5 are used Monday night to Friday night and then 6-10 are used the next Monday night to Friday night. 11-15 and 16-20 follow the same pattern. The barcode labels follow the format UWD6xx where xx is 01 to 20. The single use monthly backup tape for the SuperServer is manually scheduled in BackupExec to occur on the closest Saturday night to the start of each month and its barcode label follows the format of UMyymm with yy being the 2 digit year and mm being the 2 digit month. Although a full monthly backup is no different from a full daily backup, there isn't enough time to do both a daily and monthly on the same night and that is why the monthly gets pushed to a Saturday. The monthly backup of the SuperServer is required to satisfy Health Canada regulations for GMP data.
The third tape rotation is for all the other Windows based servers including 4 business critical desktop computers and appliances. This rotation uses the LTO4 drive and the left side magazine in the library attached to BackupExec. There is no monthly backup tape for any of the servers included in the third rotation even though it follows the same daily pattern as the SuperServer. The barcode label for the third rotation is UWD0xx with xx being 01 to 20.
The tape library attached to BackupExec does have the 23rd slot reserved for a cleaning cartridge and it can be used from the web administration page. The library sends an email notification to Darren and Norwin when it needs cleaning. Both libraries are under valid maintenance agreements with IBM as they do need regular replacements for drives robotic arms and pwer supplies. The libraries have many moving parts and they do wear out when used every day.
BC Records
FileBridge provides the new web portal for BC Records.
The URL for the new (as of October 1, 2018) BC Records portal is
- https://portal.filebridge.com/Account/Login?ReturnUrl=%2F
- Credentials: (username/password found here)
Hopefully we don't need to use this too often. The searching abilities are improved using the new website but the data conversion negates some of those niceties. (darrenf)
Conventions
Tape arrangement in library
For BRMS backups, the order of tapes in the library doesn't actually matter; BRMS will hunt for the an eligible tape. The convention we use is just for ease of administration. For x86 backups, tape order and slot arrangement matters. The jobs in BackupExec expects serial arrangement of tapes and will error out otherwise. I think this can be fixed, but meh we're moving to Veeam soon.
IBM Power / BRMS tapes
For the Power8 tapes:
- Tapes 1-11 are in the left library slot, starting from the 2nd slot (2nd from the lower left), then left to right, down to up (top right tape is therefore #11). The first slot is the I/O slot and should remain vacant.
- Tapes 12-21 are in the right library slot, starting from the right, and going right to left, down to up. The last 3 slots are "reserved" for the cleaning tape ("CLN*"), temporary backup tape ("9999") and the monthly tape. Therefore, the last tape of the rotation (3rd slot from the left in top row) should be #20.
- Extrapolate for series 22-42.
x86 / BackupExec tapes
- Same order and slot arrangement as the tape libray for IBM Power, except no temporary tape in right slot.
Periodic purchase of monthly tapes
The current pattern is quarterly i.e. to buy 2-3 monthly tapes each for the Power8 (LTO5) and x86 (LTO6).