Information Systems:Disaster Recovery Tasks on IBM i
Over Night Processing
Over night processing is split into two parts, both of which are submitted by the job scheduler. The first does all the updating, reporting, housekeeping, etc. The first thing it does is shut down all user access (except for the HMC), and all monitors that use ASW or extension data files. We call this 'application restricted state'.
The second part does a full system backup; which means the system has to first go into a restricted state. In this state, the system can't even be accessed by the HMC.
The first part starts at 1:00AM and usually finishes around 2:15AM. If it doesn't, because it is in error, or in a loop, it will be cancelled when the backup starts at 3:00AM. We will have to keep an eye on this - if the length of the process approaches two hours, we will have to change the timings.
When the backup starts the restricted state, it sets a timer of 3 hours. If it is still running when those three hours are up, the system will cancel it, and start everything back up again. It currently takes about two hours.
Application Restricted State
This is what we call it when we shut down all user access to the system (except the HMC), and end the monitor jobs.
If the system is in this state, users cannot sign on, but I.T. can sign on to the HMC.
Sign on as EODJOB and our favorite password. Key in GO DISASTER and press enter. ** Note that if you plan on running options 32 or 91, or if the system is currently in the middle of the overnight processing, this must be done on the HMC console. See the section ‘HMC Console’ for how to set up and use this.
DISASTER Disaster Menu
Select one of the following:
11. Work with active jobs
12. Work with system status
13. Work with disk status
14. Display operator messages
31. Start all processes
32. End all processes
33. Start web servers
34. End web servers
91. Power down without restart
99. Signoff
Selection or command
===>
F3=Exit F4=Prompt F9=Retrieve F12=Cancel
F13=Information Assistant F16=System main menu
System Restricted State
The command ‘End subsystem *all’ will put the system into a restricted state; which will allow a full system save.
It can only be run in the controlling sub system (QCTL); either interactively on the HMC workstation session, or when submitted to job queue QCTL (which runs batch jobs in the controlling sub system).
When run interactively, the HMC session will remain active. You will be able to respond to error messages.
Click on ‘Actions / Display Popup Keypad / SysRq’ and press enter. You will have the option to display (and answer) system operator messages, and to display the current job – which let you look at the job log.
If it is run in batch, you will be completely locked out of the system; all you will be able to see is the SRC (sustem code on the front panel of the processor). For this reason, a parameter has been added – BCHTIMLMT. This controls how long the system will stay in a batch restricted state. If there is a message that requires an answer, or a program goes into a loop, the system will sit there waiting (and you cannot do anything about it). When the batch time limit is up, the program issuing the command will be cancelled, and QCTL will be started up. This will run QSTRUP – which will start everything else.
If you know something is wrong, and don’t want to wait for time to run out, you can manually force the end of the batch restricted state.
End System Restricted State
On the green screen, get an HMC workstation sign on display.
Then, in FireFox go to -
https://hmcpower8.unipharm.local/preloginmonitor/index.jsp
Click on ‘log on and Launch the Hardware Management Console web application’
Sign on with HMCROOT, and our new favorite password.
Expand ‘Systems Management / Servers’. Click on server.
Select ‘Bart’. Expand ‘Serviceability / Control Panel Functions’.
Click on ‘(21) Activate Dedicated Service Tools’
The HMC workstation session will change to –
Dedicated Service Tools (DST) Sign On
System: BART
Type choices, press Enter.
Service tools user . . . . . . . . . . . __________
Service tools password . . . . . . . . . __________
F3=Exit F5=Change password F12=Cancel
The user is QSECOFR, and password is ‘i’ plus our favorite.
Use Dedicated Service Tools (DST)
System: BART
Select one of the following:
1. Perform an IPL
2. Install the operating system
3. Work with Licensed Internal Code
4. Work with disk units
5. Work with DST environment
6. Select DST console mode
7. Start a service tool
8. Perform automatic installation of the operating system
10. Work with remote service support
13. Work with system security
14. End batch restricted state
Selection
__
F3=Exit F12=Cancel
The option to us is ’14. End batch restricted state’.
Here is a link to ‘Batch Full System Saves to the Controlling Subsystem Using BRMS’.
http://www-01.ibm.com/support/docview.wss?uid=nas8N1019121
Disaster Menu
** Note that if you plan on running options 32 or 91, or if the system is currently in the middle of the overnight processing, this must be done on the HMC console. See the section ‘HMC Console’ for how to set up and use this.
DISASTER Disaster Menu
Select one of the following:
11. Work with active jobs
12. Work with system status
13. Work with disk status
14. Display operator messages
31. Start all processes
32. End all processes
33. Start web servers
34. End web servers
91. Power down without restart
99. Signoff
Selection or command
===>
F3=Exit F4=Prompt F9=Retrieve F12=Cancel
F13=Information Assistant F16=System main menu
The options in black can be run at any time – the ones in red must be used only for extreme occasions.
The first four are pretty much self-explanatory. Start all processes
Option 31 will start any processes that are not currently running. I used this when we were having problems with the tape drive, and I had to cancel the backup. InfoManger, Gauss, and Domino were running; ASW, extensions, Web Orders, and InfoNet were not. This option passed over the first group, and started the second. End all processes
- This must be done on the console **
Option 32 will end everything and bring the system into an application restricted state; taking about 10 minutes. It brings everything down gracefully – for example we originally had problems with Domino unless it was ended just right. In order to do a ‘save system’, the system must be in a restricted state. Use option 11 ‘Work with Active Jobs’ to make sure everything has ended before you continue with what you are doing. Knowing what we know now, it also should have been used on 16 Dec 2013 to keep everyone off of the system while document number control files were being updated. Start web servers
Option 33 will only start servers that aren’t currently running. End web servers
Option 34 could cause internal users and customers to lose what they are in the process of updating. We have occasionally had a problem with the web orders server programs getting the date formats mixed up, which caused some orders to fail in IOP with invalid dates (not all orders, as there are multiple server programs). There is no way to identify which program has the problem, so they can all be ended with this option. Then restarted of course. Power down without restart
- This must be done on the console **
Option 91 would be used when we want the computer to stay off – for hardware service, or because power will be lost. Be sure to use option 32 to end all processes first, and option 11 ‘Work with Active Jobs’ to make sure everything has ended before you continue with the power down.