Information Systems:Disaster Recovery Tasks on IBM i
Disaster Menu
- Note that if you plan on running options 32 or 91, or if the system is currently in the application restricted state, this must be done on the HMC.
To access this menu, sign on as EODJOB, with our favorite password, and press enter. (This is to get the correct library list.) Key in GO DISASTER and press enter.
DISASTER Disaster Menu Select one of the following: 11. Work with active jobs 12. Work with system status 13. Work with disk status 14. Display operator messages 31. Start all processes 32. End all processes 33. Start web servers 34. End web servers 91. Power down without restart 99. Signoff Selection or command ===> ___________________________________ F3=Exit F4=Prompt F9=Retrieve F12=Cancel F13=Information Assistant F16=System main menu
The options in black can be run at any time – the ones in red must be used only for extreme occasions.
The first four are pretty much self-explanatory.
Start all processes
This is used when the system is in an application restricted state, or when some applications (ie. Domino, Web Orders, electronic orders, etc) aren't running. This option passes over those applications that are running, and starts those that aren't.
End all processes
- This must be done on the HMC.
This will end everything and bring the system into an application restricted state; taking about 10 minutes. It brings everything down gracefully – for example we originally had problems with Domino unless it was ended just right. The system must be in this state in order to do a complete save of the ASW or extension data files. Use option 11 ‘Work with Active Jobs’ to make sure everything has ended before you continue with what you are doing. Knowing what we know now, it also should have been used on 16 Dec 2013 to keep everyone off of the system while document number control files were being updated.
Start web servers
This will start servers that aren’t currently running. Our web servers currently are WEBINTER for InfoNet, WEBSMART for Web Orders, and WEBTEST for testing.
End web servers
This option could cause internal users and customers to lose what they are in the process of updating. We have occasionally had a problem with the web orders server programs getting the date formats mixed up, which caused some orders to fail in IOP with invalid dates (not all orders, as there are multiple server programs). There is no way to identify which program has the problem, so they can all be ended with this option. Then restarted of course.
If you know there is a problem with only one web server, you can bring only it down by going to UNITY / Start uniPharm Extensions / VA Company / UWD Extensions / Web Interface. Then use the menu option to end the affected server.
Power down without restart
- This must be done on the console
This is used when we want the computer to stay off - for hardware service, or because power will be lost. Be sure to use the option to end all processes first, and the option ‘Work with Active Jobs’ to make sure everything has ended before you continue with the power down.
Over Night Processing
Over night processing is split into two parts, both of which are submitted by the job scheduler. The first does all the updating, reporting, housekeeping, etc. The first thing it does is shut down all user access (except for the HMC), and all monitors that use ASW or extension data files. We call this 'application restricted state'. Domino is left up so the program can send emails reporting on its progress. If these emails are not received, we know something may have gone wrong.
The second part does a full system backup; which means the computer has to first go into a system restricted state. In this state, the system can't even be accessed by the HMC. (A system restricted state can be either interactive or batch. If interactive, it can only be run from the HMC; which stays available. If batch, it can only be run in the QCTL subsystem, and the only access to the computer is through the control panel.)
The first part starts at 1:00AM and usually finishes around 2:15AM. If it doesn't, because it is in error, or in a loop, it will be cancelled when the backup starts at 3:00AM. We will have to keep an eye on this - if the length of the process approaches two hours, we will have to change the timings.
When the backup starts the restricted state, it sets a timer of 3 hours. If it is still running when those three hours are up, the system will cancel it, and start everything back up again. It currently takes about two hours.
Application Restricted State
This is what we call it when we shut down all user access to the system, and end the automated jobs. We cannot do a full backup of ASW and extension files if any users or programs are using them.
If the system is in this state, the only access to the system is through the HMC or through System i Navigator.
End Application Restricted State
The system my be stuck here if something happens during the overnight processing. Or it may be partially in this state. In either case, the fix is the same.
On the HMC, sign on as EODJOB with our favorite password. Key in GO DISASTER and press enter. Note that the the menu options in red must be used with extreme care!
DISASTER Disaster Menu Select one of the following: 11. Work with active jobs 12. Work with system status 13. Work with disk status 14. Display operator messages 31. Start all processes 32. End all processes 33. Start web servers 34. End web servers 91. Power down without restart 99. Signoff Selection or command ===> ___________________________________ F3=Exit F4=Prompt F9=Retrieve F12=Cancel F13=Information Assistant F16=System main menu
Option 31 will start up all processes. If any are currently running, they will not be affected.
Another option is to use System i Navigator to start up the Qinter subsystem; which will allow any terminal to sign on.
Under My Connections, open: Bart.unipharm.com / Work Management / Subsystems
Right-click on Active subsystems, click on Start subsystem. Enter QINTER, QSYS. Then click on 'OK'.
Log onto a green screen as usual.
TO PREVENT USERS FROM LOGGING ON WHILE THE SYSTEM IS UNSTABLE, key in TRANSFER and press enter. This will move your job from subsystem QINTER to QCTL. You can then end QINTER with the command ENDSBS QINTER.
System Restricted State
The command ‘End subsystem *all’ will put the system into a restricted state; which will allow a full system save.
It can only be run in the controlling sub system (QCTL); either interactively on the HMC workstation session, or when submitted to job queue QCTL (which runs batch jobs in the controlling sub system).
When run interactively, the HMC session will remain active. You will be able to respond to error messages. In Client Access, click on ‘Actions / Display Popup Keypad / SysRq’ and press enter. You will have the option to display (and answer) system operator messages, and to display the current job – which let you look at the job log.
If it is run in batch, you will be completely locked out of the system; all you will be able to see is the SRC (system code on the front panel of the processor). For this reason, a parameter has been added – BCHTIMLMT. This controls how long the system will stay in a batch restricted state. If there is a message that requires an answer, or a program goes into a loop, the system will sit there waiting (and you cannot do anything about it). When the batch time limit is up, the program issuing the command will be cancelled, and QCTL will be started up. This will run QSTRUP – which will start everything else.
If you know something is wrong, and don’t want to wait for time to run out, you can manually force the end of the batch restricted state.
End System Restricted State
On the green screen, get an HMC workstation sign on display.
Then, in FireFox go to - (If you are in FireFox now, you can right click on the following address, and select 'open ink in new window'.)
https://hmcpower8.unipharm.local/preloginmonitor/index.jsp
Click on ‘log on and Launch the Hardware Management Console web application’
Sign on with HMCROOT, and our new favorite password.
Expand ‘Systems Management / Servers’. Click on server.
Select ‘Bart’. Expand ‘Serviceability / Control Panel Functions’.
Click on ‘(21) Activate Dedicated Service Tools’
The HMC workstation session will change to –
Dedicated Service Tools (DST) Sign On
System: BART
Type choices, press Enter.
Service tools user . . . . . . . . . . . __________
Service tools password . . . . . . . . . __________
F3=Exit F5=Change password F12=Cancel
The user is QSECOFR, and password is ‘i’ plus our favorite.
Use Dedicated Service Tools (DST)
System: BART
Select one of the following:
1. Perform an IPL
2. Install the operating system
3. Work with Licensed Internal Code
4. Work with disk units
5. Work with DST environment
6. Select DST console mode
7. Start a service tool
8. Perform automatic installation of the operating system
10. Work with remote service support
13. Work with system security
14. End batch restricted state
Selection
__
F3=Exit F12=Cancel
The option to us is ’14. End batch restricted state’.
Here is a link to ‘Batch Full System Saves to the Controlling Subsystem Using BRMS’.
http://www-01.ibm.com/support/docview.wss?uid=nas8N1019121
End Applications Individually
EDI Monitor
UNITY / Start uniPharm Extensions / VA Company / EDI Menu / EDI Monitor - EDI Gateway
12/01/15 16:16:35 EDI Gateway Status EDRIU011/CTL
uniPHARM -- PRODUCTION
Monitor status: Running Waiting for next poll
Last activity: 12/01/15 16:15:21 FTP: ftp.edigateway.com
Poll interval: 15 minutes 5:00 AM to 11:00 PM In: /in
Out: /out
A-D=Query 1-4=DFU X=Clear ---- Outgoing --- -- Incoming ---
File Not Int Not Sent Waiting At SPS Archive
------- Incoming -------------
Invoice header 1 810 73,021
Invoice header 2 810 73,022
Invoice line 810 12 593,018
(ASW A/P intfc) SROLBA 810
P/O resp header 855 8 7,608
P/O resp address 855 50 15,101
P/O resp line 855 123,555
ASN header 856 21 28,230 +
F3=Exit F5=Refresh F6=Refr FTP F8=End Mon F9=Poll Now
F13=Configuration F14=Files F15=EDI Messages F16=W/W Mon Job F12=Exit
Inbound P/O data in error: F17=Fix F18=Clear
Inbound invoices in error: F19=Fix F20=Clear F21=Print F22=UOM mapping
Wait until the status is 'Running Waiting for next poll', then press F8=End Monitor. Press F5=Refresh until the status is 'Not running'.
=Email Orders
UNITY / Start uniPharm Extensions / VA Company / UWD Extensions / EDI - IMS, IMP, etc / 88 EDI Control Panel
12/01/15 16:22:58 uniPHARM -- PRODUCTION EDRIQ100/CONTROL
EDI -- Main Control Panel PRODUCTION 1.00
Action Sub System
System Status Switch
__ IMS Inbound Mail Receiving 20 Fully Active START
__ IMP Inbound Mail Parsing 20 Fully Active START
__ IOP Inbound Order Posting 20 Fully Active START
__ OMS Outbound Mail Send 20 Fully Active START
__ EMA Alerts 20 Fully Active START
Actions
ES End Subsystem SP=Start Monitor Program
SS Start Subsystem EP=End Monitor Program
OF Switch System Activity OFF
ON Switch System Activity ON
F3=Exit F5=Refresh F8=Details F10=Active F12=Previous
F14=IOP Exc Msgs On F15=IOP Exc Msgs Off
For each process you want to stop, run these actions -
OF Switch System Activity OFF EP End Monitor Program ES End Subsystem