Difference between revisions of "Information Systems:Disaster Recovery Tasks on IBM i"
m (Norwinu moved page Information Systems:IBM i Disaster Recovery Tasks to Information Systems:Disaster Recovery Tasks on IBM i without leaving a redirect) |
|||
| Line 533: | Line 533: | ||
[[Category: Disaster Recovery]] |
[[Category: Disaster Recovery]] |
||
| + | [[Category: IBM i]] |
||
Revision as of 15:43, 22 June 2016
Disaster Menu
- Note that if you plan on running options 32 or 91, or if the system is currently in the application restricted state, this must be done on the HMC.
To access this menu, sign on as EODJOB, with our favorite password, and press enter. (This is to get the correct library list.) Key in GO DISASTER and press enter.
DISASTER Disaster Menu Select one of the following: 11. Work with active jobs 12. Work with system status 13. Work with disk status 14. Display operator messages 31. Start all processes 32. End all processes 33. Start web servers 34. End web servers 91. Power down without restart 99. Signoff Selection or command ===> ___________________________________ F3=Exit F4=Prompt F9=Retrieve F12=Cancel F13=Information Assistant F16=System main menu
The options in black can be run at any time – the ones in red must be used only for extreme occasions.
The first four are pretty much self-explanatory.
Start all processes
This is used when the system is in an application restricted state, or when some applications (ie. Domino, Web Orders, electronic orders, etc) aren't running. This option passes over those applications that are running, and starts those that aren't.
End all processes
- This must be done on the HMC.
This will end everything and bring the system into an application restricted state; taking about 10 minutes. It brings everything down gracefully – for example we originally had problems with Domino unless it was ended just right. The system must be in this state in order to do a complete save of the ASW or extension data files. Use option 11 ‘Work with Active Jobs’ to make sure everything has ended before you continue with what you are doing. Knowing what we know now, it also should have been used on 16 Dec 2013 to keep everyone off of the system while document number control files were being updated.
Start web servers
This will start servers that aren’t currently running. Our web servers currently are WEBINTER for InfoNet, WEBSMART for Web Orders, and WEBTEST for testing.
End web servers
This option could cause internal users and customers to lose what they are in the process of updating. We have occasionally had a problem with the web orders server programs getting the date formats mixed up, which caused some orders to fail in IOP with invalid dates (not all orders, as there are multiple server programs). There is no way to identify which program has the problem, so they can all be ended with this option. Then restarted of course.
If you know there is a problem with only one web server, you can bring only it down by going to UNITY / Start uniPharm Extensions / VA Company / UWD Extensions / Web Interface. Then use the menu option to end the affected server.
Power down without restart
- This must be done on the console
This is used when we want the computer to stay off - for hardware service, or because power will be lost. Be sure to use the option to end all processes first, and the option ‘Work with Active Jobs’ to make sure everything has ended before you continue with the power down.
Over Night Processing
Over night processing is split into two parts, both of which are submitted by the job scheduler. The first does all the updating, reporting, housekeeping, etc. The first thing it does is shut down all user access (except for the HMC), and all monitors that use ASW or extension data files. We call this 'application restricted state'. Domino is left up so the program can send emails reporting on its progress. If these emails are not received, we know something may have gone wrong.
The second part does a full system backup; which means the computer has to first go into a system restricted state. In this state, the system can't even be accessed by the HMC. (A system restricted state can be either interactive or batch. If interactive, it can only be run from the HMC; which stays available. If batch, it can only be run in the QCTL subsystem, and the only access to the computer is through the control panel.)
The first part starts at 1:00AM and usually finishes around 2:15AM. If it doesn't, because it is in error, or in a loop, it will be cancelled when the backup starts at 3:00AM. We will have to keep an eye on this - if the length of the process approaches two hours, we will have to change the timings.
When the backup starts the restricted state, it sets a timer of 3 hours. If it is still running when those three hours are up, the system will cancel it, and start everything back up again. It currently takes about two hours.
Application Restricted State
This is what we call it when we shut down all user access to the system, and end the automated jobs. We cannot do a full backup of ASW and extension files if any users or programs are using them.
If the system is in this state, the only access to the system is through the HMC or through System i Navigator.
End Application Restricted State
The system my be stuck here if something happens during the overnight processing. Or it may be partially in this state. In either case, the fix is the same.
On the HMC, sign on as EODJOB with our favorite password. Key in GO DISASTER and press enter. Note that the the menu options in red must be used with extreme care!
DISASTER Disaster Menu Select one of the following: 11. Work with active jobs 12. Work with system status 13. Work with disk status 14. Display operator messages 31. Start all processes 32. End all processes 33. Start web servers 34. End web servers 91. Power down without restart 99. Signoff Selection or command ===> ___________________________________ F3=Exit F4=Prompt F9=Retrieve F12=Cancel F13=Information Assistant F16=System main menu
Option 31 will start up all processes. If any are currently running, they will not be affected.
Another option is to use System i Navigator to start up the Qinter subsystem; which will allow any terminal to sign on.
Under My Connections, open: Bart.unipharm.com / Work Management / Subsystems
Right-click on Active subsystems, click on Start subsystem. Enter QINTER, QSYS. Then click on 'OK'.
Log onto a green screen as usual.
TO PREVENT USERS FROM LOGGING ON WHILE THE SYSTEM IS UNSTABLE, key in TRANSFER and press enter. This will move your job from subsystem QINTER to QCTL. You can then end QINTER with the command ENDSBS QINTER.
System Restricted State
The command ‘End subsystem *all’ will put the system into a restricted state; which will allow a full system save.
It can only be run in the controlling sub system (QCTL); either interactively on the HMC workstation session, or when submitted to job queue QCTL (which runs batch jobs in the controlling sub system).
When run interactively, the HMC session will remain active. You will be able to respond to error messages. In Client Access, click on ‘Actions / Display Popup Keypad / SysRq’ and press enter. You will have the option to display (and answer) system operator messages, and to display the current job – which let you look at the job log.
If it is run in batch, you will be completely locked out of the system; all you will be able to see is the SRC (system code on the front panel of the processor). For this reason, a parameter has been added – BCHTIMLMT. This controls how long the system will stay in a batch restricted state. If there is a message that requires an answer, or a program goes into a loop, the system will sit there waiting (and you cannot do anything about it). When the batch time limit is up, the program issuing the command will be cancelled, and QCTL will be started up. This will run QSTRUP – which will start everything else.
If you know something is wrong, and don’t want to wait for time to run out, you can manually force the end of the batch restricted state.
End System Restricted State
On the green screen, get an HMC workstation sign on display.
Then, in FireFox go to - (If you are in FireFox now, you can right click on the following address, and select 'open ink in new window'.)
https://hmcpower8.unipharm.local/preloginmonitor/index.jsp
Click on ‘log on and Launch the Hardware Management Console web application’
Sign on with HSCROOT, and our new favorite password.
Expand ‘Systems Management / Servers’. Click on server.
Select ‘Bart’. Expand ‘Serviceability / Control Panel Functions’.
Click on ‘(21) Activate Dedicated Service Tools’
The HMC workstation session will change to –
Dedicated Service Tools (DST) Sign On
System: BART
Type choices, press Enter.
Service tools user . . . . . . . . . . . __________
Service tools password . . . . . . . . . __________
F3=Exit F5=Change password F12=Cancel
The user is QSECOFR, and password is ‘i’ plus our favorite.
Use Dedicated Service Tools (DST)
System: BART
Select one of the following:
1. Perform an IPL
2. Install the operating system
3. Work with Licensed Internal Code
4. Work with disk units
5. Work with DST environment
6. Select DST console mode
7. Start a service tool
8. Perform automatic installation of the operating system
10. Work with remote service support
13. Work with system security
14. End batch restricted state
Selection
__
F3=Exit F12=Cancel
The option to us is ’14. End batch restricted state’.
Here is a link to ‘Batch Full System Saves to the Controlling Subsystem Using BRMS’.
http://www-01.ibm.com/support/docview.wss?uid=nas8N1019121
End Applications Individually
EDI Monitor
UNITY / Start uniPharm Extensions / VA Company / EDI Menu / EDI Monitor - EDI Gateway
12/01/15 16:16:35 EDI Gateway Status EDRIU011/CTL
uniPHARM -- PRODUCTION
Monitor status: Running Waiting for next poll
Last activity: 12/01/15 16:15:21 FTP: ftp.edigateway.com
Poll interval: 15 minutes 5:00 AM to 11:00 PM In: /in
Out: /out
A-D=Query 1-4=DFU X=Clear ---- Outgoing --- -- Incoming ---
File Not Int Not Sent Waiting At SPS Archive
------- Incoming -------------
Invoice header 1 810 73,021
Invoice header 2 810 73,022
Invoice line 810 12 593,018
(ASW A/P intfc) SROLBA 810
P/O resp header 855 8 7,608
P/O resp address 855 50 15,101
P/O resp line 855 123,555
ASN header 856 21 28,230 +
F3=Exit F5=Refresh F6=Refr FTP F8=End Mon F9=Poll Now
F13=Configuration F14=Files F15=EDI Messages F16=W/W Mon Job F12=Exit
Inbound P/O data in error: F17=Fix F18=Clear
Inbound invoices in error: F19=Fix F20=Clear F21=Print F22=UOM mapping
Wait until the status is 'Running Waiting for next poll', then press F8=End Monitor. Press F5=Refresh until the status is 'Not running'.
Email Orders
UNITY / Start uniPharm Extensions / VA Company / UWD Extensions / EDI - IMS, IMP, etc / EDI Control Panel
12/01/15 16:22:58 uniPHARM -- PRODUCTION EDRIQ100/CONTROL
EDI -- Main Control Panel PRODUCTION 1.00
Action Sub System
System Status Switch
__ IMS Inbound Mail Receiving 20 Fully Active START
__ IMP Inbound Mail Parsing 20 Fully Active START
__ IOP Inbound Order Posting 20 Fully Active START
__ OMS Outbound Mail Send 20 Fully Active START
__ EMA Alerts 20 Fully Active START
Actions
ES End Subsystem SP=Start Monitor Program
SS Start Subsystem EP=End Monitor Program
OF Switch System Activity OFF
ON Switch System Activity ON
F3=Exit F5=Refresh F8=Details F10=Active F12=Previous
F14=IOP Exc Msgs On F15=IOP Exc Msgs Off
For each process you want to stop, run these actions -
OF Switch System Activity OFF EP End Monitor Program ES End Subsystem
FTP Orders
UNITY / Start uniPharm Extensions / VA Company / UWD Extensions / EDI - IMS, IMP, etc / FTP Order Retrieval
12/01/15 16:46:55 AIP Orders Status AIRIU010/CTL
uniPHARM -- PRODUCTION
Monitor status: Running Waiting for next poll
Last activity: 12/01/15 16:45:57 Poll interval: 3 minutes
________ ___ <-- Order type - CAN/DAT/KRL
Date Time Type File Renamed Recs
12/01/15 4:45 PM KRO 115403308.kro FTP0087242 12
12/01/15 4:36 PM KRO 115403307.kro FTP0087241 18
12/01/15 4:36 PM KRO 121372383.kro FTP0087240 51
12/01/15 4:31 PM KRO 107676727.kro FTP0087239 1
12/01/15 4:22 PM KRO 7821713.kro FTP0087238 43
12/01/15 4:18 PM KRO 12140181.kro FTP0087237 10
12/01/15 4:14 PM KRO 121821356.kro FTP0087236 1
12/01/15 4:09 PM KRO 107676726.kro FTP0087235 8
12/01/15 4:09 PM KRO 107676721.kro FTP0087234 2
12/01/15 4:05 PM KRO 12144798.kro FTP0087233 64
12/01/15 4:04 PM KRO 3072617.kro FTP0087232
F3=Exit F5=Refresh F8=End Mon F9=Poll Now
F13=Config F14=Interfaces F15=AIP Messages F16=W/W Mon Job F12=Exit
Wait until the status is 'Running Waiting for next poll', then press F8=End Monitor. Press F5=Refresh until the status is 'Not running'.
Web Servers
UNITY / Start uniPharm Extensions / VA Company / UWD Extensions / Web Interface
Each of the three web servers can be started or ended individually. Use the option 'Show HTTP server status' to make sure they have ended.
If you end all three web servers, you can also end the subsystem.
ENDSBS SBS(QHTTPSVR) DELAY(60)
Telnet
Telnet must end before QSYSWRK subsystem is ended, other wise there has been an issue with the RG guns.
ENDTCPSVR SERVER(*TELNET)
User subsystems
ENDSBS SBS(QINTER) OPTION(*IMMED) ENDSBS SBS(QBATCH) OPTION(*IMMED) ENDSBS SBS(DCRF) OPTION(*IMMED) ENDSBS SBS(UP1480BS) OPTION(*IMMED) ENDSBS SBS(TLABARCODE) OPTION(*IMMED) ENDSBS SBS(BC400SERVR) OPTION(*IMMED) ENDSBS SBS(MONITOR2) OPTION(*IMMED)
Gauss
Add libraries to your library list
ADDLIBLE LIB(DOCFOLDERS) POSITION(*LAST) ADDLIBLE LIB(DOCMGRCMD) POSITION(*LAST) ADDLIBLE LIB(DOCMGRDTA) POSITION(*LAST)
Then end Gauss, and the susbsystem it runs in
ENDLIS LISTENER(*DOCVIEW)
ENDLIS LISTENER(*SPYCS)
ENDSBS SBS(SPYCS) OPTION(*IMMED)
Domino
First try to end gracefully
ENDDOMSVR SERVER(MAGGIE) OPTION(*CNTRLD) JSC(*YES)
Wait three minutes, then force anything left to end.
ENDDOMSVR SERVER(MAGGIE) OPTION(*IMMED) JSC(*YES)
Start Applications Individually
Operator Message Monitor
This program keeps an eye on the system operator message queue, and emails any that are unexpected.
STRSBS SBSD(MONITOR)
SBMJOB CMD(CALL PGM(OPMESSAGES)) JOB(MSGMONITOR)
JOBD(EODJOB) JOBQ(MONITOR) USER(EODJOB)
User Subsystems
STRSBS SBSD(QSYSWRK)
STRSBS SBSD(QSNADS)
STRSBS SBSD(QSERVER)
STRSBS SBSD(XL_EXODUS/EXODUS)
STRSBS SBSD(XL_EX51/EXODUS51)
STRSBS SBSD(QINTER)
STRSBS SBSD(QBATCH)
STRSBS SBSD(DCRF)
STRSBS SBSD(DOCMGROBJ/SPYCS)
STRSBS SBSD(TLACS/TLABARCODE)
STRSBS SBSD(MONITOR2)
Web Servers
UNITY / Start uniPharm Extensions / VA Company / UWD Extensions / Web Interface
Each of the three web servers can be started or ended individually. Use the option 'Show HTTP server status' to make sure they have started.
Email Orders
UNITY / Start uniPharm Extensions / VA Company / UWD Extensions / EDI - IMS, IMP, etc / EDI Control Panel
12/01/15 16:22:58 uniPHARM -- PRODUCTION EDRIQ100/CONTROL
EDI -- Main Control Panel PRODUCTION 1.00
Action Sub System
System Status Switch
__ IMS Inbound Mail Receiving 20 Fully Active START
__ IMP Inbound Mail Parsing 20 Fully Active START
__ IOP Inbound Order Posting 20 Fully Active START
__ OMS Outbound Mail Send 20 Fully Active START
__ EMA Alerts 20 Fully Active START
Actions
ES End Subsystem SP=Start Monitor Program
SS Start Subsystem EP=End Monitor Program
OF Switch System Activity OFF
ON Switch System Activity ON
F3=Exit F5=Refresh F8=Details F10=Active F12=Previous
F14=IOP Exc Msgs On F15=IOP Exc Msgs Off
For each process you want to start, run these actions -
ON Switch System Activity ON SS Start Subsystem SP Start Monitor Program
Seagull Server (JWALK GUI for ASW)
SEALMS/STRLMSTCP PORT(*DFT) DAS(*NO)
EDI Monitor
UNITY / Start uniPharm Extensions / VA Company / EDI Menu / EDI Monitor - EDI Gateway
12/01/15 16:16:35 EDI Gateway Status EDRIU011/CTL
uniPHARM -- PRODUCTION
Monitor status: Not running
Last activity: 12/01/15 16:15:21 FTP: ftp.edigateway.com
Poll interval: 15 minutes 5:00 AM to 11:00 PM In: /in
Out: /out
A-D=Query 1-4=DFU X=Clear ---- Outgoing --- -- Incoming ---
File Not Int Not Sent Waiting At SPS Archive
------- Incoming -------------
Invoice header 1 810 73,021
Invoice header 2 810 73,022
Invoice line 810 12 593,018
(ASW A/P intfc) SROLBA 810
P/O resp header 855 8 7,608
P/O resp address 855 50 15,101
P/O resp line 855 123,555
ASN header 856 21 28,230 +
F3=Exit F5=Refresh F6=Refr FTP F7=Str Mon
F13=Configuration F14=Files F15=EDI Messages F16=W/W Mon Job F12=Exit
Inbound P/O data in error: F17=Fix F18=Clear
Inbound invoices in error: F19=Fix F20=Clear F21=Print F22=UOM mapping
Press F7=Start Monitor.
FTP Orders
UNITY / Start uniPharm Extensions / VA Company / UWD Extensions / EDI - IMS, IMP, etc / FTP Order Retrieval
12/01/15 16:46:55 AIP Orders Status AIRIU010/CTL
uniPHARM -- PRODUCTION
Monitor status: Not running
Last activity: 12/01/15 16:45:57 Poll interval: 3 minutes
________ ___ <-- Order type - CAN/DAT/KRL
Date Time Type File Renamed Recs
12/01/15 4:45 PM KRO 115403308.kro FTP0087242 12
12/01/15 4:36 PM KRO 115403307.kro FTP0087241 18
12/01/15 4:36 PM KRO 121372383.kro FTP0087240 51
12/01/15 4:31 PM KRO 107676727.kro FTP0087239 1
12/01/15 4:22 PM KRO 7821713.kro FTP0087238 43
12/01/15 4:18 PM KRO 12140181.kro FTP0087237 10
12/01/15 4:14 PM KRO 121821356.kro FTP0087236 1
12/01/15 4:09 PM KRO 107676726.kro FTP0087235 8
12/01/15 4:09 PM KRO 107676721.kro FTP0087234 2
12/01/15 4:05 PM KRO 12144798.kro FTP0087233 64
12/01/15 4:04 PM KRO 3072617.kro FTP0087232
F3=Exit F5=Refresh F7=Str Mon
F13=Config F14=Interfaces F15=AIP Messages F16=W/W Mon Job F12=Exit
Press F7=Start Monitor.
Other Monitor Programs
Carrier interface (Atlas)return transmissions
SBMJOB CMD(CALL PGM(ATCUT010)) JOBQ(MONITOR2)
USER(ATLAS)
Sales order allocation
SBMJOB CMD(CALL PGM(DMCUPALLOC)) JOB(ORDALLOCFL)
JOBQ(MONITOR2) USER(EODJOB)
Domino
STRDOMSVR SERVER(MAGGIE) JSC(*NO)
Gauss
Add libraries to your library list
ADDLIBLE LIB(DOCFOLDERS) POSITION(*LAST)
ADDLIBLE LIB(DOCMGRCMD) POSITION(*LAST)
ADDLIBLE LIB(DOCMGRDTA) POSITION(*LAST)
Then start Gauss and the subsystem it runs in
STRSBS SBSD(DOCMGROBJ/SPYCS)
STRLIS LISTENER(*DOCVIEW)
STRLIS LISTENER(*SPYCS)