Difference between revisions of "Information Systems:Optimizing ENDOFDAY / overnight processing"

Latest revision as of 15:18, 4 October 2018

Overnight / end-of-day (EOD) processing refers to the backup job(s) and batch jobs (e.g. ASW DIS-to-FIN, stats rebuilds) that occur each night. The overall routine and strategy require periodic review as organic data growth combined with other system factors (i.e. new backup technology, newly installed software or added overnight jobs) necessitate changes and adjustments to keep the maintenance window within an optimal duration.

Criteria for overnight processes

The following lists general criteria for an ideal overnight/backup strategy:

A full-system save (including a *SAVSYS) puts the system into restricted-state. For daily backups, a *SAVSYS may be too excessive.
Overnight jobs should be grouped together and separated into programs to allow the ability to re-run them in case they fail.

Breakdown of the backup job

Since much of everything that is saved doesn't take very long, only significant 'stages' (larger libraries) are noted here (e.g. >5m save time)

~15m for SAVSYS (IBM LIC and other core OS components)
~10m for DOCFOLDERS (Gauss)
~15m UP1480BFPL (not being backed up)
~25m for UP1480BFVA
~16m for UWDASWPRDD

XXSTATSP rebuild

XXSTATSP is a file that gets rebuilt every night, and part of this rebuild is a routine to compile sales statistics at the item-warehouse level for the past 12 months. To get this information, the rebuild uses Analyzer (balance) files. These balance files were rebuilt in mid-2017 to include data from more years (this was to fix the infamous 32.94 issue). Therefore, ANOBAL is now larger than before (probably 20% larger). The effect of this growth on the rebuild time of XXSTATSP is exponentially amplified by the program's inefficient design, which essentially does the following:

Take each item in XXITEMP.
For each item, iterate through the entirety of ANOBAL, filtering through the 66-million record file by current year and summary type with each iteration.
Get the period-to-date total in $.
Iterate through ANOBAL again to get the period-to-date quantity in number of units sold.

By using SQL and possibly creating a work table (a subset of ANOBAL that serves the purpose of the rebuild), this process could potentially be improved. At best, this rebuild took 45m. After the Analyzer rebuild and organic growth, the process now takes a whopping 1h35m. But I believe even 45m is still way longer than it needs to be.

Possibly related: BLDCSTSL2

As with the XXSTATS rebuild, BLDCSTSL2 also looks at Analyzer/ANOBAL. This process has grown from an average of 14-16m to 25-30m.

XXITEMP rebuild

The rebuild of XXITEMP has been consistent at 15-20m. However, at the beginning of every month, there is additional processing (to be examined) that extends this rebuild to 40m+.

Other notes

It is startling how fast some processes in the overnight are. Consider DIS TO FIN, which we consider some big, critical task that allows ASW to 'go into tomorrow'. It takes less than a minute.
Update: Turns out UP1480BFPL and UWDASWVOLD don't actually get backed up. I missed it as this was define in the Backup Policy, not the Backup Control Group.

Methods of Analysis

A Python script was developed to run over the info in RCSUTILPRD/LGFITXP, which holds the data for the logging utility developed by Steel Net and used extensively in the overnight processing jobs to log each step. The logging only became consistent in Mar. 2016 after Sheila settled on the overnight scheme that still persists today. The little program just makes analysis easier by calculating the time differences between two processes and outputting to the terminal:

On 2017-08-28, EODBU      had a duration of  2h:03m:16s.
On 2017-08-29, EODBU      had a duration of  2h:08m:05s.

It may be of some interest to develop a web utility in the future that looks at this file. So we can access trends in the overnight processing in a dynamic way.

For analyzing the backup jobs, the following reports were used:

GO BRMS -> Reports -> Print backup log generates QP1ALG and details each step of the backup job. The timestamps of each stage are useful.
GO BRMS -> Backup analysis is also a good tool, but a RTVDSKINF needs to be run first (batch job which takes a while).

Running SQL on qsys2.systablestat is highly informative as well, showing sizes of tables (in number of records and size of files)

History of steps

Excluded development libraries.
Cleaned up PTF save files.

@@ Line 1: / Line 1: @@
+Overnight / end-of-day (EOD) processing refers to the backup job(s) and batch jobs (e.g. ASW DIS-to-FIN, stats rebuilds) that occur each night. The overall routine and strategy require periodic review as organic data growth combined with other system factors (i.e. new backup technology, newly installed software or added overnight jobs) necessitate changes and adjustments to keep the maintenance window within an optimal duration.
-Investigating ways to shorten the total time of the overnight process (i.e. backup, ASW processing, rebuilds etc.)
-==Potential avenues to explore==
-===Excluding objects from the temporary backup===
-Currently, the temporary backup job saves the entirety of the following libraries:
+==Criteria for overnight processes==
-* RCSUTILPRD
+The following lists general criteria for an ideal overnight/backup strategy:
-* UP1480BFVA
-* UWDASWPRDD
-* WEBPRDD
+* A full-system save (including a *SAVSYS) puts the system into restricted-state. For daily backups, a *SAVSYS may be too excessive.
-The backup currently takes about 1h10m, having grown from an average of ~50m at the start of 2016. Given the libraries above, the increase in backup time can most likely be attributed to organic data growth. While the philosophy of the temporary backup (e.g. what dangers we're protecting against by doing it) should be questioned overall, an immediate fix would be to logically assemble a list of files to exclude from each of the libraries above.
+* Overnight jobs should be grouped together and separated into programs to allow the ability to re-run them in case they fail.
-===Excluding objects from the full backup===
+====Breakdown of the backup job====
+Since much of everything that is saved doesn't take very long, only significant 'stages' (larger libraries) are noted here (e.g. >5m save time)
-A full system save is cute, and the original times of ~2h drew no objections. But the entire backup will also start to grow
+* ~15m for SAVSYS (IBM LIC and other core OS components)
+* ~10m for DOCFOLDERS (Gauss)
+* ~15m UP1480BFPL (not being backed up)
+* ~25m for UP1480BFVA
+* ~16m for UWDASWPRDD
 ===XXSTATSP rebuild===
@@ Line 18: / Line 20: @@
 * Take each item in XXITEMP.
-* For each item, iterate through the entirety of ANOBAL, filtering by current year
+* For each item, iterate through the entirety of ANOBAL, filtering through the 66-million record file by current year and summary type with each iteration.
-* Get sales in $.
+* Get the period-to-date total in $.
-* Iterate through ANOBAL again to get quantity in number of units.
+* Iterate through ANOBAL again to get the period-to-date quantity in number of units sold.
-By using SQL and possibly creating a work table (a subset of ANOBAL that serves the purpose of the rebuild), this process could potentially be improved. At best, this rebuild took 45m. After the Analyzer rebuild and organic growth, the process now takes a whopping 1h35m.
+By using SQL and possibly creating a work table (a subset of ANOBAL that serves the purpose of the rebuild), this process could potentially be improved. At best, this rebuild took 45m. After the Analyzer rebuild and organic growth, the process now takes a whopping 1h35m. But I believe even 45m is still way longer than it needs to be.
-====BLDCSTSL2====
+====Possibly related: BLDCSTSL2====
 As with the XXSTATS rebuild, BLDCSTSL2 also looks at Analyzer/ANOBAL. This process has grown from an average of 14-16m to 25-30m.
@@ Line 32: / Line 34: @@
 ==Other notes==
 * It is startling how fast some processes in the overnight are. Consider DIS TO FIN, which we consider some big, critical task that allows ASW to 'go into tomorrow'. It takes less than a minute.
+* '''Update''': Turns out UP1480BFPL and UWDASWVOLD don't actually get backed up. I missed it as this was define in the Backup Policy, not the Backup Control Group.
 ===Methods of Analysis===
-A Python script was developed to run over the info in RCSUTILPRD/LGFITXP, which holds the data for the logging utility developed by Steel Net and used extensively in the overnight processing jobs to log each step. The logging only became consistent in Mar. 2016 after Sheila settled on the overnight scheme that still persists today.
+* A Python script was developed to run over the info in RCSUTILPRD/LGFITXP, which holds the data for the logging utility developed by Steel Net and used extensively in the overnight processing jobs to log each step. The logging only became consistent in Mar. 2016 after Sheila settled on the overnight scheme that still persists today. The little program just makes analysis easier by calculating the time differences between two processes and outputting to the terminal:
+ On 2017-08-28, EODBU      had a duration of  2h:03m:16s.
+ On 2017-08-29, EODBU      had a duration of  2h:08m:05s.
+It may be of some interest to develop a web utility in the future that looks at this file. So we can access trends in the overnight processing in a dynamic way.
+* For analyzing the backup jobs, the following reports were used:
+:* GO BRMS -> Reports -> Print backup log generates QP1ALG and details each step of the backup job. The timestamps of each stage are useful.
+:* GO BRMS -> Backup analysis is also a good tool, but a RTVDSKINF needs to be run first (batch job which takes a while).
+* Running SQL on qsys2.systablestat is highly informative as well, showing sizes of tables (in number of records and size of files)
+==History of steps==
+* Excluded development libraries.
+* Cleaned up PTF save files.
 [[Category: Overnight Processing]]
+[[Category: I.T. Periodic Tasks]]