Information Systems:Optimizing ENDOFDAY / overnight processing - Revision history

Norwinu at 22:18, 4 October 2018

2018-10-04T22:18:33Z

Norwinu at 23:07, 3 October 2018

2018-10-03T23:07:23Z

Norwinu: /* Potential avenues to explore */

2017-12-15T23:19:39Z

Potential avenues to explore

Norwinu: /* Methods of Analysis */

2017-12-15T19:28:17Z

Methods of Analysis

Norwinu at 19:26, 15 December 2017

2017-12-15T19:26:15Z

Norwinu: /* Breakdown of the backup job */

2017-12-15T19:03:15Z

Breakdown of the backup job

Norwinu at 17:51, 15 December 2017

2017-12-15T17:51:46Z

Norwinu: /* Excluding objects from the full backup */

2017-12-14T23:21:05Z

Excluding objects from the full backup

Norwinu: /* XXSTATSP rebuild */

2017-12-14T23:16:22Z

XXSTATSP rebuild

Norwinu: /* Other notes */

2017-12-14T23:12:26Z

Other notes

@@ Line 1: / Line 1: @@
-Overnight / end-of-day (EOD) processing refers to the system backup job(s) and batch jobs (e.g. ASW DIS-to-FIN, stats rebuilds) that occur each night. The overall routine and strategy require periodic review as organic data growth combined with other system factors (i.e. new backup technology, newly installed software or added overnight jobs) necessitate changes and adjustments to keep the maintenance window within an optimal duration.
+Overnight / end-of-day (EOD) processing refers to the backup job(s) and batch jobs (e.g. ASW DIS-to-FIN, stats rebuilds) that occur each night. The overall routine and strategy require periodic review as organic data growth combined with other system factors (i.e. new backup technology, newly installed software or added overnight jobs) necessitate changes and adjustments to keep the maintenance window within an optimal duration.
 ==Criteria for overnight processes==
@@ Line 6: / Line 6: @@
 * A full-system save (including a *SAVSYS) puts the system into restricted-state. For daily backups, a *SAVSYS may be too excessive.
 * Overnight jobs should be grouped together and separated into programs to allow the ability to re-run them in case they fail.
 ====Breakdown of the backup job====

← Older revision		Revision as of 23:07, 3 October 2018
Line 1:		Line 1:
		+	Overnight / end-of-day (EOD) processing refers to the system backup job(s) and batch jobs (e.g. ASW DIS-to-FIN, stats rebuilds) that occur each night. The overall routine and strategy require periodic review as organic data growth combined with other system factors (i.e. new backup technology, newly installed software or added overnight jobs) necessitate changes and adjustments to keep the maintenance window within an optimal duration.
−	Investigating ways to shorten the total time of the overnight process (i.e. backup, ASW processing, rebuilds etc.)
−	==Potential avenues to explore==
−	===Excluding objects from the temporary backup===
−	Currently, the temporary backup job saves the entirety of the following libraries:

		+	==Criteria for overnight processes==
−	* RCSUTILPRD
		+	The following lists general criteria for an ideal overnight/backup strategy:
−	* UP1480BFVA
−	* UWDASWPRDD
−	* WEBPRDD

		+	* A full-system save (including a SAVSYS) puts the system into restricted-state. For daily backups, a SAVSYS may be too excessive.
−	The backup currently takes about 1h10m, having grown from an average of ~50m at the start of 2016. Given the libraries above, the increase in backup time can most likely be attributed to organic data growth. While the philosophy of the temporary backup (e.g. what dangers we're protecting against by doing it) should be questioned overall, an immediate fix would be to logically assemble a list of files to exclude from each of the libraries above.
		+	* Overnight jobs should be grouped together and separated into programs to allow the ability to re-run them in case they fail.

−	===Excluding objects from the full backup===
−	The duration of the full backup is showing signs of slow growth, from about 1h55m-2h05m in Jan. 2016 to 2h10m-2h20m presently. The backup policy is to save:
−
−	* All IBM data
−	* All user data
−	* All document data
−	* All directory data
−
−	There are a lot of objects being unnecessarily backed up every night:
−
−	* Various IFS folders related to banking files, Web Orders etc. For the most part, the IFS can and should be backed up to Superserver periodically, even during the day. These are PC-formatted files which don't need to be saved as IBM objects; they can be restored by dragging and dropping in Windows Explorer!
−	* WEBPRDD (production files related to Web Orders and Infonet), is being backed up twice, once in the temporary backup and in the full system save. If DIS-TO-FIN doesn't affect Web Orders, why are we backing this library up twice?
−	* Yearly snapshots of the SROSRO file.
−	* The entire PL library is being backed up - do we need nightly backups of this? (Update: The 2 big dev libraries are not actually being backed up.)
−	* CIDOWNP is a 39GB MBR file storing raw data from carrier invoices/waybills/manifests. It will be trimmed.


Line 70:		Line 51:

	* Running SQL on qsys2.systablestat is highly informative as well, showing sizes of tables (in number of records and size of files)		* Running SQL on qsys2.systablestat is highly informative as well, showing sizes of tables (in number of records and size of files)
		+
		+	==History of steps==
		+	* Excluded development libraries.
		+	* Cleaned up PTF save files.

	[[Category: Overnight Processing]]		[[Category: Overnight Processing]]
		+	[[Category: I.T. Periodic Tasks]]

@@ Line 24: / Line 24: @@
 * WEBPRDD (production files related to Web Orders and Infonet), is being backed up twice, once in the temporary backup and in the full system save. If DIS-TO-FIN doesn't affect Web Orders, why are we backing this library up twice?
 * Yearly snapshots of the SROSRO file.
-* The entire PL library is being backed up - do we need nightly backups of this?
+* The entire PL library is being backed up - do we need nightly backups of this? (Update: The 2 big dev libraries are not actually being backed up.)
 ====Breakdown of the backup job====

@@ Line 65: / Line 65: @@
 * For analyzing the backup jobs, the following reports were used:
 :* GO BRMS -> Reports -> Print backup log generates QP1ALG and details each step of the backup job. The timestamps of each stage are useful.
 :* GO BRMS -> Backup analysis is also a good tool, but a RTVDSKINF needs to be run first (batch job which takes a while).
 [[Category: Overnight Processing]]

@@ Line 53: / Line 53: @@
 ==Other notes==
 * It is startling how fast some processes in the overnight are. Consider DIS TO FIN, which we consider some big, critical task that allows ASW to 'go into tomorrow'. It takes less than a minute.
 ===Methods of Analysis===
@@ Line 62: / Line 63: @@
 It may be of some interest to develop a web utility in the future that looks at this file. So we can access trends in the overnight processing in a dynamic way.
 [[Category: Overnight Processing]]

@@ Line 28: / Line 28: @@
 ====Breakdown of the backup job====
 Since much of everything that is saved doesn't take very long, only significant 'stages' (larger libraries) are noted here (e.g. >5m save time)
 * ~15m for SAVSYS (IBM LIC and other core OS components)
 * ~10m for DOCFOLDERS (Gauss)
-* ~15m for UP1480BFPL
+* ~15m UP1480BFPL (not being backed up)
 * ~25m for UP1480BFVA
 * ~16m for UWDASWPRDD

@@ Line 25: / Line 25: @@
 * Yearly snapshots of the SROSRO file.
 * The entire PL library is being backed up - do we need nightly backups of this?
 ===XXSTATSP rebuild===
@@ Line 53: / Line 61: @@
 It may be of some interest to develop a web utility in the future that looks at this file. So we can access trends in the overnight processing in a dynamic way.
-* For analyzing the backup jobs, the spool file generated by BRMS during every backup job has a timestamp on each object saved. This, along with using the 'Schemas' utility of Access Client Solutions determine largest files in each library, can help build a list of files to exclude.
+* For analyzing the backup jobs, GO BRMS -> Reports -> Print backup log generates QP1ALG and details each step of the backup job. The timestamp difference between subsequent steps give good indication to the duration of the save time of each library. This, along with using the 'Schemas' utility of Access Client Solutions determine largest files in each library, can help build a list of files to exclude.
 [[Category: Overnight Processing]]

@@ Line 21: / Line 21: @@
 There are a lot of objects being unnecessarily backed up every night:
-* Various IFS folders related to banking files, Web Orders etc. For the most part, the IFS can and should be backed up to Superserver periodically, even during the day.
+* Various IFS folders related to banking files, Web Orders etc. For the most part, the IFS can and should be backed up to Superserver periodically, even during the day. These are PC-formatted files which don't need to be saved as IBM objects; they can be restored by dragging and dropping in Windows Explorer!
 * WEBPRDD (production files related to Web Orders and Infonet), is being backed up twice, once in the temporary backup and in the full system save. If DIS-TO-FIN doesn't affect Web Orders, why are we backing this library up twice?
 * Yearly snapshots of the SROSRO file.

@@ Line 30: / Line 30: @@
 * Take each item in XXITEMP.
-* For each item, iterate through the entirety of ANOBAL, filtering by current year
+* For each item, iterate through the entirety of ANOBAL, filtering through the 66-million record file by current year and summary type with each iteration.
-* Get sales in $.
+* Get the period-to-date total in $.
-* Iterate through ANOBAL again to get quantity in number of units.
+* Iterate through ANOBAL again to get the period-to-date quantity in number of units sold.
-By using SQL and possibly creating a work table (a subset of ANOBAL that serves the purpose of the rebuild), this process could potentially be improved. At best, this rebuild took 45m. After the Analyzer rebuild and organic growth, the process now takes a whopping 1h35m. But I believe 45m is still longer than it needs to be.
+By using SQL and possibly creating a work table (a subset of ANOBAL that serves the purpose of the rebuild), this process could potentially be improved. At best, this rebuild took 45m. After the Analyzer rebuild and organic growth, the process now takes a whopping 1h35m. But I believe even 45m is still way longer than it needs to be.
 ====Possibly related: BLDCSTSL2====

@@ Line 46: / Line 46: @@
 ===Methods of Analysis===
-A Python script was developed to run over the info in RCSUTILPRD/LGFITXP, which holds the data for the logging utility developed by Steel Net and used extensively in the overnight processing jobs to log each step. The logging only became consistent in Mar. 2016 after Sheila settled on the overnight scheme that still persists today. The little program just makes analysis easier by calculating the time differences between two processes and outputting to the terminal:
+* A Python script was developed to run over the info in RCSUTILPRD/LGFITXP, which holds the data for the logging utility developed by Steel Net and used extensively in the overnight processing jobs to log each step. The logging only became consistent in Mar. 2016 after Sheila settled on the overnight scheme that still persists today. The little program just makes analysis easier by calculating the time differences between two processes and outputting to the terminal:
  On 2017-08-28, EODBU      had a duration of  2h:03m:16s.
  On 2017-08-29, EODBU      had a duration of  2h:08m:05s.
-Also, for analyzing the backup jobs, the spool file generated by BRMS during every backup job has a timestamp on each object saved. This, along with using the 'Schemas' utility of Access Client Solutions determine largest files in each library, can help build a list of files to exclude.
+* For analyzing the backup jobs, the spool file generated by BRMS during every backup job has a timestamp on each object saved. This, along with using the 'Schemas' utility of Access Client Solutions determine largest files in each library, can help build a list of files to exclude.
 [[Category: Overnight Processing]]