Information Systems:Optimizing ENDOFDAY / overnight processing

From uniWIKI
Revision as of 10:51, 15 December 2017 by Norwinu (talk | contribs)
Jump to navigation Jump to search

Investigating ways to shorten the total time of the overnight process (i.e. backup, ASW processing, rebuilds etc.)

Potential avenues to explore

Excluding objects from the temporary backup

Currently, the temporary backup job saves the entirety of the following libraries:

  • RCSUTILPRD
  • UP1480BFVA
  • UWDASWPRDD
  • WEBPRDD

The backup currently takes about 1h10m, having grown from an average of ~50m at the start of 2016. Given the libraries above, the increase in backup time can most likely be attributed to organic data growth. While the philosophy of the temporary backup (e.g. what dangers we're protecting against by doing it) should be questioned overall, an immediate fix would be to logically assemble a list of files to exclude from each of the libraries above.

Excluding objects from the full backup

The duration of the full backup is showing signs of slow growth, from about 1h55m-2h05m in Jan. 2016 to 2h10m-2h20m presently. The backup policy is to save:

  • All IBM data
  • All user data
  • All document data
  • All directory data

There are a lot of objects being unnecessarily backed up every night:

  • Various IFS folders related to banking files, Web Orders etc. For the most part, the IFS can and should be backed up to Superserver periodically, even during the day. These are PC-formatted files which don't need to be saved as IBM objects; they can be restored by dragging and dropping in Windows Explorer!
  • WEBPRDD (production files related to Web Orders and Infonet), is being backed up twice, once in the temporary backup and in the full system save. If DIS-TO-FIN doesn't affect Web Orders, why are we backing this library up twice?
  • Yearly snapshots of the SROSRO file.
  • The entire PL library is being backed up - do we need nightly backups of this?

Breakdown of the backup job

Since much of everything that is saved doesn't take very long, only significant 'stages' (larger libraries) are noted here (e.g. >5m save time)

  • ~15m for SAVSYS (IBM LIC and other core OS components)
  • ~10m for DOCFOLDERS (Gauss)
  • ~15m for UP1480BFPL
  • ~25m for UP1480BFVA
  • ~16m for UWDASWPRDD

XXSTATSP rebuild

XXSTATSP is a file that gets rebuilt every night, and part of this rebuild is a routine to compile sales statistics at the item-warehouse level for the past 12 months. To get this information, the rebuild uses Analyzer (balance) files. These balance files were rebuilt in mid-2017 to include data from more years (this was to fix the infamous 32.94 issue). Therefore, ANOBAL is now larger than before (probably 20% larger). The effect of this growth on the rebuild time of XXSTATSP is exponentially amplified by the program's inefficient design, which essentially does the following:

  • Take each item in XXITEMP.
  • For each item, iterate through the entirety of ANOBAL, filtering through the 66-million record file by current year and summary type with each iteration.
  • Get the period-to-date total in $.
  • Iterate through ANOBAL again to get the period-to-date quantity in number of units sold.

By using SQL and possibly creating a work table (a subset of ANOBAL that serves the purpose of the rebuild), this process could potentially be improved. At best, this rebuild took 45m. After the Analyzer rebuild and organic growth, the process now takes a whopping 1h35m. But I believe even 45m is still way longer than it needs to be.

Possibly related: BLDCSTSL2

As with the XXSTATS rebuild, BLDCSTSL2 also looks at Analyzer/ANOBAL. This process has grown from an average of 14-16m to 25-30m.

XXITEMP rebuild

The rebuild of XXITEMP has been consistent at 15-20m. However, at the beginning of every month, there is additional processing (to be examined) that extends this rebuild to 40m+.

Other notes

  • It is startling how fast some processes in the overnight are. Consider DIS TO FIN, which we consider some big, critical task that allows ASW to 'go into tomorrow'. It takes less than a minute.

Methods of Analysis

  • A Python script was developed to run over the info in RCSUTILPRD/LGFITXP, which holds the data for the logging utility developed by Steel Net and used extensively in the overnight processing jobs to log each step. The logging only became consistent in Mar. 2016 after Sheila settled on the overnight scheme that still persists today. The little program just makes analysis easier by calculating the time differences between two processes and outputting to the terminal:
On 2017-08-28, EODBU      had a duration of  2h:03m:16s.
On 2017-08-29, EODBU      had a duration of  2h:08m:05s.

It may be of some interest to develop a web utility in the future that looks at this file. So we can access trends in the overnight processing in a dynamic way.

  • For analyzing the backup jobs, GO BRMS -> Reports -> Print backup log generates QP1ALG and details each step of the backup job. The timestamp difference between subsequent steps give good indication to the duration of the save time of each library. This, along with using the 'Schemas' utility of Access Client Solutions determine largest files in each library, can help build a list of files to exclude.
Retrieved from "https://owl.unipharm.com/mediawiki/index.php?title=Information_Systems:Optimizing_ENDOFDAY_/_overnight_processing&oldid=10314"