Information Systems:Bart Hardware Issues History

Power8

Buggy CPU firmware causing massive system slowdown

Problem description

Occurrence: August 2015, Decemeber 2015

The Power8 has had two instances of severe system slowdown that affected everything from green-screen menu interaction, overnight backup and processing, and network latency. With network issues being the primary symptom, the entire network configuration in the server room and on the IBM i was examined and ruled out. This led to a network card replacement on September 11, 2015, which fixed the issue. The problem manifested itself again on December 2, 2015, and thus a recurrence pattern of about 2 months was determined. During the 2nd PMR, it was determined that the cause of the issue was buggy CPU firmware that caused improper allocation of CPU resources to the LPAR.

Outcome

A temporary fix was implemented by creating another LPAR and running it. See here for official IBM fix page. This running partition keeps the CPU allocation in check.

SAS/RAID card firmware incompatibility led to unbootable LPAR

Problem description

Occurrence: December 2-4, 2016

A maintenance upgrade was scheduled in December 2016 that involved:

1. Updating the Technology Refresh level from TR2 to TR5.
2. Updating all other PTF groups (including the cumulative group) to their latest release.
3. Upgrading the system firmware from 810.40 to 840.30.

While all the updates/upgrades were performed successfully, the LPAR could not be started, and it was discovered that the system no longer recognized the SAS/RAID cards controlling the disk array (ASP) on which the LPAR lived. This caused a complete system-down situation. Updating the firmware to an even more recent level did not help, neither did reseating the hardware cards and attempting to reinitialize them through the HMC.

Outcome

IBM had to ship out new cards and they were replaced the following day (Saturday), after which the system immediately recognized the replacement cards and the system booted successfully. End-of-day processing had to be done manually on a Saturday, but there were virtually no consequences outside of email and Web Orders being down for the earlier part of Saturday.

The likely theory is that the jump from firmware level 810.40 to 840.30 was too big, such that the newer system firmware was no longer compatible with whatever level of firmware was on the SAS cards, leading to them being unable to be recognized by the system. Our system met the prerequisites for the firmware upgrade, so this was clearly an undiscovered problem by IBM.