Information Systems:Retiring Python/open-source development on IBM i

Overview

This page discusses a project/endeavor (specifically, its cancellation) that took place between 2017-2019 to implement a post-RPG software development strategy on IBM i by using a modern programming language as well as open-source software. As a concept, this was (and this day continues to be) an approach strongly advocated by IBM for users of IBM i.

The following archived articles are related to this endeavor:

Background and History

uniPHARM uses IBM i to host its ERP. The ERP and its extensions are programmed in RPG, which is an aging language (officially known as "legacy", along with COBOL).
IBM i is a highly proprietary operating system but contains a "space" that emulates a Linux environment (PASE), complete with a shell. This is somewhat analogous to WSL.
Starting in 2017, IBM injected mass improvements to PASE, porting over major programming languages and open-source software such as MySQL/MariaDB, Python, Mono (.NET), Node.js, and nginx. They essentially created Linux within IBM i (pedants would be quick to correct this but it's the practical truth).
Along with bringing over modern programming languages so that they can be run on the i, IBM also provided XMLSERVICE, which is a system API of sorts to allow these languages to interact with IBM i programs and objects e.g. Python program can call an RPG program.

Therefore, the ability to use modern programming languages and an open-source development strategy common, combined with the proximity to the built-in Db2 database and usefuleness of XMLSERVICE to interface with the system, made this a dream situation, on paper.

Decision to abandon

There were several factors that led to the eventual demise of this endeavor and strategy:

Issues with package 'flavor'

Due to the difference in platform

Newness of technology to a legacy audience

It became apparent after a year or two of observing the IBM i OSS community that they were in a state described as a "peak of inflated expectations". Community discourse mainly pertained to how "cool" the technology was, and conferences/tech-talks would always showcase pedestrian examples, focusing more on the fact that the technology was indeed running on IBM i rather than how to use the tech to solve real-world business challenges. This was found to be a major barrier, as there was no active development community experiencing the same type of issues. As of 2021, IBM i OSS still comes off as a cute pet project for IBM to show how the platform isn't the "AS/400" of old and can be "modern". Nah, not really... -norwizzle (talk)

Performance

The Power8 is a well-spec'ed machine, but ultimately it has a single-core CPU (i.e. 1/6 activated) and spinning disk storage. While the design of IBM i may be able to skirt this sort of a legacy hardware configuration (in the age of multi-core CPUs and flash storage), these bottlenecks very obviously affected performance in PASE (the emulation taking palce may also very well be a factor). The web server (nginx) and app server (gunicorn) would be sluggish compared the built-in IBM i HTTP server which runs the sites built with WebSmart.

Database flavor differences

Db2 has a respectable marketshare amongst other relational databases, but it exists in many flavors, presumably the most popular being Db2 LUW (Linux, Unix, Windows). The one built in to IBM i is known as Db2 for i, and several issues have already been encountered that are attributed to the flavor differences. Things related row_over(), resource record numbers, and system naming vs SQL naming are just some of the issues that have caused major delays or necessitated hacky workarounds because of their particular implementation.

System availability

The IBM i server is shut down nightly between 1AM to 6AM for maintenance processes. And because it runs critical workloads, the server also cannot just be restarted for whatever reason, and extra care must always be taken in terms of where files are written, how much CPU programs are taking up, and the security impact of running programs directly alongside the ERP. For example, opening up another port on the server to host a web app in debug mode (for testing/development) to be accessible on the internet potentially exposes the entire server. It was also deemed impractical or overly complex to find workarounds to limited system availability, such as maintaining a parallel environment on x86 for testing, subscribing to free IBM i public-shared instances, or setting up chroot within PASE.

Availability of more appropriate alternatives

Over the span of this endeavor, alternatives started to appear more appropriate as a solution. These include:

Using the more common strategy of deploying Linux x86 servers and connecting remotely via ODBC to the database.
Increased feasibility of running and maintaining Linux workloads on-premise. IT has Linux administration skills in-house and a virtual environment with ample resources on which new virtual servers can be provisioned.
Using the Java programming language, which is supported on the i and has more "backing". In particular, the IBM Toolbox for Java / JTOpen software is a very mature product that serves the same purpose as XMLSERVICE, but is far superior in every way (e.g. performant, well-supported, intended for remote system access rather than for accessing the i from within PASE etc.)
Availability and eventual procurement of Iptor Integrator, which is Java-based middleware that provides a structured way of developing/running integrations.
The revelation that much of DC1 (the upgrade to ASW) uses the Java platform (XT, Netstore, Integrator).

Evolving philosophy within IT

In hindsight, part of why open-source on i was approved as an exploratory endeavor is because of the 'i' factor. That is, it may have been too much of a shock to propose open-source on x86 sintead, which is the conventional method of modern development (e.g. Linux or Windows servers). At this time, there were no Linux servers in place, no use of external databases, and no use of other programming languages for development (only RPG, CL, and WebSmart). The fact that IBM i OSS was an official project provided (now false) assurance that if new, open-source development were to be explored, doing so within IBM i was the best choice. It turned out to cause more headaches than provide gains, and it took this experience to gain the philosophy that the i is better left to run RPG/CL and that running integrations externally is no less viable than running on the same system.

Outcome

uniPHARM has continued with modern development using Iptor Integrator, Java, IBM Toolbox for Java, Python, Django, Apache, and MariaDB as the main components.
Java is the prevailing language to be used for developing back-end integrations. Python is used for Django web app development (InfoNext).
The IBM i is no longer being used to do any new development (WebSmart being the exception). All the software listed above runs in Windows (Iptor Integrator) or Linux (Apache, Django, MariaDB).
uniPHARM is no longer working with the 2 consultant firms previously contracted for the Carrier Interface / IBM i OSS project.