Information Systems:Notes on development using Iptor Integrator

Overview

This document discusses various topics regarding software development using Java and Iptor Integrator.

Topics

Not using Integrator Studio as an IDE

Iptor Integrator includes an IDE called Integrator Studio, but even 10 minutes of use is enough to indicate its inferiority as any kind of serious IDE (if you're a developer and not a power/business user). Specifically, some of its limitations are listed below.

• There are two modes, a drag-drop user mode and a Java/code mode. Except the Java mode does not show you everything. Upon inspecting the reverse-generated source (i.e. using IntelliJ's decompiler), it was discovered that additional code is added on later by Integrator Studio's build process. So even the supposed developer mode doesn't show you all the code. For example, import statements are not shown, as well as the main() function.
• Apart from the inability to see code altogether, some code that deals with proprietary features is also marked up. For example, doing something in the visual editor will translate to what looks like Java code, but appears in the decompiled source with more complicated syntax i.e. marked up.
• Often, compiler errors point to a line containing an error but the IDE doesn't suggest further information about what the error could be. So debugging is a painful process, often involving a line-by-line, trial-and-error approach to try and pinpoint the exact location of the error.
• Sometimes, the code inspector will get hung up on an error that is no longer present, and the IDE has to be restarted.
• Running a program crashes the IDE. A LOT.
• The data mapping wizard also crashes a lot, especially when you browse database libraries on the i.
• The lack of a dark theme (lol).

Personally, I find the IDE unusable for developing programs beyond the complexity of a simple sript. Therefore, I use the IDE to learn Integrator-specific capabilities by testing them out using the visual editor and then studying the generated java source. Then I code in IntelliJ IDEA. Actually, the real value in Integrator from a developer's perspective comes from the Integrator libraries e.g. net.ibs.*, not their crappy IDE -norwizzle (talk). However, these are not documented (because you're supposed to use their IDE), so their functionality has to be studied by inference. -norwizzle (talk)

Classpath length

Integrator runs on Windows, which has a well-known limitation regarding the classpath. Specifically, it can only be a certain length before you get this error upon adding more JARs to the library.

Error when starting process -classpath. Ex: java.io.IOException: Cannot run program "C:\Integrator\java\jre\bin\ibs_EventLogMonitor_Default.exe": CreateProcess error=206, The filename or extension is too long

Read this article for more information on the problem and how it was avoided.

Indispensable third-party libraries

Of course, one of the great advantages of using a modern programming language like Java is the availability of libraries so that you don't have to reinvent the wheel when programming more common tasks (or even less common for that matter). For example, libraries exist that make parsing XML, CSV and Excel files easier (not just easier, but possible, really). For reference purposes, in case one is interested to know which library was chosen (as sometimes many alternatives exist), here are the significant ones in use:

• IBM Toolbox for Java - Programming in Java for IBM i would be useless without this quintessential package.
• Apache Commons - another immensely useful library. Actually, Commons is an umbrella for a collection of many libraries. The one that have been used significantly thus far are:

• Apache POI - For reading/writing to Microsoft document formats. The XSSF is particularly useful for dealing with XLSX files e.g. the Health Canada MAR report uses this to output and stylize the report.
• Apache Commons Net - FTPClient class used for FTP connections
• Apache Commons CSV - Useful for reading/writing CSV files.

• Backblaze SDK - provides classes for managing files on Backblaze object storage. One current use is for uploading the Atlas POD images to Backblaze B2, which can serve the files as a CDN.
• JCraft - For sFTP communication, as Apache Commons Net above is limited to FTP.
• univocity - CSV parser/writer. The CSVWriter class has features that are not found in Apache Commons CSVPrinter.
• MariaDB JDBC driver - to read/write to the database.
• HikariCP - Connection Pool implementation for JDBC.
• Apache PDFBox - Reading/writing PDFs.

Potential Integrator projects

The following extensions/integrations (currently in legacy RPG/CL) may be moved to Integrator.

• System checks / process monitors: Email alerts for periodic checks on system function and business continuity.
• Carrier integration: send shipping manifest to carriers, receive acknowledgements/shipping updates/invoice files from carriers.
• EDI: Build EDI/csv documents from purchase order data; receive EDI/csv documents from EDI provider and update ERP.
• Bank EDI: generate EFT payment file from statement information, process inbound credit card transaction summary files and post to A/R, process inbound cheque-clearing file.
• Periodically-generated reports: Gilead daily transactions report, Health Canada MAR, lost sales.
• SQL data-warehousing, including offload/ETL into open-source database.
• Update lost sales codes.
• Update P/O delivery ETA, send emails.
• File rebuilds: consolidated item master, web orders item master, stats file i.e. run SQL in Java, write to Db2 on i.
• Inbound order processing: receive EDI/txt files from pharmacies (either polled via FTP, polled via email inbox, or by watching folders for new files). Send electronic invoices via FTP.
• ECU: generate catalog for customer/pharmacy POS systems (e.g. Kroll, WinRx etc.)
• Generate remittance advices, distribute via email or write back to OUTQ for Catapult/TLAshford processing.
• System cleanup functions in the overnight schedule and in purge jobs: clear outqueues, prune tables.

Questions

These questions have come up from using and dissecting Integrator without prior formal training.

ASW interaction

• What's the proper way to call ASW programs from Integrator? i.e. what are the commands to change company and set up data area etc.
• What is the process to implement a Job Control program in Integrator?
• Noticed an article referencing an IBM i Integrator component (accima.savf) that can facilitate calls from RPG/IBM i to Integrator programs - is there more info on this?

General

• Is there javadoc available for the Integrator framework libs/API? e.g. IBSFramework.jar
• There seems to be a difference in the index of the first program arg when running the same class from a Timer job vs ClassRunner. Is this by design or is it a bug? Specifically, I think Timer job takes args[0] while ClassRunner takes args[1].
• Is there any way to access the java.sql.Connection object within JDBCConnection object? This is so I can use Integrator's connection pooling functionality outside of Design Studio. getJDBCConnection() seems to be the method but it returns null.
• Can the FolderWatcher flow be explained in more detail? i.e. the path of a new file and what conditions are that make it hang in the temp folder or go to the error or incomplete folders.
• What is the recommended way to manage the hash files that get generated as part of the duplicate detection mechanism for FolderWatcher jobs? One job currently has over 250,000 of these hash files. Do we just have to delete these manually?
• Different question but same intent as the previous point - how best to use Integrator's connection pooling outside Design Studio i.e. in Java?
• What is the best way to manage the packages in IBSExternalPackages? I've been upgrading some JARs when there's version conflict with project dependencies, but I believe I had to revert once before as Integrator was not compatible with the newer version. Is there a document stating the minimum/maximum versions supported for these dependencies?
• Is the IBSUserPackages folder used? Is this folder what its name indicates i.e. the location to put additional JARs? Why is it not part of the classpath?
• Are there any caveats/considerations when developing Integrator-run programs in a different IDE? e.g. how to set classpath so that the XMLs in the config folder are read?
• Is it possible to change the content body for alert emails sent by the Monitor feature? i.e. to include more of the stack trace, or to incorporate HTML content
• How can the "forward to syslog" feature be utilized? Does JMX have to be incorporated? Or what Design Studio functions correlate to this feature in Integrator Server?
• Feature request - can ClassRunner be changed to accept runtime args?
• Feature request - can there be a Delete All button for ClassRunner logs?
• Bug report - the process for setting up queue persistence in Communicator contains a bad SQL statement (reported as ticket)