Information Systems:Managing dependencies in Iptor Integrator

Overview

This article discusses the limitations regarding Iptor Integrator and the Java classpath, as well as the solution to the problem (Maven and pre-packaged project dependencies). This does not apply if you're using Integrator within its confines i.e. use Integrator Studio to develop programs, [which I don't do].

Background

Being itself built with Java, Iptor Integrator relies on open-source Java libraries to run. An example is the Apache DBCP library for database connection pooling, which Integrator's own Connection Pooling feature is built on (This could be inaccurate, but the dbcp JAR does exist and I'm just making a point anyway -norwizzle (talk)). These open-source libraries (about 250 JARs) come pre-packaged with the Integrator installation and are found in C:\Integrator\bin\extensions\ibs\eclipse\plugins\IBSExternalPackages. Because this folder is part of the classpath, it makes sense to also put project dependencies in here. However, adding JARs to this folder gives rise to two problems:

1. The classpath becomes too long as more JARs are added. There are alreaady 242 dependencies. Empirically, the classpath issue arises at around the 300 JAR mark. Integrator's JRE has some proprietary elements so it is not easy/possible to change the classpath to perhaps use the folder/wildcard notation.
2. Version conflict between shared system and project dependencies. i.e. the project requires the same library as Integrator, but needs a newer version which is incompatible with Integrator. This is exacerbated when version conflicts exist within transitive dependencies. To give an example, let's say Integrator can only use version 4.1 of a hypothetical "logginglib", but the Backblaze B2 SDK (project dependency) requires v5.6 or higher of the lib. This is known as [dependency hell.]

Solution

There were several things tried, but the eventual solution involves 2 things:

1. The incorporation of Maven to manage project dependencies and package them into a fat jar (aka uber jar) called uwd.jar. Now, Integrator loads only uwd.jar from the classpath and there is no longer a risk of this becoming too long. Regarding Maven, IntelliJ IDEA has plugins that can be used to simplify Maven processes (e.g mvn:compile, mvn:install etc.). The pom.xml file is found in the root folder of the project.
2. Manual and careful curation of shared dependencies (libs required by both Integrator and the project). This is not an elegant process and can basically be described as follows: if I encounter an error like NoClassFoundError or NoSuchMethodError related to a shared dependency when running project code in Integrator, I check to confirm that the Integrator lib folder contains an older version of the shared dependency (uwd.jar contains the newer one). I then delete this so the new version gets used, and test (by stopping and starting the Integrator service) to see if Integrator complains about having to use the new version. If it doesn't, rejoice. If it does, then I figure something else out (Like I said, inelegant.)

Notes

• There is a IBSUserPackages folder, but it doesn't seem to get loaded whatsoever by Integrator. This is something worth asking about.
• Related to the previous point, Integrator Studio provides a mechanism so that users can add packages to the Integrator environment. Interestingly, using this feature puts the jar in an adjacent folder (plugins\IBSIntegrator), which is not part of the classpath for Timer and FolderWatcher jobs.