This article was written not to discuss what systems integration is from the theoretical side, because a lot of books, a lot of articles, good blogs have already been written about it. In writing this article, or rather a series of articles, I would like to introduce you to the issue of integration as seen through the eyes of a person whose daily life is working on integration projects, in the role of "integrator" of systems. With about 10 years of experience in the IT industry (I have worked on larger and smaller projects),I think I will be able, to present some recurrent project problems and approaches to their solution on the example of ESB provided by Mulesoft.
What is integration and why is it not the most important thing?
Integration in itself is not a new concept. Certainly integration should not be equated with the concept of ESB. Integration was before ESB became common and will be when ESB becomes out of fashion, replaced by something else - a new approach, a different methodology, which in the end will have the same task: to force a certain number of systems to communicate with each other. I can't tell you how many years it took for the IT market in Poland to reach a situation where probably every major private or public company has some kind of dedicated integration solution implemented. Take my word for it, when I started my work in IT, the common practice was to create JavaEE class applications responsible for integration, where, in order to integrate CRM with Provisioning, one created a dedicated application in EJB installed on the application server, where all communication was implemented in Java, where, at the end, one created with Java some "better" input interface than RMI for our operations. This was often SOAP with a ton of Java code generated. Whether it was good or bad is a matter of taste, it certainly accomplished its purpose as did the integration layer based on EJB 2.0 where mainly database, C++ and Cobol domain systems had adapters written in JNI responsible for communicating with services in EJB. This, too, is an integration that worked production-wise and perhaps continues to do so.
So why such a development of EAI and later ESB class tools ? The first reason is certainly marketing, because it is much easier to sell something that is a "dedicated product for the implementation of integration" than an application server with the ability to run applications in Java, which we have to write ourselves beforehand. Customers also often expect a "closed, dedicated tool" to perform certain tasks, even though we often have to work much harder with this closed dedicated tool than with creating our own solution with the hands of a few programmers, not to mention the cost. I am not against integration technologies, I think they have a lot of advantages. For me, the biggest one is the built-in number of connectors to all those SAPs, CRMs and many other systems, so that it is very quick and easy from the technical side to create and publish in an organization a service responsible for retrieving customer data from Salesforce, and you don't have to bite into a specific API, waste time learning about such an API to be able to use it at a slightly higher than typical basic level. A few more advantages could be mentioned, but it's not that ESB is a cure-all for all the integration hell of this world. It is certainly a good solution, meeting expectations and giving a lot of possibilities, but the problem of integration in itself is that from the point of view of the organization it is not the most important, from the point of view of the business it is often seen as an additional "cost", a "fad" because, after all, it is not CRM, where we can enter and manage customers, it is not Provisioning, which will set up our router or TV set-top box, it is not a transaction system, which will "push" transfers, so what is it for ? Many times I have encountered such statements, probably not only me, but anyway in the end business people who are "sponsors" of IT changes manage to convince themselves, because eventually these solutions appeared and are widely used. The second major problem of the integration itself is that often, despite sincere intentions to do it really well, to define a good and well-thought-out data model, to create a well-organized catalog of services, we are not able to do it for one simple reason - when we appear with our new integration (let it be our ESB) we often find a certain entity already. Of course, there are projects that are completely new, where the whole architecture changes, where we create our service catalog based on our new model and as part of a larger project, all domain systems have to adapt, but usually it looks like we have a large company X with systems deployed for X amount of time, in a chaotic architecture often Point-to-Point, sometimes attempts to realize the integration layer based on, for example, JavaEE applications, sometimes mixed approaches and there is a decision - we implement ESB, we move to a service architecture. Up to this point it's fine, problems arise with the pressure of time and budget for this project, where we can't create a nice lightweight structured interface in a REST approach, because our SAP has always called out SOAP services and there "development is expensive and we won't change it", "the data model must stay, even though it's bad and poorly designed". Such a project outline usually crystallizes after the first few weeks of project work, and then we already know that even though we will buy and implement a solution that has many capabilities, it won't really matter, because in many aspects we will have to keep the status quo.
A simple integration scenario in a non-ideal world
We will now discuss a simplified integration scenario, but the problem presented in it is one of the real problems we will encounter in a real project, in a typical company deciding to implement/create a new service integration layer based on an ESB class system. Let's consider a situation where, implementing an ESB, we want to create a service for retrieving user data from a CRM system, as the leading customer management system in the organization. We also have an old database-based system with customers who have not been migrated to CRM. We want to create a lightweight REST service available for our domain systems, based on the canonical data model, which will be owned by the integration department. However, we have a system that is currently integrated using SOAP, with an interface that deviates from the new target model. It must remain unchanged. All services should use SSL, in addition for rest services we want to have Http Basic Auth. The current architecture provides for the use of an external load balancer, the target architecture will additionally use Mule API Gateway, so the solution should be as easy as possible to be "plugged" into Mule API Gateway.
The general outline of the above architecture is shown in the diagram below. On it we see two systems, SAP accessible within the intranet using SOAP, connecting to the ESB using an internal load balancer, and an external system using a REST service and a load balancer dedicated to traffic from outside the internal network. On the other side, we have two Salesforce systems integrated into the ESB using the native connector provided with Mule ESB, and our "old" database system integrated via JDBC.
In the service itself, three important things affecting the course of its implementation can be distinguished:
- Converting the request to a unified interface: in the situation of having a requirement to publish a service for two different systems requiring different interfaces (REST/JSON and SOAP), we need to isolate the place responsible for the execution of all operations related to the service of the requesting system, such as mapping the request to the ESB model, possible security, if we can't execute them before the ESB. In our case, the current architecture allows us to configure SSL at the level of load balancers, but we do not have the ability to configure Http Basic Auth, so this will be implemented at the level of Mule ESB flows.
- The business logic of our service: our requirement says clearly, in the absence of an account in Salesforce, we should try to search for it in a legacy database system. It is a good practice to separate the operations directly related to the ordering channels from the logic that should be reused and have one common interface.
- Support for domain systems responsible for providing data: the last major requirement for our service, to provide a place responsible for implementing integration with domain systems responsible for providing data. Again, as in the above case, so here we should not mix business logic with the implementation of integration to domain systems for one simple reason - someday maybe Salesforce or DB System will be replaced by another CRM or another database, in which case the need for changes should not interfere with the code responsible for both the channel layer and the business logic of the service, because these components do not change.
The diagram below is a presentation of the first requirement. In it we see two systems SAP and Web App. Each of them calls the service using a different interface. Each of these systems has a dedicated application on Mule responsible for publishing/implementing the API. In the case of the SAP system, we have a sap-channel which is responsible for issuing the SOAP API and the subsequent transformation of the XML request into a REST interface, which is the output interface for the service responsible for implementing the business logic. The Web App system uses the REST interface and the canonical model (used by the Mule ESB), so there is no need to do the transformation because the model used by the Web App is compatible with the model used by the ESB. However, we have a requirement to provide HTTP Basic Auth for the Web App, and the implementation of this will be done at the webapp-channel level.
The next two requirements are the following diagram. The first major issue is the implementation of business logic, which has been elevated to a dedicated business-account application. In fact, this is where the implementation of our real account download service begins. The business-agreement application publishes a REST service, using the canonical model. The task of channel applications is to use it, either by simply calling it as in a Web App, where there is no protocol conversion, or as in the case of a sap-channel application, where we need to convert from SOAP to REST. The previously mentioned business logic is "encapsulated" within the business-account application. In this case, the logic is quite simple, if we can't find the user data in Salesforce, we have to query the DB System. Both Salesforce and DB System are domain systems with which our ESB must be integrated. As in the case of the channel and business parts, dedicated applications (adapters) responsible for integration at the level of domain systems have been prepared for implementation. The basic task of these applications is to build a request to the domain system and call it. In our case, executing a DQL (DataSense Query Language) based SELECT request using the Salesforce connector provided with Mule in the case of the salesforce-adapter, and a SQL query using JDBC for the Web App system.
These three important implementation tasks I mentioned above were assigned to the three types of applications:
- Publishing the interface for the client application, protocol conversion, data model and possible security - channel applications
- Unified interface, canonical data model and business logic - business type applications
- Conversion to a request for a domain system, system invocation, and response mapping to the canonical model - adapter type applications.
In the featured post, I introduced and discussed the concept of system integration, we discussed what system integration is and its relation to ESB class tools. In the second part, I presented, in some simplification, one of the typical scenarios, problems that we may encounter during the implementation of ESB within the existing enterprise IT. In the next post, I will discuss the implementation of the above scenario based on Mule ESB.
All posts in this series:
- [Introduction to systems integration based on Mule ESB]({{ "/introduction-to-system-integration-with-mule-esb/" | prepend: site.baseurl }})
- [Introduction to systems integration based on Mule ESB (part 2)]({{ "/introduction-to-system-integration-with-support-o-mule-esb-part-2/" | prepend: site.baseurl }})
