Migrating Camel EIP applications to AWS using Event-Driven Architecture

The Enterprise Integration Patterns is a popular way of designing integration systems. Every pattern comes with a nice infographic and guidelines on how to use it. Being implementation agnostic, it offers a solution for a re-occurring design challenge. Using EIP, architects can quickly create and communicate their designs to implementation teams and business owners. Afterwards, logical components can be mapped almost 1:1 to physical ones using a framework which supports the same patterns.

Let’s assume that an integrational process consists of the following steps:

1. An external system sends an API request or puts a file to a sftp location.
2. The inbound component transforms the input data to a message.
3. The validation filter performs schema validation using XSD.
4. The message is enriched with additional data by calling service APIs.
5. Business rules are checked.
6. Domain entities associated with the message are persisted to the DB.
7. The routing component chooses what outbound adapter to use and puts the message to a queue.
8. The outbound adapter transforms and sends the message to an external system.

Every component logs granular events with an execution context to provide visibility and audit of processing logic. If errors appear, the support team can update the context and restart the specific step.

Apache Camel and OSGi/Karaf were a popular choice for implementing EIP based architectures. Some vendors offer packaged enterprise products combining Karaf, Camel, message broker and UI to design and manage integrational flows. Promoting easy development with UI tools and backed by enterprise-grade support, such products are not very popular with developers.

Practical problems with Camel/Karaf and similar packaged products:

1. As requirements grow, process definitions become impossible to edit via UI.
2. Developing Camel components (`routes`) with XML (`configuration over coding` mantra) is slow and tedious because of verbosity of XML. Such ‘code’ is so hard to read and understand that developers tend to create many java utility classes just to avoid dealing with XML.
3. There are no declarative component contracts:
   a. EIP diagram and Camel code do not define parameters expected and results produced.
   b. Camel routes are configured using URI syntax: parameters are encoded within query.
4. Interactive debugging support is very poor, often replaced by excessive logging.
5. Karaf OSGi is heavy and has issues with re-deployments, slowing down development and leading to all-or-nothing deployments to production.
6. CI CD is inefficient: building all components takes significant time.

Being functionally correct, Karaf and Camel implementations fall behind on non-functional requirements: extensibility, cost efficiency, maintainability, and scalability. Nowadays companies utilize cloud offerings to cut operational costs and deliver new products faster.

Let’s see how AWS, event-driven architecture and microservices can be combined to implement the same processing logic.

There are several types of components:

1. Inbound service: exposing HTTP API for external systems to start processing the flow with the initial context;
2. Step Functions: executing state machine, keeping the context, and posting commands to task queues (SQS);
3. Task queues, one per handler;
4. Task handler:
   a. performing a job to process a step;
   b. posting audit events;
   c. responding back to Step Functions with the task token and updated context upon completion;
5. Audit handler: persisting audit events to the DB.

If the processing task fails, it posts a failure event with the context and the task token for the audit handler for the event to be logged to the DB. Later, the support team can re-start the processing step passing the updated context to HTTP API exposed by the handler. If the task handler crashes, the command will not be acknowledged by SQS and re-processed next time when the handler is operational again. Should a completed step be re-executed, a new workflow instance is to be created, potentially having a choice at the beginning to re-direct processing to that step.

Inbound service and task handlers can be implemented as Spring Boot apps deployed to the EKS cluster. As an intermediate stage, legacy camel solution can be decomposed and re-packaged as task handlers. However, efforts dealing with legacy code might be bigger than re-implementing everything with modern java following clean code and BDD approaches. The choice of Java and EKS comes as constraints: to be as much cloud provider agnostic as possible and utilize existing team knowledge.

To summarize, enterprise integration patterns are still actual for designing but their implementations are outdated, while AWS and event-driven architecture can be used to migrate existing Camel/Karaf solutions to the cloud.