Adopting a Modular Monolith with Event-Driven Architecture to Handle Time-Consuming APIs
Handling time-consuming APIs in a monolithic architecture can be a significant challenge, often leading to server timeouts and poor user experiences. This article explores how we addressed this issue by adopting a modular monolith architecture combined with AWS Lambda and an event-driven design. While this approach solved key challenges, it also introduced complexities that are worth considering.
Problem: Time-Consuming API Calls Causing Server Timeouts
Our monolithic application encountered frequent timeout errors due to a super time-consuming external API. These errors disrupted client experiences and consumed server resources inefficiently. The server had to wait for the API response, leaving clients without timely feedback and leading to resource contention.
Solution: Modular Monolith with Event-Driven Architecture
To address this issue, we redesigned the architecture by:
Decoupling the API Calls:
・Offloading the time-consuming API interactions to AWS Lambda, which handles the API calls asynchronously.
Event-Driven Processing:
- Once the client sends a request, the server immediately responds with
200 OK. - The request is then sent to AWS Lambda for processing.
- After the external API responds, the result is sent back to the client via an event-driven workflow.
WebSocket Connection:
・A persistent WebSocket connection ensures real-time updates to the client, keeping them informed as the workflow progresses.
System Design Overview
Here’s a simplified diagram of our system:
[Client] → [Monolith Server] → [AWS Lambda] → [External API]
↑ ↓
↳ [WebSocket Connection] ← [Event Trigger]- Client: Sends requests and receives immediate feedback via
200 OK. Updates are pushed through WebSocket. - Monolith Server: Acts as the orchestrator, handling client requests and initiating the event-driven workflow.
- AWS Lambda: Processes the API calls asynchronously and triggers events upon completion.
- Event Trigger: Publishes the API response as an event, which is consumed by the WebSocket handler.
- WebSocket: Pushes real-time updates back to the client.
Benefits of the New Approach
1. Eliminating Timeout Errors
- The server no longer waits for the API response, avoiding timeout errors and improving reliability.
2. Improved User Experience
- Clients receive an immediate acknowledgment of their request and real-time updates, enhancing interactivity.
3. Scalable Processing
- AWS Lambda’s serverless architecture allows for automatic scaling, handling varying workloads efficiently.
Challenges and Downsides
1. Integration Testing Complexity
- Testing the system end-to-end is more challenging due to the decoupled nature of the components.
- Simulating events, WebSocket updates, and Lambda behaviors requires extensive mocking and infrastructure.
2. Increased Codebase Size
- The introduction of event-driven workflows, additional modules, and WebSocket handling has led to a significantly larger codebase.
- This complexity increases development time and requires more robust documentation and team collaboration.
3. Debugging and Monitoring
- Debugging issues across Lambda, the event bus, and WebSocket layers requires advanced logging and monitoring tools.
Key Takeaways
While adopting a modular monolith and event-driven architecture helped us resolve critical issues like timeout errors and poor user experience, it also introduced challenges in terms of testing and complexity. Organizations considering a similar approach should:
- Invest in testing frameworks to handle asynchronous workflows.
- Adopt monitoring tools to track events and API interactions.
- Document the architecture thoroughly to ensure maintainability.
This solution demonstrates that even within a monolithic structure, modern architectural patterns like event-driven design can significantly enhance system reliability and user satisfaction.
