Migrating Apps to the Cloud — Shunting the Event Stream
In another blog post I wrote about modern application development, I spent a lot of time talking about orchestration vs. choreography, and how message-driven applications can be architected and written to work at massive scale. In that post, I discussed aging legacy architectures based on the “dump and poll” method — doing work, dropping a file in some location, and hoping some downstream system will notice and continue the chain of work while tightly coupled to specific communication technologies.
What about applications that are already implementing message-driven architectures? In some cases, legacy applications are already designed around the concept of an event or message stream. Countless enterprises operate core parts of their business on this model, ensuring that technologies like JMS, WebSphere, Tibco, and MSMQ are used well beyond their practical expiration dates.
Let’s take a look at a diagram that represents a pretty common scenario in an enterprise. We’ve got a number of monoliths, all dangling from a central messaging system. Messages come in from other departments or lines of business, and we process these messages, make a bunch of calls to other services within the organization, and produce output messages.
Let’s assume that this diagram belongs to a bank. All these monoliths are performing work and flowing financial transactions through the system. Now you’ve decided that you’re eager to get rid of your aging monoliths, and you’re planning on adopting microservices architecture. You want to deploy these new services to a PaaS (Platform as a Service), aka “the cloud”.
If you just had a monolith that exposed RESTful APIs, you might just “strangle” that and pinch off pieces at the seams (bounded contexts) and create a bunch of smaller REST services, then refactor the edges to talk to the newer, smaller APIs. Unfortunately, we don’t always have the leeway to refactor the edges or modify downstream consumers.
Pinching off at the seams doesn’t quite work as well in a situation where you have a monolith listening for messages and responsible for emitting certain types of messages. Additionally, let’s say you’ve got a requirement where you can’t mess with the other monoliths in the event stream just yet, so anything you do has to be completely seamless. Some of you might be shaking your heads, but such a requirement is pretty common, and often becomes more common the older the systems are. Older systems gather their own “fear barnacles” — the older it is, the less likely anyone in the organization understands the code or has access to the original requirements. Therefore, people are simply scared to mess with it. It’s making money, it isn’t broken, so leave it alone.
The solution to this particular problem is what I like to call shunting the event stream. As you might guess, it starts with the creation of a shunt. Depending on the type of technology you’re using to support your event/message stream, you probably have access to “bridges”, which are really just blobs of code that listen for certain types of messages, and forward them on to a different messaging infrastructure. If you can’t get them off-the-shelf, they’re usually pretty easy to write.
Let’s say we’re planning on taking monolith “B” out of the legacy banking event stream diagram above. We want to modernize this, and we’ve decided that it can be refactored as 4 different microservices. In order to maintain the legacy messaging contract, we’re going to insert a shunt in the event stream where the old monolith used to be. This shunt will lead to a new messaging pipeline, as diagrammed below.
In this new architecture, microservices B1-B4 are all communicating using new events, and might be embracing modern concepts like eventual consistency, event sourcing, and CQRS. There is nothing about the messages used internally within the new “B1-B4 ecosystem” that is tightly coupled to the legacy system.
The inbound portion of the shunt in the diagram is responsible for listening for the appropriate message(s) emitted by B1-B4. When it sees the right message, it does the necessary conversion and then emits legacy messages that conform to the old legacy messaging contract. The key is that the existing monoliths can’t tell the difference between the old “B” monolith and the new shunt.
In this way, we can modernize individual monoliths without having a “stop the world” moment to take everything down or perform a massive system-wide rewrite which will take too long and cost too much money.
As we do this shunting multiple times, we go back and remove the shunts that are no longer necessary, like picking up traffic diversion cones when the construction is done, only we don’t have to pay the penalty of reducing the number of open lanes.
As with all architectures, there are an infinite number of solutions for any given problem. Shunting the event stream is like a macro version of a facade pattern used for extracting microservices from a monolithic codebase and has worked extremely well on numerous occasions.
Hopefully this has provided some inspiration for how you might be able to tackle a similar problem.
