Over the past few months at Lab49, we’ve thrown ourselves into complex event processing (CEP) — aka event stream processing (ESP) — and have been formulating exactly how and when it fits into the larger, more comprehensive technology stack found in global financial services institutions. We’ve formed a number of interesting vendor partnerships, attended product training, sampled, compared, and teased apart many of the popular products, and we’ve created several CEP-based demo applications that have been shown at recent events like SIFMA.
Along the way, we’ve all learned a lot about CEP, and the more I learn, the more I dig it. The more I put CEP into practice, the more I foresee its ultimate dominance as an architectural design pattern for everyday development.
What’s fascinating to me about CEP isn’t that it’s a new idea, despite how it may be touted by vendors. Regardless of the hype, CEP isn’t the most revolutionary technology you’ve never heard of. What’s fascinating is that out from a decades-old, primordial soup of ideas, research, and trial-and-error that, in and beyond academia, has been trying to create architectural models around complex data problems with real-time constraints, enough best practices and design patterns have emerged to evolve an ecosystem of market entrants, seemingly all at once.
It’s not the first time that a bundle of quality design patterns took concrete form as a technology. Object pooling, lifetime management, transaction enlistment, and crash domains begat COM+, Microsoft Component Services, and J2EE application servers. Logging levels, external configuration, adaptable logging sinks begat log4j, syslogd, and the Logging Application Block from the Microsoft Enterprise Library. Unit-testing and test-driven development begat JUnit and its children.
These transformations have been crucial. Once developers accepted these patterns and solutions as sufficiently solved and commoditized, they were saved considerable time and attention. Freed of coding logging libraries and unit testing frameworks for the umpteenth time, developers could focus more on the business problem being solved rather than the infrastructure details required to solve it.
But these transformations didn’t really upset the gross architecture of applications. They may have changed some of the design decisions and simplified the implementations, but they didn’t fundamentally change the abstraction you would use to model a problem and architect a solution.
CEP, on the other hand, does.
Instead of storing and indexing miles of cumulative data in a persistent store to service complex queries in batch/polling fashion, CEP inverts the whole shebang, storing and indexing the complex queries before streaming data across queries without storing a lick. The transformation of a business problem from tables, rows, and polling intervals into events, filters, triggers, and real-time reactions is not only quite enabling, it changes the very way you think about how business problems can be solved and which problems may have viable solutions.
Over the next few weeks, I’ll delve a bit more into CEP and how it relates to technologies you might be more familiar with. In the meantime, check out some of the in-depth blog entries other folks from Lab49 have been writing about CEP.
