from: http://elementallinks.typepad.com/bmichelson/2006/02/eventdriven_arc.html
Event-Driven Architecture Overview
[Notice February 6, 2011: To celebrate the 5th Anniversary of the Event-Driven Architecture Overview, I've re-issued the original content in a new report. Learn more here.]
Continuing my business-driven architecture series, I just published an Event-Driven Architecture Overview for the PSGroup Research Service. In the overview, I explain key event concepts, walk through event processing flows, and identify the major implementation components of an event-driven architecture. This post excerpts the core of the paper. If you would prefer the full report, go here (free with registration).
For those unfamiliar with business-driven architecture, I believe the most viable, agile architectures will be comprised of a blend of architecture strategies, including (but not limited to) service-oriented architecture, event-driven architecture, process-based architecture, federated information, enterprise integration and open source adoption. How you blend, depends on your business.
Before I jump into the excerpt, I want to touch on the relationship between service-oriented architecture and event-driven architecture.
I believe that SOA and EDA are peers and complements.
So, I disagree with the SOA evangelists who say EDA is merely a subset (child) of SOA.
On the SOA and EDA are complements point, I see two distinct interactions:
Event-Driven SOA. In the first interaction, the occurrence of an event (a notable thing that happens inside or outside your business) can trigger the invocation of one or many services. Those services may perform simple functions, or entire business processes. This interaction between events and services is commonly referred to as event-driven SOA.
Service as Event Generator. In the second interaction, a service may generate an event. The event may signify a problem or impending problem, an opportunity, a threshold, or a deviation. Upon generation, the event is immediately disseminated to all interested parties (human or automated). The interested parties evaluate the event, and optionally take action. The event-driven action may include the invocation of a service, the triggering of a business process, and/or further information publication/syndication. In this interaction, the service is purely one of many event sources in a broader event-driven architecture.
A broader event-driven architecture stretches beyond event-driven SOA, to include real-time information flow and analysis, and complex event processing.
And now, the excerpt: key concepts, event processing flows and implementation components.
EVENT-DRIVEN ARCHITECTURE BASICS
What is an Event?
An event is a notable thing that happens inside or outside your business. An event (business or system) may signify a problem or impending problem, an opportunity, a threshold, or a deviation.
Specification and Occurrence. The term event is often used interchangeably to refer to both the specification (definition) of the event, and each individual occurrence (instance) of the event.
Define in Business Terms. For an event to be meaningful to downstream subscribers (human and automated) it is imperative that the event (name and body) is specified in business terms, not data or application terms.
What is in an Event?
Each event occurrence has an event header and event body. The event header contains elements describing the event occurrence, such as the event specification ID, event type, event name, event timestamp, event occurrence number, and event creator. These elements are consistent, across event specifications.
Event Body. The event body describes what happened. For example, if a retailer specified a low inventory threshold event, the event body would contain the information to communicate which product fell below the allowable threshold.
The event body must be fully described so any interested party can use the information without having to go back to the source system. For the low inventory threshold event, the event body would contain not only the product identifier, but also the product description, and the point in time inventory and threshold levels. To ensure events are understood by all consumers, a clear business lexicon or ontology should be used.
What is an Event-Driven Architecture?
In an event-driven architecture, a notable thing happens inside or outside your business, which disseminates immediately to all interested parties (human or automated). The interested parties evaluate the event, and optionally take action. The event-driven action may include the invocation of a service, the triggering of a business process, and/or further information publication/syndication.
Extreme Loose Coupling. By its nature, an event-driven architecture is extremely loosely coupled, and highly distributed. The creator (source) of the event only knows the event transpired. The creator has no knowledge of the event’s subsequent processing, or the interested parties. The traceability of an event through a dynamic multipath event network can be difficult. Thus, event-driven architectures are best used for asynchronous flows of work and information.
Event Processing Styles
There are three general styles of event processing: simple, stream, and complex. The three styles are often used together in a mature event-driven architecture.
Simple Event Processing. In simple event processing, a notable event happens, initiating downstream action(s). Simple event processing is commonly used to drive the real-time flow of work—taking lag time and cost out of a business.
Stream Event Processing. In stream event processing, both ordinary and notable events happen. Ordinary events (orders, RFID transmissions) are both screened for notability and streamed to information subscribers. Stream event processing is commonly used to drive the real-time flow of information in and around the enterprise––enabling in-time decision making.
Complex Event Processing. Complex event processing (CEP) deals with evaluating a confluence of events and then taking action. The events (notable or ordinary) may cross event types and occur over a long period of time. The event correlation may be causal, temporal, or spatial. CEP requires the employment of sophisticated event interpreters, event-pattern definition and matching, and correlation techniques. CEP is commonly used to detect and respond to business anomalies, threats, and opportunities.
EVENT PROCESSING
In this section, I present example event flows for each style of event processing. The flows are represented logically, broken out into four layers. Before jumping into the event flows, I describe the logical layers.
Event Flow Layers
An event flow starts with the event being generated and culminates with the execution of any downstream (event-driven) activity. The four logical layers are as follows:
Event Generators. Every event is generated from a source. The source might be an application, data store, service, business process, transmitter, sensor, or collaboration tool (IM, email). An ordinary event may be evaluated for notability by an event preprocessor (router, filter), resulting in the generation of a new notable event.
Because of the variety of event generators, not all events will be generated in the required format for event processing. In those cases, the events need to be transformed to the required (enterprise standard) format prior to being deposited in the event channel. [a]
Event Channel. The event channel, typically a messaging backbone, transports standard formatted events between event generators, event processing engines, and downstream subscribers.
Event Processing. In the event processing layer, upon receipt, events are evaluated against event processing rules, and actions are initiated. The event processing rules and actions are defined in accordance to the needs of the interested parties, not of the event generators.
The actions include invoking a service, initiating a business process, publishing the event out to a subscription hub, directly notifying humans or systems, generating a new event, and/or capturing the event for historical purposes.
Events are processed by engines. A simple engine processes each event occurrence independently. A complex engine processes new event occurrences in context of prior and future events.
Downstream Event-Driven Activity. A single event, or event correlation, may initiate numerous downstream activities. The invocation of the activity might be a push by the event processing engine (service invocation, business process initiation, notification) or a pull by subscribers of event publications. Subscribers might be humans, applications, active business processes, data warehouses, performance dashboards, and/or automated agents. Events should be published in the standard event format. Transformation to subscriber-specific formats is typically done by an enterprise integration backbone. [b]
Example Event Flows
I am presenting simple examples to illustrate the concepts and capabilities of an event-driven architecture. The inherent flexibility at each layer—what events are generated, how they are processed, and who receives them—is the real power of event. As you review our examples, think of your own business and potential event flows that can streamline work and information delivery.
Please note, for the sake of completeness, my event flow illustrations (1-3) include more actions in the processing layer, and more subscriber types than I reference in the examples. The illustrations flow from left to right, starting with event generators.
Illustration 1 - Simple Event Flow [click ON picture to enlage]
Simple Event Processing. In Illustration 1, I show a simple event processing flow for an online bookseller’s inventory position optimization. When a customer places an order for books, the bookseller immediately commits inventory via a commit inventory service. The commit inventory service shifts inventory from available to reserved, and then checks the remaining available inventory against the optimal inventory thresholds. If the stock falls below the inventory threshold, the commit inventory service generates a Low Inventory Threshold event. This event is shown in Illustration 1 as Event Q, occurrence 27.
The Low Inventory Threshold event is deposited into the event channel and received by the simple event processing engine. The processing rules for this event type (Low Inventory Threshold) dictate two actions. A re-order inventory process is initiated, and the event is published for subscription. The subscribers are the inventory buyer, and the inventory manager’s performance dashboard. The re-order inventory process could be a straight-through process, or it could require human review and approval.
Illustration 2 - Stream Event Flow [click ON picture to enlage]
Stream Event Processing. In Illustration 2, I show three stream event processing flows for a multichannel consumer electronics retailer. In the first flow, top left hand corner of Illustration 2, an RFID sensor is emitting events every time a product leaves the warehouse. The consumer electronics retailer wants to be informed when high-end products leave the warehouse. To meet this requirement, a local event filter has rules to filter out events for items priced less than $4,000. One event, Event A, occurrence 2, is for a $5,000 plasma TV. This event is reformatted to a standard event format, and placed in the event channel. [c] The simple event processing engine receives the event, and following the rules for high-end products leaving the warehouse, publishes the event. The event is subscribed to by the inventory manager’s performance dashboard.
The second and third flows originate from the order entry application shown at the bottom left hand corner of Illustration 2. For every customer order placed, an ordinary Order Placed event is generated. All ordinary Order Placed events are streamed to a data warehouse, through the event channel, and the publication action of the simple event engine. These events are shown in Illustration 2 as Event B, occurrences 1 and 2.
For high value customer orders, an order totaling more than $1,500, the consumer electronics retailer wants to immediately reward the customer with a promotion in status. To meet this requirement, a local event router evaluates the order total of each ordinary Order Placed event, and generates a new notable High Value Order event when the $1,500 condition is met. This event is shown in Illustration 2 as Event C, occurrence 1.
The local router is employed because the burden of event routing or redistribution is on the event processing system, not the source application. The source of the event should only have to say once “this thing happened,” regardless of how the event is used in the enterprise.
Once the High Value Order event is generated, it is deposited into the event channel, and processed by the simple event engine. The action associated with the processing rules for this event is to invoke the Evaluate Customer Level service. This flow is also an example of event-driven SOA.
Illustration 3 - Complex Event Flow [click ON picture to enlage]
Complex Event Processing. In Illustration 3, I show three complex event processing flows for the same multichannel consumer electronics retailer. In the first flow, top left hand corner of Illustration 3, a business-to-business order gateway is supposed to be emitting System Heartbeat events every 15 minutes. The System Heartbeat events inform IT operations the gateway is up and running. The absence of a heartbeat event indicates a failure. If the order gateway is down, business customers are likely to place an order with a competitor.
The complex event engine tracks the timestamp of the last received System Heartbeat event. If 15 minutes have elapsed, the event processing actions associated with non-arrival are initiated. In this case, the order gateway IT guy is immediately notified (paged) and a new B2B Order Gateway Failure event is generated.
This new B2B Order Gateway event is deposited into the event channel. When the B2B Order Gateway Failure event is received by the simple event engine, a publishing action is performed. The enterprise problem management system subscribes to this event for resolution tracking.
The second and third flows show two variations of fraud detection. Both flows originate with the point-of-sale application shown at the bottom left hand corner of Illustration 3. Similar to the customer order example above, for every store sale, an ordinary Store Sale event is generated (Event Y, occurrences 1-3). These events are evaluated by a local event router, producing High Value Store Sale events (Event Z, occurrence 1) for transactions more than $1,500. All of the Store Sale events, ordinary and notable (high value), are deposited in the event channel.
In the first fraud detection flow, the complex event engine checks for multiple transactions (ordinary store sale events) by the same customer (credit card), over a short amount of time (10 minutes), at different locations over a large distance (20 miles). If these conditions are met, the place account in fraudulent status service is invoked.
In the second fraud detection flow, on receipt of a High Value Store Sale event, the complex event engine does an inquiry on the customer’s past purchases to determine if this purchase should be marked suspicious. [d] If the current purchase amount deviates more than 50 percent from largest historical purchase, the event (Z, occurrence 1) is published as suspicious. A customer advocate team subscribes to these events and places calls to registered cardholders.
As you can see from my examples, the possibilities are almost limitless. I find it particularly powerful to combine services and business processes with events. The services and business processes can be event generators, event-driven actions, or both.
EVENT-DRIVEN ARCHITECTURE IMPLEMENTATION COMPONENTS
Illustration 4 - EDA Implementation Components [click ON picture to enlage]
In the event basics and event processing sections, I surfaced many of the implementation components required for an event-driven architecture. Illustration 4 shows these components in a layered architecture manner. The components can be broken out in five categories:
• Event Metadata. A good event-driven architecture has a strong metadata architecture. Event metadata includes event specifications and event processing rules. Event specifications must be made available to event generators, event format transformers, event processing engines, and subscribers. While no standards currently exist for event definition and processing notation, it is only a matter of time.
• Event Processing. The cores of event processing are the engine and the event occurrence data. Simple event engines are often homegrown. Complex event engines should be acquired from a CEP engine provider. [e] Event occurrence data is typically persisted and retained for audit and trend analysis.
• Event Tooling. Event development tools are required to define event specifications and processing rules, and to manage subscriptions. Event management tools provide administration and monitoring of the event processing infrastructure, monitoring of event flows, and visibility into event generation and processing statistics.
• Enterprise Integration. An enterprise integration backbone plays a large role in event-driven architecture. Some of the required integration services are: event preprocessing (filters, routes, and transformations), event channel transport, service invocation, business process invocation, publication and subscription, and enterprise information access.
• Sources and Targets. These are the resources of the enterprise—applications, services, business processes, data stores, people, and automated agents—that generate events and/or perform an event-driven action.
The topology configurations for these components will vary for each enterprise, based on the event flows, event occurrence volumes, enterprise integration backbone, and location/distribution of sources and targets.
[a] To date, no industry standards exist for business event definition or notation. IBM has defined a common base event for systems events. The common base event is included in the Web Services Distributed Management (WSDM) specification. The common base event format evolved from IBM’s Tivoli (systems management) product line.
[b] For human event subscribers, consider the use of Really Simple Syndication (RSS).
[c] It is an architectural decision if the local event filter should be aware of the proprietary RFID format, or if all RFID events should be transformed to a standard format (event formatter), and then evaluated by a local event filter.
[d] To facilitate the enterprise database query (customer order history), complex event processing systems have the capability to cache relevant enterprise information.
[e] Some of the providers offering event processing engines are: AptSoft, IBM, KnowNow, Progress Software (parent of Sonic), StreamBase, and Tibco.
Posted by brenda michelson on February 06, 2006 at 06:06 PM in business-driven architecture, event driven architecture / event processing, soa | PermalinkShareThisShareThis
TrackBack
TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c2a6553ef00d8345c1d3769e2
Listed below are links to weblogs that reference Event-Driven Architecture Overview:
» traceability in event driven architectures from Controlled Agility
Brenda Michelson writes a long post on event driven architecture EDA. I've been a supporter of EDA for a while now and have seen it deployed badly and well. One thing I'd like to draw attention to is extreme loose [Read More]
Tracked on February 07, 2006 at 05:45 PM
» Got events, SOA? You need business rules from Enterprise Decision Management - a Weblog
In Time for event-driven SOA Joe McEndrick summarizes a post by Brenda Michelson It seems to me, reading these, that a business rules approach to automating operational business decisions would be a very valuable element in an event-driven SOA. Joe [Read More]
Tracked on February 21, 2006 at 02:05 PM
» Well defined interfaces from Todd Biske's Blog
On an SOA mailing list I follow, someone recently posted a very simple question. What is a well-defined interface? The notion of a well-defined interface is frequently cited as a requirement for a good service, but the discussion usually heads in the... [Read More]
Tracked on February 25, 2006 at 12:31 AM
» Event-Driven Architecture Overview from dougmcclure.net
A great read on events and application of them within an Event-Driven Architecture by Brenda Michelson in her Elemental Links blog. I agree with everything shes laid out in her posting. Events, complex event processing (CEP) and the associated ... [Read More]
Tracked on March 24, 2006 at 08:05 AM
» Stop the Madness from IT Blog
Bei Gartner hat man sich auf die durchaus sinnvolle Verbindung zwischen SOA und EDA besonnen. Ereignis Orientierte Architekturen und Service Orientierte Systeme ergänzen sich in der Praxis. Ereignis Orientierung wird besonders in den Bereichen in denen es [Read More]
Tracked on June 21, 2006 at 02:49 AM
» SOA Trough of Disillusionment: Sinkhole or Poser Filter? from elemental links
It looks like we are squarely, predictably, and self-fulfillingly in the SOA Trough of Disillusionment. Analysts and press are (enthusiastically) ringing the death knell for SOA as they redirect their short attention spans to the next silver bullet/vic... [Read More]
Tracked on August 26, 2007 at 03:12 PM
» Moving the SOA Goalposts from Loosely Coupled Thinking
SOA is about loosely coupled system integration . Sorry, I meant to say that SOA is about [Read More]
Tracked on July 01, 2008 at 04:14 PM
Comments
Hey Brenda!
Thanks for the post on EDA, I was looking for some information on EDA over last couple of days. Very much agree with u on the fact that EDA is not a subset of SOA, but EDA + SOA together makes a good business driver for Event SOA.
Posted by: Vivek Kondur | February 09, 2006 at 11:54 PM
Vivek - thanks for the feedback and email exchange. good luck on your project. -brenda
Posted by: brenda michelson | February 10, 2006 at 06:02 PM
Hi Brenda,
Great article, and a great foundation for discussing event-driven applications! I'm just curious if you've ever heard of Coral8, a state-of-the art SQL-based ESP/CEP engine based on research from Stanford University? I would be very happy to give you a demonstration and explain how Coral8 customers solve many of the problems you talk about in your article.
Posted by: Mark Tsimelzon | February 14, 2006 at 01:49 AM
Hi Mark - I haven't heard of Coral8, but would definitely be interested in learning about it. Let's set something up. - brenda
Posted by: brenda michelson | February 14, 2006 at 06:56 AM
Thanks! Brenda.
Mark, few days ago I happened to stumble upon Standford's research on CEP, but didn't know that Coral8 was an implementation based on the research done @ Stanford. Thanks for the info, will sure check it out.
Posted by: Vivek Kondur | February 15, 2006 at 07:08 AM
Vivek -- Just to clarify things a little, as it may be confusing:
There are two teams working on similar problems at Stanford, although their approaches are very different:
One is the CEP team headed by Prof. David Luckham (http://pavg.stanford.edu/cep/). They approach the problem from the Complex Event Processing angle, and they developed RAPIDE.
Another is the Project STREAM team (http://www-db.stanford.edu/stream/), headed by Prof. Rajeev Motwani (our Chief Scientist) and Jennifer Widom. They look at the problem from the Event Stream Processing angle, and they developed an SQL-like language CQL (Continuous Query Language).
Now, Coral8's CCL (Continuous Computation Language) is more similar to CQL, but it also includes some elements that are closer to RAPIDE, such us the event pattern matching syntax.
Note that Coral8 did a lot of original research in this area, and the entire code base was developed from scratch, to make sure it satisfies the rigorous demands of the enterprise customers.
I hope this helps :)
Posted by: Mark Tsimelzon | February 17, 2006 at 10:33 AM
Nice summary - I think business rules can help in CEP and EDA in both deciding what to do and managing the processing as services.
http://edmblog.fairisaac.com/weblog/2006/02/got_events_soa_.html
Posted by: James Taylor | February 21, 2006 at 02:07 PM
James- I agree, business rules plays a big part in 'event'...at event generation, pre-processing, within the engines, and in some cases, even in downstream processing. -brenda
Posted by: brenda michelson | February 21, 2006 at 07:39 PM
Hi! Mark,
Thanks for clarification & the links, I had actually gone through the CEP research lead by Prof. David Luckham and missed the STREAM team's research, will check it out.
Posted by: Vivek Kondur | February 26, 2006 at 09:48 AM
There is an open-source CEP/ESP Engine called Esper at http://www.sourceforge.net, totally free.
Posted by: Thomas Bernhardt | February 27, 2006 at 09:55 PM
I think it is not SOA, but EDA that will appeal to the business... See the article below:
http://soa-eda.blogspot.com/2006/06/soa-doesnt-add-business-value-but-eda.html
Posted by: Jack van Hoof | June 17, 2006 at 05:33 AM
On target and a great article. We (CA) have been improving our EDA within our Unicenter NSM architecture to align IT Services with business processes for customers. Again, great article. It validates our work.
Posted by: Ishmael Thomas | June 22, 2006 at 08:05 PM
Agreed with all of the above that Brenda has done a very nice job here. The only slight issue I have is that the distinction among simple event processing, event stream processing, and complex event processing is still a very much debated and confusing topic, and there's little agreement in the industry about each of these terms.
But I think you really nailed the relationship between SOA and EDA. In fact, I think this issue deserves deeper exploration, which is why we have picked up on the EDA versus SOA debate, the notion of "advanced SOA," or "SOA 2.0" and have been writing about it in the Apama event processing blog. For example, http://apama.typepad.com/my_weblog/10_myths_of_eda_and_soa/index.html> Check out the "10 Myths of EDA and SOA" here
Hopefully expansion on this post and the ongoing debate will continue as EDA and event processing expands.
- Mark Palmer, General Manager, Progress Apama
Posted by: Mark Palmer | March 17, 2007 at 12:45 AM
I have been struggling with this issue for quite some time now. I appreciated the clarity of the article in defining EDA.
As an aside, I was also reading another article the Vanilla Layer Cake Theory.(http://www.ebizq.net/hot_topics/soa/features/6444.html)
I was curious what your thoughts are about getting those packaged apps plugged into this notion of EDA.
Mark O'Neal - Enterprise Architect
Tarrant County, Texas
Posted by: Mark O'eal | April 14, 2007 at 11:40 AM
Dear Brenda,
I believe when you wrote "casual," you meant "causal." I noticed the same typo on Wikipedia and in one of your whitepapers. I find your insights into SOA and EDA very refreshing; please keep up the good work!
Sincerely,
Joel
Joel Lindheimer Ph.D.
Lead Architect of Corporate R&D
Click Commerce Inc.
Posted by: Joel Lindheimer | October 19, 2007 at 06:38 PM
Joel, thanks for pointing out the typo. While I enjoy "casual" events, I did in fact mean "causal". It's fixed now (here). Thanks again, -brenda
Posted by: brenda | October 20, 2007 at 04:35 PM
Hi Brenda,
SOA is simply an outgrowth and stretching of object-oriented approach in the direction of services/contracts (invoking a method or call-back processing due to an event occurrence). I believe it's simply a hyped name of EDA. SOA is simply EDA.
Ilyas,
Application/Physical DB2 DBA
Posted by: Ilyas Shaikh | December 05, 2007 at 12:49 PM
Ilyas,
Could you please illuminate us more on how SOA is simply EDA?.
Posted by: Queen Latifah | February 19, 2008 at 02:52 PM
Hi Brenda,
Your post from February 06, 2006 was one of the main focus points for me while building MashupFactory. Its a Platform as a service - IT LifeCycle Management. Your view of holistic IT thinking played an importent role as well. check it out at http://mashupfactory.wordpress.com/ and let me know what you think.
Regards.
Yoram
Posted by: Yoram Kochol | February 20, 2008 at 04:28 AM
Brenda, I wrote a piece a few months ago entitled "SOA is from Mars and EDA is from Venus". While somewhat tongue in cheek, it also suggest the need for both kinds of thinking. If we think of SOA as largely being request response (even if the invocation channel is asynchronous) and EDA as being largely "Notify and forget" (rather a broad generalization, I agree then we see huge benefits to EDA thinking in cases where the notifier of the event doesn't have any idea what downstream or post event things are interesting. For instance if we see a major purchase from a vendor and that vendor is also a customer, the sales organization may want to use that large purchase as leverage in the sales process. The "purchasing system" doesn't have a clue that purchases might be used like that - and nor should it.
I do believe we should really think of using an eventing style of communication/invocation as our default position, an only revert to request/response when we absolutely have to - for transaction integrity, legal purposes, company policy needs, etc. There is more complexity in the event based approach, but it is many ways more natural. Many cross business functions are handled asynchronously. We have all heard, "Leave that with me, I'll look into it and get back to you." A very natural human approach that allows us to disengage from the current task and not be blocked interminably from doing other things
Posted by: Chris Bird | May 22, 2009 at 09:18 PM