jump to navigation

CEP and IMDB Contrasts March 25, 2012

Posted by Mich Talebzadeh in Complex Event Processing.

Fundamentally products like classic databases (DBMS), in-memory database (IMDB) or Complex Event Processing (CEP engines) strive or work towards *maximizing throughput and minimising latency*.

That is probably easier said than done.

A DBMS including IMDB is charged with three basic tasks. It must be able to *store data*, keep that data, and take the data out and work with it. The keyword is storage of data which involves a structure requiring physical and logical IO. In case of IMDB things are simplified (less complex indexes like Oracle TimesTen or simply in case of ASE IMDB, be content with Physical IO cost for table access to be zero). However, the nature of the beast has not changed. Data has to be stored whether on disk or in memory and has to be persistent. Access to data will require as minimum Logical IO.

In contrast, Complex Event Processing engines like Sybase ALERI, Oracle CEP or Streambase CEP are *Event Driven and memory based*. There is no requirement for either physical or logical IO. In such scenarios data from various sources is streamed in like a stream/river so to speak. You build logic to alert you if an event is happening (an event as described before). Think about it as fishing for Salmon. You may see alert for Salmon that could turn up to be carp and choose to ignore it. CEP do not have any persistent database. They are not meant to store data (hence event driven/on the spot decision). They can have adapters to flush out memory data into something like Sybase IQ as Sybase Aleri can do. As an analogy, in UK most anklers have a  bucket that keep fish in it for a short while. They catch the fish, keep it in the bucket, take few pictures and release the fish back to the stream (carp normally, you do not get Salmon in most British rivers except Scotland) . So the bucket could be the database but is not part of the process and you do not need it.

In trading when you have streams of data coming in like in algorithmic trading into CEP with rules defined, the quotes or ticks are events to themselves. Think of it as correlation of two instruments like Exxon and Shell. They are both oil related and share quotes/ratios are strongly correlated. Any significant divergence in the ratio say outside of Bollinger bands (Simple moving averages (SMA) outside of SMA + 2σ) may trigger buy or sell condition for one or other. In these scenarios one acts on alerts and may get historical data from IQ to help Situation Assessment.


No comments yet — be the first.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: