jump to navigation

in-memory databases March 26, 2012

Posted by Mich Talebzadeh in In-Memory databases, Uncategorized.

An in-memory database or IMDB could be a standalone database management system (DBMS) like Oracle’s TimesTen or a specific database, part of DBMS like Sybase Adaptive Server Enterprise (ASE, aka Sybase classic) IMDB (ASE-IMDB).

These databases rely on computer memory for data storage. This is in contrast to the traditional database management systems that rely on disk storage for storing data, even if the database is using Solid Stated Devices (SSD). IMDBs are faster than disk optimized databases since the internal optimization algorithms are simpler and execute fewer CPU instructions. Accessing data in memory provides much faster response. In applications where response time is critical, such as telecommunications network equipment, defense and certain trading systems, IMDBs are often used. Because of the nature of IMDBs, these databases tend to use more memory than their disk resident database counterparts.

In general one can categorize IMDBs as in-process and out-of-process main-memory databases. Out of process database management systems are usually, yet not necessarily, high functionality systems that implement full SQL, possibly with some dialects, security, database administration, etc. Specifically the administration is provided via a separate application (these days commonly running in a browser). These servers never provide access to data other than via SQL. Servers are not restricted in their memory and other system resources usage. Server’s footprint is usually rather high (in order of several megabytes). Servers such as ASE-IMDB and Oracle’s Times Ten can mimic the functionality of their “disk resident big brothers”. That makes it easier to use these products for caching SQL requests to a SQL back-end that maintains persistent databases. From a performance standpoint it takes time to route database requests either through some sort of an IPC mechanism (for local access), or through the network. This basically means:

  •  A lot of “out-of-the-box” functionality
  • No  need to write applications to access the database

In contrast, in-process database management systems are implemented as libraries that applications link with. This technique is sometimes referred to as “embedded database”. In-process databases can provide a native-language API, such as C/C++, Java, C#, etc., in addition (or even instead of the SQL API). In many cases, these native APIs can be more efficient than the SQL API, yet they require knowledge of the database layout and internals. IMDBs such as eXtremeDB from McObject provide API that facilitate database control functionality – system tables’ access, etc., as well as often integrate http server functions that allow applications to easily access the database through a browser. The scope of SQL is less than that found in out-of-process IMDBs, specifically security is never implemented. Most libraries carry much less overhead as they use application’s memory pools and other resources. Libraries footprint is usually much lower than those of out-of-process IMDBS, usually in order of hundreds kilobytes or less. This usually means:

  •  Less  “out-of-the box” SQL functionality, applications must be written to access      the database, compensated by providing application’s native language API
  • Higher  performance
  • Lower   footprint

Both TimesTen and ASE-IMDB and almost all current commercial IMDBs are based on what I call row-based storage implementation (RBSI) of relational model. These products are extremely effective for online transaction processing (OLTP) type applications and in majority cases that is what they deal with. This was the era of the transactional base databases in which the user — for example, a trader – was only interested in his/her own portfolio. Adding a few trades and reading them with some aggregates was fine and within the capability of these systems. Things have moved on since. Today’s systems and users deal much more with non-transactional (read) activity than with transactional (write/update) activity. An average trader today is not only interested in his/her portfolio but also interested in other portfolios and analytics, which requires sifting through millions of lines of records.

To this end, there has been another physical implementation of relational model for some time. In this physical model, data is stored in columns. This is referred to as Column Based Storage Implementation (CBSI) of Relational Model. This offering is used for analytics and data warehousing. Sybase IQ is an example of CBSI of relational model so is Lucid which is an open-source RDBMS for Analytics.

So how about another solution where you have columnar databases in memory? SAP who recently acquired Sybase Inc has already delivered columnar in-memory database technology to market via its Business Warehouse Accelerator hardware-based acceleration engine. So certainly IMDB is moving forward. Having said that, there appears to be a roadmap for Sybase IQ to be in-memory as well. Time will tell.


No comments yet — be the first.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: