Requirements

• Inifinite Scalability
• High Availability (0% downtime)
• Data Redundancy
• High Performance
• Storage Flexibility – ability to store any type of data
• Query Flexibility – ability to perform simple gets, range based gets, range based updates, and possibly complex joins

Features that a solution must have to deliver the above Requirements

• Replication – Each unit of data should be copied to multiple nodes so that if an underlying node crashes there is no data loss
• Partitioning – Data should be divided across multiple nodes based on specific keys so that the data layer is infinitely scalable
• Online node addition – Solution should support adding new nodes online, with automated data distribution upon new node addition
• Load Balancing – Queries should be load balanced between nodes
• Persistence – Should have a data persistence layer so that data is not volatile
• Caching – Should support flexible in-memory caching for increased retrieval speeds
• Tree based Indexing – To support range based queries ideally one may need tree type indexes for keys on which range queries maybe made

The holy grail of scalability is to be able to scale your data store. And as data stores go, RDBMSes seem to be the predominant choice (though that is changing – refer http://bit.ly/2lnRet). RDBMSes by their very nature, due to the features they provide (ACID compliance, Transaction safety etc) tend to be difficult to easily scale. This has resulted in the recent mushrooming of data storage options that are feature-poor but scalable out of the box (eg Voldemort, HyperTable etc)

I wanted to chronicle the list of features that a standard RDBMS provides, that we take for granted, so that I have a reference of the features that one may have to compromise on w.r.t application development in favor of easier scalability -

• Range based selects and updates – Being able to fire queries on a table specifying a range of values (eg where age >35). Typically RDBMSes use B+ Tree based indexes which support range based row selection. This in turn allows one to fire range based queries.
• Transactions – In an RDBMS one can perform multiple operations within a transaction and ensure that all of them or none of them go through. This ensures data integrity
• B+ tree indexes
• Foreign key relationships and referential integrity
• Joins and nested selects
• Aggregations (sum, avg etc)
• Advanced scripting using non-native languages (java etc)
• Stored procedures (allow encapsulation of business logic in the database layer)
• Triggers

]]> http://bhavin.directi.com/infinitely-scalable-infrastructure-and-rdbmses/feed/ 1 http://bhavin.directi.com/column-oriented-dbms/ http://bhavin.directi.com/column-oriented-dbms/#comments Wed, 22 Jul 2009 08:41:19 +0000 Bhavin Turakhia http://bhavin.directi.com/?p=174 Conventionally we take DBMS for granted as a structured data store that stores data in the form of rows. Infact most application developers can begin visualizing their data as rows in an RDBMS quite naturally.

While RDBMS serve the purposes of OLTP applications well, OLAP / data anlytics type applications have conventionally not been able to obtain the type of performance needed from RDBMses. This is where column oriented DBMS can help.

In the simplest form the difference between a conventional RDBMS and a column oriented database is that the latter stores data in a column form rather than a row form when persisted to disk. Another way to look at this is that the storage in a column oriented DBMS transposes the rows and columns of the storage in a conventional RDBMS.

For eg

This would be persisted in a conventional RDBMS as follows -

1,Bhavin,29|2,Roger,30

In a column oriented DBMS this would be persisted as -

1,2|Bhavin,Roger|29,30

It is common knowledge that the slowest piece of a DB query is its disk seek time. While the RDBMS favors queries which require fetching all data of a given row, the latter model favors queries which require aggregates. For instance – count of all users with age >20, or sum of ages of all users, and so on. These type of queries will run much faster on a column oriented DBMS due to lesser seek time required to obtain the data.

OLAP and BI applications mostly consist of data aggregation and would therefore run faster on column oriented databases.

For a list of column-oriented DBMSes refer to http://en.wikipedia.org/wiki/Column-oriented_DBMS

• MySQL supports multiple backend storage engines providing more flexibility of choice. For instance one can choose MyISAM for tables where transactions and ACID compliance does not matter, and gain a performance advantage. Or one can use a Memory storage engine for temporary in-memory tables
• InnoDB supports optional MVCC, thus providing best of both worlds
• MySQL supports native replication and shared nothing clusters
• MySQL has better integration with memcached
• MySQL uses multi-threading as opposed to process-forking, making it less heavy
• More people are using MySQL than Postgres – eg Facebook, Youtube etc
• MySQL is now owned by Sun, and despite their recent lay-offs they are a company I respect

There are many other reasons, but currently these are the ones that are relevant to the products we are working on.