31 Oct, 2009

My Ideal layered distributed clustered redundant self healing Filesystem

Posted by Bhavin Turakhia | (4) Comments

This is a collection of notes on the features that my dream filesystem would support -

• multi-Layered storage – ability to support layers of slower to faster disks and move data at block and/or file level between them atomically, keeping the most frequently read data on the fastest disks. If there is block level support for this it would be a boon for databases, where frequently accessed pages could be kept in faster SSDs and less frequently accessed pages would be stored on slower SATA drives
• Distributed – ability to have multiple clients participate and access a single virtual storage device
• Replicated – each file (or block) is replicated ‘n’ times using master-master sync/async replication
• No FSCK
• Self-healing
• Compression – native support for compression at file or block level (ideally the former)
  • Ability to access the file in its compressed form (useful where we can send out a compressed byte stream to the client and the uncompression is handled at the client)
• Snapshot capability
• Parity based RAID without write penalties (like Raid-Z)

ZFS supports many of the above features.

15 Mar, 2009

Selecting a Solid State Drive technology

Posted by Bhavin Turakhia | (4) Comments

This is my 3rd post on Solid State drive technology (read Solid State Drives vs Hard disk drives). Offlate I have been making a ton of posts on storage, given that in a high-performance, data-intensive environment, applications eventually bottleneck at the slowest component in the chain – the disk (no surprise there considering it is the only device with moving parts).

With the numerous SSD options floating around in the market, it is hard to figure out which way to go without adequate research. Infact, a wrong choice can result in slowing down your application (check Solid State Hard drives have poor random write performance).

Here are my notes on selecting your SSD platform for maximum performance -

Check the random write IOPs
Different SSDs have different performance. SSDs fare worst on random writes (4K or smaller blocks). Therefore when checking an SSD – always check the maximum random write IOPs it delivers

Single cell (SLC) vs Multi-cell (MLC)
SSDs are either single cell (SLC) or multi-cell (MLC). SLC memory has the advantage of faster transfer speeds, lower power consumption and higher cell endurance. However, since they store less data per cell, SLC costs more per megabyte of storage than MLC.

Managed Flash Technology
I first wrote about Managed Flash Technology in my earlier article on SSDs - Solid State Drives vs Hard disk drives. MFT was developed by EasyCo. MFT essentially converts multiple random writes into a single linear write by collating them and writing them out together. MFT can pretty much be used in conjunction with any SSD. EasyCo’s website has interesting benchmark comparisons of SSDs with and without MFT. For instance, on the Mtron PRO 7000, MFT increased the random IOPs for 4K blocks from 123 to a whopping 16,180

Fusion-io
Fusion-io claims to have produced the fastest SSD in the world. With Steve Wozniak on their team, I would be inclined to believe their claims. The spec sheet of their latest drive – the iodrive duo – boasts a random write performance of 180,530 IOPs. That is insane. Additionally multiple ioDrives can be configured into a RAID array for additional performance and reliability.

8 Mar, 2009

iostat and disk utilization monitoring nirvana

Posted by Bhavin Turakhia | (6) Comments

In my neverending quest of performance monitoring, I have been constantly trying to find better ways to monitor disk utilization on a server. At Directi we use the usual medley of tools at our disposal viz. iostat, sar, sysstat. I made serious progress last week, when Dushyanth from my team shared this post on IO Monitoring on Linux, by the folks over at Pythian, on our internal mailing list. Here are my notes on the subject.

Performance measurement and Capacity planning are a science. It is common practice at Directi to attempt to determine what the performance bottlenecks in any given application are. A usual generalization is to determine whether an application is cpu-bound / memory-bound / IO bound.

IO bound applications end up wasting cpu cycles, especially incase of Disk IO, since most programming languages do not have Async Disk IO support today. Therefore in order to maximize performance and optimize resource utilization one should try and reduce iowait time of a CPU and tweak a deployment to make an application cpu-bound.

When your CPU seems to be spending a lot of time on iowait you need to make some changes. However an iowait can occur either because there is a lot of Disk/Network IO taking place, or because the disk subsystem is saturated and cannot provide greater throughput. iostat allows you to determine which one it is. A regular iostat output consists of the following fields -

# iostat -dkx 60

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
dm-0              0.00     0.00  611.40  414.23 20286.60  1656.93 42.79    17.50   17.33   0.96  98.57

Explanation of the above fields:

• Device: the block device whose performance counters are being reported
• r/s and w/s: number of read and write requests issued per second to the device (in this case 611 and 414)
• rsec/s and wsec/s – number of sectors read/written per second
• rkB/s and wkB/s – number of kilobytes read/written per second
• avgrq-sz – average number of sectors per request (for both reads and writes). ie (rsec + wsec) / (r + w)
• avgqu-sz – average queue length in the monitoring interval (in this case 42.79)
• await – average time that each IO Request took to complete. This includes the time that the request was waiting in the queue and the time that the request took to be serviced by the device
• svctim – average time each IO request took to complete  during the monitoring interval
• Note: await includes svctim. Infact await (average time taken for each IO Request to complete) = the average time that each request was in queue (lets call it queuetime) PLUS the average time each request took to process (svctim)
• %util: This number depicts the percentage of time that the device spent in servicing requests. This can be calculated with the above values. In the above example the total number of reads and writes issued per second is 611 + 414 => 1025. Each request takes 0.96 ms to process. Therefore 1025 requests would take 1025 x 0.96 => 984 ms to process. So out of the 1 second that these requests were sent to the device in, 984 ms were taken to process the requests. This means the device utilization is 984/1000 * 100 => ~98.4%. As you can see in the above iostat output the %util does show ~ 98.5%

Interpreting iostat values

Lets take the above example

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
dm-0              0.00     0.00  611.40  414.23 20286.60  1656.93 42.79    17.50   17.33   0.96  98.57

• avg time that each request spent in queue (qtime) = await – svctime = 17.33 – 0.96 => 16.37 ms
• avg time tha each request spent being serviced = 0.96 ms
• so averagely each IO request spent 17.33ms to et processed of which 16.37 ms were spent just waiting in queue
• %util can be calculated as (r/s + w/s) * svctim / 1000ms * 100  => 1025*0.96/1000 * 100 => 98.5%
• This simple means that in a 1 second interval, 1025 requests were sent to disk, each of which took 0.96ms for the disk to process resulting in 984 ms of disk utilization time in a period of 1 s (or 1000 ms). This means the disk is greater than 98% utilized

On this disk subsystem, it is clear that the disk cannot process more IO requests than what it is getting

Lets take another example -

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb               6.33   139.07   46.30   19.53   526.27   634.40    35.26 0.54    8.17   3.30  21.74

• avg time that each request spent in queue (qtime) = await – svctime = 8.17 – 3.30 => 4.87 ms
• avg time tha each request spent being serviced = 3.30 ms
• so averagely each IO request spent 8.17 ms to et processed of which 4.87 ms (a little more than half) were spent waiting in queue
• %util can be calculated as (r/s + w/s) * svctim / 1000ms * 100  => 65.83 * 3.3/1000 * 100 => 21.72%
• This simple means that in a 1 second interval, 65 requests were sent to disk, each of which took 3.30ms for the disk to process resulting in 217 ms of disk utilization time in a period of 1 s (or 1000 ms). This means the disk is around 21.7 % utilized

On this disk subsystem, it is clear that the disk is not fully utilized. While due to the nature of the requests, averagely requests are spending half their time in queue, that is not so bad. This disk subsystem is capable of greater throughput.

Notes

On every Linux box the following should be graphed at 5 minute averages

• %util: When this figure is consistently approaching above 80% you will need to take any of the following actions -
  • increasing RAM so dependence on disk reduces
  • increasing RAID controller cache so disk dependence decreases
  • increasing number of disks so disk throughput increases (more spindles working parallely)
  • horizontal partitioning
• (await-svctim)/await*100: The percentage of time that IO operations spent waiting in queue in comparison to actually being serviced. If this figure goes above 50% then each IO request is spending more time waiting in queue than being processed. If this ratio skews heavily upwards (in the >75% range) you know that your disk subsystem is not being able to keep up with the IO requests and most IO requests are spending a lot of time waiting in queue. In this scenario you will again need to take any of the actions above
• %iowait: This number shows the % of time the CPU is wasting in waiting for IO. A part of this number can result from network IO, which can be avoided by using an Async IO library. The rest of it is simply an indication of how IO-bound your application is. You can reduce this number by ensuring that disk IO operations take less time, more data is available in RAM, increasing disk throughput by increasing number of disks in a RAID array, using SSD (Check my post on Solid State drives vs Hard Drives) for portions of the data or all of the data etc

1 Jan, 2009

Solid State Drives vs Hard disk drives

Posted by Bhavin Turakhia | (10) Comments

Intro

• A solid state drive stores its data in solid-state memory (Flash / SRAM / DRAM)
• Flash does not require constant power and is non-volatile while SRAM and DRAM are volatile

Speeds

• Flash maybe slower than even tradition HDDs on big file access
• Flash is considerably slower than conventional disks for small writes. This is partly due to their large erase block size of 0.5-1 MB
• SSDs are faster than HDDs for small random reads due to negligible seek time (no moving parts)
• Check the comparison table at http://www.storagesearch.com/ssd-ram-v-flash.html.  When Flash based SSDs are used for equal reads and writes they are actually slower than HDDs. However if small random reads far outweigh writes, the performance gains can be upto 100x!!?
• Download the paper – Comparison of Drive Technologies for High-Transaction Databases. Findings below -
  • HDDs: Small reads – 175 iops/s, Small writes – 280 iops/s
  • Flash SSDs: Small reads – 1075 iops/s (6x), Small writes – 21 iops/s (0.1x)
  • DRAM SSDs: Small reads – 4091 iops/s (23x), Small writes – 4184 iops/s (14x)
• Another whitepaper on Flash vs HDDs is Understanding Flash SSD Performance. Findings below -
  • Read performance: Flash outperforms hdds by a large magnitude for small block size
  • It is with write performance that Flash SSDs become problematic. The issue here is the internal structure used within the Flash storage array. This structure includes a collection of bytes called an “erase block”. When you write to a Flash SSD, the drive itself cannot just update the sectors you are changing, but must merge your changes with existing data to update a complete erase block. As Flash SSDs have gotten faster and larger, erase blocks have grown as well. Flash erase blocks used to be 16K in length. Now they are 1 Megabyte for small SSDs extending up to as large as 4 Megabytes for some models.
  • If you are doing pure reads, a Flash SSD will typically be 20x faster than a hard disk for small random reads. If you are doing pure random writes, the same drive might be 15x slower than a hard disk
  • Of pertinence is the table which shows how a small % of writes can destroy Flash SSD Performance. It is for this reason alone that Flash SSDs, by themselves, are not very effective with random update applications like on-line databases, mail queues, and other environments that involve a lot of small updates
    
• One can improve write performance of a Flash SSD using the following methods -
  • OS Write caching – OS buffers writes which eventually get written to disk making the writes appear faster
  • File systems optimized to minimize random writes – YAFFS, JFFS2.
  • Managed Flash Technology – a patent pending technology by easyco which enables Flash Drives to write clusters of random data in linear streams

Costs

• As of mid-2008, SSD prices are still considerably higher per gigabyte than are comparable conventional hard drives: consumer grade drives are typically US$2.00 to US$3.45 per GB for flash drives and over US$80.00 per GB for RAM-based compared to about US$0.12 per gigabyte for hard drives
• DRAM based SSD require more power than hard disks when operating, and need continuous power when not in use if the data needs to be persistent
• Check article Flash vs DRAM Price Projections – for SSD Buyers
  • In the first half of 2007 the difference in user price between a RAM versus Flash SSD was about 45 to 1. A year later in the first half of 2008 that ratio had changed to 25 to 1
  • However NAND has been on a steeper price decline than DRAM for its entire existence. The price of a gigabyte of DRAM declines (on average) 32% per year. There are indications that this decline may slow. Meanwhile, NAND’s price per gigabyte declines faster, at an average of 50% per year

My Conclusions

• DRAM based SSDs are crazy expensive. They serve best for volatile caches (eg, memcached pools etc). If you have servers dedicated to serve in-memory cache data, it may reduce your cost to add DRAM SSDs to these clusters since they are likely not going to bottle-neck on CPU anyways
• Flash based SSDs would work in an environment where the % of writes is low. As can be seen in some of the above benchmarks, a flash based SSD starts degrading in performance in comparison to HDDs in environments with just 5% writes. If one wants to use Flash based SSDs in environments with substantial writes, one should use special filesystems (YAFFS / JFFS2)  and/or use Managed Flash Technology
• Flash based SSDs work like a charm in a read-only or mostly read environment

5 Nov, 2007

Demystifying Storage and Building large Storage Networks – Part I

Posted by Bhavin Turakhia | (3) Comments

I have been studying storage systems over the last few months, and have been wanting to create a Training Course for it since a while now. I decided to get down to it few weeks ago and started creating a PPT for speaking at BarCampMumbai. Here are the slides which I used to give this talk. This is Part I – of what I anticipate to be a 3 part Training Course. The ppt is fairly detailed, and covers hard drives, RAID levels, and a few DAS and SAN topologies. I plan on posting a comprehensive training video of the same separately.

(Visit http://www.slideshare.net/bhavintu79/demystifying-storage for the fullscreen version)

Part II and Part III (whenever I get around to building them) will cover – Complex SAN configurations, iSCSI, NAS, Clustered Storage and other associated topics