High Performance Enterprise Data Storage and Analytics

Aamir A. Farooqui (Ph.D.)

CONFIDENTIAL

Enterprise Data storage and analytics

DB Analytics Issues: Oracle has said, in effect, that its most important technological challenge of the decade is getting solid-state memory right. Data volumes will keep going up, up, up. Exploding, less time to analyze, Technology needs to keep evolving accordingly. Data needs to be processed and analyzed at very different latencies. And theres much further to go in integrating disparate latencies. Analytic database management in the cloud hasnt been solved yet, especially for Big Data. Among the reasons are the difficulty of moving data into the cloud (unless it originated there), the slowness of moving it from node to node in shared-nothing architectures (which reduces the elasticity benefit), and above all the long and unpredictable latencies of interprocessor communication while queries are running Solid-state memory will soon be the right storage technology for a large fraction of databases, OLTP and analytic alike. Putting a disk controller in front of solid-state memory is really wasteful. It wreaks havoc on I/O rates. Generic PCIe interfaces dont suffice either, in many analytic use cases. Their I/O is better, but still not good enough. (Doing better yet is where Petascan the stealth-mode company I keep teasing about comes in.) Disk will long be useful for very large databases. Kryders Law, about disk capacity, has at least as high an annual improvement as Moores Law shows for chip capacity, the disk rotation speed bottleneck notwithstanding. Disk will long be much cheaper than silicon for data storage. And cheaper silicon in sensors will lead to ever more machine-generated data that fills up a lot of disks. Disk will long be useful for archiving. Disk is the new tape. In other news, Carson likes 10 Gigabit Ethernet, dislikes Infiniband.

Enterprise Data storage and analytics

Currently, for Teradata, cheap equates to SATA, expensive equates to Fibre Channel, and SAS 1.0 isnt used. But SAS 2.0, coming soon, will supersede both of those interfaces, as discussed below. And in a few years all of this will be pretty moot, because solid-state drives (SSDs) will be taking over. Carson thinks SSDs will have a 100X performance benefit versus disk drives, a figure that took me aback. However, hes not yet sure about how fast SSDs will mature. Also complicating things is a possible transition some years down the road from SLC (Single-Level Cell) to MLC (Multi-Level Cell) SSDs. MLC SSDs which store multiple bits of information at once, are surely denser than SLC SSDs. I dont know whether theyre more power efficient as well. One limitation of flash memory is that although it can be read or programmed a byte or a word at a time in a random access fashion, it must be erased a block at a time. Another limitation is that flash memory has a finite number of erase-write cycles. This effect is partially offset in some chip firmware or file system drivers by counting the writes and dynamically remapping blocks in order to spread write operations between sectors; this technique is called wear leveling. Another approach is to perform write verification and remapping to spare sectors in case of write failure, a technique called bad block management (BBM). Carson has become a 10 GigE bigot, and Teradata will soon certify 10 Gigabit Ethernet cards for connectivity to external systems. Carsons interest in Infiniband, never high, went entirely away after Cisco decommitted to it. Obviously, this stands in contrast to the endorsements of Infiniband for data warehousing by Oracle and Microsoft.

Enterprise Data storage and analytics

Teradata will go to 10-gigabit Ethernet for external connectivity on all its equipment, which should improve load performance. Teradata will also go to 10-gigabit Ethernet to play the Bynet role on appliances. Tests are indicating this improves query performance. Whats more, Teradata believes there will be no practical scale-out limitations with 10-gigabit Ethernet. Teradata hasnt decided yet what to do about 2.5 SFF (Small Form Factor) disk drives, but is leaning favorably. Benefits would include lower power consumption and smaller cabinets. Also on Carsons list of exciting future technologies is SAS 2.0, which at 6 gigabits/second doubles the I/O bandwidth of SAS 1.0. Carson is even excited about removing universal power supplies from the cabinets, increasing space for other components. Teradata picked Intels Host Bus Adapters for 10-gigabit Ethernet. The switch supplier hasnt been determined yet. Teradatas future lies in solid-state memory. Thats in line with what Carson Schmidt told me six months ago. To Teradatas surprise, the solid-state future is imminent. Teradata is 6-9 months further along with solidstate drives (SSD) than it thought a year ago it would be at this point. Teradata SSD highlights include: I/O speeds on random medium blocks are 520 megabytes/second, vs. 15 MB/second on their fastest disks. And thats limited by SAS 1.0, load-balanced across two devices, not the hardware itself. (2 x 300+ MB/sec turns out to be 520 MB/sec in this case.) No wonder Carson is excited about SAS 2.0. Teradata is using SAS interfaces for its SSDs, and believes thats unusual, in that other companies are using SATA or Fibre Channel. Never having had a part fail, Teradata has no real basis to make MTTF (Mean Time To Failure) estimates for its SSDs. Teradatas SSD appliance design includes no array controllers. The biggest reason is that right now array controllers cant keep up with the SSDs speed. In its SSD appliance, Teradata has abandoned RAID, doing mirroring instead via a DBMS feature called Fallback thats been around since Teradatas earliest days. (However, unlike Oracle in Exadata, Teradata

Enterprise Data storage and analytics

System Requirement:
Less than 10usec disk access time, 1 Terra Byte/hr query, low power

Solution:
10Gb Ethernet with TCP offload using BCM5771 (ASIC), SSD, DDR3, FPGA with 2 PowerPc running at 400MHz, PCIe, Properietry DB query & search engine (DB Accelerator). Fully programmable solution Demo already working on FPGA

Applications:
Data storage and analysis in data centers and access through internet.

System Block Diagram

Power PC405 SSD

Digital I/O
DDR3

Specifications

Magnetic Drive

TOE* MAC DB Acc. MAC

10Gb 10Gb 400Mhz 100Mhz 100Mhz 100Mhz 8-16GB 80GB 1TB

TCP IP

PowerPC DB Accelerator

PCIe

ISCSI

TOE

PCIe iSCSI

Red is FPGA

DDR3 SSD Magnetic Drive

*TCP Offload Engine

CONFIDENTIAL