In the quest for faster storage, Intel hopes that Optane's time has come

Since the first mainframes, computer storage technology has necessitated a three-way tradeoff between speed, persistence and capacity, choices that are manifested in today's servers in the distinction between volatile system memory and various forms of non-volatile storage, primarily magnetic disks and semiconductor flash memory. Almost a decade ago, Intel and Micron began developing a technology, 3D XPoint, that promised to bridge the gaps between, volatility and capacity, however, perfecting and commercializing the novel technology has been problematic. 

Over the past year, Intel — which is far more committed to 3D Xpoint, which it branded Optane, than its technology partner — has released Optane-based memory and storage products that finally deliver on the original promise. Unfortunately, the lengthy development timeline has suppressed demand and increased costs, with Optane drives still more than 10-times the price of an equivalently-sized flash device. Intel's vision of Optane defining a new storage tier between flash and DRAM memory has yet to be realized. However, recent product announcements from several storage vendors are finally plugging these gaps by providing a faster option than NVMe flash for performance-hungry storage buyers.

Optane penetrates the data center

As Intel's famous hierarchical pyramid illustrates, the company sees two roles for Optane: system memory and long-term storage. It addresses the former via nonvolatile DIMMs (aka persistent memory NVDIMMs) that are supported on 2nd generation Xeon scalable processors or later. Intel targets the storage layer via Optane SSDs in various form factors. 

Until recently, the high cost of Optane drives — a 750GB, 2.5-inch DC P4800X runs about $2500 versus $240 for a 960GB enterprise-grade flash drive — has limited their use to read caching in storage arrays. However, improvements in storage controller and data management software have led at least two vendors to incorporate Optane as a storage tier alongside NVMe flash devices.

Dell EMC was first to tease Optane support when it announced updates to the PowerMax portfolio last year, releasing PowerMax SCM products using dual-ported Optane SSDs last September. The secret to not wasting the expensive Optane capacity is a PowerMax OS that includes machine learning algorithms that optimize data placement based on a device's I/O metrics that feed both predictive and pattern recognition models that Dell claims are trained by "an average of 40 million data sets per array that drive over 6 billion decisions per day. The combination delivers astounding performance including a 5-times the IOPS, double the bandwidth and 26% lower latency than an all-flash configuration according to Dell-sponsored tests.

PowerMax can support arrays composed entirely of Optane, what Dell calls Storage Class Memory or SCM, (presumably to leave open the possibility of future products using different non-volatile technologies) however, the more common configuration uses SCM as a storage tier. The company's documentation recommends using SCM for only  3-12% of total capacity, however, given the typical 10:1 difference in capacity between flash and SCM drives, even such a small fraction might require that the majority of drive slots be filled with SCM.

StoreONE joins the Optane party

StoreONE (S1)  is the latest company to incorporate Optane via an update to its flash array products via the All-Flash Array.next. Like Dell, S1 treats Optane drives as a data tier, not a cache, and uses controller software that manages data placement on either Optane or low-cost QLC flash drives to maximize performance and improve flash drive durability. S1 says that designing software specifically for all-flash arrays instead of evolving legacy code from disk systems squeezes more performance from each Optane drive; up to one million read and 300,000 write IOPS from a minimal system with four Optane drives. 

In contrast to the PowerMax, S1's product is suitable for smaller organizations since the smallest configuration requires only three Optane and five QLC drives. According to the company's online pricing site, Optane arrays command a premium over conventional AFAs that decreases with a higher ratio of flash to Optane capacity.

• An entry-level 23 TB AFA using Supermicro hardware costs $48,486
• The least expensive, 23 TB AFA.next configuration with a single dual-processor controller, 3TB of Optane storage and 20TB of QLC flash runs $59,184 ($2.57 per raw TB) or 22% more.
• A 184 TB AFA, again using a Supermicro server, cost $187,053.
• A comparable single-controller Optane system, which includes 4.5TB of Optane and 179.5TB of flash storage ($1.07/raw TB), runs $196,406, a mere 5% increase.

Optane proponents argue the technology reduces the cost for arrays hosting performance-sensitive workloads like transaction processing and analytics databases since achieving the same IOPS and throughput with an all-flash configuration requires more drives and controllers. For example, S1 CTO George Crump writes that:

With regular NAND flash, they would need dozens, maybe hundreds of drives to get to one million IOPS. In many cases, the system would run out of other resources long before it broke the one million IOPS barrier."

Likewise, an Intel blog comparing configurations for Ceph block storage servers contends that (emphasis added),

When compared to the previous generation all TLC solution, “O+Q” [Optane plus QLC flash hybrid system] offers twice the storage for 10% less cost per node. This is possible because of the densely packed QLC NAND which enables less expensive and denser nodes. Additionally, the solution provides 10% better IOPS and 67% better P9999 [Four-9s or 99.99 percent] tail latency.

 

Optane's tepid adoption demonstrates that most enterprise storage vendors remain unconvinced of the technology's benefits. Aside from increased parallelism via adding drives to an array, the other accepted way of juicing storage performance uses DRAM as a cache. DRAM caching has become more effective and less expensive as prices have declined and storage software has gotten more sophisticated by using machine learning to analyze data patterns. For example, Infinidat claims that its arrays deliver most I/O directly from DRAM, which makes interposing an Optane storage tier both expensive and superfluous. 

My take

The performance gains from augmenting traditional all-flash storage with Optane drives are sizeable, however, as the S1 example illustrates, the price premium is highly variable. That said, benchmarking from the few systems available demonstrates that, like adding a turbocharger to an internal combustion engine, a little Optane goes a long way. However, Intel still faces two significant challenges to increased Optane adoption by enterprises and cloud builders:

• The declining price differential between Optane and DRAM, which is exacerbated by, 
• The lack of manufacturing efficiencies due to continued tepid adoption. 

Jim Handy, Chief Analyst at Objective Analysis, sums it up here when he says that:

Intel has a 'chicken and egg problem' with Optane. It must keep the price below that of the faster DRAM to make it appealing, but it can't make a profit at that price until it hits mass adoption.

Indeed, as Intel's most recent Q1 2020 financial results illustrate, the company's memory business unit continues to lose money since the costs of "investing in Optane" outweigh the profits from NAND flash. Nonetheless, with Dell, S1, Pure Storage perhaps HPE — which claims to have designed its Primera arrays to support SCM, but it doesn't yet include Optane in shipping products — and Pure Storage using Optane as a high-performance storage tier, it significantly shortens Intel's path to Optane profitability. 

Unfortunately for Intel, cloud providers are less likely to introduce a new storage technology when they can achieve acceptable performance by scaling across their massively distributed storage systems. However, for enterprises requiring maximum throughput on workloads that must remain on company-operated systems, Optane has finally emerged as a compelling option.