DUG Technology: Exascale Flash Storage

Making the move from HDDs to SSDs

DUG Technology switched from hard disk drives to petabytes of flash storage powered by Intel® (now Solidigm™) technology.

• DUG Technology is at the forefront of high-performance computing (HPC), combining innovative hardware and software solutions that enable clients to make use of large and complex datasets.
• To build a resilient and adaptive storage environment that enabled expansion into new markets, DUG switched from hard disk drives to petabytes of flash storage with VAST

Solidigm SSDs help make seismic data actionable 

Seismic analysis is a high-performance computing (HPC) discipline that pieces together what lies under the surface of the earth from nothing more than the reflection of sound. Useful 3D analysis requires petabytes (PB) of data and thousands of powerful computers with vast amount of SSD storage.

Oil companies don’t possess the computational resources necessary to conduct all of this analysis in-house, so they turn to companies like DUG Technology to tease out details from their mountains of data.

High-performance computing as a service 

DUG refers to this capability as HPC-as-a-service (HPCaaS): specialized, full-stack exascale computation available on demand. Traditionally, DUG’s compute-as-a-service technology was available only to specific customers, such as major oil and gas companies. However, as the market took notice of its capabilities, DUG expanded its offering to other industry verticals that use this same service to tackle a diverse set of extreme computational needs.

DUG decided to bring the same “bring-nothing-but-your-data” ease of service to businesses outside of the energy sector. DUG knew that it could serve these new industry verticals economically because of the specialized DUG McCloud service for HPC. VAST Data Universal Storage, powered by Solidigm technologies, undergirds DUG McCloud and enabled DUG to successfully break into new verticals, including academia, astrophysics, medicine and genomics, wildfire modeling, and COVID-19 research. However, getting to this point required a sea change in how DUG dealt with its storage.

Solidigm SSDs accept the challenge

For its first decade of operation, DUG had been deploying and managing HDD-based storage to deliver the scale and cost economy that its seismic workloads required. During that time, DUG thoroughly optimized its applications to make use of the capabilities of its Lustre HDD-based infrastructure, while working with their limits.

DUG had to make compromises to maximize productivity while managing cost. For example, when Lustre file system clients would hit peak throughput for a given workflow, other users sharing the same file system would suffer slowdowns. Although DUG designed its software to protect against HDD failures, the need to swap out failed drives on a weekly basis was a constant thorn in DUG’s side, pushing up costs in both time and resources .

New applications for HPCaaS called for new storage solutions

While DUG’s applications were well optimized for Lustre and HDD storage, DUG was evolving new applications that handled storage input/output (I/O) differently. These applications call for storage versatility and multitenancy. DUG needed new solutions that would support a broad set of requirements at exascale.

DUG also needed storage solutions that could handle the multiplicity of throughput requirements for different applications. DUG looked to solid state drive (SSD)-based storage to provide higher performance and reliability. However, moving to SSDs on Lustre would have been prohibitively expensive, and affordability was paramount for DUG.

To build a resilient and adaptive storage environment that enabled expansion into new markets, DUG required a new approach to storage.

Solution: VAST Data universal storage using Solidigm SSDs

DUG chose VAST Data Univeral Storage to expand its business and support the needs of a wide diversity of new markets and customers. The Universal Storage offering combines the speed and scale of a parallel file system with a new level of flash affordability and multitenancy to deliver a complete technological leap forward for DUG. VAST Data’s disaggregated shared everything (DASE) architecture also provides consistent performance by isolating non-optimized I/O so as not to impact other tenants.

With the DASE approach, VAST Data eliminates the concurrency challenges of parallel storage to deliver high performance for specific workloads that does not come at the expense of other workloads.

Beyond significantly improving the customer performance experience, VAST Data provides a combination of reliability, management, and support that is not otherwise found with legacy HPC storage technologies.

Exascale scalability with VAST Data’s DASE architecture 

VAST Data’s DASE architecture supplies exascale scalability, which enabled DUG to grow to tens of petabytes of flash storage. This architecture can quickly recover from failure, too, since there are no single points of failure. The reliability of the DASE architecture comes “for free”as a direct result of VAST Data’s data-protection efficiency and the architecture's statelessness.

Beyond resilience, VAST Data Universal Storage also simplifies DUG’s deployment and management experience. The integrated scale-out appliance consistently pushes out new features that are automatically applied while the system is online, so there’s no downtime for DUG.

Overview of VAST Data universal storage with Solidigm storage technologies

VAST Data Universal Storage provides a single, global namespace so that each application has access to all of the associated data for that workload. The VAST Data solution combines:

• All-flash drive performance
• Massive scalability
• The economics of archive storage
• The simplicity of plug-and-play network-attached storage (NAS) connectivity

Solidigm SSDs provide the hardware basis for the cost-efficiency and reliability of VAST Data Universal Storage. Solidigm pairs vertical floating-gate technology and complementary metal-oxide-semiconductor (CMOS) under-array architecture, which delivers the highest areal density (gigabytes of storage per square millimeter) in the industry for the same bits per cell.¹

This means that Solidigm QLC 3D NAND SSDs provide not only greater areal density than previous-generation triple-level cell (TLC) media, but greater areal density and higher reliability than competing quad-level cell (QLC) designs based on charge-trap technology. [1] The architectural innovations from Solidigm enable the VAST Data solution to economically store all data on flash drives.

The bottom line

Cost effectiveness and high reliability of Intel (now Solidigm) QLC 3D NAND SSDs provides the foundation for VAST Data’s architecture to reduce costs by up to 85% compared to HDDs, providing a dollar-per-gigabyte (GB) cost similar to that of HDD-based systems over 10 years. [2, 3]

To learn more, download the DUG Technology white paper below or click here to read the VAST Data exascale NAS white paper.