HPE Data Management Framework Base System 10PB 3‑server 5‑year Subscription E‑LTU

HPC/AI Applications are Data Rich, and Data Must be Protected

HPE Data Management Framework 7 creates immutable versions of files and takes snapshots of the namespace reflection. Managed via an administrator policy, recovery can be customized for lowest RTO from disk, lowest cost from tape, and/or from remote locations via S3/cloud.

Loss of the file system due to failure has catastrophic impact upon availability of the high performance compute cluster. Even when a file system has tools for repairing, the complexity and time it takes to repair a broken file system can extend the compute outage beyond acceptable SLAs.

Up to now, protecting file systems and data has been a costly investment with imposing drawbacks, including the lack of backup windows, backup utilities that are sub-optimized, optimized for PB-sized parallel file systems, and the negative impact upon performance from scanning file system metadata.

The Emergence of Exascale Computing is Challenging the Scaling Limits of Legacy HPC Storage

HPE Data Management Framework 7 manages free space in storage by automatically moving 'stale' files out of the high performance storage, creating an underlying storage space that is bigger on the inside. Administrators easily manage policy settings, alleviating the need to take brute force actions.

Perhaps the challenge is felt most acutely by storage administrators, who struggle to maintain enough free space in costly high performance storage while users independently flood the file system with new files. Increasing the storage budget and/or deleting user files aren't practical remedies.

The volume and diversity of data demanded by HPC/AI applications has fueled the growth of the "storage beast' that feeds on HPC budgets. At the same time traditional parallel file system architectures are struggling under the weight of relentless growth in the number of files and inodes.

Eventually, administrators need to work with users to prune unused files from the file system to ensure metadata performance isn't undermined. When old files are marked for removal, no data has to be moved since HPE DMF7 already preserves files and metadata in less costly back end storage.

HPC/AI Storage Environments are Diverse and Data has to be Portable

HPE Data Management Framework 7 automatically migrates files down the storage system hierarchy without administrator interaction and recalls them up to high performance storage on demand. It uses parallel data movers and the high speed network to move files faster than standard desktop utilities.

Managing HPC/AI data movement is an intimidating task. Tools aren't easy to use, they don't scale well, network pipe bandwidth is limited, and users may not have the needed skills. When data cannot be moved easily and the motivation to move it is low, the default choice is to leave it in place.

Storage systems are optimized for performance, capacity, and cost, and data is always in flight between these tiers. Application workflows demand that data follow the user and the application, and administrators are continually pressured to manage storage costs and push data down the hierarchy.

Technology migration is a common driver of data movement and HPE DMF7 future proofs against this risk. It automates the migration of back end objects from older, inefficient generation HDD/tape technologies and on to generations that have the highest density, reliability and performance.

When the file system must be retired, administrators move the file system and files onto HPE DMF7 back end devices. Once they are protected, the file system can be staged into an entirely new namespace and files can be staged into the new namespace or remain in curation by HPE DMF7.

Reducing HPC Storage Costs Means More Budget for Compute

HPE Data Management Framework 7 is the HPC storage management platform that automates data workflows and reduces HPC storage costs, so HPC customers can spend their valuable project budgets on the infrastructure resource that matters most.

The storage beast works against that goal. The storage beast encourages HPC customers to replicate files on expensive proprietary storage, scale out the most expensive storage tier to satisfy data growth, and says it's easier to leave data in place on expensive storage than it is to move data.

The primary goal of HPE DMF7 is to reduce storage costs. Instead of purchasing more expensive storage, HPE DMF7 makes it easier for HPC customers to protect, scale and move data using lower cost storage and improve utilization and performance of mission critical HPC storage resources.

The unabated growth of HPC data and the orchestration of large data sets for AI/machine learning are driving unprecedented growth in storage capacity needs. And the adoption of flash storage at a higher effective cost means that containing cost is still the primary goal for HPC storage buyers.