HPC/AI Applications are Data Rich, and Data Must be Protected
In whole or in part, data can be lost for several reasons: user error, application failure, storage system failure, and malfeasance. The adage 'just rerun the application' isn't a useful strategy in present day HPC, especially when the data from sensors, satellites and signals may be ephemeral.
Loss of the file system due to failure has catastrophic impact upon availability of the high performance compute cluster. Even when a file system has tools for repairing, the complexity and time it takes to repair a broken file system can extend the compute outage beyond acceptable SLAs.
Up to now, protecting file systems and data has been an costly investment with imposing drawbacks, including the lack of backup windows, backup utilities that are sub-optimized optimized for PB-sized parallel file systems, and the negative impact upon performance from scanning file system metadata.
HPE DMF7 creates immutable versions of files and takes snapshots of the file system state via namespace reflection. Both operations are managed via administrator policy. Recovery can be optimized for lowest RTO from disk, lowest cost from tape, and/or from remote locations via S3/cloud.
The Emergence of Exascale Computing is Challenging the Scaling Limits of Legacy HPC Storage
The volume and diversity of data demanded by HPC/AI applications has fueled the growth of the "storage beast' that feeds on HPC budgets. At the same time traditional parallel file system architectures are struggling under the weight of relentless growth in the number of files and inodes.
Perhaps the challenge is felt most acutely by storage administrators, who struggle to maintain enough free space in costly high performance storage while users independently flood the file system with new files. Increasing the storage budget and/or deleting user files aren't practical remedies.
HPE DMF7 manages free space in storage by automatically moving 'stale' files out of the high performance storage, creating an underlying storage space that is bigger on the inside. Administrators easily manage this by policy settings, alleviating the need to take brute force actions.
Eventually, administrators need to work with users to prune unused files from the file system to ensure metadata performance isn't undermined. When old files are marked for removal, no data has to be moved since HPE DMF7 already preserves files and metadata in less costly back end storage.
HPC/AI Storage Environments are Diverse and Data has to be Portable
Storage systems are optimized for performance, capacity, and cost, and data is always in flight between these tiers. Application workflows demand that data follow the user and the application, and administrators are continually pressured to manage storage costs and push data down the hierarchy.
Managing HPC/AI data movement is an intimidating task. Tools aren't easy to use, they don't scale well, network pipe bandwidth is limited, and users may not have the needed skills. When data cannot be moved easily and the motivation to move it is low, the default choice is to leave it in place.
HPE DMF7 automatically migrates files down the storage system hierarchy without administrator interaction and recalls them up to high performance storage on demand. It uses parallel data movers and the high speed network to move files faster than standard desktop utilities and in a single step.
Technology migration is a common driver of data movement and HPE DMF7 future proofs against this risk. It automates the migration of back end objects from older, inefficient generation HDD/tape technologies and on to generations that have the highest density, reliability and performance.
When the file system must be retired, administrators move the file system and files onto HPE DMF7 back end devices. Once they are protected, the file system can be staged into an entirely new namespace and files can be staged into the new namespace or remain in curation by HPE DMF7.
Reducing HPC Storage Costs Means More Budget for Compute
The unabated growth of HPC data and the orchestration of large data sets for AI/machine learning are driving unprecedented growth in storage capacity needs. And the adoption of flash storage at a higher effective cost means that containing cost is still the primary goal for HPC storage buyers.
The storage beast works against that goal. The storage beast encourages HPC customers to replicate files on expensive proprietary storage, scale out the most expensive storage tier to satisfy data growth, and says it's easier to leave data in place on expensive storage than it is to move data.
The primary goal of HPE DMF7 is to reduce storage costs. Instead of purchasing more expensive storage, HPE DMF7 makes it easier for HPC customers to protect, scale and move data using lower cost storage and improve utilization and performance of mission critical HPC storage resources.
HPE DMF7 is the HPC storage management platform that automates data workflows and reduces HPC storage costs so HPC customers can spend their valuable project budgets on the infrastructure resource that matters most.